An easy Raspberry Pi weather station using DHT11 + Python + QuestDB (with source code)
Every now and then, it’s good fun to go back into older projects, to update and rethink them. A few years ago, I’ve made a simple temperature and humidity monitor using a Raspberry Pi, Python, Portainer, Prometheus, and Grafana. You can check the full project and tutorial here. This service stack was hosted on an RPi 3B (with only 1GB of RAM), and those services were eating a lot of resources for this purpose, and since then, I’ve always wanted to improve on this design.
The previous architecture
Before, every 15 seconds, Prometheus reached for the Python API that reads the sensor data, without checking for errors like invalid values. Those values were stored on Prometheus and then read by Grafana, whenever the user requested a dashboard. All those services are great, but they all require lots of resources to operate inside an environment where we’re limited by memory and CPU.
A new architecture
For a new architecture, the main objective was to reduce the amount of overhead on this service. We’re still using docker to organize all those services, but now, with fewer resources being consumed and delivering the same value.
Now, the user interface (front-end) is handled by a simple static SPA (made with NuxtJS) served using a slim Apache container (could be Nginx or any other web server), whenever a user requests this page for the first time, it’s downloaded to the browser and runs from there, so, less stuff to process on the server side.
Now, the Python API, it’s responsible for reading data from the sensor every minute, by using a background scheduler, and storing it in a questdb instance, a Postgresql-based database that thrives on high-performance time-based storage and querying, while also using the newer libraries from Adafruit for DHT11. The API also uses Flask to expose an endpoint for reading the latest measure of the sensor, consumed by the SPA.
The SPA, served at port 80, then reaches for the Python API, served at port 81, that recovers the data from the questdb instance and returns it as a JSON payload. All those services are created using a docker-compose file for easy deployment and configuration.
Check out the full project at github.com/gmullernh/dht11-weather-station
The results
The stack now occupies and consumes fewer resources, allowing for more services to be added to the machine and a better overall optimization result. A few spikes in CPU and RAM usage occur when writing or reading data from questdb, but otherwise, the values sit around lower percentiles.
Future improvements
This design could be improved by using a lighter database engine (questdb is really good, but for this scale, could be something smaller), the Python API could be replaced with some lighter runtime language like Rust, Go or similar, and the SPA could receive some more features, like graph visualizations and more data analysis. Also, feel free to help improve the project.
