Skip to main content

Raspberry Pi Temperature Data Recorder - Part II: Data Collection

<- Part I: Hardware

In the previous part of this tutorial, we looked at how to connect a few DS18B20 digital temperature sensors to a Raspberry Pi and read their values.

The next part of the problem is periodically collect and record these measurements and store them for graphing and further analysis.

It is always possible to store measurement data in a general purpose database e.g. SQLite or MySQL and then plot the data for example with the Google charts API.

For this project however, we are going to use RRDTool, which is a special purpose database optimized for recording, aggregating and graphing time-series data. It is particularly popular for network and system monitoring applications and for example at the base of smokeping  which we used in an earlier example. However this time, we need to configure and setup our own database from scratch.

Some of the reasons why RRDTool is particularly nice for this type of application:
  • Fixed-size, fixed-interval sliding window database always stores the N most recent data-points, which means that the data does not grow unbounded and does not need to be deleted.
  • Powerful built-in graph generation (see next part)
  • Multiple levels of aggregation allow a level of granularity which naturally match the resolution of display: very fine-grain for the last N days, more course grain for the last N months or years.
  • Handles gaps in the data without skewing statistics
  • Compact and efficient storage format for time-series data
Each RRDTool database is configured with a base interval in seconds, defining the pulse of the measurement system at which the data is supposed to be sampled. For each interval, it can determine the average rate from two or more samples of a counter or as for our case, the average value of readings of a thermometer.

For each database we can define a number of data-sources, which can be sampled during each base interval. The data for each data-source is then recorded in a series of round-robin archives, each aggregating over a number of base intervals with potentially a different aggregation function (e.g. avg, max, min) and storing a different number of last N values in a round-robin or sliding window fashion.

For our temperature recorder application, we define 3 data sources, one for each of the sensors: inside, outside and heater-feed temperature. As the base clock of the system, we somewhat arbitrarily choose 5min (300 seconds) which should be a compromise between dynamic sensitivity and not trashing the CF storage card unnecessarily.

In order to be able to look at the data over different time-scales, we want to collect:
  • with 5 minute granularity (1 base interval) for  two days (12*24*2=576 samples)
  • with 15 minute granularity (3 base intervals) for  two weeks (4*24*7*2=1344 samples)
  • with 1 hour granularity (12 base intervals) for two months (24*31*2=1488 samples)
  • with 6 hour granularity (72 base intervals) for 16 months (4*31*16=1984 samples)
All the aggregations should be averaged, except for the longest one, where we want to keep average, max and min for each of the 6 hour intervals.

The following scripts creates an RRDTool database with these properties:
#!/bin/sh

rrdtool create /opt/templog/data/templog.rrd --step 300   \
DS:internal:GAUGE:600:-55:125  \
DS:external:GAUGE:600:-55:125  \
DS:heat:GAUGE:600:-55:125  \
RRA:AVERAGE:0.5:1:576    \
RRA:AVERAGE:0.5:3:1344   \
RRA:AVERAGE:0.5:12:1488  \
RRA:AVERAGE:0.5:72:1984  \
RRA:MIN:0.5:72:1984      \
RRA:MAX:0.5:72:1984

Where DS defines the 3 data-sources and RRA the different round-robin archives at various granularities, retention and aggregation types. The data sources are of type GAUGE, which means that the absolute values are used and not the rate/delta-increments as is the main mode of operation for RRDTools. The additional arguments define an update timeout (not really relevant for GAUGE types) and the estimated max/min range of the values, which in this case are the supported range of the DS18B20 sensor according to the datasheet.

The round-robin archives within the database are configured with an aggregation function (avg, max, min), a fudge-factor to define how many missing base samples we can tolerate before the aggregate itself becomes unknown as well as the number of base intervals to to be aggregated and how many of the values should be kept.

Now we need to set up a job which periodically, at least every 5min reads the temperature sensors and inserts the measurements into the database created above. The easiest/most robust way to do that on Linux is through cron, i.e. crontab -e and add the following line:
*/4 * * * * /opt/templog/python/templogger.py

The */4 setting schedules the collection job to run every 4 minutes, which is a little faster than required, but helps reduce the risk that we miss any sample period. RRDTool will automatically create an average value for each base sampling interval for which we record at least one data-point (otherwise the value is unknown). One of the advantages of RRDTool is the proper handling of missing values. Those are simply ignored and create gaps in the graphs and don't affect the aggregated values.

Assuming the files for this application are going to live in /opt/templog and are running as the user pi, we can create this directory with
sudo mkdir -p /opt/templog
sudo chown pi:pi /opt/templog

And create the following script in /opt/templog/python/templogger.py:
#!/usr/bin/python

import logging
import logging.handlers
import rrdtool
import temp_sensor
import time
import sys


def do_update():
  timestamp = time.time()
  internal = temp_sensor.TempSensor("28-000005303678")
  external = temp_sensor.TempSensor("28-000005604c61")
  heater = temp_sensor.TempSensor("28-000005610c53")
# in case of error, retry after 5 seconds
  for retry in (5, 1):
    try:
      rrdtool.update("/opt/templog/data/templog.rrd",
                     "%d:%s:%s:%s" % (timestamp,
                                     internal.get_temperature(),
                                     external.get_temperature(),
                                     heater.get_temperature()))
      return
    except:
      logging.exception("retry in %is because of: ", retry)
      time.sleep(retry * 1000)

# set up logging to syslog to avoid output being captured by cron
syslog = logging.handlers.SysLogHandler(address="/dev/log")
syslog.setFormatter(logging.Formatter("templogger: %(levelname)s %(message)s"))
logging.getLogger().addHandler(syslog)

do_update()

This script sets up logging to syslog, as we want to avoid any output to stderr in cron. We declare the 3 sensor access objects based on the ID which correspond to each sensor (see part I). Since reading a sensor can occasionally fail, we can retry a second time.

Now that we have data accumulating in the RRDTool time-series database, we will be looking at visualizing the data in the next part.

Labels: