Sunday, January 26, 2014

Raspberry Pi Temperature Data Recorder - Part III: Visualization

<- Part II: Data Collection


Once we have accumulated some data in the RRD database, we can start generating some plots with the RRDTools built-in graph function. The graph above shows the values of the DS18B20 temperature sensors over 3 unseasonably warm days in January. Because of the short cables, the sensors are not optimally placed. E.g. the inside temp sensor is relatively close to the radiator, the outside one is on the ledge, just outside the window and still behind a partially closed shutter in a narrow alley between 2 old poorly isolated buildings. According to the weather report, the current outside temperature is about 3-4 C less than what the sensor shows. From the chart, we can see that the heater feed temperature seems to fluctuate a bit, maybe a sign of hysteresis in the burner controller. The heater temperature is also lowered a bit for about 6h during each night, but this seems not to have any noticeable effect on the room temperature. There are small gaps in the graph, which are caused by read errors or other lapses in the data collection. The chart is generated by running the following command:
#!/bin/sh

rrdtool graph /var/www/temp_graph.png \
-w 1024 -h 400 -a PNG --slope-mode \
--start -3d --end now \
--vertical-label "temperature (°C)" \
DEF:in=data/templog.rrd:internal:AVERAGE \
DEF:out=data/templog.rrd:external:AVERAGE \
DEF:heat=data/templog.rrd:heat:AVERAGE \
LINE2:in#00ff00:"inside" \
LINE2:out#0000ff:"outside" \
LINE2:heat#ff0000:"heat"

A graph command roughly requires 3 types of parameters: general setup, data-source definitions and drawing commands. For more details, see the RRDTool graph documentation.

In order to view the graph on a computer, tablet or smartphone connected to the same wifi network, we can most easily export it through a web-server. Among the major web servers, lighttpd is probably the one with the smallest resource footprint. We can install it like this:
sudo apt-get install lighttpd
and then make sure that the file temp_graph.png in the web servers root directory is writable by the pi user:
sudo touch /var/www/temp_graph.png
sudo chown pi:pi /var/www/temp_graph.png
After running the rrdtool graph script, we should be able to see the graph in a browser as http://<address-of-pi>/temp_graph.png .

We could simply regenerate the graph periodically with another cron job and be done with it or we could go ahead and build a small web application, which generates one or potentially several kinds of graphs on demand.

Among the many available frameworks which help simplify building web applications in python, we somewhat arbitrarily choose web.py for its simple URL routing and request/response management.

The following simple web-app handles 2 URL servlets - graph & view each supporting a scale parameter. While graph calls rrdtool graph to generate a PNG image, view generates a web page around it including a menu to toggle the scale parameter from a daily to a yearly granularity.

#!/usr/bin/python

import os
import rrdtool
import tempfile
import web

# web.py app URL routing setup
URLS = ('/graph.png', 'Graph',
        '/view', 'Page')

RRD_FILE = '/opt/templog/data/templog.rrd'
SCALES = ('day', 'week', 'month', 'quarter', 'half', 'year')
RESOLUTIONS = {'day': '-26hours', 'week':'-8d', 'month':'-35d', 'quarter':'-90d',
  'half':'-6months', 'year':'-1y'}

class Page:
    def GET(self):
        scale = web.input(scale='day').scale.lower()
        if scale not in SCALES:
            scale = SCALES[0]
        result = '<html><head><title>Temp Logger</title></head><h4>'
        for tag in SCALES:
            if tag == scale:
                result += '| %s |' % (tag,)
            else:
                result += '| <a href="./view?scale=%s">%s</a> |' % (tag, tag)
        result += '</h4>'
        result += '<img src="./graph.png?scale=%s">' % (scale, )
        result += '</html>'
        web.header('Content-Type', 'text/html')
        return result
 
class Graph:
  def GET(self):
      scale = web.input(scale='day').scale.lower()
      if scale not in SCALES:
          scale = SCALES[0]
      fd,path = tempfile.mkstemp('.png')
      rrdtool.graph(path,
                    '-w 900',  '-h',  '400', '-a', 'PNG',
                    '--slope-mode',
                    '--start',  ',%s' % (RESOLUTIONS[scale], ),
                    '--end', 'now',
                    '--vertical-label',  'temperature (C)',
                    'DEF:in=%s:internal:AVERAGE' % (RRD_FILE, ),
                    'DEF:out=%s:external:AVERAGE' % (RRD_FILE, ),
                    'DEF:heat=%s:heat:AVERAGE' % (RRD_FILE, ),
                    'LINE2:in#00ff00:inside ',
                    'GPRINT:in:LAST:Cur\:%8.2lf',
                    'GPRINT:in:AVERAGE:Avg\: %8.2lf',
                    'GPRINT:in:MAX:Max\:%8.2lf',
                    r'GPRINT:in:MIN:Min\:%8.2lf\j',
                    'LINE2:out#0000ff:outside',
                    'GPRINT:out:LAST:Cur\:%8.2lf',
                    'GPRINT:out:AVERAGE:Avg\:%8.2lf',
                    'GPRINT:out:MAX:Max\:%8.2lf',
                    r'GPRINT:out:MIN:Min\:%8.2lf\j',
                    'LINE2:heat#ff0000:heat  ',
                    'GPRINT:heat:LAST:Cur\:%8.2lf',
                    'GPRINT:heat:AVERAGE:Avg\:%8.2lf',
                    'GPRINT:heat:MAX:Max\:%8.2lf',
                    r'GPRINT:heat:MIN:Min\:%8.2lf\j')
      data = open(path, 'r').read()
      os.unlink(path)
      web.header('Content-Type', 'image/png')
      return data
      

if __name__ == "__main__":
    web.application(URLS, globals()).run()

Running this app directly as /opt/templog/python/app.py creates a web-server listening on port 8080 and two working URLs - /view and /graph.png

The local web-server is at least very useful for testing and we could launch it as a service from /etc/init.d . But since we already have lightthpd running, we can as well serve this app through it via the fastcgi support. For that we need to modify the config in /etc/lightthpd/lighttpd.conf by adding "mod_fastcgi" to server.modules and add the following section to the file:
fastcgi.server = ( "/templogger" =>     
 (( "socket" => "/tmp/fastcgi.socket",
    "bin-path" => "/opt/templog/python/app.py",
    "check-local" => "disable",
    "max-procs" => 1
 ))
 )

After restarting the server with
sudo /etc/init.d/lighttpd restart
our app is now mapped unter  /templogger/... into the URL-space of the server, so that it can be accessed like this:


With this, we conclude the basic setup of a temperature monitoring system and any further ideas on what else could be done with this data is (for now) left as an exercise to the reader...

Friday, January 24, 2014

Raspberry Pi Temperature Data Recorder - Part II: Data Collection

<- Part I: Hardware

In the previous part of this tutorial, we looked at how to connect a few DS18B20 digital temperature sensors to a Raspberry Pi and read their values.

The next part of the problem is periodically collect and record these measurements and store them for graphing and further analysis.

It is always possible to store measurement data in a general purpose database e.g. SQLite or MySQL and then plot the data for example with the Google charts API.

For this project however, we are going to use RRDTool, which is a special purpose database optimized for recording, aggregating and graphing time-series data. It is particularly popular for network and system monitoring applications and for example at the base of smokeping  which we used in an earlier example. However this time, we need to configure and setup our own database from scratch.

Some of the reasons why RRDTool is particularly nice for this type of application:
  • Fixed-size, fixed-interval sliding window database always stores the N most recent data-points, which means that the data does not grow unbounded and does not need to be deleted.
  • Powerful built-in graph generation (see next part)
  • Multiple levels of aggregation allow a level of granularity which naturally match the resolution of display: very fine-grain for the last N days, more course grain for the last N months or years.
  • Handles gaps in the data without skewing statistics
  • Compact and efficient storage format for time-series data
Each RRDTool database is configured with a base interval in seconds, defining the pulse of the measurement system at which the data is supposed to be sampled. For each interval, it can determine the average rate from two or more samples of a counter or as for our case, the average value of readings of a thermometer.

For each database we can define a number of data-sources, which can be sampled during each base interval. The data for each data-source is then recorded in a series of round-robin archives, each aggregating over a number of base intervals with potentially a different aggregation function (e.g. avg, max, min) and storing a different number of last N values in a round-robin or sliding window fashion.

For our temperature recorder application, we define 3 data sources, one for each of the sensors: inside, outside and heater-feed temperature. As the base clock of the system, we somewhat arbitrarily choose 5min (300 seconds) which should be a compromise between dynamic sensitivity and not trashing the CF storage card unnecessarily.

In order to be able to look at the data over different time-scales, we want to collect:
  • with 5 minute granularity (1 base interval) for  two days (12*24*2=576 samples)
  • with 15 minute granularity (3 base intervals) for  two weeks (4*24*7*2=1344 samples)
  • with 1 hour granularity (12 base intervals) for two months (24*31*2=1488 samples)
  • with 6 hour granularity (72 base intervals) for 16 months (4*31*16=1984 samples)
All the aggregations should be averaged, except for the longest one, where we want to keep average, max and min for each of the 6 hour intervals.

The following scripts creates an RRDTool database with these properties:
#!/bin/sh

rrdtool create /opt/templog/data/templog.rrd --step 300   \
DS:internal:GAUGE:600:-55:125  \
DS:external:GAUGE:600:-55:125  \
DS:heat:GAUGE:600:-55:125  \
RRA:AVERAGE:0.5:1:576    \
RRA:AVERAGE:0.5:3:1344   \
RRA:AVERAGE:0.5:12:1488  \
RRA:AVERAGE:0.5:72:1984  \
RRA:MIN:0.5:72:1984      \
RRA:MAX:0.5:72:1984


Where DS defines the 3 data-sources and RRA the different round-robin archives at various granularities, retention and aggregation types. The data sources are of type GAUGE, which means that the absolute values are used and not the rate/delta-increments as is the main mode of operation for RRDTools. The additional arguments define an update timeout (not really relevant for GAUGE types) and the estimated max/min range of the values, which in this case are the supported range of the DS18B20 sensor according to the datasheet.

The round-robin archives within the database are configured with an aggregation function (avg, max, min), a fudge-factor to define how many missing base samples we can tolerate before the aggregate itself becomes unknown as well as the number of base intervals to to be aggregated and how many of the values should be kept.

Now we need to set up a job which periodically, at least every 5min reads the temperature sensors and inserts the measurements into the database created above. The easiest/most robust way to do that on Linux is through cron, i.e. crontab -e and add the following line:
*/4 * * * * /opt/templog/python/templogger.py

The */4 setting schedules the collection job to run every 4 minutes, which is a little faster than required, but helps reduce the risk that we miss any sample period. RRDTool will automatically create an average value for each base sampling interval for which we record at least one data-point (otherwise the value is unknown). One of the advantages of RRDTool is the proper handling of missing values. Those are simply ignored and create gaps in the graphs and don't affect the aggregated values.

Assuming the files for this application are going to live in /opt/templog and are running as the user pi, we can create this directory with
sudo mkdir -p /opt/templog
sudo chown pi:pi /opt/templog

And create the following script in /opt/templog/python/templogger.py:
#!/usr/bin/python

import logging
import logging.handlers
import rrdtool
import temp_sensor
import time
import sys


def do_update():
  timestamp = time.time()
  internal = temp_sensor.TempSensor("28-000005303678")
  external = temp_sensor.TempSensor("28-000005604c61")
  heater = temp_sensor.TempSensor("28-000005610c53")
# in case of error, retry after 5 seconds
  for retry in (5, 1):
    try:
      rrdtool.update("/opt/templog/data/templog.rrd",
                     "%d:%s:%s:%s" % (timestamp,
                                     internal.get_temperature(),
                                     external.get_temperature(),
                                     heater.get_temperature()))
      return
    except:
      logging.exception("retry in %is because of: ", retry)
      time.sleep(retry * 1000)

# set up logging to syslog to avoid output being captured by cron
syslog = logging.handlers.SysLogHandler(address="/dev/log")
syslog.setFormatter(logging.Formatter("templogger: %(levelname)s %(message)s"))
logging.getLogger().addHandler(syslog)

do_update()

This script sets up logging to syslog, as we want to avoid any output to stderr in cron. We declare the 3 sensor access objects based on the ID which correspond to each sensor (see part I). Since reading a sensor can occasionally fail, we can retry a second time.

Now that we have data accumulating in the RRDTool time-series database, we will be looking at visualizing the data in the next part.

Monday, January 20, 2014

Raspberry Pi Temperature data recorder - Part I: Hardware

The Raspberry Pi seems ideal for all kinds of "physical computing" applications, as it is small, cheap, low-powered and yet more powerful and feature rich than a traditional micro-controller.

One way to showcase such applications in an educational context could be to control science experiments which require long term measurements and data collection.

One of the easiest measurement sensors to connect to a Raspberry Pi is the DS18B20 digital thermometer,. It can be read out via a multi-device 1-wire bus that is directly supported by a driver in the Linux kernel. Several sensors can be connected in parallel to the same data-wire and read out individually over the bus interface by their hard-coded IDs. All we need to connect one or more DS18B20 sensors to a Raspberry Pi, is to connect the VCC pin to 3.3V, GND to GND and data to GPIO4 on the Raspberry Pi GPIO header as well as connect a 4.7k Ohm resistor between the VCC and data lines of the sensor. The sensor is available among others as a basic board mounted package as well as a water-proof assembly with a ca. 1m long isolated cable.

The whole setup can be assembled without any soldering, using some prototyping tools from Adafruit Industries, a great supplier of electronics parts for hacking, making and education, based in NYC but who also ships world-wide.

In addition to the Raspberry Pi with CF card, micro-usb cellphone charger and EW-7811Um USB wifi adapter (see configuration here), the shopping list for this project includes:
This great tutorial explains in much detail how to connect both types of DS18B20 sensors using the breadboard & breakout connector. The ribbon cable on fits one-way into the breakout connector due to the notch on one side, but should be mounted to the Raspberry Pi with the cable pointing away as in the picture above and with the different colored wire towards the edge where the CF card slot and micro-USB power connectors are.

For the experiment, we want to connect at least 3 temperature sensors to simultaneously monitor:
  • inside/room temperature (board mounted sensor)
  • outside temperature
  • heater temperature
Using the data collected for inside & outside temperature and a lot of simplifying assumptions, we could for example estimate the amount of thermal flow out of the room and thus how much heating energy the room has used during the measurement period. Or try to reverse-engineer the control function of the heater thermostat by looking at the relationship of outside, inside and heater feed temperature.

Our house is a roughly 120 year old apartment building with a radiator based central heating system. There are exposed riser pipes which distribute the hot water from the boiler in the basement to the radiators and the apartments above us. This is the place where we can connect the heater temperature sensor, as it will show the feed temperature, regardless of whether the radiator in the room is turned on or off. Tin-foil can make a great heat-conducting sleeve to connect the sensor to the tube. The outside sensor is simply stuck outside the window with the cable pinched by the window frame in less than professional manner...

Once we have carefully assembled all the parts, powered up the Raspbery Pi and made sure that none of the sensors are starting to overhead from wrong polarity, we can test if the driver is working properly:
pi@my-pi ~ $ sudo modprobe w1-gpio
pi@my-pi ~ $ sudo modprobe w1-therm
pi@my-pi ~ $ cd /sys/bus/w1/devices
pi@my-pi /sys/bus/w1/devices $ ls
28-000005303678  28-000005604c61  28-000005610c53  w1_bus_master1
pi@my-pi /sys/bus/w1/devices $ cat 28-*/w1_slave
79 01 4b 46 7f ff 07 10 0a : crc=0a YES
79 01 4b 46 7f ff 07 10 0a t=23562
83 00 4b 46 7f ff 0d 10 5b : crc=5b YES
83 00 4b 46 7f ff 0d 10 5b t=8187
5c 02 4b 46 7f ff 04 10 e6 : crc=e6 YES
5c 02 4b 46 7f ff 04 10 e6 t=37750


In order to make sure that the drivers for the GPIO 1wire interface and the DS18B20 protocol are loaded at boot-time going forward, we need to add them to /etc/modules:
sudo -s
echo w1-gpio >> /etc/modules
echo w1-therm >> /etc/modules

Then we create the following simple driver class in temp_sensor.py that is going to be used to read and record the sensor data in the second part of this tutorial:
#!/usr/bin/python

import re

class TempSensor():
    """
    Read data from DS18B20 Temperature sensor via 1-wire interface.
    """
    _DRIVER = "/sys/bus/w1/devices/%s/w1_slave"
    _TEMP_PATTERN = re.compile("t=(\d+)")

    def __init__(self, sensor_id):
        self._id = sensor_id

    def _read_data(self):
        data_file = open(self._DRIVER % (self._id, ), "r")
        try:
            return data_file.read()
        finally:
            data_file.close()

    def get_temperature(self):
        data = self._read_data()
        # data should contain 2 lines of text like this:
        # 86 01 4b 46 7f ff 0a 10 5e : crc=5e YES
        # 86 01 4b 46 7f ff 0a 10 5e t=24375
        #
        # Temperature reading is value of t= in milli-degrees C
        m = self._TEMP_PATTERN.search(data)
        if not m:
            raise IOError("Invalid data for sensor " + self._id)
        return float(m.group(1)) / 1000.0

if __name__ == '__main__':
    import sys
    for sensor_id in sys.argv[1:]:
        sensor = TempSensor(sensor_id)
        print sensor.get_temperature()

In order to test the driver, we can also execute it directly:
pi@my-pi $ ./python/temp_sensor.py 28-000005303678 28-000005604c61 28-000005610c53
23.562
7.812
39.562
Here we can also identify which ID corresponds to which sensor in our setup. Since it is winter right now, we can assume that 23.5C is the inside temperature, 7.8C the outside and 39.5C is the temperature of the heating pipe. If testing first on a workbench, we can also touch one of the sensors and see which one of the sensor readings is going up towards 37C as a consequence.

Right now, we can simultaneously measure the current temperature in each of the 3 zones, but in the next part, we are going to record time-series based measurements into a database for graphing and further analysis.

Part II: Data Collection ->

Thursday, January 16, 2014

Wi-Pi : 802.11 Networking for Raspberry Pi (EW-7811Un)

One of the most conspicuously absent standard interfaces on the Raspberry Pi is built-in support for 802.11 WiFi wireless LAN networking.

A low-cost, low-power way to remedy this is for example the Edimax EW-781Un USB WiFi adapter which plugs easily into one of the USB ports on the Raspberry Pi and is supported out of the box by the current Raspbian distribution.

It seems to be very popular for use with Raspberry Pi and is available for about $10 in many places where Raspberry Pi are sold.

WiFi Client

Connecting to an existing WiFi network is trivial, once we know the SSID and access password for the network we are trying to connect to. After plugging in the adapter, it should be automatically recognized by the linux kernel - the output of lsusb should contain an entry like this:
Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]

In order to connect to a typical home wifi network, we only need to add the following to /etc/networks/interfaces :
allow-hotplug wlan0

iface wlan0 inet dhcp
wpa-ssid <SSID>
wpa-psk <wifi-key>

substituting <SSID> with the "name" (SSID) of the wifi network and <wifi-key> wifi access code or password configured in the router. After rebooting or running sudo ifup wlan0, the interface should be connected and configured with an address from the gateway via DHCP.

WiFi Access Point

Configuring an EW-781Un adapter as a WiFi access point is not as easy, as the RTL8188CUS chipset is not supported by the standard version of hostapd. Some tutorials like here or here are explaining how to install hostapd and replace it with a custom version which supports the chipset.

For example:
sudo apt-get install hostapd

wget http://www.daveconroy.com/wp3/wp-content/uploads/2013/07/hostapd.zip
unzip hostapd.zip 
sudo mv hostapd /usr/sbin/hostapd
chomd+x /usr/sbin/hostapd

One of the simplest use-case for an access point is to connect a tablet, phone or laptop to a Raspberry Pi as a console or controller for some application which doesn't require Internet access - e.g. a robot or a monitoring device of some sorts.

To allow most standard devices to connect to the Raspberry Pi access point, we have to configure a static IP address on the interface and set up a DHCP server to push IP interface configuration to the devices which are connecting.

For a simple static IP address add the following to /etc/networks/interfaces :
allow-hotplug wlan0

iface wlan0 inet static
address 10.0.0.1
netmask 255.255.255.0

Dnsmasq is a small footprint DHCP server and DNS server/proxy for small networks connected to the Internet masquerading behind a  NAT firewall.
sudo apt-get install dnsmasq

It can also easily serve DNS names for the local network by picking up names from the static /etc/hosts file. In order to have dnsmasq server dynamic IP addresses and DNS names in a pseudo-domain .home to devices connecting to the access point on wlan0, change /etc/dnsmask.conf to the following:
local=/home/
interface=wlan0
domain=home
dhcp-range=10.0.0.2,10.0.0.255,12h

For example, to create a network called TempSensor, create the following hostapd configuration in /etc/hostapd/hostapd.conf:
interface=wlan0
driver=rtl871xdrv
bridge=br0
ssid=TempSensor
channel=1
wmm_enabled=0
wpa=2
wpa_passphrase=tempsensor
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CMMP
rsn_pairwise=CCMP
auth_algs=1
macaddr_acl=0

and set DAEMON_CONF="/etc/hostapd/hostapd.conf"
in /etc/default/hostapd and restart.