BirdBox entrance counter v3: postgres / docker / python

This post describes the software side of my v3.0 birdbox entrance hole implementation, refer to part 1 related post  for the physical build details.  To log activity, I used a basic Postgres database, which (to continue an ongoing theme) is run in a docker container to simplify setup.  I updated the python logging script to log to a database rather than a text file, and corrected some previous errors along the way...

The existing birdbox setup remains the same, its a Raspberry Pi ZeroW running Pikrellcam for motion detection and auto capture.  The LED illumination, IR cut and RasPi IR camera module etc remain unchanged.

Like before.. but with entrance hole activity logged to a database

v3.0 entrance hole counter updates...
The two other boxes that I have 'in the wild' with entrance hole counters log partial and full 'in' vs 'out' events activity to a text file.  This updates (v3.0) to a Postgres database instead.  Using a database rather than a text file opens up more possibilities, in particular allowing collected data to be accessed remotely.  I plan to pull data off for analysis on a separate machine, for example to populate activity plots on my grafana nestbox CCTV-sytle dashboard.  Having the activity data sitting in a database makes this much easier.  I've used a 'dockerised' version of postgres as its simpler to setup than installing it from scratch.

This post will cover
1) Docker Installation
2) Docker-Compose Installation
3) Add Postgres container to  Docker installation
4) Creation of a database to log activity
5) The activity logging script itself

I use a base image of Raspbian 'Lite' to start off.  This takes up less space on the SD card but may need more things installing along the way than than 'full-fat' version.  You can follow these setup steps on a Raspberry Pi ZeroW, but be prepared to wait a long time for some of them (especially for Docker Compose).  I use a spare Pi 3 or 4 for the build (much quicker) and transfer the sd card over to a Zero W when its done.  The installtion Pi is assumed to be networked. There's about a zillion how-tos online for that.

1) Docker installation

Think of Docker as an empty applications box.  Once Docker is running there are many many pre-configured applications than can be (fairly) effortlessly run without having to do lots of fiddly setup...

This is mostly lifted from these sites, with my comments added:

https://www.docker.com/blog/happy-pi-day-docker-raspberry-pi/

https://www.freecodecamp.org/news/the-easy-way-to-set-up-docker-on-a-raspberry-pi-7d24ced073ef/

Install commands (I left my hashed-out comments in)

sudo apt-get install apt-transport-https ca-certificates software-properties-common -y
curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh
sudo groupadd docker  #not actually required, groupadd: group 'docker' already exists
sudo gpasswd -a $USER docker
newgrp docker #avoids reboot
docker run hello-world  #test that it works

If it all works, you get the following message:

Hello from Docker!
This message shows that your installation appears to be working correctly. 

Run Postgres on Docker
Details here: https://dev.to/rohansawant/spin-up-a-postgres-docker-container-on-the-raspberry-pi-in-2-minutes-2klo and https://dev.to/rohansawant/installing-docker-and-docker-compose-on-the-raspberry-pi-in-5-simple-steps-3mgl

2) Docker-Compose Installation

Install proper dependencies:

sudo apt-get install libffi-dev libssl-dev
sudo apt-get install -y python python-pip  #?had issue with python 2.7.13 not being compatible with the version of docker-compose below

#edit 

sudo apt-get install -y python3 python3-pip

sudo apt-get remove python-configparser

Install Docker Compose

#sudo pip install docker-compose #takes a long time on a Pi ZeroW

# edit 12/2/21

sudo pip3 install docker-compose
#see details here

3) Add a Postgres database container to Docker:

See https://github.com/CT83/Raspberry-Pi-PostGres-Docker-Compose
Note, MySQL is another option, there are MySQL images available on Docker, or maybe install from scratch.  I went with Postgres as I like pgAdmin as a sql interpretor.

# clone the repo which contains the Compose file: 
git clone https://github.com/CT83/Raspberry-Pi-PostGres-Docker-Compose.git
# Up the container:  
cd Raspberry-Pi-PostGres-Docker-Compose
sudo docker-compose up --build -d
# note this returns an error for a file it cant find:
# ERROR: Couldn't find env file: /home/pi/Raspberry-Pi-PostGres-Docker-Compose/.env
# fix this by making a dummy file first:
touch .env  
# then re-run
sudo docker-compose up --build -d

All being well, postgres database should be running in a docker container on the host Pi, on the default port (5432).   As-is the default user is postgres, password is password@7979  You can change these in the docker-compose.yml.

pgAdmin is a handy sql interpretor/tool for managing local or remote postgres databases.  Install it on a networked PC and you should be able to remotely connect to the postgres instance running on the Pi created above.  See https://www.pgadmin.org/.  Google for loads of tutorials around this bit.

4) Creation of a database to log activity

Within pgAdmin (on a remote PC), this sql command should make the necessary database and table within it

CREATE DATABASE db_activity
    WITH 
    OWNER = postgres
    ENCODING = 'UTF8'
    LC_COLLATE = 'C.UTF-8'
    LC_CTYPE = 'C.UTF-8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

CREATE TABLE public.entrance_log
(
    sensor integer,
    state integer,
    event_time timestamp without time zone
)
WITH (
    OIDS = FALSE
)
TABLESPACE pg_default;

ALTER TABLE public.entrance_log
    OWNER to postgres; 

I'm starting basic, so there's only only one table, for the entrance hole activity.

5) The activity logging script

This is written in Python.  Need to install some extra python packages first:

#install the postgres/sql connector psycopg2
pip install psycopg2  #it does not work properly, need to run the following command first:
sudo apt-get install libpq-dev #(suggestion here: https://github.com/psycopg/psycopg2/issues/699)

The following python script creates a entry in the db_activity database in a table called entrance_log.

A simple bird  'head bob in' event breaks the outer beam[1,0], then the inner beam [2,0],  then withdraws: Inner beam whole again [2,1], then outer beam whole [1,1].
Note that in this example, Outer beam = 1, Inner beam = 2, Broken = 0, Whole = 1

Database entry

This above event as recorded in postgres database

I added some error checking code to stop the script from crashing out if a beam is triggered and it cant see the postgres database for whatever reason - it will carry on merrily logging to file instead.
I have left in the 'log to file' functionality as a backup, I'll may remove that eventually, its there as a backup.

The above event as logged to a txt file

Input comes from two GPIO input pins from the entrance hole sensor: pin 22 (outer beam) and pin 23 (inner beam).  Other pins are available....

HoleSensor.py

import RPi.GPIO as GPIO

from time import sleep

#postgres error handling from here: https://kb.objectrocket.com/postgresql/python-error-handling-with-the-psycopg2-postgresql-adapter-645
#postgres add record adapted from here: https://pynative.com/python-postgresql-insert-update-delete-table-data-to-perform-crud-operations/

# import sys to get more detailed Python exception info
import sys
# import the connect library for psycopg2
from psycopg2 import connect
# import the error handling libraries for psycopg2
from psycopg2 import OperationalError, errorcodes, errors
# import the psycopg2 library's __version__ string
from psycopg2 import __version__ as psycopg2_version

from datetime import datetime

# Change log
# 16/09/19 updated for zerocam 7
# 14/12/19  modified to write activity to a local postgres database, database implemented in docker
#           postgres error handler added so the script does not crash out if the database isnt available for whatever reason

GPIO.setmode(GPIO.BCM)          #use BCM pin numbering system
GPIO.setwarnings(False)

whichBirdcam = 'zerocam6'

#file used to log entrance actions
EntrancelogFile='/home/pi/zerocam6/logs/'+zerocam6_birdlog.txt'
OpenEntrancelogFile= open(EntrancelogFile,  'a', 0)

#function to return the current time, formatted as
# e.g. 13 Jun 2013 :: 572
def getFormattedTime():
    now = datetime.now()
    return now.strftime("%d %b %Y %H:%M:%S.") + str(int(round(now.microsecond/1000.0)))
    #note that this does not work correctly, as rounding 040 will give 40, but this modified function is only meant as a backup for the local file log

# ********* postgres stuff

# define a function that handles and parses psycopg2 exceptions
def print_psycopg2_exception(err):
    # get details about the exception
    err_type, err_obj, traceback = sys.exc_info()

    # get the line number when exception occurred
    line_num = traceback.tb_lineno

    # print the connect() error
    print '\n' "psycopg2 ERROR:", err, "on line number:", line_num
    print "psycopg2 traceback:", traceback, "-- type:", err_type

    # psycopg2 extensions.Diagnostics object attribute
    print '\n' "extensions.Diagnostics:", err.diag

    # print the pgcode and pgerror exceptions
    print "pgerror:", err.pgerror
    print "pgcode:", err.pgcode, '\n'

def logEntranceEventPostgres(sensor, state):
    dt = datetime.now()  #set the census datetime for the event

    try:

        conn = connect(
            dbname = "db_activity",
            user = "postgres",
            host = "127.0.0.1",
            password = "password@7979")

    except OperationalError as err:
        # pass exception to function
        print_psycopg2_exception(err)

        # set the connection to 'None' in case of error
        conn = None

    # if the connection was successful
    if conn != None:

        # declare a cursor object from the connection
        cursor = conn.cursor()
        print "cursor object:", cursor, '\n'

        #dt = datetime.now()
        cursor.execute('INSERT INTO entrance_log (sensor,state,event_time) VALUES (%s,%s,%s)', (sensor,state,dt,))
        conn.commit()
        count = cursor.rowcount
        print count, "record(s) inserted successfully into entrance_log table"

        # close the cursor object to avoid memory leaks
        cursor.close()

        # close the connection object also
        conn.close()

    #append the same timestamp data to file for good measure
    #formattedTime_old = dt.strftime("%d %b %Y %H:%M:%S.") + str(int(round(dt.microsecond/1000.0)))  #note that this is NOT correct since it 'rounds' a 041ms to 41
    formattedTime = dt.strftime('%d %b %Y %H:%M:%S.%f')[:-3]  #this works correctly
    #write to file until we figure out if the same data is being captured
    OpenEntrancelogFile.write(str(sensor) + "," + str(state) + "," + formattedTime + "\n")

#setup GPIOs
detect_OUTER = 22 #6    #set GPIO pin for Outer photransducer (input)
detect_INNER = 23 #12    #set GPIO pin for Inner photransducer (input)

print 'detect_OUTER = ' + str(detect_OUTER)
print 'detect_INNER = ' + str(detect_INNER)

#Constants
#OUTER_BEAM = 1
#INNER_BEAM = 2

#WHOLE = 1
#BROKEN = 0

# setup GPIO pins:
GPIO.setup(detect_OUTER, GPIO.IN)   #set Outer GPIO Phototransducer as input
GPIO.setup(detect_INNER, GPIO.IN)   #set Inner GPIO Phototransducer as input


#indicate the point the program started in the log
OpenEntrancelogFile.write("### recordBird_v3 starting up at:" + getFormattedTime() + "\n")
print "============================================"
print whichBirdcam.upper() + ": Starting up entrance hole counter script..."
sleep (0.5)

#Set initial state of WasBroken for both beams:
OUTER_WasBroken = False
INNER_WasBroken = False

# LED status check
# LEDstate= GPIO.input(detect_INNER)

print ""
print "detect_OUTER status = " + str(GPIO.input(detect_OUTER))
print "detect_INNER status = " + str(GPIO.input(detect_INNER))
print ""


#When the detector          'sees' IR led, the detector pin is 0/LOW/False
#When the detector does not 'see ' IR led, the detector pin is 1/HIGH/True

def checkStatus():
    if GPIO.input(detect_OUTER):  #if OUTER detector does not see IR led, print error, GPIO.input = HIGH
        print "OUTER beam detect failure!, status = " +str(GPIO.input(detect_OUTER))
        #quit()
    else:
        print "OUTER beam detect - passed :) | Status = "+str(GPIO.input(detect_OUTER))

    if GPIO.input(detect_INNER):  #if INNER detector does not see IR led, print error, GPIO.input = HIGH
        print "INNER beam detect failure!, status = " +str(GPIO.input(detect_INNER))
        #quit()
    else:
        print "INNER beam detect - passed :) | Status = "+str(GPIO.input(detect_INNER))
        print "============================================"
        print ""

def status2():

    print "============================================"
    print "OUTER_IsWhole = "+str(OUTER_IsWhole)
    print "OUTER_WasBroken = "+str(OUTER_WasBroken)
    print ""
    print "INNER_IsWhole = "+str(INNER_IsWhole)
    print "INNER_WasBroken = "+str(INNER_WasBroken)
    print "============================================"
    print ""

checkStatus()

# (x,y)
#  x=beam   (1=Outer,2=inner)
#  y=state  (1=Whole,0=Broken)

while (True):
    OUTER_IsWhole = (GPIO.input(detect_OUTER) == 0)  #read current state of beam
    INNER_IsWhole = (GPIO.input(detect_INNER) == 0)  #read current state of beam
    #print ""
    #print "INNER BeamIsWhole = " + str(INNER_IsWhole)
    #print "OUTER BeamIsWhole = " + str(OUTER_IsWhole)

    sleep(0.05)

    if (not OUTER_IsWhole and not OUTER_WasBroken): #if OUTER beam is broken [FALSE], and OUTER_WasBroken=FALSE (ie default value)
        #GPIO.output(greenLED, 1) #greenLED output to HIGH
        OUTER_WasBroken = True
        print "(OUTER,Broken)"+ getFormattedTime()
        status2()
        #logEntranceEvent(1,0)
        logEntranceEventPostgres(1,0)

    if (OUTER_IsWhole and OUTER_WasBroken): #if Outer beam is whole [TRUE] and OUTER_WasBroken=TRUE
        #GPIO.output(greenLED, 0) #greenLED output to LOW
        OUTER_WasBroken = False
        print "(OUTER,Whole)"+ getFormattedTime()
        status2()
        #logEntranceEvent(1,1)
        logEntranceEventPostgres(1,1)

    if (not INNER_IsWhole and not INNER_WasBroken): #if INNER beam is broken [FALSE], and INNER_WasBroken=FALSE (ie default value)
        #GPIO.output(redLED, 1) #redLED output to HIGH
        INNER_WasBroken = True
        print "(INNER,Broken)"+ getFormattedTime()
        status2()
        #logEntranceEvent(2,0)
        logEntranceEventPostgres(2,0)

    if (INNER_IsWhole and INNER_WasBroken): #if INNER beam is whole [TRUE] and INNER_WasBroken=TRUE
        #GPIO.output(redLED, 0) #redLED output to LOW
        INNER_WasBroken = False
        print "(INNER,Whole)"+ getFormattedTime()
        status2()
        #logEntranceEvent(2,1)
        logEntranceEventPostgres(2,1)

GPIO.cleanup()

To make this script run in the background, add an entry to /etc/rc.local as follows:

# open a text editor in the command window
sudo nano /etc/rc.local

# after this section in the rc.local file 
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi

# add the following
python /home/pi/zerocam6/HoleSensor.py &

Birdsy - Monitor wildlife and auto classify bird species

I recently added a wifi IP camera to add to my collection of home-brew wildlife monitoring kit, kindly supplied by Birdsy: https://www.birdsy.com/ .  This IP camera livestreams to the 'cloud' where artificial intelligence (AI) software is used to classify the bird species.  Video clips are saved to your own secure section of the Birdsy website...

Here is a screenshot of some recent captures off my Birdsy webpage:

A selection of captured, species classified clips as presented on the Birdsy webpage

You can that the bird AI is fairly good, it struggles with non-avian species, but given the company is called 'Birdsy' we can let that go...

As with many internet CCTV streaming applications you can review your live camera from anywhere with internet access and a web browser or via a dedicated phone app.  I can now watch my camera live at work too now...

My Birdsy camera setup

The 'value add' is its auto species classification, which is prett neat.  This is the first Goldfinch it classified, I downloaded it direct from the Birdsy website and uploaded to YouTube:

A mouse or two regularly compete with a Robin, the most frequently classified visitor.

As most people will set this up outdoors, you'll need a decent outdoors wifi connection, and access to a power socket.  I've run a 10m DC extension cable to reach the nearest plug socket (amazon link here).  I already have good wifi coverage outside for half a dozen or so other home-brew cameras.

10m 12v extension cable

The Birdsy camera is configured fairly easily via a smartphone/tablet app. You can dial up or down the resolution of the videa stream, presumably to adjust for better/poorer  wifi signals.

It *is* possible to run this with a dedicated wired ethernet connection, this would require either an additional network cable to be run, or once PoE (power over ethernet) setup with the network and power split out (power dropped to 12v) at the camera end.

Here is a screenshot of the configuration app:

IP camera vs home brew
Until recently, all my nature-watching has been with home-made setups using a Raspberry Pi as a processing unit, with either an attached webcam or raspberry pi camera module (or both).  A cursory glance around the web will find many people using IP cameras like this one from Birdsy instead... so what's the difference?

An IP camera, or 'Internet Protocol' camera is an all-in-one device that connects to a wired or wireless network and generates a video stream.  Camera configuration is usually achieved via a smartphone/tablet app or web browser interface.  The camera itself usually does not deal with video storage or motion capture (I'm oversimplifying, as some do bits of both).  They exist to provide a video 'stream' to some other application, e.g. CCTV motion capture software, and/or dedicated storage device, eg a digital video recorder/DVR.  IP cameras often have built in day/night modes, infra-red cut switching (IR cut) and often an integrated microphone.  They also tend to be expensive.

Most of my home made kit is Raspberry pi minicomputer with attached camera module (like the camera bit from your mobile phone).  All of the functionality mentioned above is possible but require fiddly configuration, the addition of extra components, e.g.illumination, IR cut, microphone and some programming (depending on your aims this may be minimal).  While more fiddly and requiring more technical insight, this approach costs a lot less to setup.

The cool thing about the Birdsy camera, isn't the camera, its the cloud-based image recognition system that categorises the bird video clips based on bird species being filmed.  Given that I get a lot of non-avian species I'm hoping that the AI algorithm will be extended to include non-avian species.  I'm sure there are research applications to auto identification of bird species.

As this is the first dedicated IP camera I've had to play with I plan on investigating other creative things than can be done with the video stream, so more at a later date..

Mini HDMI cable camera for nests in awkward places

This camera setup is designed to go into small spaces where my other setups would not fit

Its basically the same setup I use in my birdboxes, with a Raspberry Pi + Raspberry Pi v2 IR camera module + IR cut + some IR LEDs, with the addition of an adaptor which allows you to turn an HDMI cable into a camera cable extension.  I used a 3 meter HDMI cable (Amazon basics) as I wanted to get the camera into an otherwise inaccessible bit of the shed where a Treecreeper had been building a nest.

The HDMI cable adapter with the camera attached has a couple of spare cables in it, so I was also able to power four IR leds and switch an IR cut via the one HDMI cable.  The IR cut is the round black thing on top of the Raspberry Pi camera board that switches it from visible to IR light sensitive modes.

If I was to do this again I would add a few white LEDs that would switch on when the IR leds were off (it's possible do the IR on/vis off and vice versa  by using the same GPIO pin with NPN and PNP transistor combined).  As-is, having the IR cut is a bit superfluous since without the IR light it's pretty dark in there, and a 'day mode' doesn't see much without any visible light illumination - one for a future post (when I've modified it!).

You can get the HDMI adaptor direct from it's French maker via Tindie, or via UK all-round supplier of shiny must-haves Pimoroni (hint it's a bit cheaper to go direct to Tindie).

Here is a screen capture of the Treecreeper nest

The camera cable is poked through a gap in the shed structure into a void space created by the corner cladding on the outside.

Here is the bird bringing nesting material in...

This gives some idea of the length of the cable

I sort of threw together the perfboard sitting on top, so don't look too hard...  I've also used a longer than normal camera ribbon cable so that it comfortably reaches the HDMI adaptor.

I've enlarged it below to give show how the Pi end of the HDMI adaptor works.

Unfortunately our Treecreeper didn't get as far as egg laying, I can only assume that she found a preferable site elsewhere.  I do wonder if the bench saw in the shed put her off.... But it does mean that I've now got a mini camera for tricky places for next year...

Birdbox camera dashboard now with live weather data

I added a live weather widget to my grafana birdbox camera monitoring dashboard, in case I can't be bothered to look out of the window.

Dark Sky weather data 

I did an earlier post that described getting my camera feeds into this setup here.
I used info on Michael Green's blog post here as inspiration for my setup.

This is the updated version, not a lot going on apart from the wind rattling the windows...

The weather bit is in the top left corner, it pulls weather data from an online weather visualisation app called Dark Sky.  It's added via a short piece of modifiable html code into an iframe panel on the grafana dashboard that runs in a docker container on a Raspberry pi.  It was a bit of a pain to get running , so I'm going to document it in case anyone else wants to do this.
Note that the 'accelerometer' style graphs are populated by a temperature and humidity sensor in one of the bird boxes themselves.

Steps / considerations
I'm running grafana version v6.3.6 in a docker container on a Raspberry pi.  If you go to the grafana website you'll see a darksky plugin that can also be used... I didn't go that route, instead opted for (what I thought) would be simpler, namely adding a short bit of html to a text panel in html mode (see my original post on this) , this proved not to be the case...  The iframe embed method is described in this helpful blog post.

In theory, you just add the iframe html into a new grafana panel and set it to txt/html mode.
It's quite configurable, e.g. the colour of the text and background/font used and temperature units used - I'm in the UK so have set it to degrees celsius.
You'll need to substitute the XXXX and YYYY for your latitude and longitude so that the weather data is relevant to your location.  I sourced my location via this handy postcode to lat/long converter

<iframe id="forecast_embed" frameborder="0" height="245" width="100%" bgColor="transparent" src="//forecast.io/embed/#lat=XXXX&lon=YYYY&units
=uk&color=#ced6cb&text-color=#ced6cb"></iframe>

The fiddly bit.

So in theory its as simple as dropping in a bit of html - no it isnt...
There's a configuration option that needs to be changed, otherwise an ifram wont render, and you just get a string of text.  In the grafana server admin screen, if you scroll down to the [panels] section there is a setting called 'disable_sanitise_html = false', this needs to be changed to ' = true' or your iframe wont load.  You would think that changing it directly here would be possible?  ...but it isn't

Note - I've updated it in my version.  the default is 'false'

How to fix it depends on how you're running garafana.  If you've installed grafana locally, then you just need to directly edit the grafani.ini file, which is located in /etc/grafana/grafana.ini.

In my case I've set it up in a docker container which makes editing the file a bit more tricky- editing it directly does not work either but I'll describe the process as its handy to know how to connect to a running docker container and modify ts contents:

SSH into the Rassberry pi running Docker, and by definition your grafana container
Find the container ID of the grafana container

docker container ls -a

Connect to the container as root

docker exec -it <container-name> bash

To edit the grafana.ini file, you'll need to install a txt editor inside the docker container first

apt-get update
apt-get install nano

Then edit the file in nano, updating the disable_sanitise_html = false to true

nano /etc/grafana/grafana.ini

Save the file, exit the container.

SO.. that should fix it?  WRONG.  The config screen in the web interface as described above will still show this setting as = false.  Loads of forums recommend restarting the docker container, which has not effect either.  Running the command 'service grafana-server restart' within the docker container also has not effect.

The solution..
The trick is to pass 'disable_sanitise_html = true' to a docker image as an environmental variable at the point that a docker image is spun up into a container.  To make this work, I committed my existing grafana container back to an image file (this is essentially making a backup/snapshot of a container),  then re-launched the new local image file with the environmental variable added:

#run newly saved image file with environmental variable set
docker run -d -p 3000:3000 -e GF_PANELS_DISABLE_SANITIZE_HTML=true  grafana:1_Nov

Note that I wanted to keep my modification to the stock docker image for my modified version of grafana.  If you want to apply it to the stock grafana image on dockerhub.com then use this command to spin up a new container:

#or add environmental variable option to the image on docker hub:
docker run -d --name=grafana -p 3000:3000 -e GF_PANELS_DISABLE_SANITIZE_HTML=true grafana/grafana:6.3.6

I've squished the DarkSky iframe down a bit on my dashboard, if you make it bigger within its grafana panel, you get this which has some pretty cool animated effects too

Birdbox camera dashboard + environmental monitoring

I made a CCTV-type monitoring screen to present multiple birdbox cameras together + environmental monitoring data, using a Grafana / Docker container on a Raspberry pi...

The screenshot below shows all five of my currently active bird boxes.  The foot in the top Left box  belongs to a blue tit that spends quite a lot of time in there the moment (roosts too).  The temperature graph and widgets are populated by a temp + humidity sensor in the bottom R box that logs to its own internal environment every minute - the data really is an in-box box environment monitor.

EDIT Nov 2019: New blog post describing addition of web-sourced weather data to grafana

Prerequisites.. quite a lot of groundwork.  This project relies on various stuff already existing:

(1) Multiple birdbox camera streams
Most of mine use Raspberry Pi ZeroW mini computers + v2 camera modules running motion capture & streaming software called pikrellcam.  The top L one is a motion jpeg stream from a webcam on an early first raspberry pi model that has been going strong since 2014, the camera window could do with a clean though, our early evening roosting blue tit is present in this one:

(2) Some environmental data.
The aim is to present environmental data relevant to to the camera feeds.  The bottom R bird box has a sensor that logs its internal temp & humidity every minute to a mysql database, and is used to populate the temperature plot:

Temperature plot over time, current humidity and temps also showing

Equally there are plenty of free sources of weather data that could be used to provide a feed - it would be fun to see what the weather is locally too.

(3) A 'video dashboard'
The main point of this post.  The dashboard runs on the newest model from the Raspberry pi foundation, the Model 4B (4Gb ram version).  This has faster wired networking, more ram available, and generally more processing ooph.  I would NOT advocate putting one in a bird box unless you're planning on keeping roosting birds warm over the winter.  Mine sits 'headless' (ie with no monitor/keyboard etc) in the lounge behind the TV.  I've set it up with fixed IP address (wired, not wireless), and is accessed via SSH from a laptop.  I'm not going into detail for that as there are countless 'how-to's' out there.  Lets call this machine 'Hub-Pi'.

Hub-Pi setup as follows

Docker
Docker provides a seamless(ish) way of running 'OS-level virtualization software in packages called containers'.  Basically there are a lot of freely available pre-configured software 'images' that allow rapid deployment of software without having to fiddle about with setup.  I've used a Docker image of a dashboard application called Grafana.

Install Docker on Hub-Pi:  I used the guide on this site, up to the bit about 'swarming' (ignore from then).

Docker works by first downloading a specified image.  It then creates a live 'container' from that which is what you 'do stuff with'.  The 'docker run...' command specifies how the resulting container is configured.  If a container is deleted then all configuration data within the container is lost and you need to re-create a container from the original image and loose all your work in the process.  You can 'commit' a container back to a fresh image at any time,  again Google is your friend.

Install Grafana Docker image: Docker images are supposed to run on any hardware running Docker... sort of.  A base image is built with respect to the architecture of the CPU of the machine it runs on.  This basically means that Raspberry Pis are different to PCs (Arm vs  AMD64).  I'm not totally clear on the difference, but an image built for one may not work on the other.  Many images are built for both CPU architecture and Docker intuitively picks the right one to run.  Mostly.

On writing this, the most recent Grafana image is 6.4.3.  The Arm (is for Raspberry pi) version is somehow broken, so I had to revert back to version 6.3.6, which runs fine.  The Docker command to pull down the right Grafana image and create a Docker container that runs it is...

docker run -d --name=grafana -p 3000:3000 grafana/grafana:6.3.6

If you omit the :6.3.6 bit you'll get the current image which may be fixed in a future version, but didi not work for me.

This command is saying:

1. Pull down the v6.3.6 of the Grafana image from DOckerHub.
2. Run it (make container) and forward port 3000 to the host computers port 3000.
3. Assuming the container runs, if you navigate to the Pi's IP address in a web browser that you fixed earlier, using port 3000 (http://HubPiIPaddressHere:3000), if its connected to a screen & keyboard use localhost:3000 directly then you get to the Grafana main login screen where you can assign a new password

Addition of camera feeds to Grafana
To make a camera feed in Grafana, got to..
Add Panel, then choose 'Text' from the visualisation options.
Based on the two types of video feds that I currently have, I edited the text in html mode as shown below.

My network video streams are of two flavours:
(1) Using webcams to generate a motion JPEG (MJPEG) stream, add the following to the html:
<img src="http://XXX.XXX.X.XX:YYY/?action=stream">
where XXX.XXX.X.XX is the IP address of the remote source and YYYY is the port it streams from

(2) Using Raspberry Pi v2 camera modules and pikrellcam software, add the following to the html:
<a href="http://XXX.XXX.X.XX" target="_blank"/a>
<img src="http://XXX.XXX.X.XX/mjpeg_stream.php"">

Where XXX.XXX.X.XX -  the IP address of the remote Pi running pikrellcam.  This also adds a convenient hyperlink to the video in the dashboard to the main pikrellcam page for that camera where motion captured video can be reviewed.

Once a video feed is added you can manually drag it about and resize it to fit your screen. Simple.

Potential developments...
This is an early version.  I plan to add in local weather data, e.g. from https://www.wunderground.com/.  I have entrance hole activity counter on one of my boxes (soon three), so it would be good to see a plot of activity associated with each video feed too.

UPDATE Nov 2019: see a new post which discribes the addition of a live weather feed
https://nestboxtech.blogspot.com/2019/11/birdbox-camera-dashboard-now-with-live.html

Bird box activity counter v3: Build, detect & log activity

Update Dec 2019:  see part 2 of this series that deals with logging entrance hole activity to a postgres database and text file:

I've built entrance hole detectors into two existing bird boxes, lets call those counter versions 1 + 2.
This year, the v.2 counter box had two consecutive Great Tit nesting sessions, 8 chicks fledgling in total (7 and 1).  With the data it records, it can generate informative visuals like this one that shows daily counts of 'in events' for nesting sessions 1 and 2.  Can you work out which was the more successful 2019 brood?

Nest box activity for two consecutive nesting sessions in Spring 2019

I re-visited this from an earlier version (v1), this post describes version 3.  It uses the same principle as a commercial product from Schwegler that displays a local count - their product doesn't do anything else and will set you back approx £65.

Commercial counter displayed on the of the box front

My detector uses a pair of infrared (IR) beams & detectors that are offset from each other giving an 'outer' and an 'inner' beam.  A bird coming in will break the outer beam before the inner one and vice versa for a bird exiting the box.  Beam disruption events are logged to a text file on the integral  Raspberry pi Zero W which also runs video motion capture software, controls the lighting etc.

By having two detectors, its possible to differentiate between 'in' vs 'out' events.  Sensor noise such as a bird popping its head in from the outside or a spider jumping up and down on one IR LED can be ignored.  This is better than a 'one beam' approach that would produce noisy data that would be impossible to clean.

My design is an evolved version of a project in the 'Raspberry Pi projects' book by Robinson & Cook.  Don't buy the book for this project as it uses obsolete hardware (something called a 'PiFace', I used that in my v1 counter from 2014), the python code is chock full of mistakes, the code also isn't available to download on the publisher's website.

I reckon my v3 version is simpler to do and I've fixed the code side of things too 😏

Parts / equipment
2x 5mm IR LEDs buy here
2x IR phototransistor QSE113 buy here
Some 0.25 watt resistors: 1x 100ohm, 2x 1k ohm, 2x 10k ohm
Small piece of stripboard (grandly(!) referred to as the 'interface board' below)
1.6mm / 2.4mm heat shrink tubing
Wire.. I buy one roll of black 2 core firework shooting wire every few years, and a box of ethernet cable provides virtually endless colour-coded twisted pairs of low gauge wire.
Raspberry pi Zero W + power source

For the box:
Plywood (front of bird box): 1x 18mm piece and 2x 3mm pieces
30mm flat drill bit for the entrance hole
Router with a narrow cutter for the cable runs, e.g. this 3.2mm one.

The idea is to create a plywood sandwich for the v3 counter, with the 18mm ply between two 3mm pieces.  In contrast, v1 and v2 used more layers of plywood and was more fiddly to make.

How to make this....
Start off by clamping the three plywood pieces together.  Drill the entrance hole through all three ( I use a 30mm hole) then put the inner and outer 3mm plywood pieces to one side.
Drill 4 small 'guide holes' through the inner 18mm piece only, in approx a square (ish) arrangement around the entrance hole.  Use these as guide to route a diagonal channel on either side.  The channels need to intersect the guide holes   You want an 'X' shape  with one / and \ channel cut on each side, some of these guide holes will also double up as cable conduits.  Cut a couple of cable channels for the LED and phototransducer as shown below.  Its important that the cable channels are not full thickness.

Cut a recessed chamber on the 'outer' face of the 18mm ply piece that is deep enough to fit a small piece of stripboard (no smaller than 8 holes along the top, 10 on the side).  Drill another small hole through the ply in the recess that connects to a channel that runs up to the top of the inner face.  Power and GPIO connections to the Raspberry Pi come out via this route to the Pi in the top of the bird box.

Routing channels for the phototransistor and LEDs is a bit fiddly, here's a closeup of my efforts.  Any over cut/gappy bit around the detector can be filled with blu-tac or wood filler.

Before making the 'Interface board' I prototyped it out to make sure it worked

Breadboard fiddling

The resistors are as follows:
1) 100 ohm for the in-parallel LED circuit (220 ohm may also be okay)
2) 10k ohm pull up resistor to 5V for the transistor collector
3) 1k ohm resistor for the transistor collector GPIO connection

A handy hint for making stuff with IR LEDs: You cant see when they're on, however if you point a digital camera (with a screen on it) at them, then the camera can see it.

The breadboard prototype evolved into this shrunk down version on stripboard:

'Interface' resistor board

I use 2 core cable to wire the phototransistor emitter and collector legs as shown above, and a couple of twisted pair wires from a piece of ethernet cable for the LEDs, using different colour twisted pair wires for each LEDs - this make is easier to tell them apart.  I'm in the habit of using the stripy one of the pair for the + and the solid colour for the - pole (helps avoid soldering stuff in-situ the wrong way around).

Wire the transistors and LEDs cables first, then slot them into their routed channels and solder in-situ to the prepared interface board as shown above.

There are 4 connections to be made coming off the interface board: (1)5v and (2) ground [GND] (3) GPIO_1 for inner beam, (4) GPIO_2 for outer beam connections.  For the 5V and GND I use two core cable again, and coloured twisted pair for the GPIOs - as before, having a stripy one and solid coloured one for the GPIO connections helps you tell which is which.

How you inerface this with your raspberry pi is up to you, these 4 wires can be connected directly to the appropriate GPIO pins.  In my boxes, the 5V and GND will connect direct to the recom switching regulator (R-78B5.0-1.5) that drops the box's 12v feed to 5v. 

In this setup the IR LEDs are 'always on'.  I cant forsee a situation where I may need to turn them off but you could alternatively wire the LEDs via a second GPIO for power and power them on/off programatically.

The holesensor.py python script writes a single line to a log file if one of the beams transitions from broken to whole or vice versa.

The logfile logic is as follows:  BEAM,STATE,event time
Outer beam = 1; Inner beam = 2
Whole = 1; Broken =0

In this excerpt from the log, a bird has come into the box:
Outer beam broken, event_time
Inner beam broken, event_time
Outer beam whole, event_time
Inner beam whole, event_time

### recordBird_v3 starting up at:21 Sep 2019 09:47:35.990
1,0,21 Sep 2019 09:48:04.100
2,0,21 Sep 2019 09:48:04.446
1,1,21 Sep 2019 09:48:05.180
2,1,21 Sep 2019 09:48:05.228

The python script 'holesensor.py' that records this is detailed below.
On my 'to do' list is to modify this to record to a database rather than a static text file. 
Having this recorded to a database opens up the possibility of connecting remotely to this box and directly querying its on-board database.

The 'log to file' version is below.  I'll write up the graphing functions in another post...

import RPi.GPIO as GPIO
from time import sleep
import time
import datetime

# Change log
# 16/09/19 updated for zerocam 7

GPIO.setmode(GPIO.BCM)          #use BCM pin numbering system
GPIO.setwarnings(False)

whichBirdcam = 'zerocam7'

#file used to log entrance actions
EntrancelogFile='/home/pi/testpizero/logs/zerocam7_birdlog.txt'
OpenEntrancelogFile= open(EntrancelogFile,  'a', 0)

#function to return the current time, formatted as
# e.g. 13 Jun 2013 :: 572
def getFormattedTime():
    now = datetime.datetime.now()
    return now.strftime("%d %b %Y %H:%M:%S.") + str(int(round(now.microsecond/1000.0)))

#generate and record an event to file
def logEntranceEvent(sensor, state):
    OpenEntrancelogFile.write(str(sensor) + "," + str(state) + "," + getFormattedTime() + "\n")


#setup GPIOs

detect_OUTER = 22 #6    #set GPIO pin for Outer photransducer (input)
detect_INNER = 23 #12    #set GPIO pin for Inner photransducer (input)

print 'detect_OUTER = ' + str(detect_OUTER)
print 'detect_INNER = ' + str(detect_INNER)

#Constants
#OUTER_BEAM = 1
#INNER_BEAM = 2

#WHOLE = 1
#BROKEN = 0

# setup GPIO pins:
GPIO.setup(detect_OUTER, GPIO.IN)   #set Outer GPIO Phototransducer as input
GPIO.setup(detect_INNER, GPIO.IN)   #set Inner GPIO Phototransducer as input


#indicate the point the program started in the log
OpenEntrancelogFile.write("### recordBird_v3 starting up at:" + getFormattedTime() + "\n")
print "============================================"
print whichBirdcam.upper() + ": Starting up entrance hole counter script..."
sleep (0.5)

#Set initial state of WasBroken for both beams:
OUTER_WasBroken = False
INNER_WasBroken = False

# LED status check
# LEDstate= GPIO.input(detect_INNER)

print ""
print "detect_OUTER status = " + str(GPIO.input(detect_OUTER))
print "detect_INNER status = " + str(GPIO.input(detect_INNER))
print ""


#When the detector          'sees' IR led, the detector pin is 0/LOW/False
#When the detector does not 'see ' IR led, the detector pin is 1/HIGH/True

def checkStatus():
    if GPIO.input(detect_OUTER):  #if OUTER detector does not see IR led, print error, GPIO.input = HIGH
        print "OUTER beam detect failure!, status = " +str(GPIO.input(detect_OUTER))
        #quit()
    else:
        print "OUTER beam detect - passed :) | Status = "+str(GPIO.input(detect_OUTER))

    if GPIO.input(detect_INNER):  #if INNER detector does not see IR led, print error, GPIO.input = HIGH
        print "INNER beam detect failure!, status = " +str(GPIO.input(detect_INNER))
        #quit()
    else:
        print "INNER beam detect - passed :) | Status = "+str(GPIO.input(detect_INNER))
        print "============================================"
        print ""

def status2():

    print "============================================"
    print "OUTER_IsWhole = "+str(OUTER_IsWhole)
    print "OUTER_WasBroken = "+str(OUTER_WasBroken)
    print ""
    print "INNER_IsWhole = "+str(INNER_IsWhole)
    print "INNER_WasBroken = "+str(INNER_WasBroken)
    print "============================================"
    print ""

checkStatus()

# (x,y)
#  x=beam   (1=Outer,2=inner)
#  y=state  (1=Whole,0=Broken)

while (True):
    OUTER_IsWhole = (GPIO.input(detect_OUTER) == 0)  #read current state of beam
    INNER_IsWhole = (GPIO.input(detect_INNER) == 0)  #read current state of beam
    
    sleep(0.01)

    if (not OUTER_IsWhole and not OUTER_WasBroken): #if OUTER beam is broken [FALSE], and OUTER_WasBroken=FALSE (ie default value)
        OUTER_WasBroken = True
        print "(OUTER,Broken)"+ getFormattedTime()
        status2()
        logEntranceEvent(1,0)


    if (OUTER_IsWhole and OUTER_WasBroken): #if Outer beam is whole [TRUE] and OUTER_WasBroken=TRUE
        OUTER_WasBroken = False
        print "(OUTER,Whole)"+ getFormattedTime()
        status2()
        logEntranceEvent(1,1)


    if (not INNER_IsWhole and not INNER_WasBroken): #if INNER beam is broken [FALSE], and INNER_WasBroken=FALSE (ie default value)
        INNER_WasBroken = True
        print "(INNER,Broken)"+ getFormattedTime()
        status2()
        logEntranceEvent(2,0)


    if (INNER_IsWhole and INNER_WasBroken): #if INNER beam is whole [TRUE] and INNER_WasBroken=TRUE
        INNER_WasBroken = False
        print "(INNER,Whole)"+ getFormattedTime()
        status2()
        logEntranceEvent(2,1)

GPIO.cleanup()

Here is a 'production version' of this being setup.  I swapped out the camera unit from an existing box.  You can see the IR leds in this camera - the're invisible to humans/birds

2x entrance hole IR beams - Digital camera view only!