Thursday 26 December 2019

BirdBox entrance counter v3: postgres / docker / python

This post describes the software side of my v3.0 birdbox entrance hole implementation, refer to part 1 related post  for the physical build details.  To log activity, I used a basic Postgres database, which (to continue an ongoing theme) is run in a docker container to simplify setup.  I updated the python logging script to log to a database rather than a text file, and corrected some previous errors along the way...

The existing birdbox setup remains the same, its a Raspberry Pi ZeroW running Pikrellcam for motion detection and auto capture.  The LED illumination, IR cut and RasPi IR camera module etc remain unchanged.

Like before.. but with entrance hole activity logged to a database

v3.0 entrance hole counter updates...
The two other boxes that I have 'in the wild' with entrance hole counters log partial and full 'in' vs 'out' events activity to a text file.  This updates (v3.0) to a Postgres database instead.  Using a database rather than a text file opens up more possibilities, in particular allowing collected data to be accessed remotely.  I plan to pull data off for analysis on a separate machine, for example to populate activity plots on my grafana nestbox CCTV-sytle dashboard.  Having the activity data sitting in a database makes this much easier.  I've used a 'dockerised' version of postgres as its simpler to setup than installing it from scratch.

This post will cover
1) Docker Installation
2) Docker-Compose Installation
3) Add Postgres container to  Docker installation
4) Creation of a database to log activity
5) The activity logging script itself

I use a base image of Raspbian 'Lite' to start off.  This takes up less space on the SD card but may need more things installing along the way than than 'full-fat' version.  You can follow these setup steps on a Raspberry Pi ZeroW, but be prepared to wait a long time for some of them (especially for Docker Compose).  I use a spare Pi 3 or 4 for the build (much quicker) and transfer the sd card over to a Zero W when its done.  The installtion Pi is assumed to be networked. There's about a zillion how-tos online for that.

1) Docker installation

Think of Docker as an empty applications box.  Once Docker is running there are many many pre-configured applications than can be (fairly) effortlessly run without having to do lots of fiddly setup...

This is mostly lifted from these sites, with my comments added:

Install commands (I left my hashed-out comments in)
sudo apt-get install apt-transport-https ca-certificates software-properties-common -y
curl -fsSL -o && sh
sudo groupadd docker  #not actually required, groupadd: group 'docker' already exists
sudo gpasswd -a $USER docker
newgrp docker #avoids reboot
docker run hello-world  #test that it works

If it all works, you get the following message:
Hello from Docker!
This message shows that your installation appears to be working correctly. 
Run Postgres on Docker
Details here: and

2) Docker-Compose Installation

Install proper dependencies:
sudo apt-get install libffi-dev libssl-dev
sudo apt-get install -y python python-pip  #?had issue with python 2.7.13 not being compatible with the version of docker-compose below
sudo apt-get install -y python3 python3-pip

sudo apt-get remove python-configparser

Install Docker Compose

#sudo pip install docker-compose #takes a long time on a Pi ZeroW
# edit 12/2/21
sudo pip3 install docker-compose
#see details here

3) Add a Postgres database container to Docker:

Note, MySQL is another option, there are MySQL images available on Docker, or maybe install from scratch.  I went with Postgres as I like pgAdmin as a sql interpretor.

# clone the repo which contains the Compose file: 
git clone
# Up the container:  
cd Raspberry-Pi-PostGres-Docker-Compose
sudo docker-compose up --build -d
# note this returns an error for a file it cant find:
# ERROR: Couldn't find env file: /home/pi/Raspberry-Pi-PostGres-Docker-Compose/.env
# fix this by making a dummy file first:
touch .env  
# then re-run
sudo docker-compose up --build -d

All being well, postgres database should be running in a docker container on the host Pi, on the default port (5432).   As-is the default user is postgres, password is password@7979  You can change these in the docker-compose.yml.

pgAdmin is a handy sql interpretor/tool for managing local or remote postgres databases.  Install it on a networked PC and you should be able to remotely connect to the postgres instance running on the Pi created above.  See  Google for loads of tutorials around this bit.

4) Creation of a database to log activity

Within pgAdmin (on a remote PC), this sql command should make the necessary database and table within it

    OWNER = postgres
    LC_COLLATE = 'C.UTF-8'
    LC_CTYPE = 'C.UTF-8'
    TABLESPACE = pg_default

CREATE TABLE public.entrance_log
    sensor integer,
    state integer,
    event_time timestamp without time zone
TABLESPACE pg_default;

ALTER TABLE public.entrance_log
    OWNER to postgres; 

I'm starting basic, so there's only only one table, for the entrance hole activity.

5) The activity logging script

This is written in Python.  Need to install some extra python packages first:
#install the postgres/sql connector psycopg2
pip install psycopg2  #it does not work properly, need to run the following command first:
sudo apt-get install libpq-dev #(suggestion here:

The following python script creates a entry in the db_activity database in a table called entrance_log.

A simple bird  'head bob in' event breaks the outer beam[1,0], then the inner beam [2,0],  then withdraws: Inner beam whole again [2,1], then outer beam whole [1,1].
Note that in this example, Outer beam = 1, Inner beam = 2, Broken = 0, Whole = 1

Database entry
This above event as recorded in postgres database

I added some error checking code to stop the script from crashing out if a beam is triggered and it cant see the postgres database for whatever reason - it will carry on merrily logging to file instead.
I have left in the 'log to file' functionality as a backup, I'll may remove that eventually, its there as a backup.

The above event as logged to a txt file
Input comes from two GPIO input pins from the entrance hole sensor: pin 22 (outer beam) and pin 23 (inner beam).  Other pins are available....
import RPi.GPIO as GPIO

from time import sleep

#postgres error handling from here:
#postgres add record adapted from here:

# import sys to get more detailed Python exception info
import sys
# import the connect library for psycopg2
from psycopg2 import connect
# import the error handling libraries for psycopg2
from psycopg2 import OperationalError, errorcodes, errors
# import the psycopg2 library's __version__ string
from psycopg2 import __version__ as psycopg2_version

from datetime import datetime

# Change log
# 16/09/19 updated for zerocam 7
# 14/12/19  modified to write activity to a local postgres database, database implemented in docker
#           postgres error handler added so the script does not crash out if the database isnt available for whatever reason

GPIO.setmode(GPIO.BCM)          #use BCM pin numbering system

whichBirdcam = 'zerocam6'

#file used to log entrance actions
OpenEntrancelogFile= open(EntrancelogFile,  'a', 0)

#function to return the current time, formatted as
# e.g. 13 Jun 2013 :: 572
def getFormattedTime():
    now =
    return now.strftime("%d %b %Y %H:%M:%S.") + str(int(round(now.microsecond/1000.0)))
    #note that this does not work correctly, as rounding 040 will give 40, but this modified function is only meant as a backup for the local file log

# ********* postgres stuff

# define a function that handles and parses psycopg2 exceptions
def print_psycopg2_exception(err):
    # get details about the exception
    err_type, err_obj, traceback = sys.exc_info()

    # get the line number when exception occurred
    line_num = traceback.tb_lineno

    # print the connect() error
    print '\n' "psycopg2 ERROR:", err, "on line number:", line_num
    print "psycopg2 traceback:", traceback, "-- type:", err_type

    # psycopg2 extensions.Diagnostics object attribute
    print '\n' "extensions.Diagnostics:", err.diag

    # print the pgcode and pgerror exceptions
    print "pgerror:", err.pgerror
    print "pgcode:", err.pgcode, '\n'

def logEntranceEventPostgres(sensor, state):
    dt =  #set the census datetime for the event


        conn = connect(
            dbname = "db_activity",
            user = "postgres",
            host = "",
            password = "password@7979")

    except OperationalError as err:
        # pass exception to function

        # set the connection to 'None' in case of error
        conn = None

    # if the connection was successful
    if conn != None:

        # declare a cursor object from the connection
        cursor = conn.cursor()
        print "cursor object:", cursor, '\n'

        #dt =
        cursor.execute('INSERT INTO entrance_log (sensor,state,event_time) VALUES (%s,%s,%s)', (sensor,state,dt,))
        count = cursor.rowcount
        print count, "record(s) inserted successfully into entrance_log table"

        # close the cursor object to avoid memory leaks

        # close the connection object also

    #append the same timestamp data to file for good measure
    #formattedTime_old = dt.strftime("%d %b %Y %H:%M:%S.") + str(int(round(dt.microsecond/1000.0)))  #note that this is NOT correct since it 'rounds' a 041ms to 41
    formattedTime = dt.strftime('%d %b %Y %H:%M:%S.%f')[:-3]  #this works correctly
    #write to file until we figure out if the same data is being captured
    OpenEntrancelogFile.write(str(sensor) + "," + str(state) + "," + formattedTime + "\n")

#setup GPIOs
detect_OUTER = 22 #6    #set GPIO pin for Outer photransducer (input)
detect_INNER = 23 #12    #set GPIO pin for Inner photransducer (input)

print 'detect_OUTER = ' + str(detect_OUTER)
print 'detect_INNER = ' + str(detect_INNER)


#WHOLE = 1

# setup GPIO pins:
GPIO.setup(detect_OUTER, GPIO.IN)   #set Outer GPIO Phototransducer as input
GPIO.setup(detect_INNER, GPIO.IN)   #set Inner GPIO Phototransducer as input

#indicate the point the program started in the log
OpenEntrancelogFile.write("### recordBird_v3 starting up at:" + getFormattedTime() + "\n")
print "============================================"
print whichBirdcam.upper() + ": Starting up entrance hole counter script..."
sleep (0.5)

#Set initial state of WasBroken for both beams:
OUTER_WasBroken = False
INNER_WasBroken = False

# LED status check
# LEDstate= GPIO.input(detect_INNER)

print ""
print "detect_OUTER status = " + str(GPIO.input(detect_OUTER))
print "detect_INNER status = " + str(GPIO.input(detect_INNER))
print ""

#When the detector          'sees' IR led, the detector pin is 0/LOW/False
#When the detector does not 'see ' IR led, the detector pin is 1/HIGH/True

def checkStatus():
    if GPIO.input(detect_OUTER):  #if OUTER detector does not see IR led, print error, GPIO.input = HIGH
        print "OUTER beam detect failure!, status = " +str(GPIO.input(detect_OUTER))
        print "OUTER beam detect - passed :) | Status = "+str(GPIO.input(detect_OUTER))

    if GPIO.input(detect_INNER):  #if INNER detector does not see IR led, print error, GPIO.input = HIGH
        print "INNER beam detect failure!, status = " +str(GPIO.input(detect_INNER))
        print "INNER beam detect - passed :) | Status = "+str(GPIO.input(detect_INNER))
        print "============================================"
        print ""

def status2():

    print "============================================"
    print "OUTER_IsWhole = "+str(OUTER_IsWhole)
    print "OUTER_WasBroken = "+str(OUTER_WasBroken)
    print ""
    print "INNER_IsWhole = "+str(INNER_IsWhole)
    print "INNER_WasBroken = "+str(INNER_WasBroken)
    print "============================================"
    print ""


# (x,y)
#  x=beam   (1=Outer,2=inner)
#  y=state  (1=Whole,0=Broken)

while (True):
    OUTER_IsWhole = (GPIO.input(detect_OUTER) == 0)  #read current state of beam
    INNER_IsWhole = (GPIO.input(detect_INNER) == 0)  #read current state of beam
    #print ""
    #print "INNER BeamIsWhole = " + str(INNER_IsWhole)
    #print "OUTER BeamIsWhole = " + str(OUTER_IsWhole)


    if (not OUTER_IsWhole and not OUTER_WasBroken): #if OUTER beam is broken [FALSE], and OUTER_WasBroken=FALSE (ie default value)
        #GPIO.output(greenLED, 1) #greenLED output to HIGH
        OUTER_WasBroken = True
        print "(OUTER,Broken)"+ getFormattedTime()

    if (OUTER_IsWhole and OUTER_WasBroken): #if Outer beam is whole [TRUE] and OUTER_WasBroken=TRUE
        #GPIO.output(greenLED, 0) #greenLED output to LOW
        OUTER_WasBroken = False
        print "(OUTER,Whole)"+ getFormattedTime()

    if (not INNER_IsWhole and not INNER_WasBroken): #if INNER beam is broken [FALSE], and INNER_WasBroken=FALSE (ie default value)
        #GPIO.output(redLED, 1) #redLED output to HIGH
        INNER_WasBroken = True
        print "(INNER,Broken)"+ getFormattedTime()

    if (INNER_IsWhole and INNER_WasBroken): #if INNER beam is whole [TRUE] and INNER_WasBroken=TRUE
        #GPIO.output(redLED, 0) #redLED output to LOW
        INNER_WasBroken = False
        print "(INNER,Whole)"+ getFormattedTime()


To make this script run in the background, add an entry to /etc/rc.local as follows:

# open a text editor in the command window
sudo nano /etc/rc.local

# after this section in the rc.local file 
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"

# add the following
python /home/pi/zerocam6/ &

Saturday 21 December 2019

Birdsy - Monitor wildlife and auto classify bird species

I recently added a wifi IP camera to add to my collection of home-brew wildlife monitoring kit, kindly supplied by Birdsy: .  This IP camera livestreams to the 'cloud' where artificial intelligence (AI) software is used to classify the bird species.  Video clips are saved to your own secure section of the Birdsy website...

Here is a screenshot of some recent captures off my Birdsy webpage:
A selection of captured, species classified clips as presented on the Birdsy webpage
You can that the bird AI is fairly good, it struggles with non-avian species, but given the company is called 'Birdsy' we can let that go...

As with many internet CCTV streaming applications you can review your live camera from anywhere with internet access and a web browser or via a dedicated phone app.  I can now watch my camera live at work too now...

My Birdsy camera setup
The 'value add' is its auto species classification, which is prett neat.  This is the first Goldfinch it classified, I downloaded it direct from the Birdsy website and uploaded to YouTube:

A mouse or two regularly compete with a Robin, the most frequently classified visitor.

As most people will set this up outdoors, you'll need a decent outdoors wifi connection, and access to a power socket.  I've run a 10m DC extension cable to reach the nearest plug socket (amazon link here).  I already have good wifi coverage outside for half a dozen or so other home-brew cameras.

10m 12v extension cable

The Birdsy camera is configured fairly easily via a smartphone/tablet app. You can dial up or down the resolution of the videa stream, presumably to adjust for better/poorer  wifi signals.

It *is* possible to run this with a dedicated wired ethernet connection, this would require either an additional network cable to be run, or once PoE (power over ethernet) setup with the network and power split out (power dropped to 12v) at the camera end.

Here is a screenshot of the configuration app:

IP camera vs home brew
Until recently, all my nature-watching has been with home-made setups using a Raspberry Pi as a processing unit, with either an attached webcam or raspberry pi camera module (or both).  A cursory glance around the web will find many people using IP cameras like this one from Birdsy instead... so what's the difference?

An IP camera, or 'Internet Protocol' camera is an all-in-one device that connects to a wired or wireless network and generates a video stream.  Camera configuration is usually achieved via a smartphone/tablet app or web browser interface.  The camera itself usually does not deal with video storage or motion capture (I'm oversimplifying, as some do bits of both).  They exist to provide a video 'stream' to some other application, e.g. CCTV motion capture software, and/or dedicated storage device, eg a digital video recorder/DVR.  IP cameras often have built in day/night modes, infra-red cut switching (IR cut) and often an integrated microphone.  They also tend to be expensive.

Most of my home made kit is Raspberry pi minicomputer with attached camera module (like the camera bit from your mobile phone).  All of the functionality mentioned above is possible but require fiddly configuration, the addition of extra components, e.g.illumination, IR cut, microphone and some programming (depending on your aims this may be minimal).  While more fiddly and requiring more technical insight, this approach costs a lot less to setup.

The cool thing about the Birdsy camera, isn't the camera, its the cloud-based image recognition system that categorises the bird video clips based on bird species being filmed.  Given that I get a lot of non-avian species I'm hoping that the AI algorithm will be extended to include non-avian species.  I'm sure there are research applications to auto identification of bird species.

As this is the first dedicated IP camera I've had to play with I plan on investigating other creative things than can be done with the video stream, so more at a later date..