Babysafe is an IoT solution designed to prevent adults from forgetting their babies in the rear seat of their cars.
In Israel, from the start of 2008 and up until mid-2013 the total number of cases where people forgot their babies in the car was 205. This could be easily prevented with today’s technology. Babysafe is our way of doing that.
A Raspberry Pi 3 with a thermal camera (MLX90640) connected to it via I2C is installed in the car. The Raspberry Pi is connected via Bluetooth to an OBDII unit in the car. The Raspberry Pi reads the RPM from the car to know if the car is on or off. If the car is off it looks for a baby in the back seat via the thermal camera. If a baby is detected for a long period of time after the car is turned off, a message is sent to the cloud.
In the cloud we have an IoT Hub which receives the messages from the Raspberry Pi. When a message is received, the IoT Hub forwards it to a logic app in the cloud which reads the device id the message came from and locates the owner of this device to send them a notification to the mobile app.
The mobile app is used to connect the device id with the owner. The registration is sent to a database in the cloud. The app receives push notification when the owner forgets their baby in the car.
We have a Raspberry Pi 3 connected to the internet via WIFI. To the Raspberry Pi we attached a Qwiic IR Array MLX90640 thermal camera to the I2C bus and connected the Raspberry Pi to the On-Board Diagnostics unit in the car via Bluetooth.
On the Raspberry Pi we installed Raspbian OS which is Linux based. We installed .NET Core 2 runtime for Linux, so we’ll be able to run .NET Core applications.
The application was developed in C# on Windows, compiled and published for Linux and ARM architecture. Then we use the .NET Core runtime we previously installed on the Raspberry Pi to run the application.
The application starts with the car engine. It keeps checking the car status every two minutes and sends a message to the cloud with the status. If at some point, we recognize that the car is off we immediately look for the baby. If no baby is detected, we send the data to the cloud and shutdown. Otherwise we wait another two minutes to give the parent some time between turning off their car and getting the baby and then we look again for it. After that we send a message to the cloud indicating the situation and we shutdown.
In the cloud we have an IoT Hub configured to receive and send messages from and to devices. We used the SDK provided by Microsoft to create an instance of the hub in our code and used it to send the messages to the cloud.
Each message is a json with two fields and an additional property. The two fields are the car status which can be on or off and a presence field which can be true or false depending on if there’s a baby in the car or not, or null if this information is not available.
The additional property indicates that there’s a baby forgotten in the car. This property is sent only if the car is off and the baby was and still in the car.
The camera is used to detect the baby presence in the car. We took a C++ application that is built over the C++ driver provided by the makers of the camera and changed it to suit our need. The result was an application that initializes the camera and captures an image. The image is stored in a text file with specific format.
In the main application we created an image class. The class holds a matrix of pixels, each pixel is represented by the RGB color. Then we developed a helper class that can take the output of the C++ application and create an object of type image. In the image class we wrote a function that can recognize a presence in a given image. It counts the number of adjacent red (hue between 0 and 60) pixels in the image and if it exceeds a specific threshold (which was set by trying a lot of images and situations) then it positively identifies a presence in the image. We also added a functionality for the helper class to create jpeg images from the image object. This was used for testing and to help us understand what we see.
The main application holds a process that runs the C++ application. Each time we need to take an image the process is called and with the helper class we create the image object, then we ask the object if there’s a presence in the image.
The On-Board Diagnostics unit is plugged in the car. It communicates with the Raspberry Pi over Bluetooth. We used a Python application that can read and write data to the OBD to read the RPM of the engine.
When the car is off the RPM is zero and when the engine is running, and the car is parked it indicates around 700 revolutions per minute. We set the threshold to zero. When the Python application runs it initializes the connection with the OBD and then sends the command to read the RPM.
In the main application we hold a process that can run the Python application and each time we need to read the RPM we run the process and look for the result in the output. We parse the output and return car status on if the result is greater than zero and off otherwise.
We have six resources in the cloud: Azure IoT Hub, Azure Service Bus, Azure Logic App, Azure Function, Azure Database and Firebase Cloud Messaging service.
The IoT Hub was used to read messages sent from the Raspberry Pi to the cloud. It can filter the messages received by specific properties. We used this ability to filter messages that need action from our side, these are the messages which said that someone left her or his baby in the car. Said messages were inserted to a queue for further processing.
Azure Service Bus holds a queue that stores messages from the IoT Hub for further processing.
We created a logic app that is triggered by items inserted to a queue and configured it to listen to the queue we used in the IoT Hub to store the messages. When a message is inserted, the logic app flow is started.
The flow consists of steps. The first step is to parse the message receive and extract the device id that sent it. Then we send the device id to function in the Azure Database that return the FCM (Firebase Cloud Messaging) token saved for this device. Then we create an HTTP Rest call to the FCM service with said token and this sends a notification to the owner of this device.
The Azure Database was used to store FCM tokens of each device we have. In the database we have a stored procedure the takes a device id as input and finds a record in the database with this device id. This was used by the logic app as we explained earlier to find the token of the owner of this device.
The Azure Functions we developed was used in the mobile app. We send an HTTP Rest call with the device id and the token. The function connects to the Azure Database and inserts or updates the token for the specified device id.
The Firebase Cloud Messaging or FCM is a service that can send push notifications to mobile apps. It’s owned by Google. It can send a broadcast or a specific user notification. Each user is assigned an FCM token that is used to send them specific notifications. The token is assigned when the user connects to the service and is refreshed periodically.
We developed a mobile app that can receive push notifications. The app was developed for Android in C# and Xamarin. The notifications were sent via the Firebase Cloud Messaging service by Google. The app has a login screen which is used to register the device id with token generated by the FCM service for the user. The user is required to sign in with device id as username.
When the user logs in, a Rest call is made to the Azure Function we have with the device id and the FCM token to store them in the Azure Database. Each time the token is refreshed the same call is made again.
Microsoft offer Windows 10 IoT which can be installed on the raspberry Pi. Using Visual Studio, you can develop applications of type Universal Windows App or UWP. This can be easily deployed remotely to the Raspberry Pi via Visual Studio. This is easy because you develop in C# and you can remote debug the code. On the other hand, Raspbian is a Linux based OS developed specifically for Raspberry Pi, which has its own advantages.
In the first take we chose Windows. Unfortunately, the one-click deployment didn’t work. After we couldn’t fix the problem we switched to Raspbian. In Linux we tried to get the samples of the languages provided by Microsoft for the IoT Hub SDK to run but each one had a different issue. The only sample we were able to run was the one using .NET Core on Windows. So, we reinstalled Windows 10 IoT and determined to solve the deployment problem.
When we finally solved the problem turns out the SDK for UWP throws an “Unauthorized Exception” except we already were able to run the .NET Core SDK with the same credentials. One solution is develop a .NET Core application and deploy manually but that means transferring the files manually to the Raspberry Pi, and there was no easy way of doing that other than removing the SD card from the Raspberry Pi and inserting it to the laptop and transfer the files from the desktop we are developing on to the laptop and then copying them to the SD card.
In Linux we can use FTP and SSH which enables easy communication with the Raspberry Pi but how can we run a .NET Core application on Linux? Turn out Microsoft offers .NET Core runtime that can do that. So, we reinstalled Raspbian and the .NET Core runtime on the Raspberry Pi. We used PowerShell on the developing machine to publish the code with all the required dependencies and transferred the files via FTP client. Using SSH we were able to run the application and the Azure IoT Hub SDK worked as expected.
The device manufacturer offered a driver for it in C++ and a sample. The problem is they used Arduino native libraries. Fortunately, someone had opened an issue for the manufacturer regarding using the device with Raspberry Pi and some other guy provided a working sample with Raspberry Pi libraries.
The sample he provided was in C++, it initializes the device and takes images every specific period. We wanted to use this as an external library and write an API for it. We wanted to make C# wrappers for this C++ code, but we couldn’t accomplish that because the program used Raspberry Pi specific libraries which could only be compiled in Linux and to use this as a library in .NET code we needed it as a Dynamically Linked Library (.dll). Linux compiles it to a (.a) file.
We tried to find a way to convert the (.a) file to a (.dll) one but this is not possible. So, we tried a different approach. This time we changed the program to initialize the device, take a picture and save the output to a file and compiled to an executable file (in Linux). Then we created a process in .NET that takes the path to the executable and runs it on the machine.
In the .NET application we read the output of the process and parse it to verify the execution was as expected. Then we process the output file that was created.
When we wanted to connect to the OBD unit in the car, we looked up how to connect to Bluetooth devices in C#. We found a way to do that, but it used Windows API, and .NET Core doesn’t know Windows API.
After what we went through with the MLX90640, we thought why not find something that can run on Linux and do the same as we did with the camera. So, we found a Python program that connects to the OBD and submits commands to it. We changed the program to initialize the Bluetooth connection and send a command to read the RPM then print the output in a way that was convenient for us to read.
In our main application we again created a process for the Python program and each time we wanted to read the RPM we parse the output and extract the result.
Running a process in C# is not hard. You create an object holding the information of the process you want to run and call Run(). But we also wanted to read the output of the process. There’s a method for that, ReadToEnd(). The problem is each time we reached this method the program was stuck.
Researching this showed that the method used a buffer to store the output it reads. The buffer has fixed length and our C++ program had a very large output. So, every time the buffer is filled the process can’t continue to read and the program is stuck because ReadToEnd() can’t continue.
The solution was to empty the buffer as ReadToEnd() fills it. This way the process can continue to insert data to it and we receive the output a bit by bit.
Neither one of us had any experience with push notifications before. Turns out you can’t do that by your own and you need a service like Google Cloud Messaging. Such services offer you a couple of functions that you override in your mobile app that handle creating the instance of the service in the app and deciding what to do when a message is received.
The two options we had were Google Cloud Messaging and Firebase Cloud Messaging, both are owned by Google. We tried following all the tutorials for both service and for C#-Xamarin and Java applications. Each guide offered a different insight. But all of them were used to send a broadcast and we needed a specific user notification.
Even doing the broadcast notification wasn’t so easy, and we had our share of issues with it. Finally, we got it to work with Java and FCM. Then we as C# enthusiasts wanted to use Xamarin so we switched to C# and it worked there as well.
After that we had to read the documentation for the FCM API and figure out how to send a notification to a specific user, that’s when we discovered the token issue and knew we had to store a token for each user and update it each time it gets refreshed.
We finally figured it out and were able to send a push notification to specific users, but the arriving notifications were silent. A little bit of Google helped with that.