Deep learning examples on Raspberry 32/64 OS
On this page, we focus on the software. Ultimately, you should be able to perform the examples we present here yourself. They all written in the C language, one of the fastest computer languages. All use only a Raspberry Pi, no additional hardware is required. We start with an overview. Later, detailed hands-on gives you all information on how to run these examples yourself.
The overview speaks for itself. The highest frame rate measured comes from a Raspberry 64-bit OS overclocked to 1950 MHz. The lowest is the standard 32-bit Raspbian at 1500 MHz. Frame rates are only based on model run time (interpreter->Invoke()). Grabbing and preprocessing of an image are not taken into account, nor plotting output boxes or texts. This way, models working with pictures instead of video streams can also be measured. The actual frame rate will be slightly lower when playing video's or using a camera.
Regularly, we get the question if we have an SD image of a Raspberry Pi 4 with pre-installed frameworks and deep-learning examples.
We are happy to comply with this request. Please, find a complete working Raspberry Pi 4 dedicated to deep learning on our GitHub page. Download the zip file from our Mega site, unzip and flash the image on a 16 GB SD-card, and enjoy!
This application detects multiple objects in a scene. The most commonly used models are the SSD (Single Shot Detection) and YOLO (You Only Looks Once). We have some examples on GitHub of the YOLO version, but here the TensorFlow Lite SSD is explored as being one of the fastest. The COCO SSD MobileNet v1 recognizes 80 different objects. It can detect up to ten objects in a single scene. Note also, the 64-bit version is suitable for both the Raspberry 64-OS as for Ubuntu 18.04 or 20.04.
This application tries to detect the outline of multiple objects in a scene. It is done by with so-called semantic segmentation: a neural network attempts to associate every pixel in a picture with a particular subject. Tensorflow Lite has one segmentation model capable of classifying 20 different objects. Keep in mind that only reasonable sized objects can be recognized, not a scene of a highway with lots of tiny cars. Just like the previous example, the 64-bit version is suitable for both the Raspberry 64-OS as for Ubuntu 18.04 or 20.04.
This application estimates a person's pose in a scene. The deep learning model used recognizes elbows, knees, ankles, etc. TensorFlow Lite supports two models, a single and a multi-person version. We have only used the single-person model because it gives good results when the person is centred and in full view in a square-like image. The multi-pose variant lacks the robustness in our tests. Only during optimal conditions, this model came with an acceptable outcome. As usual, the 64-bit version is suitable for both the Raspberry 64-OS as for Ubuntu 18.04 or 20.04.
The most well known is, of course, the classifications of objects. Google hosts a wide range of TensorFlow Lite models, the so-called quantized models in their zoo. The models are capable of detecting 1000 different objects. Square images have been used to train the models. Therefore, the best results are with a square-like input image. Our C++ example supports all TensorFlow Lite models from the zoo, as you shall see. Now also, the 64-bit version is suitable for both the Raspberry 64-OS as for Ubuntu 18.04 or 20.04.
Face mask detection.
This application detects if a person is wearing a face mask or not. Two phases are involved in this. First, all the faces are recognized and marked in the picture. Next, every face will be examined by a second deep learning network, which will predict if the person is wearing a mask.
The face recognition software is form Linzaer (linzai). The detection of the face mask is from Baidu. Both are in the public domain. You have to install two frameworks on the Raspberry Pi: ncnn and Paddle Lite. Unfortunately, Paddle Lite can not be installed on a Raspberry Pi with a 32-bit operating system, only on a 64-bit OS, or on a Raspberry Pi 3. Ubuntu 18.04 or 20.04 on a Raspberry Pi 4 is also a possibility. The frame rate depends on the number of detected faces and can be calculated as follows: FPS = 1.0/(0.04 + 0.011 x #Faces), when overclocked to 1950 MHz.
Face mask detection 2.0.
Version 2.0 of the face mask detection application. Now, it's not the usual cascade of the two deep learning models, one face recognition and a second one that detects the masks, such as the cast in the previous Paddle Lite version.
Using a TensorFlow Lite re-trained MobileNet V2 SSD model, it recognizes three classes: no maks, a mask, and wearing a mask incorrectly. The latter category is not very convincing, given the small size of training samples.
The model recognizes not only the white masks but also the black, coloured and fancy masks. An example is a girl with the monkey mask.
Although it can detect more faces/masks in the same scene, the best result is still one face in front of the camera.
This application detects faces in a video stream. It is fast, over the 80 FPS on a bare Raspberry Pi 4 with a 64-bit OS.
We have examples of three frameworks. Installation of the MNN or ncnn is necessary before running the app. The OpenCV app, on the other hand, has no other dependencies. As you can see in the graph below, the MNN 8 bit quantized models are very fast.
This application recognizes faces from a video stream. Faces and their landmarks will be detected by an MtCNN network, or with a RetinaFace network. The later one outperforms MtCNN but takes a few mSecs extra time. Once found, it will look for the matching face in a database.
As the recognition network, we use ArcFace. It is by far the best network that can recognize a face from a large data set.
The last step is an optional anti-spoofing check.
When not founded, a face is added in the dataset automatically (optional). We have limited the database to 2000 items for now. If you reach higher numbers, let us know.
Face recognition with mask detection.
This application is identical to the above face recognition app. The only difference is the additional face mask recognition. As is known, recognition becomes very uncertain when a mask covers a large part of the face. An additional warning can help now.
A well-known family of deep learning models detecting different objects in a scene are the YOLO networks. YOLO stands for You Only Look Once. Joseph Redmon developed the original YOLO model. Releasing new versions, the model evolves. After YOLOV3, Joseph Redmon stopped his work on the project. He has ethical and moral objections to certain potential negative side-effects of his work. Think of the military embracing deep learning techniques. However, Alexey Bochnkovskiy continues working on new ideas for YOLO. They release Yolo version 4 in the summer of 2020. A few months later, another group of researchers around Glenn Jocher, how worked on porting YOLOV3 for PyTorch, released his version of YOLO, calling it version 5. All versions are on our GitHub page.
If TensorFlow Lite segmentation was still a simple model, Yolact is of a completely different order. It is complex and very precise. Even objects of the same class are distinguished from each other, as can be seen from the different zebras in the picture. The input dimension is large (550x550 pixels). The model distinguishes 80 different classes. All this comes with a price; it is not fast on a bare Raspberry Pi 4.
Super-resolution tries to enlarge images without losing detail. A blurred area in an image, such as clouds, is smoothed out, while high-detail and high-contrast areas are best enlarged with sharp edges. Deep learning is widely used to solve the often contrasting demands of super-resolution.
Here the award-winning RealSR solution from the Tencent YouTu Lab is presented. In a word, the results of this deep learning model are stunning. Details that have never been noticed in the small image are suddenly clearly visible.
The only drawback is the use of the Vulkan drivers due to the use of the ncnn framework. Unfortunately, it is only possible to run the software on a Jetson Nano. For the Raspberry Pi, we will have to wait until the preliminary Vulkan driver is further developed.
In this section, we'll give you all the steps needed to implement a deep learning model on your Raspberry Pi 4. It is more complicated than simply downloading a file, as you can see from the checklist below.
- Operating system
- Installing OpenCV
- Installing Code::Blocks
- Installing framework
- Downloading software
- Compiling software
- Running the example
Read this tutrial carefully before downloading any software.
32 or 64 bit operation system.
The first decision to make is the operating system you want to use. Most models work on either 32 or 64 bit operating systems. Obvious, the models will run faster on a 64-bit OS. If you already have a 32-bit Raspbian operating system and want a taste of deep learning, keep using it and download a relatively small ncnn framework. If you like to run more complex models, you can migrate to the 64-bit operating system. Keep in mind that transfer involves a brand new installation of your Raspberry Pi. You will lose all software on your SD card. Better to use a new card for your adventure. There are examples, such as face mask detection, that only work on a 64-bit RPi. The guide is found here: Install 64 bit OS on Raspberry Pi 4.
The next decision is overclocking. Increasing the clock frequency on your Raspberry Pi board will make applications run faster. It has a price, heat. Without a heatsink, the Raspberry protects itself in no time by automatically lowering the clock frequency. If you don't have a heatsink or other cooling on your RPi, don't overclock. With proper cooling, you could consider overclocking. We always run his RPi at about 1900 MHz. Sometimes 1950 MHz. Higher frequencies will definitely crash your Raspberry Pi when performing demanding tasks such as deep learning apps. Despite some enthusiastic auteurs you want to make you believe that they reach 2200 MHz. They simply manage to start up the Raspberry at that frequency and then immediately write their article. Safe overclocking is covered here.
You must have OpenCV installed on your Raspberry Pi to work with video and other image-related tasks. Some may have OpenCV already on their RPi. Please check if your version is C++ compatable with the following commands. All three must show some location on your disk.
If your package is installed with $ sudo apt-get install or $ pip it will not work as these are Python versions. And as you know, speed and Python don't go hand in hand. In that case, install OpenCV all over again.
We have written many guides on how to install OpenCV on your RPi. It doesn't matter which version you use, as long as it is version 4, for example, this manual about version 4.5.0. Pay attention to which operating system you are using, 32 or 64 bit. Each requires a different installation.
Time to check your operating system if your are not sure. Run the command uname -a and verify your version with the screen dump below.
You have to check your compiler version also with the command gcc -v. It must also be an aarch64-linux-gnu version, as shown in the screenshot. If you have a 64-bit operating system, but your gcc version is different from the one given above, reinstall the whole operating system with the latest version.
The 32-bit version gives the output shown below.
You need a good IDE to write your C ++ program. You could use Geany as it comes with the Raspbian OS. However, Geany cannot handle projects, only individual files. You end up messing with Make to integrate all the different files into one executable. Secondly, Geany has limited debug tools.
We are going to use Code::Blocks. The IDE can handle multi-file projects and has excellent debug functions such as variable, thread or CPU registry inspection. The IDE is relatively easy and intuitive to understand. With the following command in your terminal, you can install Code::Blocks.
$ sudo apt-get install codeblocks
Before starting straight away with a deep learning learning network, it might be an idea to check your installation and your skills so far by doing one of the examples on this page. It also gives you more information on how to handle errors. You'll find it comforting to have James Bond riding his motorcycle on your screen. At least then you know you're halfway through with the software.
We are now going to install the Face Mask Detection software, as one of our favoured examples on GitHub. In the dependencies list, found in the Readme.md, you read that the application uses two frameworks: ncnn and Paddle-Lite. The ncnn framework detects faces in an image. Paddle-Lite determines whether or not the person is wearing a mask. Also important, due to Paddle-Lite, this application can only run on a Raspberry Pi with a 64-bit operating system. So let's start. If you haven't a 64-bit operating system, you better start all over again by installing the proper OS and OpenCV version. Next install, as suggested, the ncnn framework according to the page mentioned in the Readme.md. Of course, if you have ncnn already on your RPi, no need to install it again. Last, install Paddle-Lite if you haven't done it already. Be sure to use our installation guides because all examples expect to find the directory structure as used in these guides.
At this point, you have on the Raspberry Pi the next software running. It will take about 22.2 GByte of disk space.
- 64-bit operating system
- OpenCV 64-bit
Downloading the deep learning example.
Once all software is available, you can download the software from the GitHub page. In the Readme.md you will find the instructions. For the sake of clarity, we will repeat them here.
# create a folder called software (if not already done)
$ mkdir ~/software
$ cd ~/software
# create a folder, for instance, face to store the app
$ mkdir face
$ cd face
# download the package
$ wget https://github.com/Qengineering/Face-Mask-Detection-Raspberry-Pi-64-bits/archive/master.zip
$ unzip -j master.zip
# you may want removing files not used any more
$ rm master.zip
$ rm LICENSE.txt
$ rm README.md
If you had used the download button on GitHub, you end up with the zip file in your Downloads folder. With some trial and error, you certainly will get the file extracted to a location of your choice.
Compiling the software.
The next step is compiling the software so it can run on your Raspberry Pi. Open Code::Blocks and load the project file (* .cbp), in this case MaskUltra.cbp. It not only loads the CPP files but also set all environment options correctly. Select the Release option and compile the example.
Running the software.
Hopefully, the compilation is successful and you can run the Face Mask Detection application.
Most deep learning C++ examples work with a video stream. We often provide an mp4 movie illustrating the functionality of the app. However, if you want to use a camera in your application, it's done by just altering one line of code. Look for the line where the VideoCapture is declared. Here, modify the input by replacing the name of the movie by a number. Below you see a code snippet.
Working with Code::Blocks always involves the same steps. All our GitHub projects have project files (* .cbp). Once loaded in Code::Blocks, all environment options are automatically set correctly.
Follow this procedure if you want to start with a blank project. First, you load your source code into the IDE. Second, you have to give the folder where the necessary headers are. Do this with the menu option Project → Build options.
Select tab sheet Seach directories and under Compiler give the locations where the used headers are. Note, do not select the Debug or Release option, but use instead of the project name as can be seen below.
The next step is to specify the libraries used and the linker flags. Again, use menu option Project → Build options, but now select tab sheet Linker settings. Here the linker settings are shown as used in the Face detection app.
Just like the headers, you must give the location of the used libraries now. Do this in tab sheet Search directories, tab Linker.
You are ready with most settings. However, some applications may require some additional command line parameters during startup. Give these in menu option Project → Set programs' arguments. Select your target (Debug or Release) and given the arguments just like you would on the command line. Below some example.
Instead of using the GUI, you can modify all settings in the Code::Blocks project file (*.cbp). The file is in XML format and is very readable. Before you make any modification, close the project in Code::Blocks first. This way any change you made will be loaded in Code::Blocks when you reopen your project.