Install ncnn deep learning framework on a Jetson Nano.
Last updated: April 25, 2023
Introduction.
This page will guide you through the installation of Tencent's ncnn framework on a Jetson Nano. Because the ncnn framework targets mobile devices, like Android phones, it has no CUDA support. However, most Android phones use the Vulkan API for low-level access to their GPU's. The ncnn framework can use Vulkan routines to accelerate the convolutions of a deep learning model. The Jetson Nano has Vulkan support which ncnn will be using. More information about the software structures can be found here and here. The given C ++ code examples are written in the Code::Blocks IDE for the Nano. We only guide you through the basics, so in the end, you can build your application. For more information about the ncnn library, see: https://github.com/Tencent/ncnn. Perhaps unnecessarily, but the installation is the C ++ version. It is not suitable for Python.
RTTI.
RTTI stands for Run-Time Type Identification. It is a C ++ mechanism used at runtime to get the type and memory size of an object, yet not been defined. Normally, a programmer knows the type of variable and can allocate memory that holds the object in advance. Obtaining memory from an operating system with all its processes and threads can be a relatively time-consuming operation. Modern C compilers know how much memory is required and one call to memory management is sufficient. It's one of the main advantages over Python, which is oblivious to memory requirements until it hits the line of code with a variable.
It is best not to use RTTI if you want to write the fastest possible code. It is also the case in the ncnn framework.
By default, it is compiled with a -fno-rtti flag that prevents the use of RTTI. Compiled with this flag, custom-defined layers like those found in YOLOV5 are only usable if the remaining program is still not using RTTI.
Sometimes it is not possible to avoid RTTI. Especially in mature code that requires a new function without rewriting all the code. OpenCV uses the RTTI mechanism in some places.
There is a problem here. When you compile ncnn with OpenCV, the compiler returns an error on the -fno-rtti flag. Removing the flag sometimes works with the ncnn code, depending on the type of DNN used.
At this point, the used build flag -D NCNN_DISABLE_RTTI=OFF becomes clear. It tells the compiler ncnn will allow RTTI. It means that you can now run ncnn in its full functionality without getting into problems with OpenCV. Or, for that matter, any other piece of software using RTTI.
Performance-wise, you will not notice any difference on your Jetson Nano.
CMake 3.18.4.
The ncnn framework uses Glslang for OpenGL compilation. The latest version of this compiler requires CMake version 3.18.4 or higher. Since the Jetson Nano ships with version 3.16, we need to upgrade CMake. The only way to do this is from scratch. There are no aarch64 repositories for CMake, making installation easy. The whole building will take a while, be patient.
# a fresh start
$ sudo apt-get update
$ sudo apt-get upgrade
# install dependencies
$ sudo apt-get install libssl-dev build-essential libtool autoconf unzip wget
# download ncnn
$ mkdir ~/temp
$ cd ~/temp
$ wget https://cmake.org/files/v3.18/cmake-3.18.4.tar.gz
$ tar -xzvf cmake-3.18.4.tar.gz
$ cd cmake-3.18.4/
# build cmake
$ ./bootstrap
$ make -j4
$ sudo make install
# reboot to complete
$ sudo reboot
# check the version
$ cmake --version
# you can delete the cmake directory if you wish
$ cd ~
$ sudo rm -rf ~/temp
Dependencies.
With the new CMake 3.18.4 installed, the next step is to build the actual ncnn framework.
It requires protobuf to load ONNX models and, of course, the Vulkan SDK.
The installation of ncnn on a Jetson Nano with a Linux Tegra operating system is as follows.
# install dependencies
$ sudo apt-get install libprotobuf-dev protobuf-compiler libvulkan-dev
# download ncnn
$ git clone --depth=1 https://github.com/Tencent/ncnn.git
# download glslang
$ cd ncnn
$ git submodule update --depth=1 --init
# prepare folders
$ mkdir build
$ cd build
# build 64-bit ncnn for Jetson Nano
$ cmake -D CMAKE_TOOLCHAIN_FILE=../toolchains/jetson.toolchain.cmake \
-D NCNN_DISABLE_RTTI=OFF \
-D NCNN_BUILD_TOOLS=ON \
-D NCNN_VULKAN=ON \
-D CMAKE_BUILD_TYPE=Release ..
$ make -j4
$ make install
# copy output to dirs
$ sudo mkdir /usr/local/lib/ncnn
$ sudo cp -r install/include/ncnn /usr/local/include/ncnn
$ sudo cp -r install/lib/*.a /usr/local/lib/ncnn/
# once you've placed the output in your /usr/local directory,
# you can delete the ncnn directory if you wish
$ cd ~
$ sudo rm -rf ncnn
If everything went well, you will get two folders. One with all header files and one with the libraries as shown in the screen dumps.
Please note also the folder with the examples. Many different types of deep learning are covered here. The references to the actual deep learning models can sometimes cause errors due to version changes in the ncnn library. We recently received the following repository from nihui with the latest models: https://github.com/nihui/ncnn-assets/tree/master/models.
Benchmark.
We have done some benchmarking with and without Vulkan support, to see how well ncnn is performing. On average, you will get a 57% performance boost when ncnn is using Vulkan. Which is even better than the increase of MNN with CUDA. Note, during the test the Jetson Nano CPU was overclocked to 2014.5 MHz and the GPU to 998.4 MHz.
Model | CPU (mSec) | Vulkan (mSec) |
| SqueezeNet | 57.0 | 14.0 |
| MobileNetV1 | 32.6 | 15.9 |
| MobileNetV2 | 26.9 | 27.8 |
| ResNet | 128.7 | 45.0 |
| GoogleNet | 85.4 | 43.8 |
| ShuffleNet | 27.8 | 14.4 |
Github C++ example