Jetson Nano

The Jetson Nano is a deep learning development kit from Nvidia. Let’s take a quick look at what it can do!

Terminology: Jetson Nano is the module that the heatsink is mounted on. This module is then connected to a PCB, which has a barrel jack, USB ports, Ethernet, etc. This full assembly is the Jetson Nano Developer Kit.

Setup – Parts

• Mandatory
  • Jetson Nano Developer Kit: I got the B01 variant of the 4 GB version. The A02 is an older version with different connectors and layout.
  • Micro SD Card: I got a 256 GB one, but the minimum is 32 GB.
  • Keyboard and mouse: I’m using a Logitech MK360, which only needs a single USB dongle for both.
  • Camera: Nvidia recommends a Logitech C270 camera
  • Power supply: 5 V, up to 4 Amps, barrel connector
• Optional
  • Case: To protect the developer kit
  • WiFi & Bluetooth module: Helpful if you want to browse the web or download code on the developer kit
  • Fan: The case comes with a fan, but it’s always on. The one from Noctua only turns on when Nano starts to get hot.

Setup – Procedure

• Programming the Jetson
  • The instructions can be found here. It’s very similar to programming a Raspberry Pi; burn the operating system onto an SD card, then put the card into the Jetson.
  • The Jetson can be powered by a USB micro cable, connected to J28. I don’t recommend this, since the Nano consumes a lot of power. Instead, first make sure there is a jumper on J48. Then, connect the power supply to the barrel jack. Now, the Jetson is powered by the power supply instead of over USB

• Optional
  • The Jetson, by default, can only be turned on and off by connecting/disconnecting the power supply (or going through the operating system). This is very inconvenient. A way around this is to jumper pins 5 and 6 on J50. Then, short pins 11 and 12 on J50 to turn the system on or off.
  • The WiFi/Bluetooth module is placed under the Jetson Nano module, onto J18 as shown in the figure above. Remove the two screws holding the module to the development kit, then push the two metal clips attached to the ends of J2 outwards. This will allow you to remove the Nano. Before placing the module into J18, remove the screw near J18, and attach the antennas to the WiFi/Bluetooth module. After inserting the module into J18, hold and screw it down. Reassemble the development kit. Here’s a helpful video.
  • The connector for the fan should connect to J15

Jetson AI Fundamentals Course

Nvidia has a free course for learning the basics of deep and transfer learning; you just need to create an account. The tutorial does an excellent job walking you through the process, so I’ll just point out some things I found interesting or important.

The Jetson Nano operates in “headless” mode, which means you don’t need a keyboard, mouse or monitor attached to it. It connects to a host PC over a USB micro cable, and you communicate with the Jetson through your computer. The Jetson creates a Jupyter Notebook which you can access through your browser.

Connection Problems

Before diving in, I wanted to address a problem I ran into. For me, my host PC kept losing connection to Jupyter Notebook. Reading through the documentation, I found the following:

On my Windows computer, I went into the network settings and made the following changes:

After that, connectivity was nice and reliable.

Jupyter Notebook

You connect to the headless Nano over SSH. I used putty to do this. The Nano’s IP address, when connected to the host PC through USB, is 192.168.55.1

Then, when connected to the Nano, you have to run a shell script to start the Jupyter Notebook:

Now, we can go to the provided URL and use the provided password to access the notebook!

Classification

Let’s look at one of the Jupyter Notebooks. On the left hand side of the window you’ll directories; go to classification > classification_interactive.ipynb

Go through the file from beginning to end, executing each cell (except the last one; that’ll shut down the camera and neural network).

The second to last cell will create a GUI for training the neural network:

Here, you can see what the camera sees. In category, select thumbs_up or thumbs_down, then do the corresponding gesture in the frame of the camera. Click the add button to take a photo. Do this numerous times for both gestures.

After you’ve taken many photos, you can start training the neural network. Select on the amount of epochs to train the network for, then click train. The number of epochs will go down one by one as the training runs; when epochs reaches zero, training has completed.

The image above shows that the neural network can recognize my thumbs up and down gestures pretty well! Here’s a couple of things to note:

• Take a lot of photos with your hands at different positions on the screen, and at various distances. The more diverse your photos are, the better the end result
• Try to have various backgrounds and lighting conditions. Changing such ambient information will teach the neural network to ignore it, and focus on your hand

Classification II

Let’s look under the hood. When you run the classification Jupyter notebook, you are creating an instance of a fully trained neural network. In this case, we’re using ResNet-18, a variant of the ResNet neural network that has 18 layers. Then, we’re using transfer learning to retrain the network with the photos we took. Transfer learning is implemented using PyTorch, a library dedicated to machine learning (in Python). The advantage of PyTorch is that it takes advantage of the Nano’s GPU hardware.

You may have heard that GPUs are very important for machine learning applications. But why?

CPUs are designed for general purpose applications, and they’re capable of training and running neural networks, but they’re very slow at it. This of course is a problem if you want your training to take hours instead of days, or want to run classification on a live video feed.

GPUs, on the other hand, are designed for a very specific purpose: rendering graphics. To accomplish this, GPUs are designed for parallel processing, which just so happens to be useful for machine learning. Examine the neural network below. Remember that neurons within a single layer have no connection to each other. This means that the outputs of neurons on that layer can be computed independently of each other, so they can all be computed in parallel.

While CPUs aren’t very good at parallel processing, GPUs are great at it. By taking advantage of parallel processing, training and application of neural networks can be sped up by a hundred or thousand fold.

In our case, the Nano has a CPU with 4 cores, and a GPU with 128 cores! PyTorch takes advantage of the hardware by using the GPU to perform computations, rather than the CPU. If the CPU were to run the neural network, it would calculate the output of a single neuron, then move on to the next one, then to the next one, computing one at a time. The GPU, on the other hand, can compute over a hundred neurons at once. Also note that the CPU has other responsibilities besides the neural network, such as running the operating system, so that’s another advantage to using the GPU.

Hello AI World

Hello AI World is a repo that contains files and instructions on how to run image classification, image detection and image segmentation. The files include network models, such as ResNet and GoogleNet, but also Python and C++ files to run the networks, and update them if so desired. The repo contains detailed instructions, so I’ll only show a couple of things I though were cool here.

Note: the neural networks are executed using TensorRT, which optimizes the networks for performance. Transfer learning to update these networks, while not explored here, is done using PyTorch.

Running Examples

The repo provides instructions for first time setup. I ran my examples out of docker, so each time you want to run the docker, you have to call the run shell script:

Inside the docker, go to build/aarch64/bin. Here, you’ll see several executables you can run. Let’s take a look at some of them.

Camera Test

For these examples, you’ll have to provide an input to the executable, such as a file to examine or a video stream. In most cases, the argument is ./dev/video0, which is my webcam. For video-viewer, this simply shows what the camera sees.

Image Classification

To run image classification, run the following command: ./imagenet ./images/black_bear.jpg
The command will feed the following image to GoogleNet, a pretrained image classification network:

As you can see, the neural network correctly identified the image as a black bear. Let’s try feeding the network a video feed:

The most certain output of the neural network is shown in the camera feed, 31.05% certain that the image is an apron. This is, of course, incorrect; I’d say the correct answer is a t-shirt. This does raise some interesting questions though; there are many things in the frame, so how can we be sure what the neural network is identifying?

To be generous, let’s look at the top answers of the neural network, which is shown on the left. The correct answer is highlighted: t-shirt, with 9.7% certainty. So the good news is that the correct answer is in the top 15 answers (out of, I believe, 1000). The other answers provide some interesting, and possibly amusing insights: I do indeed use this t-shirt for my pajamas, (2.9% certainty), and the design is Japanese, so kimono is kind of relevant (5.6% certainty). But let’s not get too generous…

Image Detection

While image classification will label an image as one thing, image detection scans the image and tries to identify as many things as possible, while also determining coordinates for each of those items. Let’s take a look at an example:

Here, you can see the image has me, my tie and my TV all identified and highlighted. Pretty cool, right!?

If you look at the terminal on the left, you can see the objects identified in the frame, as well as bounding boxes, which shows the position and size of the objects in the frame.

Image Segmentation

As the name suggests, segmentation divides a provided image in a meaningful way, simplifying it for a computer to work with. For example, for self driving purposes, a picture of the road (seen from the driver’s perspective) can be broken down into lanes, traffic signals, people, obstacles, etc.

Above is an example of image segmentation on a human body. The neural network used is Multi-Human, which is designed to break a person down into pieces (head, torso, arm, leg, etc.). Here’s another example from the repo:

The neural network used here is DeepScene, which is designed to work in nature. The image shows how a robot might use image segmentation to stay on the trail.

Conclusion

In this post, we saw what the Jetson Nano is capable of. We saw that a deep neural network can be used to classify, detect and segment images, and how neural networks can be updated for specific applications. Through the Jetson AI Fundamentals course, and Hello AI World, we can learn and apply the basics of machine learning to real world situations!