Snaps - xlnx-nlp-smartvision Snap for Certified Ubuntu on Xilinx Devices

This page provides usage information and release notes for the xlnx-nlp-smartvision snap for the KV260 available from the Canonical snap store at http://snapcraft.io.

This snap is currently only compatible with the Ubuntu 20.04 image. For 22.04, the NLP Smartvision accelerated application will be available in Q3 as a debian package.

The KV260 Ubuntu image is available at https://ubuntu.com/download/amd-xilinx

For detailed information about installing Ubuntu on the KV260, please refer to the https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2037317633 page.

Table of Contents

Introduction

The xlnx-nlp-smartvision (Natural Language Processing Smartvision) snap continuously detects keywords spoken by the user and showcases keyword-based dynamic switching between multiple vision tasks and/or changes the display properties. This snap is an adaptation of the NLP Smartvision Accelerated Application released for the 2020.2.2 KV260 PetaLinux platform.

https://www.xilinx.com/products/app-store/kria/nlp-smartvision.html

https://xilinx.github.io/kria-apps-docs/main/build/html/docs/nlp-smartvision/nlp_smartvision_landing.html

This snap currently only supports the KV260 Vision AI Starter Kit.

Vitis Platform

The xlnx-nlp-smartvision snap includes ready-to-use pre-built firmware, models, and application software as detailed in the “Snap Contents” section below. For reference, the NLP SmartVision accelerated application firmware is based on the kv260_ispMipiRx_DP Vitis Platform. The platform sources and build instructions can be found at the kv260-vitis github repository.

The instructions for building this platform can be found here:

 https://xilinx.github.io/kria-apps-docs/2020.2/build/html/docs/build_vivado_design.html

 The instructions for inserting the nlp-smartvision accelerated application overlay into the platform can be found here:

 https://xilinx.github.io/kria-apps-docs/2020.2/build/html/docs/build_accel.html

Snap Contents

This snap contains the nlp-smartvision firmware (.bit.bin, .xclbin, .dtbo, .json), DPU models, and the nlp-smartvision application.

Though the snap is designed and tested with only the NLP Smartvision application in mind, the DPU included in the firmware bitstream can, once loaded with xlnx-config, be used with other models and custom-built applications. It can also be used with the xlnx-vai-lib-samples snap as mentioned in the Using the DPU with xlnx-vai-lib-samples Snap section below.

The compatible DPU models are B3136 v1.3.1. Additional models for the DPU version provided in this firmware can be found on xilinx.com with the following filename:

<model name>-DPUCZDX8G_ISA0_B3136_MAX_BG2-1.3.1-r241.tar.gz

For example, to download the yolov3_voc_tf model, use the following url: https://www.xilinx.com/bin/public/openDownload?filename=yolov3_voc_tf-DPUCZDX8G_ISA0_B3136_MAX_BG2-1.3.1-r241.tar.gz

Installation

There are two ways to install the nlp-smartvision snap:

  • xlnx-config method (Recommended)

  • Manual method

Regardless of which method is used, the xlnx-config snap MUST be installed prior to installation and use of the xlnx-nlp-smartvision snap. In addition, the xlnx-config.sysinit utility must have been run to initialize the system properly. For more information related to xlnx-config, please refer to the xlnx-config snap developer page.

xlnx-config Method

Using the xlnx-config method is the easiest way to install the snap application and firmware. This method will download and install the snap from the snap store, then copy the nlp-smartvision firmware into the /lib/firmware/xilinx directory so it can be loaded into the PL using xlnx-config --xmutil loadapp. To install with this method, run xlnx-config with the --snap and --install options, specifying the snap name

1 sudo xlnx-config --snap --install xlnx-nlp-smartvision

You will be prompted to confirm copying the firmware into the /lib/firmware/xilinx directory.

Manual Method

Using the manual install method requires three steps.

First install the snap. This will install the software application and install the firmware files into the snap data directory:

1 sudo snap install xlnx-nlp-smartvision

Next, run xlnx-config to identify the name of the Platform Assets Container (PAC):

1 2 3 4 5 6 7 8 9 10 11 ubuntu@kria:~$ xlnx-config -q kv260 Platform Asset Containers (PAC) found in the system: | PAC |Installed| kv260 Assets Directory --------------------------------------------------------------------------------------------------------------- | nlp-smartvision | | /snap/xlnx-nlp-smartvision/current/hwconfig/nlp-smartvision/kv260 --------------------------------------------------------------------------------------------------------------- "Installed" indicates the Kria firmware in the assets directory matches that found in /lib/firmware/xilinx * To install firmware, use: xlnx-config --install <platform name>

Finally, install the firmware in to /lib/firmware/xilinx by referencing the PAC name:

1 2 3 4 ubuntu@kria:~$ sudo xlnx-config --install nlp-smartvision Installing firmware for nlp-smartvision on the kv260 Copying /snap/xlnx-nlp-smartvision/current/hwconfig/nlp-smartvision/kv260 to /lib/firmware/xilinx/nlp-smartvision Done. Use sudo xlnx-config --xmutil listapps/loadapps/unloadapp to manage Kria firmware assets

Loading the Firmware

Once the snap is installed and the firmware is properly installed into the /lib/firmware/xilinx folder, the accelerator bitstream must be loaded with xmutil:

First, list the available accelerators:

1 2 3 4 ubuntu@kria:~$ sudo xlnx-config --xmutil listapps Accelerator Base Type #slots Active_slot nlp-smartvision nlp-smartvision XRT_FLAT 0 -1

If there is another active accelerator loaded, be sure to unload it before proceeding:

1 2 ubuntu@kria:~$ sudo xlnx-config --xmutil unloadapp Accelerator successfully removed.

Next, load the nlp-smartvision accelerator:

1 2 ubuntu@kria:~$ sudo xlnx-config --xmutil loadapp nlp-smartvision Accelerator loaded to slot 0

The system is now ready to run the nlp-smartvision application.

Usage

The xlnx-nlp-smartvision snap is made up of five sub-applications. Each sub-application can be executed by adding the sub-app name to xlnx-nlp-smartvision, separated by a “.” as show below:

1 2 3 xlnx-nlp-smartvision.info xlnx-nlp-smartvision.set-mic xlnx-nlp-smartvision.nlp-smartvision [OPTION] [arg1] [arg2]

info

For general snap usage information, system information, and available models and apps,  run the following command: 

1 xlnx-nlp-smartvision.info

set-mic

A USB microphone or webcam with microphone input is required to run the nlp-smartvision application. To setup the default microphone, run the following command and select the desired microphone form the list:

1 2 3 4 5 6 7 ubuntu@kria:~$ xlnx-nlp-smartvision.set-mic Microphones available in the System: card 1: C525 [HD Webcam C525], device 0: USB Audio [USB Audio] card 2: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio] Which card number do you want to use?1

This creates  .asoundrc in the user’s $HOME/snap/xlnx-nlp-smartvision/current/ directory that sets the default microphone. 

nlp-smartvision

The usage info for the nlp-smartvision sub app is shown below:

1 2 3 4 5 6 7 8 9 10 11 ubuntu@kria:~$ xlnx-nlp-smartvision.nlp-smartvision Usage: /snap/xlnx-nlp-smartvision/x1/bin/nlp-smartvision [OPTION] [arg1] [arg2] -h (or) --help help -m (or) --mipi test the application with live video from mipi camera -u (or) --usb test the application with live video from USB camera -f (or) --file-audio <testing_list>.txt test the keyword spotting with audio files listed in the .txt file -t (or) --test <sample_image> <model> test the DPU with sample images. Input is Model and sample jpeg Supported models are densebox_640_360, yolov2_voc_pruned_0_77 & plate_detect -v (or) --verbose use along with -m or -u to print fps and kws latency

For more information on using the --file-audio option, please refer to https://github.com/Xilinx/nlp-smartvision#file-based-testing-and-accuracy-measurement-of-kws-only

Running the nlp-smartvision Application

Once the nlp-smartvision firmware has been loaded successfully, you can run the demo.

Required Hardware

  • USB Webcam OR

  • MIPI Camera + USB Microphone

  • Monitor

Running the Demo from the Ubuntu Desktop

These steps assume that you are executing them from within a terminal opened in the Ubuntu GNOMOE Desktop environment (i.e. Using monitor/keyboard/mouse connected to the KV260). For running the demo over the USB UART or SSH, please refer to the Running the Demo Remotely section below before proceeding with the steps in this section.

First, make sure you select the microphone you want to use for command processing using the set-mic utility. In the example below, a USB camera and a discrete microphone are detected:

1 2 3 4 5 6 7 ubuntu@kria:~$ xlnx-nlp-smartvision.set-mic Microphones available in the System: card 1: C525 [HD Webcam C525], device 0: USB Audio [USB Audio] card 2: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio] Which card number do you want to use?1

 

Next, depending on whether you want to use the USB camera or MIPI camera , start the demo application with the appropriate options:

1 2 3 4 5 # For USB Camera xlnx-nlp-smartvision.nlp-smartvision -u # For MIPI Camera xlnx-nlp-smartvision.nlp-smartvision -m

 

After a few moments, a video should appear in a window on the desktop and the selected microphone should be active.  You can use the following keywords to interact with the demo: 

The three “tasks” are implemented by switching between three different machine learning network models on the DPU:

Task

Network Model

Task

Network Model

Face Detection

cf_densebox_wider_360_640_1.11G_1.2

Objection Detection

dk_yolov2_voc_448_448_0.77_7.82G_1.2

License Plate Detection

cf_plate-detection_320_320_0.49G_1

To exit the application, use Ctrl-C from the command line.

Running the Demo Remotely

For users wanting to run the demo remotely over the USB UART or SSH, the Ubuntu desktop environment must be shut down so the application can take ownership of the display.

To shutdown the desktop, use the following command: 

1 sudo systemctl isolate multi-user.target

Now, when the demo application is run, the video output will appear in full screen mode on the connected monitor.

To restart the desktop after the demo, use the following command:

1 sudo systemctl isolate graphical.target

Using the DPU with xlnx-vai-lib-samples Snap

The DPU configuration provided by the nlp-smartvision firmware is compatible with the xlnx-vai-lib-samples snap. Once the nlp-smartvision firmware is loaded, you can use the xlnx-vai-lib-samples test applications to evaluate any of the pre-built models in the Xilinx models zoo. Please see the xlnx-vai-lib-samples snap developer page for more information.

Troubleshooting

Audio Device Errors

Error message: “audio open error: No such file or directory”

Depending on how your sound “cards” are enumerated in the system, you may need to select the default recording device. To do this, use the set-mic sub-application:

1 2 3 4 5 6 7 ubuntu@kria:~$ xlnx-nlp-smartvision.set-mic Microphones available in the System: card 1: C525 [HD Webcam C525], device 0: USB Audio [USB Audio] card 2: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio] Which card number do you want to use?1

DPU Not Found

Error message: “dpu_controller.cpp:44] Check failed: !the_factory_methods.empty()”

If the bitstream is not programmed prior to running the nlp-smartvision app, the application will fail. Make sure you load the nlp-smartvision firmware before running the app.

1 2 3 4 5 6 ubuntu@kria:~$ sudo xlnx-config --xmutil listapps Accelerator Base Type #slots Active_slot nlp-smartvision nlp-smartvision XRT_FLAT 0 -1 ubuntu@kria:~$ sudo xlnx-config --xmutil loadapp nlp-smartvision Accelerator loaded to slot 0

dpu.xclbin Not Found

Error Message: “open(/var/snap/xlnx-nlp-smartvision/<version>/assets/dpu.xclbin) failed.”

Since the consumer snap (xlnx-nlp-smartvision) "connects" to the producer snap (xlnx-config) to share information, it’s important for xlnx-config to be installed first, so that the connection to xlnx-config:assets is made.

See the Required Interfaces Section below to resolved the issue.

Required Interfaces

The snap requires the following interfaces to be connected. These should be auto connected at installation:

  • xlnx-config: assets

  • media-control

  • alsa

  • camera

You can confirm the connection is made with the following command:

1 2 3 4 5 6 7 8 9 10 ubuntu@kria:~$ snap connections xlnx-nlp-smartvision Interface Plug Slot Notes alsa xlnx-nlp-smartvision:alsa :alsa - camera xlnx-nlp-smartvision:camera :camera - content[assets] xlnx-nlp-smartvision:assets xlnx-config:assets - home xlnx-nlp-smartvision:home :home - media-control xlnx-nlp-smartvision:media-control :media-control - network xlnx-nlp-smartvision:network :network - opengl xlnx-nlp-smartvision:opengl :opengl - x11 xlnx-nlp-smartvision:x11 :x11 -

If for some reason you need to connect the snaps manually, the following commands can be used: 

1 2 3 4 sudo snap connect xlnx-nlp-smartvision:assets xlnx-config:assets sudo snap connect xlnx-nlp-smartvision:media-control    sudo snap connect xlnx-nlp-smartvision:alsa sudo snap connect xlnx-nlp-smartvision:camera

Release Notes

Date

Snap Revision

Snap Version

Notes

Date

Snap Revision

Snap Version

Notes

12/1/21

6

1.0

Initial Public Release