This page provides usage information and release notes for the xlnx-nlp-smartvision snap for the KV260 available from the Canonical snap store at http://snapcraft.io.

Table of Contents

Introduction

The xlnx-nlp-smartvision (Natural Language Processing Smartvision) snap continuously detects keywords spoken by the user and showcases keyword-based dynamic switching between multiple vision tasks and/or changes the display properties. This snap is an adaptation of the NLP Smartvision Accelerated Application released for the 2020.2.2 KV260 PetaLinux platform.

https://www.xilinx.com/products/app-store/kria/nlp-smartvision.html

https://xilinx.github.io/kria-apps-docs/main/build/html/docs/nlp-smartvision/nlp_smartvision_landing.html

This snap currently only supports the KV260 Vision AI Starter Kit.

Vitis Platform

The xlnx-nlp-smartvision snap includes ready-to-use pre-built firmware, models, and application software as detailed in the “Snap Contents” section below. For reference, the NLP SmartVision accelerated application firmware is based on the kv260_ispMipiRx_DP Vitis Platform. The platform sources and build instructions can be found at the kv260-vitis github repository.

The instructions for building this platform can be found here:

 https://xilinx.github.io/kria-apps-docs/2020.2/build/html/docs/build_vivado_design.html

 The instructions for inserting the nlp-smartvision accelerated application overlay into the platform can be found here:

 https://xilinx.github.io/kria-apps-docs/2020.2/build/html/docs/build_accel.html

Snap Contents

This snap contains the nlp-smartvision firmware (.bit.bin, .xclbin, .dtbo, .json), DPU models, and the nlp-smartvision application.

• The firmware and models are from the 2020.2.2 release: https://github.com/Xilinx/kv260-firmware/tree/release-2020.2.2_k26/nlp-smartvision

• The application code can be found here: https://github.com/Xilinx/nlp-smartvision/tree/rel_v2020.2.2_20.04

Though the snap is designed and tested with only the NLP Smartvision application in mind, the DPU included in the firmware bitstream can, once loaded with xlnx-config, be used with other models and custom-built applications. It can also be used with the xlnx-vai-lib-samples snap as mentioned in the Using the DPU with xlnx-vai-lib-samples Snap section below.

Installation

There are two ways to install the nlp-smartvision snap:

• xlnx-config method (Recommended)

• Manual method

xlnx-config Method

Using the xlnx-config method is the easiest way to install the snap application and firmware. This method will download and install the snap from the snap store, then copy the nlp-smartvision firmware into the /lib/firmware/xilinx directory so it can be loaded into the PL using xlnx-config --xmutil loadapp. To install with this method, run xlnx-config with the --snap and --install options, specifying the snap name

You will be prompted to confirm copying the firmware into the /lib/firmware/xilinx directory.

Manual Method

Using the manual install method requires three steps.

First install the snap. This will install the software application and install the firmware files into the snap data directory:

Next, run xlnx-config to identify the name of the Platform Assets Container (PAC):

Finally, install the firmware in to /lib/firmware/xilinx by referencing the PAC name:

Loading the Firmware

Once the snap is installed and the firmware is properly installed into the /lib/firmware/xilinx folder, the accelerator bitstream must be loaded with xmutil:

First, list the available accelerators:

If there is another active accelerator loaded, be sure to unload it before proceeding:

Next, load the nlp-smartvision accelerator:

The system is now ready to run the nlp-smartvision application.

Usage

The xlnx-nlp-smartvision snap is made up of five sub-applications. Each sub-application can be executed by adding the sub-app name to xlnx-nlp-smartvision, separated by a “.” as show below:

info

For general snap usage information, system information, and available models and apps,  run the following command: 

set-mic

A USB microphone or webcam with microphone input is required to run the nlp-smartvision application. To setup the default microphone, run the following command and select the desired microphone form the list:

This creates  .asoundrc in the user’s $HOME/snap/xlnx-nlp-smartvision/current/ directory that sets the default microphone. 

nlp-smartvision

The usage info for the nlp-smartvision sub app is shown below:

For more information on using the --file-audio option, please refer to https://github.com/Xilinx/nlp-smartvision#file-based-testing-and-accuracy-measurement-of-kws-only

Running the nlp-smartvision Application

Once the nlp-smartvision firmware has been loaded successfully, you can run the demo.

Required Hardware

• USB Webcam OR

• MIPI Camera + USB Microphone

• Monitor

Running the Demo from the Ubuntu Desktop

First, make sure you select the microphone you want to use for command processing using the set-mic utility. In the example below, a USB camera and a discrete microphone are detected:

Next, depending on whether you want to use the USB camera or MIPI camera , start the demo application with the appropriate options:

After a few moments, a video should appear in a window on the desktop and the selected microphone should be active.  You can use the following keywords to interact with the demo: 

Open

The three “tasks” are implemented by switching between three different machine learning network models on the DPU:

To exit the application, use Ctrl-C from the command line.

Running the Demo Remotely

For users wanting to run the demo remotely over the USB UART or SSH, the Ubuntu desktop environment must be shut down so the application can take ownership of the display.

To shutdown the desktop, use the following command: 

Now, when the demo application is run, the video output will appear in full screen mode on the connected monitor.

To restart the desktop after the demo, use the following command:

Using the DPU with xlnx-vai-lib-samples Snap

The DPU configuration provided by the nlp-smartvision firmware is compatible with the xlnx-vai-lib-samples snap. Once the nlp-smartvision firmware is loaded, you can use the xlnx-vai-lib-samples test applications to evaluate any of the pre-built models in the Xilinx models zoo. Please see the xlnx-vai-lib-samples snap developer page for more information.

Troubleshooting

Audio Device Errors

Error message: “audio open error: No such file or directory”

Depending on how your sound “cards” are enumerated in the system, you may need to select the default recording device. To do this, use the set-mic sub-application:

DPU Not Found

Error message: “dpu_controller.cpp:44] Check failed: !the_factory_methods.empty()”

If the bitstream is not programmed prior to running the nlp-smartvision app, the application will fail. Make sure you load the nlp-smartvision firmware before running the app.

dpu.xclbin Not Found

Error Message: “open(/var/snap/xlnx-nlp-smartvision/<version>/assets/dpu.xclbin) failed.”

Since the consumer snap (xlnx-nlp-smartvision) "connects" to the producer snap (xlnx-config) to share information, it’s important for xlnx-config to be installed first, so that the connection to xlnx-config:assets is made.

See the Required Interfaces Section below to resolved the issue.

Required Interfaces

The snap requires the following interfaces to be connected. These should be auto connected at installation:

• xlnx-config: assets

• media-control

• alsa

• camera

You can confirm the connection is made with the following command:

If for some reason you need to connect the snaps manually, the following commands can be used: 

Release Notes