Table of Contents

1 Revision History


This Getting Started Guide complements the 2017.2 version of the ZCU102 reVISION platform. For other versions, refer to the reVISION Getting Started Guide overview page.

Change Log:


2 Introduction


The Xilinx reVISION stack includes a broad range of development resources for platform, algorithm and application development. This includes support for the most popular neural networks including AlexNet, GoogLeNet, VGG, SSD, and FCN. Additionally, the stack provides library elements including pre-defined and optimized implementations for CNN network layers, required to build custom neural networks (DNN/CNN). The machine learning elements are complemented by a broad set of acceleration ready OpenCV functions for computer vision processing. For application level development, Xilinx supports industry standard frameworks and libraries including Caffe for machine learning and OpenCV for computer vision. The reVISION stack also includes development platforms from Xilinx and third parties, including various types of sensors. For more information go to the Xilinx reVISION webpage.


3 Overview


The below figure shows a block diagram of the ZCU102 reVISION single sensor design:



A simple command line based application controls the design over a serial terminal emulator. It constructs a video pipeline graph consisting of one source, one accelerator (optional), and one sink. It is responsible for initializing the capture, m2m, and display pipelines as well as managing the video buffer flow through the pipeline stages.

3.1 Platform


The ZCU102 reVISION platform supports the following video interfaces:

Sources:

Sinks:

3.2 Design Examples


The platform ships with 5 file I/O and 2 live I/O design examples demonstrating popular OpenCV functions accelerated on the programmable logic:

Live I/O:


File I/O:


4 Software Tools and System Requirements


4.1 Hardware


Required:

Optional (only needed for live I/O examples):

4.2 Software


Required:

4.3 Licensing


Steps to generate the license:
  1. Log in here with your work E-mail address (If you do not yet have an account, follow the steps under Create Account)
  2. Generate a license from “Create New Licenses” by checking "SDSoC Environment, 60 Day Evaluation License"
  3. Under system information, give the host details.
  4. Proceed until you get the license agreement and accept it.
  5. The License (.lic file) will be sent to the email-id mentioned in the login details.
  6. Copy the license file locally and give the same path in the SDSoC license manager.

4.4 Compatibility


The reference design has been tested successfully with the following user-supplied components.

Monitors:
Make/ModelNative Resolution
Viewsonic VP2780-4K3840x2160
Acer S277HK3840x2160
Dell U2414H1920x1080

HDMI Sources:
Make/ModelResolutions
Nvidia Shield TV3840x2160, 1920x1080
OTT TV BOX M8N3840x2160, 1920x1080, 1280x720
Roku 2 XS1920x1080, 1280x720
TVix Slim S1 Multimedia Player1920x1080, 1280x720

USB3 Cameras:
Make/ModelResolutions
ZED stereo camera3840x1080, 2560x720
See3CAM_CU301920x1080, 1280x720

DisplayPort Cables:


5 Design File Hierarchy


The Zynq UltraScale+ MPSoC reVISION Platform zip file is released with the binary and source files required to create Xilinx SDx projects and build the sample applications. The provided samples include 5 file I/O examples and 2 live I/O examples. The file I/O examples read an input image file and produce an output image file whereas the live I/O examples take live video input from a video source and output live video on a display.

For the advanced user who wants to create their own platform, a PetaLinux BSP is included as well as the sources for the video_lib library which provides APIs to interface with video sources, sinks, and accelerators. Basic README files are provided in the respective directories.

Pre-built SD card images are included that enable the user to run the two live I/O examples applications on the ZCU102 board.

The top-level directory structure:
zcu102_[es2_]rv_ss
├── hw
│   └── zcu102_[es2_]rv_ss.dsa
├── IMPORTANT_NOTICE_CONCERNING_THIRD_PARTY_CONTENT.txt
├── README.txt
├── samples
│   ├── file_IO
│   │   ├── bilateral_fileio
│   │   ├── harris_fileio
│   │   ├── opticalflow_fileio
│   │   ├── steoreolbm_fileio
│   │   └── warptransform_fileio
│   └── live_IO
│       ├── optical_flow
│       └── stereo_vision
├── sd_card
│   ├── optical_flow
│   └── stereo_vision
├── sw
│   ├── a53_linux
│   |   ├── boot
│   |   ├── image
│   |   ├── inc
│   |   ├── lib
│   |   └── qemu
│   ├── petalinux_bsp
│   ├── prebuilt
│   ├── sysroot
│   ├── video_lib
│   └── zcu102_[es2_]rv_ss.spfm
└── zcu102_[es2_]rv_ss.xpfm


6 Installation and Operating Instructions


6.1 Board Setup


Required:

Optional:

Jumpers & Switches:



6.2 Extract the design zip file



Download and unzip the reference design zip file matching your silicon version (ES2 or production).
For Linux, use the unzip utlity.
For Windows, make sure that the reference design zip file is unzipped in a directory path which contains no spaces. Use the 7zip utility and follow the steps below.


6.3 Run the Application


Run the Dense Optical Flow sample application


root@plnx_aarch64:~#
The files on your sd_card are present in directory /media/card/. That directory is already specified in the PATH environment variable, so you are not required to "cd" to that directory.
root@plnx_aarch64:~# of2.elf
The system initializes and displays the command line menu. The menu appears, allowing you to select the video input source, and to turn on/off the Optical Flow processing.

So, if you run the app as shown above, with no command-line switches, you'll get the defaults: MIPI camera input passed through to Display Port output. You should see live video from your MIPI camera on the output monitor. The menu displayed is shown below.
Video Control application:
------------------------
Display resolution: 3840x2160
 
--------------- Select Video Source ---------------
1 : MIPI CSI2 Rx (*)
2 : HDMI Input
3 : USB Webcam
4 : Virtual Video Device
 
--------------- Select Filter Type ----------------
5 : Optical Flow (*)
 
--------------- Toggle Filter Mode ----------------
6 : Filter OFF/HW (off)
 
--------------- Exit Application ------------------
0 : Exit
 
Enter your choice :
Enter your choice : 6
 
...
 
--------------- Toggle Filter Mode ----------------
4 : Filter OFF/HW  (hardware)
 
...
 
Enter your choice :
When Optical Flow processing is activated, the output shows bright colors where there is the greatest motion from one input frame to the next, and black where there is no motion. The optical flow algorithm returns 2 signed numbers at each pixel position, representing up or down motion in the vertical direction, and left or right motion in the horizontal direction. The brightness of the output, from black up to bright color, indicates the magnitude of the motion, and the color indicates the direction. +/- vertical motion is mapped onto the V color component, and +/- horizontal motion is mapped onto the U color component. To see a nice graph of the range of colors this produces, refer to the wikipedia page on YUV colors.


A number of command-line switches are available for selecting video input, filter mode, and video output when the app is launched. To do this you must know the ID number of the choices for each one of these catagories. Note that these IDs differ from the menu command numbers discussed above.
# of2.elf -S
VIDEO SOURCE          ID
MIPI CSI2 Rx          0
  HDMI Input          1
  USB Webcam          2
Virtual Video Device  3
 
# of2.elf -M
FILTER MODE           ID
        off           0
   hardware           2
These ID numbers are used with the "-s" and "-m" switches (lower case) on the command line to select the video input and the filter mode when the app is launched.

# of2.elf -s 1

# of2.elf -s 1 -m 2

# of2.elf -s 0 -m 2

To control the video output, the command line switch is "-d". ID "0" selects Display Port output, and ID "1" selects HDMI output. There is no query command for these IDs, because they do not vary.

# of2.elf -s 1 -m 2 -d 1

These switches are independent of each other and may be used in any combination and in any order. If you do not specify a switch, the default ID "0" is used" : i.e. "MIPI" input, filter mode "off", "Display Port" output.

The desired image resolution in terms of width and height (in pixels) of the input video may be selected with the "-i" switch. In general the output resolution is the same as the input resolution. If the resolution is possible, meaning the input source is capable of supplying video in that resolution, it will use that resolution. Otherwise is will refuse to run and display a message to that effect.

# of2.elf -i 1920x1080

# of2.elf -s 1 -i 1920x1080 -m 2 -d 1

Note that the USB camera mentioned in section 3.4 above, the See3CAM_CU30 from e-con Systems, has an unusual pixel format called "UYVY". The pixel format describes the ordering of the luma and chroma data stored in memory for each pixel. By far the most common 4:2:2 pixel format is "YUYV" which is the default for the reVISION platform and for the MIPI and HDMI video input sources. To use the USB See3CAM_CU30 camera as source for Optical Flow, this special pixel format must be specified by attaching "@UYVY" to the input resolution WIDTHxHEIGHT string.
# of2.elf -s 2 -i 1920x1080@UYVY

Note that ONLY Display Port output (not HDMI output) will function with the UYVY format. Also, if you attempt to select the MIPI or HDMI inputs with the UYVY format, the system will refuse to do so.

Run the Stereo Vision sample application



In general, the steps and details explained above in the Optical Flow tutorial apply here in the same way.

However, the stereo vision demo is special in several ways. First, you MUST use the ZED stereo camera connected to the USB video input. Second, and quite particular to this app, the width of the input image resolution is twice the width of the output resolution. The input actually consists of two images side-by-side, the synchronized left and right stereo input supplied by the camera. Two cases are possible: 2560x720 in to 1280x720 out, and 3840x1080 in to 1920x1080 out. For this we need to use the input resolution switch "-i" and the output resolution switch "-o". The default 3840x2160 output resolution is not supported by the Stereo Vision app. Also, you may NOT toggle between modes OFF and HW in this case, because in the pass-through case the output and input must be the same resolution, not true when Stereo Vision processing is active.

The other special thing about this app is that a configuration file corresponding to the camera you have connected to your system must be used. Each StereoLabs ZED camera has a unique parameters file associated with it. This text file comes from StereoLabs, and must be present on the SD Card for the Stereo Vision demo to work properly. You need the file unique to your camera, identified by its Serial Number (found on the ZED camera box and also on a black tag near the USB plug of the ZED camera itself). This number will be, e.g., S/N 000012345. The parameter file for that camera would be named SN12345.conf. To download your parameter file, enter this URL into your browser:
http://calib.stereolabs.com/?SN=12345 (using your serial number in place of 12345)
This will download your configuration file to your computer. Copy this file to the SD Card root directory. Also, you must specify this file on the command line when you run the app, as :
--filter-sv-cam-params /media/card/SN12345.conf

# stv.elf -s 2 -m 2 -d 1 -i 3840x1080 -o 1920x1080 -filter-sv-cam-params /media/card/SN12345.conf

# stv.elf -s 2 -m 2 -d 1 -i 2560x720 -o 1280x720 -filter-sv-cam-params /media/card/SN12345.conf
The stereo block-matching algorithm calculates depth based on binocular parallax, similar to the way human eyes perceive depth. The depth map is coded in false colors. Objects far away appear deep blue. Closer and closer objects appear in rainbow succession green, yellow, orange, red, purple and finally white at about two feet from the camera in the 720p case, and about five feet away in the 1080p case. Any object closer than that cannot be tracked, and smooth areas with no texture in the image cannot be tracked, and show up as black. Areas with a lot of detail (especially with lots of vertical edges) are tracked best. It is normal that a large area on the left is black - this is 128 pixels wide, representing the range of the horizontal search for best match between the right and left binocular images.

Commandline Options


The full list of supported command line switches is:
-d, --dri-card N                      DRI card number N (default N=0)
-h, --help                            Show this help screen
-i, --input-format WxH[@FMT]          Input Width'x'Height@Fmt
-o, --output-format WxH[-HZ][@FMT]    Output Width'x'Height-Hz@Fmt
-f, --fps N/D                         Capture frame rate
-I, --non-interactive                 Non-interactive mode
-S, --list-sources                    List video sources
-L, --list-filters                    List filters
-M, --list-filter-modes               List filter modes
-s, --video-source                    Video source ID
-l, --filter                          Set filter
-m, --filter-mode                     Set filter mode
-P, --plane <id>[:<w>x<h>[+<x>+<y>]]  Use specific plane
-b, --buffer-count                    Number of frame buffers
    --filter-sv-cam-params            File for stereo camera parameters
The dri-card switch should be set corresponding to the connected monitor, where 0 corresponds to Display Port and 1 to HDMI.

The list commands - S, M and L, output a list of sources, modes and filters, and their IDs that then can be set directly from the commandline using a corresponding s, m or l option. Example list output:
    VIDEO SOURCE        ID
    MIPI CSI2 Rx        0
      HDMI Input        1
      USB Webcam        2
      USB Webcam        3
Virtual Video De        4
 
          FILTER        ID
    Optical Flow        0
 
     FILTER MODE        ID
             off        0
        hardware        2
The input and output options allow specifying the resolution and pixel format for source and sink device. The pixel format needs to be specified as fourcc code.
The plane option allows selecting a specific plane and its resolution, which does not have to match the screen resolution. I.e. if the resolution provided in to the output option is greater (in both dimensions) than the resolution provided to the plane option, the input video (which has to match the plane resolution) is displayed in a window on the screen. This enables pass-through of the Zed camera video on HDMI monitors (the Display Port hardware does not support cases where plane resolution != output resolution).
A list of valid planes and their supported video formats can be obtained through
modetest -M xilinx_drm -D amba:xilinx_drm_hdmi -p
CRTCs:
id      fb      pos     size
30      35      (0,0)   (3840x2160)
  3840x2160 30 3840 4016 4104 4400 2160 2168 2178 2250 flags: phsync, pvsync; type: driver
  props:
 
Planes:
id      crtc    fb      CRTC x,y        x,y     gamma size      possible crtcs
26      0       0       0,0             0,0     0               0x00000001
  formats: BG24
  props:
        5 type:
                flags: immutable enum
                enums: Overlay=0 Primary=1 Cursor=2
                value: 0
27      0       0       0,0             0,0     0               0x00000001
  formats: YUYV
  props:
        5 type:
                flags: immutable enum
                enums: Overlay=0 Primary=1 Cursor=2
                value: 0
        24 alpha:
                flags: range
                values: 0 256
                value: 256
28      0       0       0,0             0,0     0               0x00000001
  formats: YUYV
  props:
        5 type:
                flags: immutable enum
                enums: Overlay=0 Primary=1 Cursor=2
                value: 0
        24 alpha:
                flags: range
                values: 0 256
                value: 256
29      30      35      0,0             0,0     0               0x00000001
  formats: AR24
  props:
        5 type:
                flags: immutable enum
                enums: Overlay=0 Primary=1 Cursor=2
                value: 1
        24 alpha:
                flags: range
                values: 0 256
                value: 256
        25 bg_color:
                flags: range
                values: 0 16777215
                value: 16711680

7 Tool Flow Tutorials


Firstly, the SDx Development Environment, version 2017.2, must be installed and working on your host computer, either the Linux or the Windows version.
Further, it is assumed that you have already downloaded zcu102_[es2_]rv_ss.zip and extracted its contents (see section 5.2).
The top-level directory name of the platform is 'zcu102_[es2_]rv_ss'.

Linux:
export SYSROOT=<local>/zcu102_[es2_]rv_ss/sw/sysroot
 
Windows:
Start->Control Panel->System->Advanced->Environment Variables
Create environment variable SYSROOT with value <local>\zcu102_[es2_]rv_ss\sw\sysroot

7.1 Build the Optical Flow sample application


The following steps are virtually identical whether you are running the Linux or Windows version of SDx.
















7.2 Build the Stereo Vision sample application



7.3 Build the File IO sample applications

















sigma_color: 7.72211 sigma_space: 0.901059
elapsed time 9133271
Minimum error in intensity = 0
Maximum error in intensity = 1
Percentage of pixels above error threshold = 0.00168789 Count: 35


8 Platform Details


8.1 Vivado Hardware Design


The Vivado hardware design is packaged inside the DSA located at zcu102_[es2_]rv_ss/hw/zcu102_[es2_]rv_ss.dsa. The DSA also includes the hpfm file that describes the available AXI interfaces, clocks, resets, and interrupts. To open the hardware design in Vivado, run the following command from the tcl console:
% open_dsa zcu102_[es2_]rv_ss/hw/zcu102_[es2_]rv_ss.dsa

8.2 PetaLinux BSP


The PetaLinux BSP is located at zcu102_[es2_]rv_ss/sw/petalinux_bsp. The hdf file exported from the corresponding Vivado project (see 8.1) is available in the project-spec/hw-description/system.hdf subfolder inside to the PetaLinux BSP. To configure and build the PetaLinux BSP, run the following commands:
% petalinux-config --oldconfig
% petalinux-build

8.3 Video Library


The Xilinx software library libvideo is used to interface with V4L2 capture devices, DRM display devices, and SDSoC based hardware accelerators. A prebuilt version of this library is available at zcu102_[es2_]rv_ss/sw/a53_linux/lib/libvideo.a. The public API is available at zcu102_[es2_]rv_ss/sw/a53_linux/inc/video_lib/. The file filter.h describes the filter_s struct that needs to be implemented by SDSoC hardware accelerators along with callback functions.

The libvideo sources are provided as XSDK project and are located at zcu102_[es2_]rv_ss/sw/video_lib/. Perform the following steps from the SDx GUI to build libvideo.a:


9 Other Information


9.1 Known Issues



9.2 Limitations




10 Support


To obtain technical support for this reference design, go to the:


11 References


Additional material that is not hosted on the wiki: