/
reVISION Getting Started Guide 2017.4 rev2

reVISION Getting Started Guide 2017.4 rev2

Oct 22, 2019

 

Table of Contents

1 Revision History


This Getting Started Guide complements the 2017.4 rev2 version of the ZCU102 and ZCU104 reVISION platforms.
For other versions, refer to the reVISION Getting Started Guide overview page.

Change Log:
rev2

  • ZCU102 and ZCU104 boards are supported

  • Samples now use the GStreamer framework, included with the reVISION platform

  • Sample design examples are built as GStreamer plugins

  • Sample apps are included, to exercise the sample plugins

  • Modified the samples directory structure, with a new workspace structure provided for the user

  • Improved and augmented many features of the xfopencv library


rev1

  • Update to 2017.4 SDSoC tools version

  • Update to 2017.4 xfOpenCV libraries version

  • Update to 2017.4 IP version

  • Cascade platform interrupts to PS GIC using AXI interrupt controller

  • Enable HP2 port in platform

  • Use Sony IMX274 v4l2 subdevice driver

  • Add filter2d live IO sample

  • Minor fixes and improvements

 

 

2 Introduction


The Xilinx reVISION stack includes a range of development resources for platform, algorithm and application development. This includes support for the most popular neural networks including AlexNet, GoogLeNet, VGG, SSD, and FCN. Additionally, the stack provides library elements including pre-defined and optimized implementations for CNN network layers, required to build custom neural networks (DNN/CNN). The machine learning elements are complemented by a broad set of acceleration ready OpenCV functions for computer vision processing. For application level development, Xilinx supports industry standard frameworks and libraries including Caffe for machine learning and OpenCV for computer vision. The reVISION stack also includes development platforms from Xilinx and third parties, including various types of sensors. For more information go to the Xilinx reVISION webpage.

 

3 Overview


The below figure shows a block diagram of the reVISION single sensor design:

  • video sources (or capture pipelines) are highlighted in blue color

  • computer vision accelerators implemented as memory-to-memory (m2m) pipelines in red color and

  • video sinks (or output/display pipelines) in green color

 



3.1 Platform


The ZCU102/ZCU104 single-sensor reVISION platform supports the following video interfaces:

Sources:

  • USB2/3 camera up to 1080p60 or stereo 1080p30

    • The USB controller is part of the processing system (PS). It uses the standard Linux Universal Video Class (UVC) driver.

  • HDMI Rx up to 4k60

    • The HDMI capture pipeline is implemented in the programmable logic (PL) and consists of HDMI Rx Subsystem, Video Processing Subsystem (Scaler only configuration), and Frame Buffer Write. The HDMI Rx subsystem receives and decodes the HDMI data stream from an HDMI source and converts it to AXI4-Stream. The Video Processing Subsystem converts the incoming color format (one of RGB, YUV444, YUV422) to YUV422 and optionally scales the image to the target resolution. The Frame Buffer Write IP writes the YUV422 stream to memory as packed YUYV format. The HDMI capture pipeline uses the V4L Linux framework.

  • MIPI CSI via optional FMC card up to 4k60

    • The MIPI capture pipeline is implemented in the PL and consists of Sony IMX274 image sensor, MIPI CSI2 Subsystem, Demosaic, Gamma, Video Processing Subsystem (CSC configuration), Video Processing Subsystem (Scaler only configuration), and Frame Buffer Write. The IMX274 image sensor provides raw image data over the camera sensor interface (CSI) link. The MIPI CSI2 Subsystem receives and decodes the incoming data stream to AXI4-Stream. The Demosaic IP converts the raw image format to RGB. The Gamma IP provides per-channel gamma correction functionality. The VPSS-CSC provides color correction functionality. The VPSS-Scaler converts the RGB image to YUV422. The Frame Buffer Write IP writes the YUV422 stream to memory as packed YUYV format. The MIPI capture pipeline uses the V4L Linux framework.

Sinks:

  • HDMI Tx up to 4k60

    • The HDMI display pipeline is implemented in the PL and consists of a Video Mixer and HDMI Tx Subsystem. The Video Mixer is configured to read one ARGB and two YUYV layers from memory. In the provided design examples, only a single YUYV layer is used. The video layers are then composed and alpha-blended into a single output frame which is sent to the HDMI Tx Subsystem via AXI4-Stream. The HDMI Tx Subsystem encodes the incoming video into an HDMI data stream and sends it to the HDMI display. The HDMI display pipeline uses the DRM/KMS Linux framework.

  • DP Tx up to 4k30

    • The DP display pipeline is configured for dual-lane mode and is part of the PS. It includes a simple two-layer blender with run-time programmable color format converters per layer. The two layers are always full screen matching the target display resolution. The DP display pipeline uses the DRM/KMS Linux framework.

 

3.2 Design Examples


File I/O:
These are the simplest design examples. Typically they will read a frame of video from a standard image file using a standard OpenCV call (such as cv::imread()), process that frame with a call to an xfopencv function, and output the result to a file, (e.g., using cv::imwrite()). They illustrate use of five different xfopencv HW accelerated versions of popular OpenCV functions.

  • Bilateral Filter

  • Harris Filter

  • Dense Optical Flow

  • Stereo Vision (Depth Detection)

  • Warp Transformation


Live I/O:
These examples input and output live video.

  • Dense Optical Flow - requires LI-IMX274MIPI-FMC or HDMI source or See3CAM_CU30 USB camera

    • This algorithm uses two successive images in time, and calculates the direction and magnitude of motion at every pixel position in the image. The calculation is a simple implementation of the Lucas–Kanade method for optical flow estimation. The optical flow algorithm returns two signed numbers at each pixel position, representing up or down motion in the vertical direction, and left or right motion in the horizontal direction. The brightness of the false-color output, from black up to bright color, indicates the magnitude of the motion, and the color indicates the direction.

 

  • Stereo Vision (Depth Detection) - requires ZED USB stereo camera

    • This algorithm uses two side-by-side images from the stereo camera taken at the same moment in time, and calculates the depth, or distance from the camera, at every pixel position in the image. The stereo block-matching algorithm calculates depth based on binocular parallax, similar to the way human eyes perceive depth. The depth map is coded in false colors. Objects far away appear deep blue. Closer and closer objects appear in rainbow succession green, yellow, orange, red, purple and finally white, closest to the camera.

 

  • Filter2D - requires LI-IMX274MIPI-FMC or HDMI source or See3CAM_CU30 USB camera

    • Convolution is a common image processing technique that changes the intensity of a pixel to reflect the intensities of the surrounding pixels. This is widely used in image filters to achieve popular image effects like blur, sharpen, and edge detection. The implemented algorithm uses a 3x3 kernel with programmable filter coefficients.

 

  • Triple - combine the above three designs in a single project (ZCU102 only)

    • All three designs are available at once in HW. The test application provided sets up three pipelines, from three independent video sources, via the three HW accelerated plugins, to three planes of the video mixer for output on the HDMI display.


Below table shows the performance matrix of the live I/O samples on the supported platforms:

 

ZCU102

ZCU104

filter2d

2160p30

2160p30

optical_flow

2160p52

2160p30

stereo

1080p16

720p18

Note: Work to bring the performance on the ZCU104 up to par with the ZCU102 is ongoing.

 

4 Software Tools and System Requirements

 

4.1 Hardware


Required:

  • ZCU104 Evaluation Board, or

  • ZCU102 Evaluation Board

    • rev 1.0 with ES2 silicon or

    • rev 1.0 with production silicon

  • Micro-USB cable, connected to laptop or desktop computer for the terminal emulator

  • SD card (ZCU102) or

  • micro-SD card (ZCU104)


Optional (needed for live I/O examples):

 

4.2 Software


Required:

 

4.3 Licensing

 

  • Important: Certain material in this reference design is separately licensed by third parties and may be subject to the GNU General Public License version 2, the GNU Lesser General License version 2.1, or other licenses.
    The Third Party Library Sources zip file provides a copy of separately licensed material that is not included in the reference design.

  • You will need only the SDSoC license to build the design. You can evaluate for 60-days or purchase it here.


Steps to generate the license:

  1. Log in here with your work E-mail address (If you do not yet have an account, follow the steps under Create Account)

  2. Generate a license from “Create New Licenses” by checking "SDSoC Environment, 60 Day Evaluation License"

  3. Under system information, give the host details.

  4. Proceed until you get the license agreement and accept it.

  5. The License (.lic file) will be sent to the email-id mentioned in the login details.

  6. Copy the license file locally and give the same path in the SDSoC license manager.

 

4.4 Compatibility


The reference design has been tested successfully with the following user-supplied components.

Monitors:

Make/Model

Native Resolution

Viewsonic VP2780-4K

3840x2160

Acer S277HK

3840x2160

Dell U2414H

1920x1080


HDMI Sources:

Make/Model

Resolutions

Nvidia Shield TV

3840x2160, 1920x1080

OTT TV BOX M8N

3840x2160, 1920x1080, 1280x720

Roku 2 XS

1920x1080, 1280x720

TVix Slim S1 Multimedia Player

1920x1080, 1280x720


USB3 Cameras:

Make/Model

Resolutions

ZED stereo camera

3840x1080, 2560x720

See3CAM_CU30

1920x1080, 1280x720


DisplayPort Cables:

  • Cable Matters DisplayPort Cable-E342987

  • Monster Advanced DisplayPort Cable-E194698

 

 

5 Design File Hierarchy


The Zynq UltraScale+ MPSoC reVISION Platform zip file is released with the binary and source files required to create Xilinx SDx projects and build the sample applications. The sample applications are built as GStreamer plugins and test designs to exercise them. The provided samples include five file I/O examples and four live I/O examples. The file I/O examples read an input image file and produce an output image file whereas the live I/O examples take live video input from a video source and output live video on a display.

The zcu102_rv_ss.zip or zcu102_es2_rv_ss.zip or zcu104_rv_ss.zip zipfile is provided, containing the reVISION Platform. This is the directory structure:

  • hw contains the .dsa file describing the hardware platform.

  • petalinux_bsp contains device tree info, hardware description files, and other system setup files. An advanced user has the option of creating their own platform.

  • samples contains sample app code. Each sample directory has a .json file describing the build process. These are the SDx sample apps that appear in the "Template" dialog when creating a new project using the reVISION platform. The file_IO projects are self-contained. The live_IO projects are more complex, and are built in several steps. See section 7.

  • sd_card contains pre-built SD card images that enable the user to run the live I/O example applications on the ZCU10x board.

  • sw contains software - bootloaders and other code and support files for the processors on the ZCU10x target board.

  • workspaces contains a workspace directory structure you may use to build the live_IO samples. See section 7.

 

zcu102_rv_ss (or zcu102_es2_rv_ss, or zcu104_rv_ss) ├── hw │ └── zcu102_es2_rv_ss.dsa ├── petalinux_bsp ├── samples │ ├── file_IO │ │ ├── bilateral_fileio │ │ ├── harris_fileio │ │ ├── opticalflow_fileio │ │ ├── steoreolbm_fileio │ │ └── warptransform_fileio │ └── live_IO │ ├── filter2d │ ├── optical_flow │ ├── stereo │ └── triple ├── sd_card │ ├── filter2d │ ├── optical_flow │ ├── stereo │ └── triple ├── sw │ ├── a53_linux │ │ ├── boot │ │ ├── image │ │ ├── inc │ │ └── qemu │ ├── prebuilt │ ├── sysroot │ └── zcu102_es2_rv_ss.spfm ├── workspaces │ ├── ws_f2d │ │ └── gst │ │ ├── allocators │ │ ├── apps │ │ ├── base │ │ └── plugins │ ├── ws_of │ │ └── gst │ │ ├── allocators │ │ ├── apps │ │ ├── base │ │ └── plugins │ ├── ws_sv │ │ └── gst │ │ ├── allocators │ │ ├── apps │ │ ├── base │ │ └── plugins │ ├── ws_triple │ │ └── gst │ │ ├── allocators │ │ ├── apps │ │ ├── base │ │ └── plugins │ └── ws_video │ ├── video_cmd │ └── video_lib └── zcu102_es2_rv_ss.xpfm

 

 

6 Installation and Operating Instructions

 

6.1 Board Setup


Required:

  • Connect power supply to the 12V power connector.

  • Display

    • Connect a DisplayPort cable to DisplayPort connector on the board; connect the other end to a monitor OR

    • Connect an HDMI cable to HDMI Output (top HDMI connector) on the board; connect the other end to a monitor.

    Note: Certain monitors have multiple HDMI ports supporting different HDMI standards. Make sure you choose an HDMI 2.0 capable port (if available) for 4k60 performance.
    Note: Make sure you only connect either DisplayPort or HDMI Output on the board, not both, otherwise the design might malfunction.

 

  • Connect micro-USB cable to the USB-UART connector; use the following settings for your terminal emulator:

    • Baud Rate: 115200

    • Data: 8 bit

    • Parity: None

    • Stop: 1 bit

    • Flow Control: None

 

  • Insert SD card (FAT formatted) with pre-built image copied from one of the following directories:

    • optical_flow: ./<platform>/sd_card/optical_flow

    • stereo: ./<platform>/sd_card/stereo

    • filter2d: ./<platform>/sd_card/filter2d

    • triple: ./<platform>/sd_card/triple


Optional:

  • Connect an HDMI cable to HDMI Input (bottom HDMI connector) on the board; connect the other end to an HDMI source

  • Connect the See3CAM_CU30 or ZED USB camera to the USB3 micro-AB connector via the Xilinx USB3 micro-B adapter

  • Connect the LI-IMX274MIPI-FMC module to the FMC connector on the board (use the HPC0 connector on the ZCU102)
    Note: Vadj needs to be set to 1.2V for correct operation of the daughter card. If the FMC card does not function, please follow the instructions explained in Answer Record AR67308 for rev 1.0 and beyond to check and/or set Vadj.


ZCU102 Jumpers & Switches:

  • Set boot mode to SD card

    • SW6[4:1]: off,off,off, on

  • Configure USB jumpers for host mode. The drawing shows the area on the board near the USB connector.

    • J110: 2-3

    • J109: 1-2

    • J112: 2-3

    • J7: 1-2

    • J113: 1-2



ZCU104 Jumpers & Switches:

  • Set boot mode to SD card

    • SW6[4:1]: off,off,off, on



6.2 Extract the design zip files


Download and unzip the reference design zip file matching your silicon version (see Section 4.2).

  • For Linux, use the unzip utlity.

  • For Windows, make sure that the reference design zip file is unzipped in a directory path which contains no spaces. Use the 7zip utility and follow the steps below. If you need 7zip, get it here 7zip. When prompted to confirm file replace, select ‘Auto Rename’

 

7 Tool Flow Tutorials


The SDx Development Environment, version 2017.4, must be installed and working on your host computer, either the Linux or the Windows version.

This guide will walk you through the process of building the sample designs. In step 6.2 above, you unzipped your platform files, and noted the exact directory paths.

The path to the extracted platform will be needed to tell SDx where your custom platform resides. You need to set the SYSROOT environment variable to point to a directory inside the platform. The platform root directory is abbreviated to <platform> below and needs to be replaced with your local path.

  • IMPORTANT: Before starting SDx, set the SYSROOT environment variable to point to the Linux root file system, i.e the sysroot top directory you just unzipped.

Linux: export SYSROOT=<platform>/sw/sysroot Windows: Start->Control Panel->System->Advanced->Environment Variables Create environment variable SYSROOT with value <platform>\sw\sysroot

You can also set SYSROOT for all projects in your SDx Environment by opening the Menu 'Window' → 'Preferences' and adding 'sysroot' variable to 'C/C++' → 'Build' → 'Environment'.



The platform ships with five file IO and three live IO design examples demonstrating popular OpenCV functions accelerated on the programmable logic. A fourth live I/O example shows how to combine the other three live I/O designs into one design, allowing the three accelerated functions to reside and run in parallel in the FPGA.

With this release of reVISION the live IO sample design examples are based on GStreamer. See GStreamer The open source GStreamer framework code is included with the reVISION platform, and design examples are built as GStreamer plugins. Code for test applications is provided as well, allowing you to compile apps that will set up and run video pipelines using the plugins. Pipelines may be run using the gst-launch-1.0 utility, or by your own app compiled against the gstreamer libraries. An example test app called gstdemo is provided for each of the platform samples. The four sample <names> are filter2d, optical_flow, stereo, and triple. See the ./workspaces/<name>/gst/apps/<name>. directory for each sample.

A GStreamer plugin is a shared library. In the case of the reVISION sample designs, the GStreamer plugin consists of two linked parts. These "top" and "bottom" parts are separate shared libraries produced by separate project builds. The top part is the GStreamer plugin itself, containing the code for interfacing with the GStreamer framework. See the ./workspaces/<name>/gst/plugins/<name> directory.
The top part links with the bottom part which contains the code for the HW accelerated function(s). This bottom project generates the BOOT.BIN file containing the programmable logic used for the HW function(s). These are SDx projects: See the ./samples/live_IO/<name> directory.

7.1 Build the Live_IO Optical Flow sample application


The following steps are virtually identical whether you are running the Linux or Windows version of SDx.

There is a ./workspaces/... folder structure already set up for the four live_IO samples as part of the platform :

├── workspaces │ ├── ws_f2d │ ├── ws_of │ ├── ws_sv │ ├── ws_triple


You should copy these workspaces to the directory where you want to work. Look at the optical_flow workspace area supplied with the platform. All files under ./gst/ are supplied exactly as shown. The ./opticalflow directory is the SDx project you will create to build the low level accelerator code - note that you'll create this 'opticalflow' SDx project directly under the ws_of workspace. Note that ./gst/ is also directly under ./ws_of :

├── ws_of │ ├── gst │ │ ├── allocators │ │ │ ├── gstsdxallocator.c │ │ │ └── gstsdxallocator.h │ │ ├── apps │ │ │ └── optical_flow │ │ │ └── main.c │ │ ├── base │ │ │ ├── gstsdxbase.c │ │ │ └── gstsdxbase.h │ │ └── plugins │ │ └── optical_flow │ │ ├── gstsdxopticalflow.cpp │ │ └── gstsdxopticalflow.h │ └── opticalflow │ └── src │ ├── optical_flow_sds.cpp │ └── optical_flow_sds.h


For a given workspace, such as ./ws_of/, the arrangement of these subdirectories must be preserved. This is because the various projects depend on each other in that they need to know the paths to each other's include files and library files. As long as you keep this structure, you're OK - i.e. you may copy the ./ws_of/ tree with everything just as shown, and put it anywhere you want to work.

If you are working on Linux, there is no restriction on where you put these workspaces. Some people may want to work directly in the ./workspaces/ directory under the platform itself, and others may want to copy it elsewhere so that the original area remains untouched.

If you are working on Windows there is a restriction, i.e. file path lengths are restricted to 256 characters. The Xilinx build process creates some very deep directory structures with long names as it goes through the build process. You are advised, therefore, to keep the path to the workspace as short as possible. E.g. C:\ws_of\...

  • Start SDx and select workspace ./ws_of. Make sure you use the same shell to run SDx as the one where you have set $SYSROOT.

  • Close the Welcome screen and select 'File'→'Import'→'General'→'Existing Projects into Workspacel'→'Next'.

 

 



  • In the Import dialog, to the right of 'Select root directory', click 'Browse'.

 



  • By default you're already in the directory you want ./workspaces/ws_of, just click 'OK'.

 



  • You should see a list of projects, with gstdemo, gstsdxallocator, gstsdxbase, and gstopticalflow selected, click 'Finish'.

 



  • Back at the main window, the four imported projects appear in the Project Explorer pane. Now select 'File'→'New'→'SDx Project'... from the menu bar.

 



  • This brings up the Project Type dialog box, with Application Project selected, click 'Next'.

 



  • In the 'Create a New SDx Project' dialog, enter Project name 'opticalflow', click 'Next'.

 



  • In the Platform dialog, click 'Add Custom Platform', find your way to the top directory where you unzipped the reVISION platform, called, for example, zcu102_rv_ss. Click 'OK'.

 

 



  • Back in the Platform dialog, the new platform appears in the list, but is not selected. Select it, then click 'Next'.

 



  • In the System configuration dialog, under Output type, select 'Shared Library', click 'Next'.