Edge AIoT & Microelectronics (EdgeAIoTM) Engineering Skills for Gifted Students
P4L3.1 AI Model Quantization, Compilation, and Inference with Vitis-AI and PYNQ
Previous Lesson
❖ Overview of Deep Learning Processor (DPU)
❖ DPU design and implementation on FPGA
1. Vitis flow of DPU development
2. Vivado flow of DPU development
❖ PYNQ-DPU overview and implementation
❖ Image classification and object detection (inference) using a pre-trained Resnet50
and Yolov3 models.
Lab4L3.1: Image classification and object detection (inference) using a pre-trained
ResNet50 and Yolov3 models on PYNQ
2
Today’s Lesson
❖ Overview of Vitis-AI,
❖ Virtual machine, Docker, and Docker Container
❖ Model Zoo
❖ AI model quantization and evaluation
❖ AI model compilation for DPU
❖ AI inference/object detection on FPGA
❖ Real Time Streaming Protocol (RTSP)
Lab: Real-time Yolo Object Detection with RTSP Streaming on PYNQ using
Ultra96 FPGA board
3
Intended Learning Outcomes
At the end of this phase, students are expected to:
1. Familiarized with Vitis-AI and its tools
2. Describe AI quantization and compilation
3. To set up a virtual machine (VM)
4. Explain Docker and be able to run a Docker container
5. Be able to quantize and compile an AI model for FPGA deployment
6. Describe RTSP and be able to implement it
7. Be able to deploy the Yolo AI model and run it on FPGA
4
Further Reading/Recommended Books
[1] L. Crockett, et al. “Exploring Zynq MPSoC: With PYNQ and Machine Learning
Applications”. Strathclyde, 2020. [Online]. Available: https://www.zynq-mpsoc-
book.com
[2] S. Sun, J. Zou, Z. Zou, and S. Wang, Experience of PYNQ: Tutorials for
PYNQ-Z2. Cham, Switzerland: Springer, 2023. [Online]. Available:
https://books.google.com/books/about/Experience_of_PYNQ.html?id=4WqxE
AAAQBAJ
[3] DPU-PYNQ [Online]. Available: https://github.com/Xilinx/DPU-PYNQ
[4] Vitis AI User Guide (UG1414): https://docs.amd.com/r/1.4.1-English/ug1414-
vitis-ai/Deep-Learning-Processor-Unit
[5] Vitis-AI Repository. [Online]. Available: https://github.com/Xilinx/Vitis-AI
5
Review of AI Inference on Xilinx FPGA
❖ Involves three steps (including hardware and software developments)
6
AI Development Vitis Flow on FPGA
7
Software Side Processes in FPGA AI Development
❖ Compiled model is loaded to the target device for execution in DPU
8
Vitis-AI Overview
9
Vitis-AI Development Environment
❖ Are set of tools and libraries given by
Xilinx for AI for accelerated AI
development on Xilinx FPGAs.
❖ Distributed via Docker or Linux sources
❖ Includes machine learning
frameworks, Vitis AI models,
development modules, and DPU
overlays.
https://github.com/Xilinx/Vitis-AI/tree/v3.5/
https://xilinx.github.io/Vitis-AI/3.5/html/docs/install/install.html
10
https://docs.amd.com/r/en-US/ug1414-vitis-ai/Vitis-AI-Overview 11
Structure of AI Development on Xilinx FPGA
❖ Software is developed on Vitis AI while the hardware is on Vitis/Vivado
12
Vitis-AI Software (Model) Development Processes
13
Vitis-AI Optimizer
With world-leading model compression technology, you can achieve an impressive reduction
in model complexity, ranging from 5x to 50x, while experiencing minimal accuracy
degradation.
14
Vitis-AI Optimizer
❖ Optimization is optional
❖ Improves latency, power
consumption, and memory
usage
15
Quantization
✓ Approximates of float-point or real numbers to integers
✓ Limits precision to enable efficient storage, processing, and hardware acceleration
Example: Float: 3.08×4 =12.32 (requires 3 integer multiplications)
Integer: 3 x 4 =12 (requires 1 integer multiplications)
16
Vitis-AI Quantizer
✓ Converts 32-bit floating-point weights and activations to fixed-point formats like INT8
✓ Significantly reduces computational complexity while preserving prediction accuracy.
✓ Reduces memory bandwidth demand, faster processing speed and improved power
efficiency compared to the floating-point model.
17
Vitis-AI Quantizer
18
Quantization
Flow in Vitis-AI
19
Vitis-AI Quantizer: Quantization Flow
Install using docker:
[docker] $ conda activate vitis-ai-pytorch
Running quantization:
python resnet18_quant.py --
quant_mode calib --subset_len 200
Evaluate quantized model:
python resnet18_quant.py --quant_mode test
Generate To generate the XMODEL, ONNX, .pt for
compilation:
python resnet18_quant.py --quant_mode test --subset_len 1
--batch_size=1 --deploy
20
Vitis-AI Compiler
✓ The Vitis AI compiler maps the AI model to a highly efficient instruction set and
dataflow model (x.model).
✓ Performs sophisticated optimizations such as layer fusion, instruction scheduling,
and reuses on-chip memory as much as possible.
21
Vitis-AI
Compiler
Workflow
22
Vitis-AI Compiler
✓ Meticulously analyzes and visualizes AI applications to identify bottlenecks
✓ Optimally allocate computing resources across different devices.
✓ Empowers developers with comprehensive insights for efficient performance tuning.
23
Vitis-AI Library
✓ Is a collection of high-level libraries and APIs designed for efficient AI
inference with DPUs.
✓ Seamlessly integrating with the Xilinx Runtime (XRT) and built upon the
Vitis AI runtime with Vitis runtime unified APIs.
✓ Ensures a smooth and unified experience.
✓ Allows you to focus more on developing your applications rather than the
underlying hardware.
24
Vitis-AI Library
25
Vitis-AI Runtime
✓ Software serving as an interface between Vitis AI libraries and the
accelerator, by leveraging the Xilinx-runtime (XRT)
The following are the features of the AI runtime API:
❖ Asynchronous submission of jobs to the accelerator
❖ Asynchronous collection of jobs from the accelerator
❖ C++ and Python implementations
❖ Support for multi-threading and multi-process execution
26
Vitis-AI Runtime Stack (VART Stack)
27
What is Docker?
✓ An open-source platform for packaging applications and all their dependencies into
lightweight, portable containers.
✓ Containers are standalone, executable software packages that include code, runtime, system
tools, libraries, and settings needed to run an application consistently across different
environments.
Key Usages of Docker
❖ Simplifies Development
❖ Consistent Deployment
❖ Resource Efficiency
❖ Scalability
❖ Isolation and avoiding dependency conflicts.
28
Vitis-AI Containers
✓ Vitis AI release uses containers to distribute the AI software.
✓ The release consists of the following components:
❖ Tools container
✓ Containers distributed through Docker Hub (https://hub.docker.com/u/Xilinx )
✓ Unified compiler flow
✓ Pre-built conda environment to run frameworks
E.g., conda activate vitis-ai-tensorflow for TensorFlow-based flows
✓ Versal Runtime tools
29
https://github.com/Xilinx/Vitis-AI
Vitis AI Host (Developer) Machine Requirements
30
Vitis-AI Model Zoo
❖ Is a curated collection of finely tuned deep learning models designed to
accelerate the deployment of AI inference on AMD platforms.
❖ Encompasses diverse applications, such as ADAS/AD, video surveillance,
robotics, and data centers.
❖ Equips developers with powerful tools and optimized models to unlock the
advantages of deep learning acceleration.
31
Vitis-AI Model Zoo
Vitis AI Model Zoo (https://github.com/Xilinx/Vitis-AI/tree/v3.5/model_zoo)
32
Vitis-AI Model Zoo: Links
Model Zoo Details & Performance table online:
https://xilinx.github.io/Vitis-AI/3.5/html/docs/reference/ModelZoo_Github_web.htm
Model Zoo Github:
https://github.com/Xilinx/Vitis-AI/tree/v3.0/model_zoo
Vitis AI Copyleft Model Zoo.
https://github.com/Xilinx/Vitis-AI-Copyleft-Model-Zoo
Documentation:
https://xilinx.github.io/Vitis-AI/3.5/html/docs/workflow-model-zoo.html
Spreadsheet:
https://xilinx.github.io/Vitis-AI/3.5/html/_downloads/ff9554ff9ff6240811c20ede15113dbd/ModelZoo_Github.xlsx
33
Deploy a Custom Model on PYNQ-DPU After
Compilation
❖ Copy and paste your custom PYNQ-DPU folder (containing all the
hardware/model files and the pynq setup files) to the /home/xilinx
directory on the board.
❖ Copy and paste the pynq_dpu in the notebooks folder of the PYNQ-DPU
GitHub to the Jupyter_notebooks folder on the board.
❖ Open a terminal in the Jupyter environment, and run
>> pip3 install /home/xilinx/DPU-PYNQ-3.5 --no-build-isolation
❖ Run the desired application for the inference
34
What is a Virtual Machine?
• Virtual Machine (VM):
➢ Emulation of a computer system
➢ Allow you to run another operating system (guest OS) on your host OS
➢ Examples: Oracle VirtualBox, VMware, etc.
• In our course, we will use the Ubuntu Linux (18.04) OS, you can:
➢ Dual boot the Ubuntu OS along with your existing OS
➢ Or run the Ubuntu OS inside the Oracle VirtualBox (Suggested)
35
Oracle Virtual Machine VirtualBox
❖ Is a cross-platform virtualization application.
❖ Extends the capabilities of your existing computer
to run multiple OSes at the same time.
❖ Simple but also very powerful.
❖ Supports Windows, Linux, MacOS etc
Url: https://www.virtualbox.org/wiki/Downloads
36
Why is Virtualization Useful?
❖ Running multiple operating systems simultaneously
❖ Easier software installations and distribution.
❖ Testing and disaster recovery (using snapshot).
❖ Infrastructure consolidation (Virtualization can
significantly reduce hardware and electricity costs)
37
Oracle VBox Provided Features
❖ Portability
❖ Guest Additions: shared folders, seamless windows, 3D virtualization
❖ Comprehensive hardware support
✓ Guest multiprocessing (SMP),
✓ USB device support, Multiscreen resolutions, etc.
❖ Multigeneration branched snapshots
❖ Remote machine display (uses VirtualBox Remote Desktop Extension (VRDE)
❖ VM groups
❖ Clean architecture and unprecedented modularity
https://www.virtualbox.org/manual/ch01.html 38
Some Terminology
❖ Host operating system (host OS): Primary OS on which the VM is
installed.
❖ Guest operating system (guest OS): The OS running inside the VM
❖ Guest Additions: Software packages to be installed inside a VM to
improve the performance of the guest OS and to add extra features.
❖ Snapshot: Record of the VM state at a particular time.
39
Guest Additions
✓ Consists of device drivers and system applications that optimize the
guest OS for better performance and usability.
✓ Provide these features:
❖ Time synchronization
❖ Mouse pointer integration
❖ Shared clipboard
❖ Shared folders
❖ Automated logins
❖ Better video support
❖ Drag and Drop
❖ Seamless windows
❖ Generic host/guest communication channels
https://www.virtualbox.org/manual/ch04.html 40
Ubuntu and Linux System
❖ OS consists of Kernel and File System
❖ Linux is a Unix-like Kernel developed by
Linus Torvalds in 1991
❖ It is Open source
❖ Ubuntu is a distribution of Linux
41
Linux File System
❖ Is a hierarchical structure that
organizes all files and directories
❖ Treats all software and devices as
files
42
Linux File System
❖ The Linux filesystem hierarchy is well documented in
the Linux Filesystem Hierarchy Standard (FHS) :
https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf
❖ At the top is the root directory (/) that holds everything
❖ Inside the root directory, the /home directory stores the users’ personal
directories (i.e. the home directories, one for each user)
❖ The /etc directory contains system configurations
43
Linux File System
44
Root Privilege
❖ On every Linux system, the root account is a special user with
administrative rights
❖ Some Linux commands requires the root privilege
❖ It is possible to run a command with the root privilege without logging into
the root account, if your account is in the sudoers list
❖ Simply by adding sudo (stands for superuser do) before the command,
then type your user password:
❖ sudo command
45
Linux
Commands
https://www.loggly.com/wp-
content/uploads/2015/05/Linux-Cheat-
Sheet-Sponsored-By-Loggly.pdf
46
Exploring Files and Directories
❖ Type the command ‘ls’ to prints the names of the files and
directories in the current directory
❖ Type ‘ls -l’ to print the result in long format
❖ From left to right:
➢ The file permissions
➢ The number of links
➢ Owner name
➢ Owner group
➢ File size
➢ Time of last modification
➢ The file or directory name
47
Creating a Directory
❖ To create a new directory, we use the command ‘mkdir’
(short for make directory):
❖ Since “new_folder” is a relative path, the new directory is created in
the current working directory
48
Ownership of Files/Directories
❖ Every files/directories are associated with 3 classes of users:
1. User: The owner. By default, it is the user who created the file/directory
2. Group: The owner group. A user-group that contains multiple users, all users in
the same group have the same group permissions to the file/directory
3. Other: Any other users who have access to the file/directory
(Not the owner nor those users in the owner group)
Permissions
❖ Every files/directories are assigned 3 types of permissions for all
3 classes of users:
1. Read (r): Permission to open and read a file, or to open a directory
2. Write (w): Permission to modify the contents of a file, or to add/remove/rename
the files stored in a directory
3. Execute (x): Permission to run a file (as a program), or to ‘cd’ into a directory
Note: The ‘r’ and ‘w’ permission for a directory work only if ‘x’ permission is also set
50
Permissions
❖ Permissions can be checked using ‘ls -l’
d: a
directory
-: a file Permission Owner Group User Other
s Owner Group
r: read
w: write
x: execute
-: no permissions
51
Changing Permissions
❖ Permissions can be changed using the ‘chmod’ (change mode) command
❖ Syntax:
➢ chmod [ugoa…] [-+=] permissions filename
➢ u: user, g: group, o: other, a: all (same as ugo)
➢ -: Removes the specific permissions
➢ +: Adds the specific permissions
➢ =: Sets the permissions to the specific permissions
(if no permissions after ‘=’, all permissions are removed)
52
Changing Permissions
❖ For example, ‘chmod u+x hello_world.sh’ would add the Execute
permission to ‘hello_world.sh’ for ‘eeuser’
❖ Alternatively, ‘chmod 777 hello_world.sh’ sets all permissions
53
Major Linux Text Editors
❖ Vim:
Powerful, modal editor favored by experienced users for speed and
customization. Steep learning curve, but highly efficient.
❖ Nano
Simple, user-friendly terminal editor suitable for beginners and quick
edits. Easy navigation and commands.
❖ Gedit
Default graphical text editor in many GNOME-based Linux distros.
Simple GUI editor with syntax highlighting.
❖ GNU Emacs
Highly extensible and customizable, with built-in tools for programming,
writing, and more. Supports GUI and CLI.
54
Lab4L4.1: Real-time Yolo Object Detection with RTSP
Streaming on PYNQ using Ultra96 FPGA board
55
Happy learning, please ask questions !!!
18
56
