Introduction to Deep Learning in Signal Processing &

Communications with MATLAB

Dr. Amod Anandkumar

Pallavi Kar
Application Engineering Group, Mathworks India

© 2019 The MathWorks, Inc.

1
Different Types of Machine Learning
Type of Machine Learning Categories of Algorithms

• Output is a choice between classes

Classification
(True, False) (Red, Blue, Green)
Supervised
Learning
• Output is a real number
Develop predictive Regression
(temperature, stock prices)
model based on both
Machine input and output data
Learning

Unsupervised • No output - find natural groups and

Clustering
Learning patterns from input data only

Discover an internal
representation from
input data only

2
What is Deep Learning?

3
Deep learning is a type of supervised machine learning in which a
model learns to perform classification tasks directly from images, text,
or sound.

Deep learning is usually implemented using a neural network.

The term “deep” refers to the number of layers in the network—the

more layers, the deeper the network.

4
Why is Deep Learning So Popular Now?

Human
Accuracy

Source: ILSVRC Top-5 Error on ImageNet 5

Vision applications have been driving the progress in deep learning
producing surprisingly accurate systems

6
Deep Learning success enabled by:

• Labeled public datasets

• Progress in GPU for acceleration

AlexNet VGG-16 ResNet-50 ONNX Converter

• World-class models and PRETRAINED
MODEL
PRETRAINED
MODEL
PRETRAINED MODEL MODEL CONVERTER

connected community Caffe GoogLeNet

TensorFlow-
IMPORTER PRETRAINED Keras Inception-v3
MODEL IMPORTER MODELS 7
Deep Learning is Versatile MATLAB Examples Available Here

8
Many Network Architectures for Deep Learning
Directed Acyclic
Series Network Recurrent Network
Graph Network

and more
(GAN, DQN,…)

AlexNet ResNet
LSTM
YOLO R-CNN
9
 Convolutional Neural Networks
Edges
Shapes
Objects

A lot of data
is required

Convolution layers Fully-connected

layers 10
 Deep Learning Inference in 4 Lines of Code

• >> net = alexnet;

• >> I = imread('peacock.jpg')
• >> I1 = imresize(I,[227 227]);
• >> classify(net,I1)
• ans =
• categorical
• peacock

11
 Understanding network behavior using visualizations

Filters
…
Deep Dream
Activations

▪ Custom visualizations
– Example: Class Activation Maps
(See blog post)

12
Visualization Technique – Deep Dream

deepDreamImage(...
net, ‘fc8', channel,
'NumIterations', 50, ...
'PyramidLevels', 4,...
'PyramidScale', 1.25);

Synthesizes images that strongly activate

a channel in a particular layer

Example Available Here

13
What is Training?

Images
Network
Training
Labels
Trained Deep
Neural Network
Large Data Set

During training, neural network architectures learn features directly

from the data without the need for manual feature extraction
14
 What Happens During Training?
AlexNet Example

Training Data Labels

Layer weights are learned

during training
15
 Visualize Network Weights During Training
AlexNet Example

Training Data Labels

First Convolution
Layer

16
Visualize Features Learned During Training
AlexNet Example

Sample Training Data Features Learned by Network

17
Visualize Features Learned During Training
AlexNet Example

Sample Training Data Features Learned by Network

18
Deep Learning Workflow

ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH
DATA DATA MODELS SYSTEMS

Files Data Augmentation/ Hardware-Accelerated Desktop Apps

Transformation Training

Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems

Sensors Import Reference Network Visualization Embedded Devices and

Models Hardware

19
Deep Learning Challenges
Data
▪ Handling large amounts of data
▪ Labeling thousands of signals, images & videos
▪ Transforming, generating, and augmenting data (for different domains)
Training and Testing Deep Neural Networks
▪ Accessing reference models from research
▪ Understanding network behaviour
▪ Optimizing hyperparameters
▪ Training takes hours-days
Rapid and Optimized Deployment
▪ Desktop, web, cloud, and embedded hardware

20
Working with Signal Data

Signals
Sequence Fully Connected
Labels Layers Layers Trained Model

Signals Transform Train

Convolutional Trained Model
Labels Network (CNN)
21
 Long Short Term Memory Networks from RNNs
▪ Recurrent Neural Network that carries a memory cell throughout the process
▪ Sequence Problems – Long term dependency does not work well with RNNs

c0 C1 Ct

22
RNN to LSTM

Cell state

Input gate
Forget gate

Update

Output gate

23
 Example: Speaker Gender Recognition Using Deep Learning
LSTM Network for Audio based Speaker Classification
audioDatastore
1. Train
Training Data

− Male

− Female

Time-varying
features:
Mozilla Common
- MFCC
Voice Dataset - Pitch
- ...

2. Predict
Test Data

− Male

− Female

24
Some audio and speech applications following CNN workflows

27
https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf

28
Solution2: Speech Command Recognition with Deep Learning(MATLAB)
Integrated Analytic
Preprocessed Models and Algorithms
Raw Data Systems
Data

...

29
43
44
45
46
Data augmentation allows training more advanced networks and
generating more robust models

Original Simulate:
Dataset - lower voice
- varying level of environmental interference
- recording hardware effects
- distortions due to use of different microphones
- different speaker vs microphone positions or room sizes in
enclosed environments
- ... 50
 % Cycle continuously and automatically through files in datastore

Serial
mfccFile = zeros(numel(ads.Files),numMfcc)
while hasdata(ads)
[data,info] = read(ads);
% do something with data
end

% Partition and explicit parfor parallel processing pattern

N = numpartitions(ads);

parfor index=1:N
subads = partition(ads,N,index);
Parallel

while hasdata(subads)
data = read(subads);
% do something with data
end
end

% Tall array and "implicit" parallel processing pattern

T = tall(ads);

dcOffsets = cellfun(@(x) mean(x), T,'UniformOutput',false);

gather(dcOffsets);
51
Package Speech Recognition App

52
53
BREAK

54
Deep Learning Challenges
Data
Not a deep learning expert
✓ Handling large amounts of data
▪ Labeling thousands of signals, images & videos
▪ Transforming, generating, and augmenting data (for different domains)
Training and Testing Deep Neural Networks
✓ Understanding network behavior
▪ Accessing reference models from research
▪ Optimizing hyperparameters
▪ Training takes hours-days
Rapid and Optimized Deployment
▪ Desktop, web, cloud, and embedded hardware

55
BREAK

56
57
Speech 2 text intro

58
59
60
61
62
Audio Labeler

▪ Work on collections of
recordings or record new audio
directly within the app
▪ Navigate dataset and playback
interactively
▪ Define and apply labels to
– Entire files
– Regions within files

▪ Import and export audio folders,

label definitions and datastores

63
Apps for Data Labeling

64
Label Images Using Image Labeler App

65
 Accelerate Labeling With Automation Algorithms

Learn More

66
 Perform Bootstrapping to Label Large Datasets

Images
Videos Train
Network
Labeling Apps Ground Truth

Improve More Propose Labels

Network Ground Truth

Automation
Algorithm

67
Example – Semantic Segmentation Available Here

▪ Classify pixels into 11 classes

– Sky, Building, Pole, Road,
Pavement, Tree, SignSymbol,
Fence, Car, Pedestrian, Bicyclist

▪ CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in
video: A high-definition ground truth database." Pattern Recognition Letters Vol 30, Issue
2, 2009, pp 88-97. 68
Example: Singing Voice Separation

▪ Source separation
▪ Based on U-Net architecture

69
Synthetically generating labeled data

70
Modulation Classification with Deep Learning

▪ Generate synthetic modulated signals

▪ Apply channel impairments
▪ Train a CNN to classify modulation types

71
Deep Learning Challenges
Data
✓ Handling large amounts of data
✓ Labeling thousands of signals, images & videos
✓ Transforming, generating, and augmenting data (for different domains)
Training and Testing Deep Neural Networks
✓ Understanding network behavior
▪ Accessing reference models from research
▪ Optimizing hyperparameters
▪ Training takes hours-days
Rapid and Optimized Deployment
▪ Desktop, web, cloud, and embedded hardware

72
Transfer Learning
8 lines of MATLAB Code
%% Create a datastore
Load Training Data 1 imds = imageDatastore(‘Data’,...
'IncludeSubfolders',true,'LabelSource','foldernames');
2 num_classes = numel(unique(imds.Labels));
%% Load Referece Network
Load Pre-Trained
Network 3 net = alexnet;
4 layers = net.Layers
%% Replace Final Layers
5 layers(end-2) =
Replace Final Layers fullyConnectedLayer(num_classes,'Name',[‘fc5']);
6 layers(end) = classificationLayer('Name','classOut');
%% Set Training Options & Train Network
7 trainOpts = trainingOptions('sgdm’,...
Train Network on 'MiniBatchSize', 64);
New Data
8 net = trainNetwork(imds, layers, trainOpts);
73
Tune Hyperparameters to Improve Training

Many hyperparameters
▪ depth, layers, solver options,
learning rates, regularization,
…

Techniques
▪ Parameter sweep
▪ Bayesian optimization

74
Keras-Tensorflow Importer

75
Model Exchange and Co-execution

MATLAB Python

Model Exchange Co-execution

76
Model Exchange With Deep Learning Frameworks

TensorFlow-
PyTorch Caffe2 TensorFlow
Keras

MXNet
ONNX MATLAB

Cognitive
Core ML Chainer Caffe
Toolkit

ONNX = Open Neural Network Exchange Format

77
Interoperate With Deep Learning Frameworks – Use Cases

Deployment
PyTorch Caffe2 TensorFlow

Export

Augmentation
MXNet and
ONNX MATLAB Optimization

Import

Cognitive Visualization
Core ML Chainer and
Toolkit
Debugging

ONNX = Open Neural Network Exchange Format

78
Model Exchange With Deep Learning Frameworks

Caffe Model Importer

▪ importCaffeLayers

▪ importCaffeNetwork
TensorFlow-Keras Model Importer
▪ importKerasLayers

▪ importKerasNetwork
ONNX Converter
▪ importONNXNetwork

▪ exportONNXNetwork
ONNX Converter
79
Model Exchange and Co-execution

MATLAB Python

Model Exchange
Co-execution

80
MATLAB-Python Co-Execution – The ‘How’
Call Python file from Call TensorFlow commands from
MATLAB MATLAB

TF_code.py

81
MATLAB-Python Co-Execution – Method A

1. Copy the code into a .PY file

2. Wrap entry point in a function
import tensorflow as tf
from tf import keras

def myTFCode():
for x in y:
a.foo()

def foo(x):
TF_code.py return x + 1

3. Add module to Python path and then call

from MATLAB via:
py.myModule.myTFCode();

82
Deep Learning on CPU, GPU, Multi-GPU and Clusters
H O W T O TA R G E T ?

Single Single CPU

CPU Single GPU

Single CPU, Multiple GPUs

On-prem server with Cloud GPUs

GPUs (AWS)
83
MATLAB Containers for NVIDIA GPU Cloud & DGX

84
Deep Learning Challenges
Data
✓ Handling large amounts of data
✓ Labeling thousands of signals, images & videos
✓ Transforming, generating, and augmenting data (for different domains)
Training and Testing Deep Neural Networks
✓ Accessing reference models from research
✓ Understanding network behaviour
✓ Optimizing hyperparameters
✓ Training takes hours-days
Rapid and Optimized Deployment
▪ Desktop, web, cloud, and embedded hardware

85
Deploying Deep Learning Application

Embedded Hardware Desktop, Web, Cloud

Application
logic

Code Application
Generation Deployment
86
MATLAB Production Server is an application server that publishes
MATLAB code as APIs that can be called by other applications

Analytics Development

MATLAB MATLAB
Compiler SDK Deploy

Package Code / test Access

Enterprise
Application
Integrate
MATLAB Production Server
Data sources /
Worker processes
applications
<>
Request
Broker Mobile / Web
Application

3rd party
Scale and secure
dashboard

87
MATLAB support for Cloud

88
89
IAAS to PAAS

90
Automatic Cluster Creation

91
Get Public IP for access

92
Server Machine VM access
• MPS console endpoint: https://xxx.xx.xx.xx

93
Deploying Deep Learning Application

Embedded Hardware Desktop, Web, Cloud

Application
logic

Code Application
Generation Deployment
94
Solution- GPU/MATLAB Coder for Deep Learning Deployment
Target Libraries

TensorRT &
GPU Coder cuDNN
Application Libraries
logic

Intel
MKL-DNN
Library
MATLAB
Coder
ARM
Compute
Library
95
 Deep Learning Deployment Workflows

INFERENCE ENGINE DEPLOYMENT INTEGRATED APPLICATION DEPLOYMENT

Trained
Pre- Post-
DNN processing processing

cnncodegen
codegen

Portable target code

Portable target code

96
 Single Image Inference on Titan Xp using cuDNN

TensorFlow (1.8.0)

MXNet (1.2.1)

GPU Coder (R2018b)

PyTorch (0.3.1)

Intel® Xeon® CPU 3.6 GHz - NVIDIA libraries: CUDA9 - cuDNN 7 97

NVIDIA Hardware Support Package (HSP)
Simple out-of-box targeting for NVIDIA boards:

INFERENCE ENGINE DEPLOYMENT

Jetson Drive platform

Trained
DNN

cnncodegen

Portable target code

98
 Summary- MATLAB Deep Learning Framework
DEPLOYMENT
Intel
MKL-DNN
Library

Access Data Design + Train

NVIDIA
TensorRT &
cuDNN
Libraries

▪ Manage large image sets ▪ Acceleration with GPU’s

▪ Automate image labeling ▪ Scale to clusters ARM
▪ Easy access to models Compute
Library

99
Summary
✓ Create and validate Deep learning models
✓ Automate ground truth labeling
✓ Access large amount of data from cluster/cloud
✓ Interoperability with Deep learning frameworks

✓ Visualization and hyperparameter tuning

✓ Seamlessly scale training to GPUs, clusters and cloud
✓ Deployment on embedded targets and web services

100
Available Here 101
102
103