Sound Source Localizer

Joren Lauwers, Changping Chen

{jorenl, ccp0101}@mit.edu
6.111 Introductory Digital Systems Laboratory, Final Project
Prof. Gim P. Hom

Abstract
As our final project, we built a sound source
localizer - a system that determines the
position of human speech, and displays it
on a two-dimensional map on a screen,
using only cheap microphones and
on-the-fly calculation on a Nexys4 FPGA
board. The sounds picked up by the
microphones are compared to determine
how much time the sound spent traveling
through the air, which corresponds to how
far the microphones are away from each Figure 1: Schematic representation of our
other. This report details the process of how finalized set-up. 1. Sound source 2.
we approached the design of this system, Microphone A 3. Microphone B 4. VGA
some technical choices and limits involved, Display 5. Microphone preamplifiers 6.
and some mathematical background that Nexys4 FPGA board
brought us to the working set-up in Figure 1.
requires only two microphones, because the

Goals FPGA can determine the location of the

speaker by measuring the delay between
Our primary goal was to locate the origin of audio samples sent to the output speaker
sound in a room using only microphone and those received from microphones. The
input. Initially, we were thinking of using information you get is the total travel time of
three stationary microphones, and using the sound from point to point, not the
only the signal delay between them to differences between them. This makes the
locate the sound, in a manner similar to mathematics behind this also simpler.
what GPS does with satellites. After
discussions with instructors, we proposed to We kept the three-microphone idea as our
first use a music playing speaker as our stretch goal, and eventually implemented an
sound source. This setup in-between approach: with three
microphones, localizing free human speech,
but in a triangular set up, where we keep computes a position based on these lag
one microphone at the sound source. We values, and a VGA display system to
find that this yields higher accuracy than visually show the results. For the controlled-
with controlled output through a sound mode of operation, an SD card
loudspeaker. reader module reads waveform data from a
memory card, and plays it through a
As shown in Figure 1, our finalized test Pulse-Width Modulation (PWM) based
set-up consisted of two stationary output, which is connected to an external
microphones, a fixed distance apart (we speaker. This section goes into more detail
used 1000mm) at eye level and a voice for each of these modules.
microphone carried by the speaker, all fed
into the FPGA through external Circuit: Audio Input Preamplifier
pre-amplifiers. The calculated origin of the
sound picked up by the microphones is then As described in our initial proposal, we built
displayed on a monitor. three identical preamplifiers for microphone
signals on a single breadboard. We
purchased a set of 3.5mm stereo jack
Modules terminals1 that allow us to connect a
microphone input to a breadboard, and then
As outlined in the block diagram in Figure 2, followed the circuit diagram for an op-amp
the digital part of our system consists of an
Analog-to-Digital converter which reads the
microphone signals, two time-shift
correlation modules that continuously 1
4 Pack CESS 3.5mm 1/8 Stereo Balanced
compute how much one signal lags behind Female Jack to AV Screw Video Balun Terminal
https://www.amazon.com/CESS-Stereo-Balanced
the other, a triangulation module which -Female-Terminal/dp/B017CBTLJK
based audio preamplifier from an online unnecessary filtering and delay, so we
guide2 reproduced in Figure 3. opted to purchase three cheap condenser
microphones similar to the those on the
headset in our labkit.

Input Sampling with ADC

File: xadc_reader.v - author: Changping

The Artix-7 FPGA on the Nexys 4 board

features two 12-bit analog-to-digital
Figure 3: Microphone pre-amplifier circuit converters4 . The two ADCs can be
configured to simultaneously sample two
We learned from the XADC datasheet that it external-input analog channels, at a
internally uses a 1V reference voltage. To sampling rate of 1MSPS. Initially we
enable bipolar mode3, the DC offset of our configured the XADC to sample two
preamplification stage should be set at microphone inputs connected to vaux3 and
0.5V, and that its output range should be vaux11 , using Simultaneous Sampling
0.5V ± 0.5V . Therefore we added a voltage mode, triggered by a 50 KHz tick. This
divider with 50kΩ and 9kΩ resistors ensures both channels are sampled
between 3.3V and GND to obtain roughly precisely at the same time.
0.5V DC voltage, and connected it to V OUT .
We observed on an oscilloscope that the Ideally to implement our stretch goal of
gain of our amplifier is appropriate for three microphones, we want to sample all
normal speech volume up to one meter three channels at the same time. However
away. V OUT is the final output of the the XADC only has two cores, and as such
preamplifier, and connected to positive input can only sample from two channels
pins of Nexys 4 XADC. simultaneously. We enabled its internal
input multiplexer, which automatically
After we built the amplifier, we first tried switches to another pair of channels when
using high quality stage vocal microphones. the current pair of channels are acquired,
However, by design they only pick up audio sampled, and stored. We selected channel
in a short range (roughly 10cm in our 3, 11, 2, and 10 through the IP wizard in de
experiment). We also suspect that their vivado design suite, since only these
internal construction introduces channels have physical I/O pins on the
Nexys4 board. A sequence of two
2
LM358 microphone amplifier
https://lowvoltage.wordpress.com/2011/05/2 4
XA Artix-7 FPGAs Overview
1/lm358-mic-amp/
3 http://www.xilinx.com/support/documentatio
7 Series FPGAs and Zynq-7000 All
Programmable SoC XADC Dual 12-Bit 1 MSPS n/data_sheets/ds197-xa-artix7-overview.pdf
Analog-to-Digital Converter User Guide ,
https://www.xilinx.com/support/documentation/us http://www.xilinx.com/support/documentatio
er_guides/ug480_7Series_XADC.pdf n/user_guides/ug480_7Series_XADC.pdf
conversions yields samples on all selected window size of W ​ . It’s a common method of
channels. comparing the similarity of two time series
when one lags behind another. This module
Because the XADC can sample at a rate is used later to calculate the time delay
much faster than our requirement of 50KHz, between when a signal is sent and the
we can treat a sequence of sampled data as same signal is received.
if they were sampled simultaneously, if we
minimize the delay between the two The cross correlation of two inputs A and B
conversions. To achieve this, we made sure over a finite sample window of W where B
to trigger another conversion immediately lags behind A by m samples, is given by
W−1
after the first conversion in a sequence is
C (n, m) = ∑ A[n + k] · B[n + k + m]
completed: k=0
We want to calculate the cross-correlation
// start conversion if score for each possible lag m , and then use
// (a) there's more channel to be sampled
the lag associated with the highest score to
// in a sequence (eoc & (~eos))
// (b) new sample needed per 50KHz calculate the difference in time of arrival.

wire​ xadc_convst_in ​=​ (xadc_eoc ​& Notice that C (n, m) depends on the future
(​~​xadc_eos)) ​|​ clk_sample_tick; value B[n + W − 1 + m] and the current value
A[n] , so we need to store at least m + W
To read digital values from XADC
samples, or S = m + W . The requirement on
post-conversion, we wrote a module to
m is discussed in the next section.
sequentially read its status registers at
designated memory addresses. These
values are fed into cross-correlation Another implication from the equation above
modules. is that our score will be delayed by S
sampling cycles, but this delay is negligible
in human perception.
Cross-correlation
File: cross_correlation.v - author:
Changping Example
In the example below, we choose W=5 and
The purpose of the correlation modules is to S=10. Suppose there exists a signal
compare two input signals, and determine
how much (in terms of number of samples, 0 0 0 0 0 1 2 3 4 5 4 3 2 1 0 0 0
which corresponds to time) one lags behind 0
the other.
This signal arrives at mic 1 after a delay of 2
Theory sampling cycles, and at mic 2 after 5
This module calculates the cross correlation sampling cycles. At one point, the sample
score (dot product) of two input signals buffer for the two mics will contain:
when one is time shifted by a lag m over a
Mic 1: 2 3 4 5 4 3 2 1 0 0
Mic 2: 0 0 1 2 3 4 5 4 3 2 implement audio buffers using Block RAMs
which has higher capacity and low latency.
Based on this dataset, the relative delay of One caveat we encountered is that using
signal received at mic 2 compared to mic 1 BRAM requires minimum of two cycle
is 3 cycles. If we calculate the cross delays (one cycle to program read address,
correlation score for m ∈ [0, 5] , we expect and another cycle for data return). To deal
the score for m = 3 to have be the highest. with this, we set up our BRAM at the system
clock rate of 100 MHz, and only perform
Here: calculation one in every four cycles,
C (n, 3) = (2)(2) + (3)(3) + (4)(4) + (5)(5) + (4)(4) effectively reducing its frequency from 100
. MHz to 25 MHz. This ensures the memory
Observe that with each streaming input delay is hidden, and requested data
sample, the difference in the new and old appears to be immediately available on the
correlation score for the same lag m ​ is next calculation cycle.
C (n, m) − C (n − 1, m) =
W−1 The cross correlation module is a state
= ∑ A[n + k]B[n + k + m] − machine, where the state is (op, m) . op
k=0
W−1 specifies the operation it’s currently
∑ A[n − 1 + k]B[n − 1 + k + m] processing. The main two operations are
k=0
= A[n + W − 1]B[n + W − 1 + m] − OP SUB and OP ADD , the former subtracts
A[n − 1]B[n − 1 + m] from correlation score the product of the
oldest samples, and the latter adds to the
Therefore, in each cycle, we only need two correlation score the product of the newest
multiplications, one subtraction and one samples. Each operation is repeated for 200
addition to compute the next value. distinct m . Therefore, the total number of
cycles for updating scores given a new
sample is roughly 400 cycles.
Implementation
To implement the cross correlation module, Each correlation module stores two streams
initially we wanted to write a parallel module of audio samples, and total of two
that can calculate multiple correlation correlation modules are needed. We need
scores simultaneously for many different to store S samples in an audio stream,
time steps. However, that requires parallel which is double the correlation window W
reads from audio buffers, and requires using (see theory section). We find the empirical
limited registers. limit of W = 1024 , above which Vivado fails
to compile.
In order to increase the accuracy, we want
to use a large enough window size, but we
The input samples have a width of 8 bits
still want minimal latency. So we wrote our
(signed), and the output scores are stored
cross correlation module in sequential
as 32 bit signed numbers.
fashion, where in each cycle it issues one
read from each audio buffer and computes
one multiplication. This allows us to
 Since we were aiming for 8-bit audio output,
Correlation Output Filter we used a pulse window width of 256 clock
cycles of the 100Mhz system clock, implying
File: cor_filter.v - author: Changping a PWM rate of around 390Khz, well above
the audible range. The duty cycle - the
Theoretically, the peak of the correlation amount of time along this window that the
score distribution corresponds to the true PWM output is held high - is then varied
delay of a signal. However, we found that between 0 and 256. FIgure 5 gives an
due to noise in the environment, the peak is example of what a PWM waveform looks
not necessarily stable. This translates into a like.
lot of jitter in triangulation result.

Music Player
Hence, we coded a very simple filtering
module with configurable parameters using File: music_player.v sd_controller.v -
switches. The filter specifies that the input author: Changping
delay must stay within a specified range for
at least a specified amount of cycles. Any The music player module plays a music
delay that fails to pass is rejected and does from SD card. We used ​sd_controller.v
not reach triangulation stage. This file to read sectors from SD card, and
introduces some visible level of stability. connected it to a Block RAM-backed 1KB
FIFO so that we can aggressively read from
In retrospect, a better filter would be an SD card in advance, and always maintain at
algorithm that tracks multiple peaks, and least one sector (512 bytes) buffer in
selects one that corresponds to previously memory. On each sampling clock cycle, one
selected peak even if there exists a peak byte is popped from FIFO and written to a
with a higher score. This filter would be audio PWM module for playback.
more tolerant of erroneous global peak.
On a MacBook, we wrote a simple Python
Audio Output script to dump length and raw bytes from a
8-bit @ 44KHz .wav file onto an SD card as
File: pwm.v - author: Joren
a raw block device.

The Nexys4 board has a mono audio output

jack, and requires a Pulse-Width Modulation
(PWM) module to play audio. The output of
the PWM pin is passed through two low
pass filters that smooth out the pulses of
above-audible frequencies to a variable
output voltage.
 derivation of the law of cosines in the
Triangulation triangle △ASB :

File: triangulate.v - author: Joren

b2 = a2 + d2 − 2 · a · d · cos(α)
⇔ x = a · cos(α) = a2+ d2−b2
2·d

And from pythagoras’ theorem in the right

triangle △SAS′ :

y2 + x2 = a2
⇔ y = √a2 − x2

This means we can compute x,y on the

FPGA without the need for trigonometric
Figure 5: geometric representation of our functions. It does however, require a square
situation root operation which, unlike multiplication
and addition, is not a built-in hardware
The cross correlation module computes how operation and requires a dedicated module
long a sound took to travel from the source
S to the two microphones A and B . Since Square root module
sound travels at a constant speed in an
environment, the corresponding distances File: sqrt.v - author: Joren
a, b can be computed using a simple scalar
multiplication: Our square root implementation is based off
a verilog implementation 5 of a binary
a = c · ma and b = c · mb search approach found online. It computes
a guess for the root, by flipping up a single
Where ma, mb is the amount of samples the bit each iteration, from most significant to
least significant, and computing the square
signal at A, B lags behind the source. Since
g2 of that guess each time. If g2 is still less
we used a 50Khz sampling rate, and
than n, the bit stays high and the guess is
distance units of 1 · 10−4m , we used a
updated, otherwise, it stays low and the
constant scaling factor c = 68 , since sound guess is not affected.
travels 6.8mm each 1/50th of a second.
When determining whether the next bit, at
Given these two distances a and b , and position i, of the current guess g should be
the distance d between the two high, it is easy to compute the square of this
microphones A, B which is known, the potential better guess:
triangulation module computes the cartesian
coordinates x, y of the sound source. The
5
computation relies on the following
https://github.com/tchoi8/verilog/blob/master/exa
mples/sqrt.vl
(g′)2 that is much higher, as sound travels
= (g + (1 << i))2 68 · 10−4m per sample.
= g2 + (1 << i)2 + 2 · g · (1 << i)
= g2 + (1 << 2i) + (g << 1 << i) Display
If (g′)2 < n then g ← g′ File: vga.v, - author: Joren

Note that g2 was computed in the previous In order to understand the results of our
iteration of the algorithm, or is 0 in the first real-time cross-correlation and triangulation
iteration. All other operations involved are calculations, we included a representation
simple additions and bit shifts. of the setup, with its microphones and the
location of the sound source. We also
Since the square of an n-bit number has at added some real-time insight into the results
most n/2 significant digits, the algorithm of the calculations beyond just the final x,y
starts with i = n/2 , decreases i by one each coordinate.
iteration, and consistently computes the root
of an n-bit unsigned integer in n/2 cycles.

Note that, when computing the square root

of a number that is not a perfect square, this
algorithm will return the floor of the square
root.

Theoretical accuracy

All computation modules store distance

values as 16-bit integers. The squares of
these quantities are stored in 32-bit
Figure 6: Example of screen layout with two
registers. Given the nature of our set-up, x
correlation waveforms, indicators for two
needs to be a signed coordinate, since it is
stationary microphones (black) and one
negative for all points S left of the main
speaker (red)
stationary microphone. The y coordinate is
always positive.
VGA Display Signals
Since we’re only doing integer arithmetic, Our vga module is based on the one
we picked our unit of distance to be provided with lab4 on the course website6,
1 · 10−4m , and in theory, the calculations but with some minor modifications. The vga
could triangulate coordinates with this module generates the proper hsync, vsync
accuracy. It is worth reconsidering this and red, blue, green video signals. While we
choice of units to allow for larger maximum built a working xvga module (1024x768
coordinates, since there is a margin of error 6
introduced by the cross-correlation module
web.mit.edu/6.111/www/f2016/handouts/labs/labk
it_lab4.v
resolution), we intentionally decided to stick
with the default 640x480 resolution with a Correlation distribution waveforms
60Hz refresh rate, since that allows for a
File: sprite_waveform.v - author: Joren
25Mhz pixel clock7, easily obtained by
running the 100Mhz system clock through a
As described in the section about the cross
quarter divider. Larger resolutions would
correlation module, it computes a
require a separate video clock domain. This
cross-correlation score for all possible lag
allows components that work in the visual
values m​ . We initially just chose the lag
system, like waveform renderers and
value corresponding to the highest
sprites, to operate on the same clock
correlation score as the correct answer, but
domain as the triangulation and audio
this turned out to be unreliable. This is why
sampling modules with which they have to
we started experimenting with filters on our
interact.
cross-correlation data.

To better visualize what is really happening,

we added a waveform sprite. At the start of
Screen layout a VGA frame, the waveform sprites stores
one run of score values from the correlation
We implemented a simple sprite pipeline, module in registers, to render on the screen.
where a pixel signal encoding the color
corresponding to a current x,y position on From this, we learned that these score vs.
the screen is passed on between sprites. delay distributions show several peaks.
Where the video chip on the 6.111 labkit Higher pitched, periodic sounds (whistles,
supports 24bit color resolution with 8 bit for tones, …) produce more peaks. This makes
each channel, the built in ADC’s tied to the sense, as those lag values that are shifted
Nexys4 VGA port are limited and only by a multiple of the dominating period also
support 4 bit for each channel, giving us a get high correlation scores. Had we realized
12 bit color resolution. this earlier, we could have made more
specific filters for a steadier result.
On the display, we show:
● The position of the two stationary Analysis of how this distribution changed
microphones for reference over time was much easier to do when
● The moving triangulated position of directly visible on the display, compared to
the sound source analyzing in (non-interactively) using the
● Waveforms indicating the current Internal Logic Analyzer (ILA), mostly it was
correlation score distribution very hard to determine the evolution of the
distribution with time.

http://www-mtl.mit.edu/Courses/6.111/labkit/vga.s
html
 output shown is the start of a “string pluck”
sound generated by a Karplus-Strong String
Lessons Learned synthesizer, which resembles white noise
and as such has a significant random
high-frequency component. While the
Sampling from several channels
different delays between the signals are
We initially expected much more trouble clearly visible, and the two microphone
sampling from three microphones waveforms look very similar, they are
simultaneously. This turned out to be very significantly different from the raw output
simple due to the wizards built into the wave.
Vivado design suite, and the multiplexer
built into the XADC.
Understanding which the filtering effects at
Third microphone vs. sound output work could make location of a
from FPGA non-microphone sound source work just as
well, but we did not have the time to look
The localization is currently much more into this.
reliable when using a third microphone and
your voice as a sound source, than when

using the sound output from the FPGA. This Challenges due to periodicity of
is surprising, since the latter gives very sound
accurate information of exactly what sound
From the correlation score distributions, we
is being produced, and at what time.
learned that the score develops peaks for
different lag values a fixed distance apart,
One possible explanation would be that the
that depends on dominant frequencies of
process of converting a waveform through
the sound being picked up. When a sound
PWM, reproducing it through a speaker, and
is very periodic, the correlation module gets
then picking it up using a microphone acts
confused since the waveforms “align” at
as a significant filter on the signal. Figure 7
several places, each shifted by one period.
compares the output wave and the recorded
signals on two microphones. The sound
One solution could be to try and filter out the
period component of the sound, and focus
on correlating the irregularities. Another
interesting solution could be to return to our
initial idea of using three stationary
microphones, since the difference in arrival
time between two microphones that are
near each other could often be less than
one period of the sound. This requires
changes to our correlation module (faster
sampling, shorter window, …) that we did
not get to experiment with.

Division of work
Joren: preamplifiers, triangulation, and VGA
display modules
Changping: cross-correlation, music player,
and ADC sampling modules.