Advanced Neural Implants
and Control
Daryl R. Kipke
Associate Professor
Department of Bioengineering
Arizona State University
Tempe, AZ 85287
[email protected]
Approved for Public Release, Distribution Unlimited: 01-S-1097
The Underlying Premise…
The ability to engineer reliable,
high-capacity direct interfaces to the
brain and then integrate these into a
host of new technologies will cause
the world of tomorrow to be much
different than that of today.
However…
� There are some serious scientific barriers between
where we stand today and where we can stand in
the future.
• How do we establish permanent and reliable interfaces to
selected areas of the central nervous system?
• How do we use these interfaces to directly and reliably
communicate at high rates with the brain?
Applied Neural Implants and Control
Systems Science & Visualization &
Signal Processing Modeling
He (BME) Farin (CSE)
Hoppensteadt (Math & Nelson (CSE)
EE) Razdan (CSE)
Kipke (BME) Smith (Math) INFO
Si (EE)
Project Director Neural & Tissue Tissue Culture &
Kipke (BME) Engineering Analysis
Kipke (BME) Capco (Bio)
Advisory Massia (BME) Massia (BME)
Committee Panitch (BME) Pauken (Bio) BIO
Raupp, Rousche (BME)
Hoppensteadt,
Farin
Materials Synthesis MEMS
& Bioactive Shen (EE)
Coatings Pivin (EE)
Ehestraimi (BME) Li (EE) MICRO
Massia (BME)
Panitch (BME)
Raupp (ChemE)
Primary Goals of the
BIO:INFO:MICRO Project
� Develop new neural implant
technologies to establish
reliable, high-capacity, and long-
term information channels
between the brain and external
world. VizMod
SysSci
TisClt
NeuEng MEMS
� Develop real-time signal Mat'lSyn
processors and system
controllers to optimize
information transmission
between the brain and the
external world.
Systems-level Approach…
Feedback control signals
Subject
Neural system
(global)
local
External Adaptive Neural
World Controller Implant
Controlled
neural
plasticity
Objective 2: Optimize Objective 1: Optimize neural
Adaptive Controller interface
Topics
� Project overview
� Towards the Development of Next-
Generation Neural Implants (BIO, MICRO,
and INFO)
� Bioactive Coatings to Control the Tissue
Responses to Implanted Microdevices
� Modeling the Device-Tissue Interface
� Direct Cortical Control of an Actuator
� Neural Control of Auditory Perception
� Wrap-up
Focus on Next-Generation
Neural Implants
Feedback signals: local
host response
Subject
Neural system
(global)
local
External Neural Neural
World Controller Implant
Controlled
Info. Signals: electrical neural
& chemical plasticity
Objective 2: Optimize Objective 1: Optimize neural
Adaptive Controller interface to achieve reliable, two-way,
high-capacity information channels.
…and “self-diagnostic”
Fundamental Problem of Implantable
Microelectrode Arrays
� Brain often encapsulates the device with scar tissue
� Normal brain movement may cause micro-motion at the tissue-
electrode interface
� Proteins adsorb onto device surface
� Useful neural recordings are eventually lost
Electrode 1
Electrode N
Implant Failure
Month N
Implant Month 1
3rd-Generation Neural Implants
Technology
Spectrum
1st-generation 2nd-generation 3rd-generation
Microwires Silicon arrays Neural Implants
Desired Properties
• Very high channel count
(<1000)
• Bioactive coatings
• Flexible
• Engineered surfaces
• Controlled biological
response
• Integrated electronics
“Brain-centered” Design of Neural Implants
Initial conceptual designs
recording site through hole
Standard Perforated Probe
Simple Bioactive Probe
Differential Bioactive Probe bioactive gel
e.g. corticosteroid flexible
NGF e.g. GABA
polyimide
substrate
cross-section (A-A) cross-section (B-B)
A A B B
A bioactive gel
B
A B
through hole connecting recording site bond pads
channel
Polymer-substrate Neural Implants
• 2-D planar devices can be bent into 3-D structures
• Increases insertion complexity
Holes to
promote
integration 90 degree
with neuropil angles
Recordings From Polymer-substrate
Neural Implants
One Day Post-op
Chan. 9
Chan. 10
Lost most unit activity
after 7 days – Most likely
due to failure to properly
close dural opening.
Flexible Neural Implants Present
Surgical Challenges
� While the “micro-motion” hypothesis suggests that flexible
neural implants should be more stable, the same flexibility
presents significant new surgical challenges.
“Difficult” insertion “Easy” insertion
Rdr2, 9-00 Rdr3, 9-00
Using Dissolvable Coatings to
Stiffen the Neural Implant
� Dip-coat microdevice with polyethylene glycol (PEG)
• Provides mechanical stiffening prior to implant
• Quickly dissolves when in contact with tissue
First insertion of coated microdevice into Second insertion of coated microdevice
gelatin -- Device easily penetrates into gelatin – The device is too flexible to
material penetrate material because the PEG has
dissolved.
Micromachined Surgical Devices
Silicon Knife/Inserter
PEG
Vacuum nozzle
Insertion aid
Flexible probe
Vacuum Actuated Knife/Inserter
Exploratory Functionality
Passive Other Active Devices
Polymer Substrate
Surface Engineering (Thermal, Magnetic, Strain, etc.)
• Magnetic/thermal
stimulation
• Drug delivery channels Bioactive Component
Electrical Active Fluid Microchannels
Storage Structures
Recording/Stimulating FET Devices,
Surfaces ChemFETs
• Active micro-
manipulation of probes
Mechanical Signal
Transfer Structures Processing Termination
Currently...
Internal Review
Feasibility Studies
Insertion Aids
Multiple Dimensions and Forms
Implant Coatings and Surface Modifications
Parylene-N,C Photo-crosslinked
Cl Polyimides
O O
Cl
C C
N N O
C C
O O n
smooth porous
Surface Plasma Treatments NH2 NH2 NH2 NH2
(NH3 - Amination)
Advanced Neuro-Device Interfaces
Passive
NH2 NH2 NH2 NH2
Chemical/Electronic
Amplification
ion beam
metal
modified region
site or interdigits
Active
polymer (PI/P-C)
release layer
or substrate
Silicon FETs?
Topics
� Project overview
� Towards the Development of 3rd-Generation
Neural Implants (BIO, MICRO, and INFO)
� Bioactive Coatings for Controlled
Biological Response (BIO, MICRO, and INFO)
� Modeling the Device-Tissue Interface
� Direct Cortical Control of an Actuator
� Neural Control of Auditory Perception
� Wrap-up
Approach
Engineer the neural implant surface in order to control
both the material response and the host response.
Advanced biomaterials and
micro-devices for long-term
implants (BIO, MICRO, INFO)
Models and 3-D visualization Cellular and biochemical
of device-tissue dynamics response characterization
(BIO, INFO) (BIO, MICRO)
Factors Limiting Chronic
Soft Tissue Implants
� Inability to control cellular interactions at
biomaterial-tissue interface
� Initial adsorption of biological proteins
• Non-selective cellular adhesion
� Unavoidable “generic” foreign body reactions
• Inflammation
• Fibrous capsule formation
Potential Solution
� Engineer surface for minimal protein adsorption
and selective cell adhesion
• Cell-resistant polymer coatings
• Synthetic: Polyethylene Glycol, Polyvinyl Alcohol
• Natural: Polysaccharides, Phospholipids
• Surface immobilization of biologically active
molecules
• Mimic biochemical signals of extracellular matrix
• Cell binding domains for integrin receptors
Biomimetic Surface Modification
O O O
O O O O O O
O OH OH OH O
HO HO HO OH HO OH HO
OH
HO NH
N OH HO NH
2 N OH
2
NTF NTF
Material Surface
Recombinant NGF Fusion Protein
Active or inactive plasmin
Degraded plasmin
Factor IIIa
degradable substrate
substrate
substrate
Plasmin
Human b-NGF cleavage
Human b-NGF
Fibrin
plasmin
Fibrin
Bioactive Functionality
Methods 6-hour diffusion in rat cortex
Fluorescence Intensity Profile
250
200
NeuroTrace� DiI tissue-labeling paste,
Pixel Value
150
inverted fluorescent microscope with
100
5 0
FITC/rhodamine filter cube 0
0 2 0 4 0 6 0 8 0
Distance (microns)
100 120 140 160
Topics
� Project overview
� Towards the Development of 3rd-Generation
Neural Implants (BIO, MICRO, and INFO)
� Bioactive Coatings to Control the Tissue
Responses to Implanted Microdevices (BIO, MICRO,
and INFO)
� Modeling the Device-Tissue Interface
(BIO, MICRO, and INFO)
� Direct Cortical Control of a Motor Prosthesis
� Neural Control of Auditory Perception
� Wrap-up
The Device-Tissue Interface
Neural Interface:
Micro-device, Neurons, Glia, Extracellular Space
The Goal is to Characterize, Predict, and Control
the Device-Tissue Interface
Biophysical
Tissue State Device Function
Model of the
(e.g., encapsulation, Device-Tissue (e.g., impedance
excitability) spectrum)
Interface
• Integrate bioelectrical, histological and biochemical data
• Optimize electrode specifications
Visualization of the Chronic Device-Tissue
Interface With Confocal Microscopy
A B
C D
In vivo Visualization of the Chronic
Device-Tissue Interface
Multi-Domain Continuum Model
• Tissue is two (or more) coupled
volume-conducting media
• Electrode is boundary condition
r
At each "point" r in space:
r
volume fraction fe / i ( r )
r
potential Fe / i ( r , t )
r
conductivity tensor Ge / i ( r )
membrane parameters
a, C, gL , etc.
Equations for a Multi-Domain
Continuum Model
Volume conductor equations (conservation of current)
- fe� � (Ge�Fe ) = +� I memi + I app
i
- fi �� ( Gi �Fi ) = -I memi i = index over intracellular domains
Membrane potential(s) and membrane current(s)
� ¶Vi �
Vi = F i - F e I memi = ai � Ci + Iioni �
Ł ¶t ł
Fe / i = potential (mV) a i = surface to volume ratio (cm -1 ) Vi = membrane potential (mV)
Ge / i = conductivity (mS/cm) I memi = membrane current (mA/cm ) 3
Ci = membrane capacitance (mF/cm 2 )
f e / i = volume fraction I app = applied current (m A/cm3 ) I ioni = membrane current (mA/cm 2 )
Levels of Modeling
Numerical
Analytical
Multiple intracellular domains
A single intracellular domain
Voltage-dependent conductances
Passive membrane conductance
I ioni = � g ij � q ijk (Vi - E j )
I ion = g L (V - E L )
j k
¶qijk q - q (Vi )
¥
= - ijk ijk
¶t t ijk (Vi )
Complex electrode geometry Simple electrode geometry
Tissue inhomogeneous and Tissue assumed homogenous and
anisotropic isotropic
under construction much progress
Bi-domain Model for the
Microcapillary Bioreactor
Write BCs and assume: j = j1eiwt � Fe / i ( x, t) = F1e / i ( x;w )eiwt
100 Hz
Fe
Calculate profiles
Fi
F 1
e/ i ( x;w) V
EL
in bioreactor
...and predict Z (w )
...and impedance...
as tissue parameters
Z (w ) =
Z fe / i , Ge / i ,a , C, g L , EL
F ( L;w) - F ( 0;w )
1
e
1
e are experimentally
j1 manipulated
w
Recap
� Focused & integrated effort
• BioMEMS…Neural
Engineering…Materials…
Computational BIO
Neuroscience…Cellular
Biology…Visualization
� Why are we so excited?
INFO
• We have the very real
potential of characterizing MICRO
the biological responses to
neural implants and then
engineering new classes
of microdevices to provide
a permanent high-capacity
interface to the brain
Why the BIO, INFO, and
MICRO Program?
� Wide-open Challenges
• Characterizing and modeling the biological (cellular and
chemical) responses around a neural implant
• Controlling the dynamic biological responses around a neural
implant.
• Designing, fabricating, and using “advanced” neural implants
� Collaboration Possibilities
• Additional functionalities for implantable microdevices of the
class that we are working on.
• Exploring fundamentally new types of tissue-device interfaces.
• Complementary studies of the neural interface (experimental
and analytical)
• Confocal microscopy of the neural interface
• Sharing technologies, procedures, insights, etc…
• New emergent ideas…
Systems-level Analysis of Advanced
Neuroprosthetic Systems
Feedback control signals
Subject
Neural system
(global)
local
External Adaptive Neural
World Controller Implant
Controlled
neural
plasticity
Objective 2: Optimize Objective 1: Optimize neural
Adaptive Controller interface
Systems-level Approach for Advanced
Neuroprosthetic Systems
Feedback control signals
Subject
Neural system
(global)
local
External Adaptive Neural
World Controller Implant
Controlled
neural
plasticity
Objective 2: Develop Objective 1: Optimize neural
adaptive controller to interface
optimize system
performance.
Advanced Neuroprosthetic Systems
External World
Neuroprosthetic System
Sensory
Transduction &
Pre-processing Motor Movement
Commands
High-Level
Sensory
Neural
Integration
Computation
Perception,
Decision,
� Underlying System Principles Detection
•Two-way communication with targeted neural systems
•Harness neural plasticity to our advantage
•Appropriately balanced “wet-side” and “dry-side” computation
Approach
� Four Project Areas
�Direct neural control of actuators
�Detection of novel sensory stimuli through
monitoring neural activity
�Neural control of behavior
�Investigate signal transformations from
ensembles of single neurons to local field
potentials to EEG.
Topics
� Project overview
� Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
� Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
� Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
� Direct Cortical Control of a Motor Prosthesis
(BIO, MICRO, and INFO)
� Neural Control of Auditory Perception
� Wrap-up
Direct Cortical Control of Actuators
External World
Neuroprosthetic
System External Actuator
Goal: Control Robotic Arm or
arm-related Virtual Reality
Sensory actuator
Transduction &
Pre-processing Motor Movement
Commands
Sensory High-Level
Integration Neural
Computation
Perception,
Decision,
Detection
Fundamental Questions
� What are “optimal” real-time signal processing
strategies for precise 3-D control of external, arm-
related actuators in the presence of sensory
distractions and/or physical perturbations to the
arm?
� To what extent can we use composite neural
signals [neuronal (unit) recordings, local field
potentials, and brain-surface recordings] for control
signals?
� How do we take advantage of inherent or
controlled neural plasticity in order to optimize
system performance?
Experimental Preparation
• Train monkeys to perform tracking and/or reaching tasks.
• Record cortical responses with multichannel neural
implants.
• Measure arm movement in 3-D space.
Chronic Neural Recordings
� Multi-channel neural implants in motor and sensorimotor cortical areas.
� Eventually: Sub-dural electrodes for local potentials
Perievent Histograms Target 1, reference = C_rel, bin = 20 ms
dsp009b dsp034a dsp046a
40
10 100
20
0 0 0
Extracellular recordings 40
-0.2 0 0.2
dsp012a
0.4 0.6
150
-0.2 0 0.2
dsp037a
0.4 0.6
15
10
-0.2 0 0.2
dsp051a
0.4 0.6
Offline
100
20 5
50
0 0 0
-0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6
dsp018a dsp040a dsp057a
20 15
Analysis
40
10
10 20
5
0 0 0
-0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6
dsp024a dsp042a dsp058a
20 80
Neural
40
20 10 40
0 0 0
-0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6
dsp025a dsp042b
Recording
40
20 20
0 0
-0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6
dsp030a dsp045a
System 10
0
30
20
10
0
-0.2 0 0.2 0.4 0.6 -0.2 0 0.2 0.4 0.6
Time (sec) Time (sec)
Real-time Actuator
Signal Control
Processing
Direct Cortical Control of Movement
Green ball: Target Yellow ball: Actual hand position, or
hand position estimated from cortical
responses
m0602pa
Topics
� Project overview
� Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
� Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
� Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
� Direct Cortical Control of a Motor Prosthesis (BIO, MICRO,
and INFO)
� Neural Control of Auditory Perception(BIO,
MICRO, and INFO)
� Wrap-up
Neural Control of Auditory Perception
External World
Neuroprosthetic
System
Goal: Control
auditory perception
Sensory
Transduction &
Pre-processing Motor Movement
Commands
Sensory High-Level
Integration Neural
Computation
Perception,
Decision,
Detection
Fundamental Questions
� To what extent can we control auditory-mediated behavior using
intra-cortical microstimulation (ICMS) through the neural interface?
Source Received
Signal Transmitter Channel Receiver Signal
Stimulator Neural Auditory
Interface Cortex
� What are the information transmission characteristics of the
multichannel neural implant in high-level cortical areas using
ICMS?
� Channel capacity (bits per second)
� Channel reliability
� Channel resolution
� How can we optimize information transmission
� Implant designs, Neural implant locations, Signal encoding strategies,
Controlled neural plasticity
Chronic Neural Recordings
� Multi-channel neural implants in primary auditory cortex
Extracellular recordings
in auditory cortex
Estimation of
Neural Neuronal
Offline Response
Recording
Analysis Properties
System
Algorithm Selection
Sounds
Electrical Signal
Stimulation Encoder
to Aud. Ctx.
Behavioral performance to both sounds and
cortical electrical stimulation
Auditory Behavior
• Lever-press sound or ICMS discrimination task
• Center paddle hit starts trial, 2-tone pair presented
• Reward obtained by signaling the correct stimulus
sequence
left center right
rat
Frequency Selectivity in Auditory Cortex
Frequency 80
dsp002a 6.
80
dsp002b 11.
80
dsp010b 24.
response areas 60
3.
60
5.5
60
12.
40 40 40
1 2 5 10 20 30 1 2 5 10 20 30 1 2 5 10 20 30
dsp012a 42. dsp018b 21. dsp018c 44.
80 80 80
60 60 60
21. 10.5 22.
40 40 40
1 2 5 10 20 30 1 2 5 10 20 30 1 2 5 10 20 30
dsp018d 22. dsp020a 20. dsp024a 20.
80 80 80
60 60 60
11. 10. 10.
40 40 40
1 2 5 10 20 30 1 2 5 10 20 30 1 2 5 10 20 30
dsp024b 56.
80
Sound 60
28.
Level 40
1 2 5 10 20 30
Freq.
Signal Encoding Algorithm:
Frequency Selectivity
ICMS pattern is based u5b 8
solely on frequency 80 6
Spikes
60
selectivity of neurons dB
40
4
recorded on an electrode 2
1 5 10 30 0
u32a
8
kHz
80
6
dB 60
4
Spikes
40
1 5 10 30 0
kHz
Behavioral Performance
Ricms6
Rat Behavioral Performance
RICMS 6
100
90
80
Percent Correct
70
60
50
40
30
20
10
00
00
00
00
00
00
6/
6/
6/
6/
6/
6/
/0
/1
/2
/0
/1
/2
09
09
09
10
10
10
Training day
Implanted
Cortical
Electrodes
Expected Results to ICMS Stimuli
Begin ICMS
100
D % due
%
to ICMS
Trial #
Auditory trial =
ICMS Algorithm1 =
ICMS Algorithm2 =
Behavioral Curve
RICMS 6 10/25 (Only Session)
100
80
Percentage
60
audPercent,
40
icmsPercent,
20
0 100 200
Trial
Alternative Signal Encoding Algorithm:
Cortical Activation Pattern
For a given electrode, the unit firing pattern is used as a
template for ICMS delivery
Auditory
Sound on
Stimulus
Response
Raster
Matching ICMS
‘ pattern’
***Procedure is simultaneously duplicated on each active electrode
Recap
� Focused & integrated effort
• Neural Engineering…Signal
Processing…Systems
Neurophysiology…Visualization BIO
� Why are we so excited?
INFO
• We have the very real
potential of developing new MICRO
classes of neuroprosthetic
systems to explore our ability
to interact directly with the
brain.
BIO, INFO, and MICRO…
� Wide-open Challenges
• Appropriate mathematical constructs for describing neural
encoding and decoding.
• Advanced data visualization techniques for understanding this
new class of neural data.
• Understanding signal transformations as a function of the
spatial and temporal scale of the neural data.
� Collaboration Possibilities
• Exploring new signal encoding and decoding strategies for
particular neuroprosthetic applications.
• Sharing technologies, procedures, insights, etc…
• New emergent ideas…
Topics
� Project overview
� Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
� Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
� Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
� Direct Cortical Control of a Motor Prosthesis (BIO, MICRO,
and INFO)
� Neural Control of Auditory Perception(BIO, MICRO, and
INFO)
� Wrap-up
Project Challenges
� Scientific
• Overcoming engineering and scientific hurdles.
• Identifying and fostering strategic alliances with appropriate
external groups.
• Crossing disciplines
� Management
• Strategic planning
• Resource allocation
• Open and effective communication among the diverse project
team
• Team-building: Maintaining enthusiasm, energy, and focus
after the initial “honeymoon” period
“Insanely Intense
Interdisciplinary” Research
“pieces of a puzzle” “easy synergism”
INFO
BIO
•Hard work
•Open minds
INFO •Honesty Breakthrough
MICRO •Top-notch research Science
MICRO BIO
What Does the Future Hold?
“Perhaps within 25 years there will be some new ways to put
information directly into our brains. With the implant technology that
will be available by about 2025, doctors will be able to put something
like a chip in your brain to prevent a stroke, stop a blood clot, detect
an aneurysm, help your memory or treat a mental condition. You
may be able to stream (digital) information through your eyes to the
brain. New drugs may enhance your memory and fire up your
neurons.” -- Dr. Arthur Caplan,
Director of the Center of Bioethics,
University of Pennsylvania
Arizona Republic, Dec 27, 1998.
Acknowledgments
� ASU Colleagues
• 13 co-PI’s, 5 research faculty, numerous graduate
and undergraduate students.
� Arizona State University administration
• Seed funding from Department, College, and
University
• Significant cost-share on this project
� DARPA Program Managers
• Eric Eisenstadt, Abe Lee, and Gary Strong
