Grid Dynamics Whitepaper

Multimodal

biomedical AI:
applications and
challenges in pharma,
medtech, and healthcare
[Link]
Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Contents

Introduction 3

Understanding multimodal AI that thinks like humans 4

6 opportunities for leveraging multimodal AI in pharma,

medtech, and healthcare 6

1. Enriching research data with decentralized clinical trials 6

2. Monitoring ‘hospitals at home’ remotely 8

3. Detecting outbreaks and monitoring a pandemic 9

4. Accelerating clinical trials with digital twins 10

5. Expanding the scope of virtual healthcare assistants 11

6. Providing personalized precision healthcare 12

C hallenges — Data collection and augmented analysis 13

O ur team at Grid Dynamics can help 14

Prepare to grow 15

Ab out Grid Dynamics 1 6

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 2

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Introduction
The healthcare environment — with its packed In this whitepaper, we’ll dig deeper into:
work days, demanding practice, time pressures,
and emotional intensity — can put physicians and
other clinicians at high risk of burnout, a rampant What multimodal AI is and
issue reported by more than 3 out of 5 physicians.
Healthcare professional (HCP) burnout impacts
1 how it’s more impactful than
single-modal AI in healthcare.
the entire healthcare lifecycle as pharma and
medtech companies engage heavily with HCPs —
the people patients trust the most when making
How multimodal AI
decisions, and who also play a vital role in
guarantees reduced HCP
clinical trials. 

supervision in clinical trials,

Fortunately, in this age of machine learning (ML) remote monitoring and care,
and artificial intelligence (AI), there are many
promising solutions that can help minimize
human intervention within healthcare. While a
2 pandemic surveillance,
digital twin technology,
virtual health assistants, and
doctor is limited by the number of patients they personalized medicine —
can treat physically in their lifetime, AI can learn a win-win for healthcare,
from millions of patients' experiences and pharma, and medtech.
decades of clinicians’ knowledge, dramatically
supporting doctors in reaching better outcomes
by improving diagnosis and treatment.

How Grid Dynamics can

Traditionally, AI models have focused on deliver omnichannel
processing information from a single modality solutions and prevent some
like text, image, or speech. With the ongoing
explosion of biomedical data as the internet of
3 blockers around data quality
and modeling that limit the
medical things (IoMT) market size is projected to execution of multimodal AI
grow from USD 30 billion in 2021 to USD 187 billion in healthcare.
in 2028 — it is now possible for AI to harness the
combined power of these multiple modalities to
get more robust and accurate results. Biomedical
data from large biobanks, electronic health
records, medical imaging, wearable and ambient
biosensors, and the lower cost of genome and
microbiome sequencing have set the stage for
multimodal AI solutions that capture the
complexity of human health and disease.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 3

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Understanding
multimodal AI that
thinks like humans
Did you know that all human experiences are
multimodal? 

Let’s say you visit a restaurant. You sit at a

specific table, feel the napkin and cutlery, smell
the aroma, and taste the different flavors of the
food. Once you combine these sensory inputs
from multiple modalities, you can conclude
whether you liked the restaurant and would leave
a good review. 

Similarly, when an AI model leverages multimodal

data like text, image, audio, video, etc., it better
understands the context and distinguishes
between nuances, resulting in more accurate and
coherent responses. This is especially vital for
healthcare and pharma where precision is key
that necessitates increased human supervision in
most activities. 

For example, a machine learning model examines

thousands of lung cancer CT scans to learn how to
identify lung cancer from medical images. Such
single-modality tools increasingly rely on medical
imaging, and tend to weigh other factors with less
importance. There are many ways doctors can
determine if someone has lung cancer or is at risk
of developing lung cancer — a persistent cough,
chest pains, loss of appetite, family history,
genetics, etc. If AI harnesses a complete picture of
a patient’s symptoms and health history, it could
identify lung cancer or other diseases earlier with
greater precision and in a much shorter time frame.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 4

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

But are we using multimodal AI in existing healthcare, medtech, and pharma settings?
Yes, we are, and here’s an example.

North Carolina-based Bionic Health raised $3M for its AI health clinic. People

can get bloodwork and other diagnostics carried out and monitored, and then

an AI — built on OpenAI’s GPT-4 and other large language and machine

learning models — provides personalized insights based on the data points

retrieved from these assessments. 

GPT4, a multimodal AI solution (technically bimodal as it uses text and image

inputs and emits text output) is 10 times more advanced than its single-modal

predecessor, GPT-3.5.

Data modalities Opportunities

Omics Precision health

Metabolites,
immune status, Digital clinical trials
biomarkers

Microbiome
Hospital-
at-home

EHR/Scans

Pandemic
surveillance

Wearable
biosensors

Digital twins

Ambient sensors

Virtual health coach

Environment

Acosta J et al, September 2022

Multimodal AI can help deliver effective solutions that perform certain clinical tasks
comparable to or even better than humans, allowing HCPs to focus on the more
critical tasks, opening doors to new opportunities for product development and
innovation in pharma, medtech, and healthcare.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 5

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

6 opportunities for
leveraging multimodal AI
in pharma, medtech,
and healthcare
1. Enriching research data with
decentralized clinical trials
Pharmaceutical companies conduct gold-standard
randomized clinical trials to investigate and provide
evidence that supports novel inventions in clinical
medicine. These costly and time-consuming trials require
exhaustive planning and execution throughout the whole
process, usually taking many years. Despite this, we see
notable under-representation of several groups in these
trials due to geographic, sociocultural, and economic gaps
in enrollment. This under-representation limits the
generalizability of results and adds to the existing
discrepant research.

Decentralized clinical trials present a new opportunity for

the pharma industry to overcome these limitations. It will
not only reduce barriers to participant enrollment and
retainment but also promote engagement and optimize
trial measurements and interventions. At the same time,
effective data handling can enhance the granularity of the
information obtained from participants, thereby increasing
the success of these studies.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 6

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Data from wearable technologies — heart rate,

sleep, physical activity, electrocardiography,
oxygen saturation, and glucose monitoring— and
smartphone-enabled self-reported
questionnaires can monitor clinical trial
participants, identify adverse events, and
determine trial outcomes. The results can also
prove beneficial in improving the way these
decentralized clinical trials work. 

Remote trial monitoring has become more

prevalent since Covid-19 mostly using a single
wearable device. Effectively integrating data from
multiple wearable sensors with clinical data, and
delivering precise reports through a headless
frontend would allow enhanced remote
monitoring, supporting HCPs to track the progress
in real-time. For pharma, this drastically helps
reduce the associated costs and speeds up the
trials by changing how HCPs and participants
work together.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 7

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

2. Monitoring ‘hospitals at home’ remotely

Recent progress with biosensors, continuous Above all, it enhances the patient experience and
monitoring, and analytics raises the possibility of helps them leverage omnichannel care while
simulating the hospital setting in a person’s feeling the comfort, convenience, and emotional
home. This creates significant new product support of being with family members at home.

opportunities for the medtech industry, promises

Wearable sensors, ambient sensors, and
reduced care costs for patients, minimizes the
electronic health records (EHRs) of patients work
need for HCP supervision, and helps prevent
in unison to make remote patient monitoring
nosocomial infections and medical errors that
successful.
occur in medical facilities.

Affordable, non-invasive devices Combining wearable data with EHRs Ambient sensors placed within the
like smartwatches can rapidly and using Fast Healthcare patient’s home environment — in a
accurately measure several Interoperability Resources, a global room, on a wall, or mirror — include
physiological metrics. industry standard for exchanging video cameras and microphones to
healthcare data, to fetch depth cameras and radio signals
information about patients’ that can further improve remote
underlying diseases and health care. Integrating data from these
conditions could create a more multiple modalities and sensors
personalized remote monitoring brings a promising shift in remote
experience for patients and caregivers. patient monitoring. For example,
combining information from
ambient sensors like depth cameras
and microphones with wearable
data from accelerometers that
measure physical activity can
enhance the reliability of fall
detection systems while keeping a
low false alarm rate.

To make ‘hospitals at home’ a success, A robust analytics platform enables us to leverage

randomized trials of multimodal AI-based remote a wide range of data sources, painlessly migrate
monitoring versus hospital admission are to the cloud, and increase our speed to insights.
required to show no impairment to safety. So, HCPs and caregivers can swiftly act on real-
Predicting likely deterioration and having a time insights available at their fingertips.
system to intervene is also necessary.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 8

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

3. Detecting outbreaks and

monitoring a pandemic
Over the last few years, COVID-19 has highlighted the need
for effective infectious disease surveillance. Many
countries have successfully integrated multimodal data
from migration maps, mobile phone utilization, and health
delivery data to forecast the spread of the outbreak and
identify potential cases. While this presents new
innovation opportunities for medtech firms, it also ensures
advanced pandemic preparedness in healthcare, limiting
HCP burnout that forced many professionals to quit their
jobs during COVID-19.

A recent study has demonstrated how tracking resting

heart rate and sleep minutes using wearable devices
improves surveillance of influenza-like illness in the USA.
Following suit, an app-based research program analyzed a
diverse set of data from wearables to allow for rapid
detection of the emergence of influenza, coronavirus, and
other fast-spreading viral illnesses. A follow-up study from
this program showed that jointly considering participants'
self-reported symptoms and sensor metrics improved
performance relative to either modality alone. 

Delivering commercial multimodal solutions that use at-

home testing and reporting with sensor-based monitoring
will promote self-care among patients with mild and
moderate symptoms, reducing overcrowding in ERs and
HCP burnout. Similarly, many other commercial use cases
of multimodal AI models in pandemic preparedness and
response have been tested with promising results.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 9

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

4. Accelerating clinical trials with digital twins

Pharma industry relies heavily on clinical trials to Integrating data from multiple sources to develop
support clinical interventions. We have already digital twin models using AI tools have long been
discussed how clinical trials deal with the considered in precision oncology and
challenges of health inequities and HCP cardiovascular health. Additionally, an open-source
engagement. Moreover, successful trials only help modular framework can support the development
10 out of 100 in a given demographic owing to the of medical digital twin models. From a commercial
limited number of participants. These trials often standpoint, [Link] has developed and tested
take place in an ideal, controlled setting to digital twin models that leverage diverse sets of
measure the efficacy rather than the effectiveness clinical data to enhance clinical trials for
of the intervention in a real-world scenario. Alzheimer’s disease and multiple sclerosis.

The development of accurate and useful digital

So how can pharma companies bridge these twin technology depends upon our ability to
gaps in clinical trials using multimodal AI? 
collect large and diverse multimodal data.
Considering the complexities behind the human
The answer lies in a complimentary trial body and mind, this would constitute omics and
approach that leverages ‘digital twins’ that physiological sensor data to clinical and
can fill the knowledge gaps using large sociodemographic data. It will likely require large
amounts of data to model and predict with collaborations across health systems, research
high precision how a certain therapeutic groups, and industries, such as the Swedish Digital
intervention would benefit or harm a Twins Consortium.
particular patient.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 10

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

5. Expanding the scope of virtual

healthcare assistants

AI-powered virtual healthcare assistants are yet another

addition to healthcare consumerism — a movement that
has made the delivery of health services more efficient,
convenient, and cost-effective — forcing the industry to
explore better ways to connect providers and patients
while taking some burden away from HCPs. Today, most
virtual assistants target a particular condition or use case
with rule-based approaches and predefined app-led
dialog. For example, the majority of existing applications
of these narrowly focused virtual health assistants are in
diabetes care.

But with ample data from genome sequencing, other omics

layers, EHRs, continuous monitoring of blood biomarkers
and metabolites, biosensors, and other relevant
biomedical data, medtech firms have an opportunity to
integrate these multiple data sources, creating
individualized customer profiles. Feeding this multimodal
data to AI models will facilitate the development of
broadly focused personalized virtual health assistants to
promote behavior change, answer health-related
questions, triage symptoms, or communicate with
healthcare providers when appropriate. 

The fusion between increasingly sophisticated multimodal

learning with rapidly advancing generative AI has the
potential to scale and deliver accurate and personalized
health coaching.

In a disability context, a Danish startup Be My Eyes

is already leading the way. Be My Eyes connects
people who are blind or have low vision with
volunteers for help with hundreds of daily life tasks
like identifying a product or navigating an airport.
With the bimodal capability (image and text) of
GPT-4, they began developing a GPT-4 powered
Virtual Volunteer™ within the Be My Eyes app that
can generate the same level of context and
understanding as a human volunteer.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 11

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

6. Providing personalized precision healthcare

In 2021, sequencing saved an infant’s life from the While compiling data from these modalities remains
same condition that this child’s elder sibling had challenging, overcoming this problem is paramount
succumbed to a decade ago. From the start of the to saving HCP effort for more crucial tasks.
sequencing to the diagnosis, the process took just
13 hours — a figure, unimaginable until recently, Our partners at AWS, Microsoft, and Google
that highlights the remarkable progress in offer omics-focused cloud infrastructures so
sequencing technology and how quickly it can you can process and analyze biomedical data
provide answers at lower costs. Large amounts of
at scale, creating value propositions in
fine-grained biological data using novel
healthcare and pharma
technological developments can be easily
leveraged now. These are collectively referred to
Combining omics data with EHRs and imaging
as the ‘omes’, and include genome, proteome,
data increases our understanding of human
transcriptome, immunome, epigenome,
health, promoting precise and individualized
metabolome, and microbiome.

preventive, diagnostic, and therapeutic strategies

Though each omics dataset holds value in in healthcare.
different clinical and research settings, their
combined power can help HCPs provide
personalized care with greater precision.

Health Centers

Omics Clinical Time

notes series
Drug Treatment Clinical
discovery decisions trials

Demographic/ Imaging
lab values
Target Diagnosis/ Phenotyping/
identification prognosis Subgrouping

P re ci s io n Health Ou tcom es I n fo r m at io n Commo ns

I n fo r m at io n trans fo r m at io n /m er gi n g/mod el i n g

Multimodal precision health; the flow of information

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 12

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Challenges — Data collection

and augmented analysis
Multimodal AI applications open the way for Usually, model development requires
endless possibilities and experimentation retrospective data in an offline setting, where the
throughout the healthcare pharma lifecycle, data scientists have complete control of the data.
creating new product and service opportunities It is then easier to perform data curation and
due to the explosion of multimodal data from preprocessing to create a cohort that fits the
IoMT. But technological limitations block such model’s design. But to ensure an omnichannel
developments and inhibit AI models’ success in experience within a healthcare setting, we also
commercial and clinical settings. This gap need to go a step ahead by implementing the
between development and deployment is known algorithm to reflect recent clinical practice.  

as the “AI chasm,” — an unfortunate spot where

Over the last few decades, many national
most projects end up, never to be heard from
biobanks have collected multimodal data from
again. If you are a pharma or medtech company,
thousands of participants to accelerate precision
you know the pain already. 

healthcare. These datasets span across several

Technology and health sectors should collaborate axes, including sample size, the depth of
and address the underlying technical challenges phenotyping, the length and intervals of follow-
so AI models can be actualized to depict ongoing up, the degree of interaction between
and historical clinical research. A robust participants, the heterogeneity and diversity of
technology infrastructure allows us to leverage the participants, the level of standardization and
data from multiple modalities and achieve better harmonization of the data, and the amount of
diagnostic performance across different medical linkage between data sources. In many cases,
tasks. The encouraging news is that all healthcare there is a need for large sample sizes ranging
data is inherently multimodal at its source level. from hundreds of thousands to millions to train AI
But no amount of technical sophistication can models. The costs of achieving rapid and deep
derive information not present in the data.
phenotyping with thousands of participants
become unsustainable unless automated data
So the key to establishing a successful multimodal
collection methods are in place.
AI infrastructure is the efficient collection,
curation, and harmonization of healthcare data to
prepare it for augmented analysis.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 13

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Our team at Grid Dynamics can help

Our data science and AI teams at Grid Dynamics can help ease your technical burden. We
understand that proper data governance for various business use cases is key to reducing
operational costs and accelerating the process of making critical business decisions.

Our Data Governance framework This is when our Data Quality AI-powered anomaly detection
helps you efficiently handle the framework comes in handy and automates data quality and helps
amount, variety, and complexity of helps increase confidence in the scale it to thousands of data
large datasets. We take advantage available healthcare data using data pipelines with minimal effort. From
of the latest advancements in the monitoring, inspection, and statistical process control to deep
automation of analytics with cleansing. So you can track errors learning algorithms, AI learns the
reporting, machine learning, and and duplicates, find root causes of relevant data profiles in real time,
artificial intelligence to fully errors and perform automated fixes. uncovering hidden defects or
automate data pipelines. unusual patterns. A rich user
interface helps tune and monitor
data quality metrics and profiles,
allowing data scientists to achieve a
deeper understanding of the data.

All our solution accelerators integrate well with our cloud- Once your aggregated data layer is configured, our Pharma
native Analytics Platform or IoT Analytics Platform that Next Best Action Engine helps deliver the best-in-class
enables pharma-focused state-of-the-art data processing multimodal commercial solution using advanced analytical
and multimodal AI models. These models support FDA- and digital tools, enabling your pharma business to
and HIPAA-compliant genomic and patient data analytics, engage more meaningfully with customers (HCPs) across
clinical trial insights to promote drug discovery, the world. Streamlined customer journeys powered by
commercial recommendations for optimized sales optimized messaging will help establish better
engagements, and IoMT data for personalized patient engagement with HCPs, adding a positive boost to the
experience and remote patient monitoring — all with a 10X sales cycle.
speed-to-market and efficiency boost. To set you up for
success, we’ll optimize these processes with data quality
automation and stream processing to achieve relevant,
real-time insights from multiple sources.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 14

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

Prepare to grow
At this stage, your multimodal
transactional engine may be ready to
launch, but our job is not yet done. Our
composability experts will help you build
a complete ecosystem that integrates
your data and analytics capabilities with
composable solutions like headless CMS
and Search, catalyzing an omnichannel
experience for your employees, HCPs,
and patients. So you can experiment
with various multimodal AI solutions and
scale your capabilities to meet the
demand in healthcare.

Choose Grid Dynamics as your agile co-

innovation partner.

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 15

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

About Grid Dynamics

Grid Dynamics is a global digital goods companies in the world, as well as leading
engineering company that co-innovates brands in the digital commerce, manufacturing,
with the most respected brands in the world finance, healthcare, and high tech sectors.

to solve complex problems, optimize

Our secret sauce? We hire the top 10% of global
business operations, and better serve
engineering talent and employ our extensive
customers. Driven by business impact and
expertise in emerging technology, lean software
agility, we create innovative, end-to-end
development practices, a high-performance product
solutions in digital commerce, AI, data, web
and agile delivery culture, and strategic partnerships
UI and UX, and cloud to help clients grow.

with leading technology service providers like

Headquartered in Silicon Valley, with Google, Amazon, and Microsoft.

delivery centers located throughout the

In 2019, Forrester named Grid Dynamics a leader
globe, Grid Dynamics is known for
among midsize agile development service providers.
architecting revolutionary digital technology
In 2020, Grid Dynamics went public and is trading on
platforms for 7 of the 25 largest retailers in
the NASDAQ under the GDYN ticker.
the US and 3 of the 10 largest consumer

The Forrester wave™

Midsize Agile Development Service

Providers Q2 2019

Stronger strategy

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 16

Grid Dynamics | Multimodal biomedical AI: applications and challenges in pharma, medtech, and healthcare

About Grid Dynamics

Key facts
Offices across the US, Mexico, UK, Netherlands,
Switzerland, India, and Central and Eastern Europ

4,000 employees in Q2 202

Forrester Leader Midsize Agile Software


Development Service Provider Q2 201

Proprietary starter kits developed in collaboration

with AWS, Google Cloud, Microsoft Azure, and
others.

Areas of expertise
Experience engineering
Web UI Mobile UX AR/VR

Data Science and AI

Search Personalization Supply chain IoT

Platform engineering
Microservices MACH Composable

Data engineering
Big data Streaming MLOps

Cloud and DevOps

CICD AIOps SRE QE

Clients

Copyright © 2023 Grid Dynamics Holdings, Inc. All rights reserved. 17

trusted engineering partner for digital transformation

Grid Dynamics Holdings, Inc.

5000 Executive Parkway,

Suite 520 / San Ramon, CA

650-523-5000

[Link]