Whitepaper

Databricks AI
Security Framework
(DASF)
Version 1.0

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 1
 Table of Contents

Executive Summary 3

1 Introduction 5
1.1 Intended audience 6
1.2 How to use this document 7

2 Risks in AI System Components 9

2.1 Raw Data 13
2.2 Data Prep 16
2.3 Datasets 19
2.4 Data Catalog Governance 20
2.5 Machine Learning Algorithms 22
2.6 Evaluation 24
2.7 Machine Learning Models 25
2.8 Model Management 27
2.9 Model Serving and Inference Requests 29
2.10 Model Serving and Inference Response 37
2.11 Machine Learning Operations (MLOps) 41
2.12 Data and AI Platform Security 42

3 Understanding Databricks Data Intelligence Platform AI Risk Mitigation Controls 44

3.1 The Databricks Data Intelligence Platform 44
Mosaic AI 46
Databricks Unity Catalog 47
Databricks Platform Architecture 48
Databricks Platform Security 49
3.2 Databricks AI Risk Mitigation Controls 50

4 Conclusion 66

5 Resources and Further Reading 68

6 Acknowledgments 70

7 Appendix: Glossary 72

8 License 84

Authors

Omar Khawaja Arun Pamulapati Kelly Albano

Vice President and Field Chief Senior Staff Security Field Engineer Product Marketing Manager
Information Security Officer
DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0
 Executive Summary

Machine learning (ML) and generative AI (GenAI) are transforming the

future of work by enhancing innovation, competitiveness and employee
productivity. However, organizations are grappling with the dual challenge
⟶ Executive
Summary of leveraging artificial intelligence (AI) technologies for opportunities while
managing potential security and privacy risks, such as data breaches and
Introduction
regulatory compliance.
Risks in AI System
Components
Adopting AI also raises regulatory considerations, exemplified by President Joe Biden’s
Executive Order (E.O. 14110) and NIST’s AI Risk Management Framework, underlining the
Understanding
Databricks Data importance of responsible governance and oversight. The evolving legal and regulatory
Intelligence Platform
AI Risk Mitigation landscape, combined with uncertainties around ownership accountability, leaves data, IT
Controls
and security leaders navigating how to effectively harness generative AI for organizational
Conclusion benefits while addressing perceived risks.

Resources and
The Databricks Security team created the Databricks AI Security Framework (DASF)
Further Reading
to address the evolving risks associated with the widespread integration of AI globally.
Unlike approaches that focus solely on securing models or endpoints, the DASF adopts a
Acknowledgments
comprehensive strategy to mitigate cyber risks in AI systems. Based on real-world evidence
Appendix: indicating that attackers employ simple tactics to compromise ML-driven systems, the
Glossary
DASF offers actionable defensive control recommendations. These recommendations can

License
be updated as new risks emerge and additional controls become available. The framework’s
development involved a thorough review of multiple risk management frameworks,
recommendations, whitepapers, policies and AI security acts.

The DASF is designed for collaboration between business, IT, data, AI and security teams
throughout the AI lifecycle. It addresses the evolving nature of data science from a
research-oriented to a project-based discipline, facilitating structured conversations on
security threats and mitigations without needing deep expertise crossover. We believe
the document will be valuable to security teams, ML practitioners and governance officers,
providing insights into how ML impacts system security, applying security engineering
principles to ML, and offering a detailed guide for understanding the security and
compliance of specific ML systems.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 3
 The DASF walks its readers through the 12 foundational components of a generic data-
centric AI system: raw data, data prep, datasets, data and AI governance, machine
learning algorithms, evaluation, machine learning models, model management, model
serving and inference, inference response, machine learning operations, and data and AI
platform security. Databricks identified 55 technical security risks that arise across these
⟶ Executive components and dedicated a chapter describing the specific component, the associated
Summary
risks and the available controls we recommend you leverage. We also provide a guide to
Introduction each AI and ML mitigation control — its shared responsibility between Databricks and your
organization, and the associated Databricks technical documentation available to learn how
Risks in AI System
Components
to enable said control.

The framework concludes with Databricks’ final recommendations on how to manage and
Understanding
Databricks Data deploy AI models safely and securely, which are consistent with the core tenets of machine
Intelligence Platform
AI Risk Mitigation learning adoption: identify the ML business use case, determine the ML deployment model,
Controls
select the most pertinent risks, enumerate threats for each risk and choose which controls
Conclusion to implement. We also provide further reading to enhance your knowledge of the AI field and
the frameworks we reviewed as part of our analysis. While we strive for accuracy, given the
Resources and
Further Reading
evolving nature of AI, please feel free to contact us with any feedback or suggestions. Your
input is valuable to us. If you want to participate in one of our AI Security workshops, please
Acknowledgments contact [email protected]. If you are curious about how Databricks approaches security,
please visit our Security and Trust Center.
Appendix:
Glossary

License

Security & Trust Center

Your data security is our priority

Learn more ⟶

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 4
 01
Introduction

Machine learning (ML) and generative AI (GenAI) are revolutionizing the future of work.
Organizations understand that AI is helping to build innovation, maintain competitiveness
and improve the productivity of their employees. Equally, organizations understand that
Executive
Summary
their data provides a competitive advantage for their artificial intelligence (AI) applications.
Leveraging these technologies presents opportunities but also potential risks. There is a risk
⟶ Introduction of security and privacy breaches, as the data sent to an external large language model (LLM)
could be leaked or summarized. Several organizations have even banned the use of ChatGPT
Risks in AI System
Components due to sensitive enterprise data being sent by users. Organizations are also concerned about
potential hazards such as data loss, data confidentiality, model theft, and risks of ensuring
Understanding
Databricks Data
existing and evolving compliance and regulation when they use their data for ML and GenAI.
Intelligence Platform
AI Risk Mitigation
Without the proper access controls, users can use generative AI models to find confidential
Controls
data they shouldn’t have access to. If the models are customer-facing, one organization
might accidentally receive data related to a different organization. Or a skilled attacker can
Conclusion
extract data they shouldn’t have access to. Without the auditability and traceability of these
Resources and models and their data, organizations face compliance risks.
Further Reading

AI adoption also brings a crucial regulatory dimension, emphasizing the need for thoughtful
Acknowledgments
oversight and responsible governance. In October 2023, President Biden issued an Executive
Order on safe, secure and trustworthy artificial intelligence, emphasizing the responsible
Appendix:
Glossary development and use of AI technologies. The National Institute of Standards and Technology
(NIST) recently published its Artificial Intelligence Risk Management Framework (AI RMF) to
License
help federal agencies manage and secure their information systems. It provides a structured
process for identifying, assessing and mitigating cybersecurity risks. Gartner’s 2023 Security
Leader’s Guide to Data Security report1 predicts that “at least one global company will
see its AI deployment banned by a regulator for noncompliance with data protection or AI
governance legislation by 2027.” With ownership accountability and an ever-evolving legal
and regulatory landscape, data, IT and security leaders are still unclear on how to take
advantage of generative AI for their organization while mitigating any perceived risks.

The Databricks Security team developed the Databricks AI Security Framework (DASF) to
help organizations understand how AI can be safely realized and risks mitigated as the global
community incorporates AI into more systems.

DATABRICKS 1
Gartner, Security Leader’s Guide to Data Security, Andrew Bales. September 7, 2023.
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 5
 The DASF takes a holistic approach to mitigating AI security risks instead of focusing only
on the security of models or model endpoints. Abundant real-world evidence suggests
that attackers use simple tactics to subvert ML-driven systems. That is why, with the DASF,
we propose actionable defensive control recommendations. These recommendations
are subject to change as new risks are identified and new controls are made available. We
Executive reviewed many risk management frameworks, recommendations, whitepapers, policies and
Summary
acts on AI security. We encourage the audience to review such material, including some of
⟶ Introduction the material linked in the resources section of this document. Your feedback is welcome.

Risks in AI System
Components 1.1 Intended audience

Understanding The Databricks AI Security Framework is intended to be used by data and AI teams
Databricks Data
Intelligence Platform collaborating with their security teams across the AI/ML lifecycle. Traditionally, the skill sets
AI Risk Mitigation
Controls of data scientists, data engineers, security teams, governance officers and DevSecOps
engineering teams did not overlap. The communication gap between data scientists and
Conclusion
these teams was manageable, given the research-oriented nature of data science and its

Resources and primary focus on delivering information to executives. However, as data science transforms
Further Reading
into a project-based discipline, it becomes crucial for these teams to collaborate.

Acknowledgments The guidance in this document provides a way for disciplines to have structured
conversations on these new threats and mitigations without requiring security engineers to
Appendix:
Glossary become data scientists or vice versa. We mostly did this work for our customers to ensure
the security and compliance of production ML use cases on the Databricks Data Intelligence
License
Platform. That said, we believe that what we have produced will be helpful to three major
audience groups:

Security teams (CISOs, security leaders, DevSecOPs, SREs) can use the DASF
to understand how ML will impact the security of systems they may be asked to
secure, as well as to understand some of the basic mechanisms of ML.

ML practitioners and engineers (data engineers, data architects, ML engineers,

data scientists) can use the DASF to understand how security engineering and,
more specifically, the “secure by design” mentality can be applied to ML.

Governance leaders, risk officers and policymakers can use the DASF as
DATABRICKS a detailed guide into a risk mindset to learn more about the security and
AI SECURITY
FRAMEWORK compliance of specific ML systems.
(DASF)
VERSION 1.0 6
 If you are new to GenAI, you can build foundational knowledge, including large language
models (LLMs), with four short videos in this Generative AI Fundamentals course created by
Databricks. In this free training, you will learn what generative AI is, what the main generative AI
applications are, and their capabilities and potential applications across various domains. It will
also cover the limits and risks of generative AI technologies, including ethical considerations.
Executive
Summary
1.2 How to use this document
⟶ Introduction

The Databricks AI Security Framework is designed for collaborative use throughout the AI
Risks in AI System
Components lifecycle by data and AI teams and their security counterparts referenced above. The DASF is
meant to foster closer collaboration between these teams and improve the overall security
Understanding of AI systems. The concepts in this document are applicable for all teams, even if they do not
Databricks Data
Intelligence Platform use Databricks to build their use cases. That said, we will refer to documentation or features
AI Risk Mitigation
Controls in Databricks terminology where it allows us to simplify our language or make this document
more actionable for our direct customers. We hope those who do not use Databricks will be
Conclusion
able to follow along without issue.

Resources and
Further Reading First, we suggest that organizations find out what type of AI models are being built or being
used. As a guideline, we define model types broadly as the following:
Acknowledgments

Predictive ML models. These are traditional structured data machine learning

Appendix:
Glossary models trained on your enterprise tabular data. They are typically Python models
packaged in the MLflow format. Examples include scikit-learn, XGBoost, PyTorch
License and Hugging Face transformer models.

State-of-the-art open models made available by Foundation Model APIs. These

models are curated foundation model architectures that support optimized
inference. Base models, like Llama-2-70B-chat, BGE-Large and Mixtral-8x7B,
are available for immediate use with pay-per-token pricing, and workloads that
require performance guarantees and fine-tuned model variants can be deployed
with provisioned throughput. We subcategorize these models’ usage patterns
as Foundation Model APIs to LLMs and retrieval augmented generation (RAG),
pretraining, and fine-tuning use of LLMs.

External models (third-party services). These are models that are hosted
outside of Databricks. Endpoints that serve external models can be centrally
DATABRICKS
AI SECURITY
governed and customers can establish rate limits and access control for them.
FRAMEWORK Examples include foundation models such as OpenAI’s GPT-4, Anthropic’s
(DASF)
VERSION 1.0 Claude and others. 7
 Second, we recommend that organizations identify where in their organization AI systems are
being built, the process, and who is responsible. The modern AI system lifecycle often involves
diverse stakeholders, including business stakeholders, subject matter experts, governance
officers, data engineers, data scientists, research scientists, application developers,
administrators, AI security engineers, DevSecOps engineers and MLSecOps engineers.
Executive
Summary We recommend that those responsible for AI systems begin by reviewing the 12 foundational
components of a generic data-centric AI system and the types of AI models, as outlined in
⟶ Introduction
Section 2: Risks in AI System Components. This section details security risk considerations

Risks in AI System
and potential mitigation controls for each component, helping organizations reduce overall
Components
risk in their AI system development and deployment processes. Each security risk is mapped
to a set of mitigation controls that are ranked in prioritized order, starting with the perimeter
Understanding
Databricks Data security to data security. These guidelines apply to providers of all AI systems, whether built
Intelligence Platform
AI Risk Mitigation from scratch or using third-party tools and services, and encompass both predictive ML
Controls
models and generative AI models.
Conclusion
To further refine risk identification, we categorize risks by model type: predictive ML models,

Resources and
RAG-LLMs, fine-tuned LLMs, pretrained LLMs, foundation models and external models. Once
Further Reading
the relevant risks are identified, teams can determine which controls are applicable from
the comprehensive list in Section 3: Understanding Databricks Data Intelligence Platform
Acknowledgments
AI Risk Mitigation Controls. Each control is tagged as “Out-of-the-box,” “Configuration”
Appendix: or “Implementation,” helping teams estimate the effort involved in the implementation of
Glossary
the control on the Databricks Data Intelligence Platform, with reference links to relevant

License
documentation provided.

Our experience shows that implementing these guidelines helps customers build secure and
functional AI systems.

When I think about what makes a good accelerator, it’s all about making things smoother,
more efficient and fostering innovation. The DASF is a proven and effective tool for
security teams to help their partners get the most out of AI. Additionally, it lines up with
established risk frameworks like NIST, so it’s not just speeding things up – it’s setting a
solid foundation in security work.
DATABRICKS
AI SECURITY
FRAMEWORK Riyaz Poonawala
(DASF) Vice President of Information Security
VERSION 1.0 8
 02
Risks in AI System Components

The DASF starts with a generic AI system in terms of its constituent components and works
through generic system risks. By understanding the components, how they work together
and the risk analysis of such architecture, an organization concerned about security can
Executive
Summary
get a jump start on determining risks in its specific AI system. The Databricks Security team
considered these risks and built mitigation controls into our Databricks Data Intelligence
Introduction Platform. We mapped the respective Databricks Platform control and link to Databricks
product documentation for each risk.
⟶ Risks in AI System
Components

AI System Components
Understanding
Databricks Data
Intelligence Platform Governance
AI Risk Mitigation
Controls
2 3 3
4 Serving
Data Prep Datasets Develop and ævaluate Model Infrastructure
Conclusion
4 1T
ETL
Training 5 Custom models
9
Clean data
AlgoritÊm

Inference requests
Validation External models
Resources and Exploratory data 2

Prompt/RAG
analytics (EDA) 6 4
Further Reading 7 FineÝtuninÚ
Featurization Test Evaluation and pretrained
Joins, aggregations, model
transformations, etc. Model serving
Featur–
Acknowledgments extraction
4
4 8
2 Model
1 Catalog Management

Inference response
Mode
l AI Gateway
1T assets
Appendix: Raw Data
Features
Glossary Your data for RAG
Indexes Vector searcÊ and feature
/
function lookup
Models
Monitor 5
License Logs 1T
New ML and RLHF data

DataOps ModelOps DevSecOps

1 6
11 Operations and Platform 12

Figure 1: Foundational components of a generic data-centric AI system.

# AI component number Number of risks
Numbers in orange indicate risks identified in that specific system.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 9
 Data operations (#1-#4 in Figure 1) include ingesting and transforming data and
ensuring data security and governance. Good ML models depend on reliable data
pipelines and secure DataOps infrastructure.

Model operations (#5-#8 in Figure 1) include building predictive ML models,

Executive
acquiring models from a model marketplace, or using LLMs like OpenAI or
Summary Foundation Model APIs. Developing a model requires a series of experiments and a
way to track and compare the conditions and results of those experiments.
Introduction

⟶ Risks in AI System
Model deployment and serving (#9 and #10 in Figure 1) consists of securely
Components building model images, isolating and securely serving models, automated scaling,
rate limiting, and monitoring deployed models. Additionally, it includes feature
Understanding
Databricks Data
and function serving, a high-availability, low-latency service for structured data
Intelligence Platform in retrieval augmented generation (RAG) applications, as well as features that are
AI Risk Mitigation
Controls required for other applications, such as models served outside of the platform or
any other application that requires features based on data in the catalog.
Conclusion

Operations and platform (#11 and #12 in Figure 1) include platform vulnerability
Resources and
Further Reading management and patching, model isolation and controls to the system, and
authorized access to models with security in the architecture. Also included is
Acknowledgments operational tooling for CI/CD. It ensures the complete lifecycle meets the required
standards by keeping the distinct execution environments — development,
Appendix: staging and production — for secure MLOps.
Glossary

License
In our analysis of AI systems, we identified 55 technical security risks across the 12
components based on the AI model types deployed by our customers (namely, predictive ML
models, generative foundation models and external models as described above), customer
questions and questionnaires, security reviews of customer deployments, in-person CISO
workshops, and customer surveys about AI risks. In the table below, we outline these basic
components that align with steps in any AI system and highlight the types of security risks
our team identified.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 10
 SY S T E M SY S T E M P OT E N T I A L
S TAG E COMPONENTS (F I G U R E 1) S E C U R I T Y R I S KS

19 specific risks:
1 Raw data → 1.1 Insufficient access controls →
2 Data preparation → 1.2 Missing data classification →
Data
3 Datasets → 1.3 Poor data quality →
operations
4 C
 atalog and 1.4 Ineffective storage and encryption →
governance → 1.5 Lack of data versioning →
Executive 1.6 Insufficient data lineage →
Summary
1.7 Lack of data trustworthiness →
1.8 Data legal →
Introduction 1.9 Stale data →
1.10 Lack of data access logs →
⟶ Risks in AI System
2.1 Preprocessing integrity →
Components
2.2 Feature manipulation →
2.3 Raw data criteria →
Understanding
2.4 Adversarial partitions →
Databricks Data
Intelligence Platform
AI Risk Mitigation 3.1 Data poisoning →
Controls 3.2 Ineffective storage and encryption →
3.3 Label flipping →

Conclusion 4.1 L
 ack of traceability and transparency of
model assets →
4.2 Lack of end-to-end ML lifecycle →
Resources and
Further Reading

14 specific risks:
Acknowledgments
5 ML algorithm → 5.1 Lack of tracking and reproducibility of experiments ⟶
6 Evaluation → 5.2 Model drift ⟶
Appendix:
Model
7 Model build → 5.3 Hyperparameters stealing ⟶
Glossary operations
8 Model management → 5.4 Malicious libraries ⟶

6.1 Evaluation data poisoning ⟶

License
6.2 Insufficient evaluation data ⟶

7.1 Backdoor machine learning/Trojaned model ⟶

7.2 Model assets leak ⟶
7.3 ML supply chain vulnerabilities ⟶
7.4 Source code control attack ⟶

8.1 Model attribution ⟶

8.2 Model theft ⟶
8.3 Model lifecycle without HITL ⟶
8.4 Model inversion ⟶

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 11
 SY S T E M SY S T E M P OT E N T I A L
S TAG E COMPONENTS (F I G U R E 1) S E C U R I T Y R I S KS

15 specific risks:
9 M
 odel Serving — 9.1 Prompt inject →
inference requests → 9.2 Model inversion →
Model 10  odel Serving —
M 9.3 Model breakout →
deployment inference responses →
Looped input →
and serving 9.4

9.5 Infer training data membership →

Executive 9.6 Discover ML model ontology →
Summary
9.7 Denial of service (DOS) →
9.8 LLM hallucinations →
Introduction 9.9 Input resource control →
9.10 Accidental exposure of unauthorized data to models →
⟶ Risks in AI System
10.1 Lack of audit and monitoring inference quality →
Components
10.2 Output manipulation →
10.3 Discover ML model ontology →
Understanding
10.4 Discover ML model family →
Databricks Data
Intelligence Platform 10.5 Black-box attacks →
AI Risk Mitigation
Controls

7 specific risks:
Conclusion 11 ML operations → 11.1 L
 ack of MLOps — repeatable enforced standards →
12 ML platform →
Operations 12.1 L
 ack of vulnerability management →
Resources and
Further Reading
and 12.2 L
 ack of penetration testing and bug bounty →
platform 12.3 L
 ack of incident response →
12.4 U
 nauthorized privileged access →
Acknowledgments
12.5 P
 oor SDLC →
12.6 L
 ack of compliance →
Appendix:
Glossary

License The 12 foundational components of a generic data-centric AI/ML model and risk
considerations are discussed in detail below.

Note: We are aware of nascent risks such as energy-latency attacks, rowhammer attacks,
side channel attacks, evasion attacks, functional adversarial attacks and other adversarial
examples, but these are out of scope for this version of the framework. We may reconsider
these and any new novel risks in later versions if we see them becoming material.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 12
 2.1 Raw Data

Data is the most important aspect of AI systems because it provides the foundation that all
AI functionality is built on. Raw data includes enterprise data, metadata and operational data.
It can be semi-structured or unstructured such as images, sensor data, documents. This
data can be batch data or streaming data. Data security is paramount and equally important
for ensuring the security of machine learning algorithms and any technical deployment
Executive
Summary particulars. Securing raw data is a challenge in its own right, and all data collections in an AI
system are subject to the usual data security challenges and some new ones. A fully trained
Introduction
machine learning (ML) system, whether online or offline, will inevitably encounter new input

⟶ Risks in AI System data during normal operations or retraining processes. Fine-tuning and pretraining of LLMs
Components
further increases these risks by allowing customizations with potentially sensitive data.

Understanding
Databricks Data
Intelligence Platform RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S
AI Risk Mitigation
Controls
RAW DATA 1.1

Conclusion Insufficient access controls DASF 1 S

 SO with IdP and MFA to authenticate and limit who can
access your data and AI platform
Effective access management is fundamental
to data security, ensuring only authorized DASF 2 S
 ync users and groups to inherit your organizational roles
Resources and to authorize access to data
individuals or groups can access specific
Further Reading
datasets. Such security protocols encompass DASF 3 R
 estrict access using IP access lists to limit IP addresses
authentication, authorization and finely tuned that can authenticate to your data and AI platform
access controls tailored to the scope of access
Acknowledgments required by each user, down to the file or record DASF 4 R
 estrict access using private link as a strong control that
level. Establishing definitive governance policies limits the source for inbound requests
for data access is imperative in response to DASF 5 C
 ontrol access to data and other objects for permissions
Appendix: the heightened risks from data breaches and
model across all data assets to protect data and sources
Glossary regulations like the General Data Protection
Regulation (GDPR) and the California Consumer DASF 51 S
 hare data and AI assets securely
Privacy Act (CCPA). These policies guard
against unauthorized use and are a cornerstone
License Applicable AI deployment model:
of preserving data integrity and maintaining
customer trust. Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Data operations →
Pre-trained LLMs: | Foundational models: | External models:

RAW DATA 1.2

Missing data classification DASF 6 C

 lassify data with tags as it is ingested into the platform
aligning with the organization’s governance requirements
Data classification is critical for data governance,
enabling organizations to effectively sort and
categorize data by sensitivity, importance and Applicable AI deployment model:

criticality. As data volumes grow exponentially, Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
prioritizing sensitive information protection, risk Pre-trained LLMs: | Foundational models: | External models:
reduction and data quality becomes imperative.
Classification facilitates the implementation of
appropriate security measures and governance
policies by evaluating data’s risk and value. A
robust classification strategy strengthens data
governance, mitigates risks, and ensures data
integrity and security on a scalable level.

DATABRICKS Data operations →

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 13
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

RAW DATA 1.3

Poor data quality DASF 7 E

 nforce data quality checks on batch and
streaming datasets
Data quality is crucial for reliable data-driven
decisions and is a cornerstone of data governance. DASF 21 Monitor data and AI system from a single pane of glass
Malicious actors threaten data integrity, accuracy
DASF 36 S
 et up monitoring alerts
and consistency, challenging the analytics and
decision-making processes that depend on
high-quality data, just as a well-intentioned user Applicable AI deployment model:
with poor-quality data can limit the efficacy of
an AI system. To safeguard against these threats,
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Executive Pre-trained LLMs: | Foundational models: | External models:
Summary organizations must rigorously evaluate key data
attributes — accuracy, completeness, freshness
and rule compliance. Prioritizing data quality
enables organizations to trace data lineage, apply
Introduction data quality rules and monitor changes, ensuring
analytical accuracy and cost-effectiveness.

Data operations →
⟶ Risks in AI System
Components

Understanding RAW DATA 1.4

Databricks Data
Ineffective storage and encryption DASF 8 E
 ncrypt data at rest
Intelligence Platform
AI Risk Mitigation DASF 9 E
 ncrypt data in transit
Insecure data storage leaves organizations
Controls
vulnerable to unauthorized access, potentially DASF 5 C
 ontrol access to data and other objects for metadata
leading to data breaches with significant legal, encryption across all data assets
financial and reputational consequences.
Conclusion Encrypting data at rest can help to render the
data unreadable to unauthorized actors who Applicable AI deployment model:
bypass security measures or attempt large- Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
scale data exfiltration. Additionally, compliance
Resources and
with industry-specific data security regulations
Pre-trained LLMs: | Foundational models: | External models:
Further Reading
often necessitates such measures.

Data operations →
Acknowledgments

Appendix: RAW DATA 1.5

Glossary
Lack of data versioning DASF 10 V
 ersion data and track change logs on large-scale
datasets that are fed to your models
When data gets corrupted by a malicious user by
License
introducing a new set of data or by corrupting a
data pipeline, you will need to be able to roll back Applicable AI deployment model:
or trace back to the original data. Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Data operations → Pre-trained LLMs: | Foundational models: | External models:

RAW DATA 1.6

Insufficient data lineage DASF 11 C

 apture and view data lineage

Because data may come from multiple sources DASF 51 S

 hare data and AI assets securely
and go through multiple transformations over
its lifecycle, understanding data transparency
Applicable AI deployment model:
and usage requirements in AI training is
important to risk management. Many compliance Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

regulations require organizations to have a clear Pre-trained LLMs: | Foundational models: | External models:

understanding and traceability of data used for

AI. Data lineage helps organizations be compliant
and audit-ready, thereby alleviating the
operational overhead of manually creating the
trails of data flows for audit reporting purposes.

DATABRICKS Data operations →

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 14
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

RAW DATA 1.7

Lack of data trustworthiness DASF 10 V

 ersion data and track change logs on large-scale
datasets that are fed to your models
Attackers may tamper with or poison raw input
data (training data, RAG data, etc). Adversaries
DASF 54 S
 hare data and AI assets securely
may exploit public datasets, which often resemble
those used by targeted organizations. To mitigate Applicable AI deployment model:
these threats, organizations should validate data
sources, implement integrity checks, and utilize
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

AI and machine learning for anomaly detection. Pre-trained LLMs: | Foundational models: | External models:

Data operations →
Executive
Summary

RAW DATA 1.8

Introduction D
 elete records from datasets and retrain models to
Data legal DASF 12

forget data subjects

Intellectual property concerns of training data
⟶ Risks in AI System and and legal mandates — such as those from DASF 29 B
 uild MLOps workflows to track models and trace data
GDPR, CCPA and LGPD — necessitate the capability sources and lineage to retrain models with the updated
Components
of machine learning systems to “delete” specific dataset by following legal constraints
data. But you often can’t “untrain” a model; DASF 27 P
 retrain a large language model (LLM) to only use the
during the training process, input data is data that is allowed with LLMs for inference
Understanding
encoded into the internal representation of the
Databricks Data
model, characterized by elements like thresholds
Intelligence Platform
and weights, which could become subject to Applicable AI deployment model:
AI Risk Mitigation
legal constraints. Tracking your training data
Controls Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
and retraining your model using clean and
ownership-verified datasets is essential for
Pre-trained LLMs: | Foundational models: | External models:

meeting regulatory demands.

Conclusion
Data operations →

Resources and
Further Reading
RAW DATA 1.9

Stale data DASF 13 U

 se near real-time data for fault-tolerant, near real-time
Acknowledgments
data ingestion, processing and machine learning, and AI
When downstream data is not timely or accurate, for streaming data
business processes can be delayed, significantly
Appendix: affecting overall efficiency. Attackers may DASF 7 E
 nforce data quality checks on batch and streaming
Glossary deliberately target these systems with attacks datasets
like denial of service, which can undermine the
model’s performance and dependability. It’s
Applicable AI deployment model:
License crucial to proactively counteract these threats.
Data streaming and performance monitoring Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

help protect against such risks, maintaining Pre-trained LLMs: | Foundational models: | External models:

the input data integrity and ensuring they are

delivered promptly to the model.

Data operations →

RAW DATA 1.10

Lack of data access logs DASF 14 A

 udit actions performed on datasets

Without proper audit mechanisms, an

organization may not be fully aware of its risk Applicable AI deployment model:
surface area, leaving it vulnerable to data Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
breaches and regulatory noncompliance. Pre-trained LLMs: | Foundational models: | External models:
Therefore, a well-designed audit team within
a data governance or security governance
organization is critical in ensuring data security
and compliance with regulations such as GDPR
and CCPA. By implementing effective data
access auditing strategies, organizations can
maintain the trust of their customers and protect
DATABRICKS
their data from unauthorized access or misuse.
AI SECURITY
FRAMEWORK Data operations →
(DASF)
VERSION 1.0 15
 2.2 Data Prep

Machine learning algorithms require raw input data to be transformed into a representational
form they can understand. This data preparation step can impact the security and
explainability of an ML system, as data plays a crucial role in security. Data preparation
Executive includes the following tasks:
Summary

Introduction

1 | Cleaning and formatting data includes handling missing values or outliers,

⟶ Risks in AI System
Components ensuring data is in the correct format and removing unneeded columns.

Understanding
2 | Preprocessing data includes tasks like numerical transformations,
Databricks Data aggregating data, encoding text or image data, and creating new features.
Intelligence Platform

|
AI Risk Mitigation
Controls 3 Combining data includes tasks like joining tables or merging datasets.

Conclusion
4 | Label data includes tasks like identifying raw data (images, text files, videos,
and so on) and adding one or more meaningful and informative labels to

Resources and provide context so an ML model can learn from it.

Further Reading
5 | Validating and visualizing data includes exploratory data analysis to ensure
Acknowledgments data is correct and ready for ML. Visualizations like histograms, scatter plots,
box and whisker plots, line plots, and bar charts are all useful tools to confirm
Appendix:
Glossary data correctness.

License

Companies need not sacrifice security for AI innovation. The Databricks AI Security
Framework is a comprehensive tool to enable AI adoption securely. It not only maps
AI security concerns to the AI development pipeline, but makes them actionable for
Databricks customers with practical controls. We're pleased to have contributed to the
development of this valuable community resource.
DATABRICKS
AI SECURITY
FRAMEWORK Hyrum Anderson
(DASF) CTO
VERSION 1.0 16
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

DATA PREP 2.1

Preprocessing integrity DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Preprocessing includes numerical
transformations, data aggregation, text or DASF 2 S
 ync users and groups to inherit your organizational roles
Executive image data encoding, and new feature creation, to access data
Summary followed by combining data by joining tables DASF 3 R
 estrict access using IP access lists to limit IP addresses
or merging datasets. Data preparation involves that can authenticate to your data and AI platform
cleaning and formatting tasks such as handling
Introduction missing values, ensuring correct formats and DASF 4 R
 estrict access using private link as a strong control that
removing unnecessary columns. limits the source for inbound requests

Insiders or external actors can introduce errors DASF 5 C

 ontrol access to data and other objects for permissions
⟶ Risks in AI System or manipulate data during preprocessing or from model across all data assets to protect data and sources
Components the information repository itself. DASF 7 E
 nforce data quality checks on batch and streaming
Data operations → datasets for data sanity checks and automatically detect
anomalies before they make it to the datasets
Understanding
Databricks Data DASF 11 C
 apture and view data lineage to capture the lineage all
Intelligence Platform the way to the original raw data sources
AI Risk Mitigation DASF 15 E
 xplore datasets and identify problems
Controls
DASF 52 S
 ource Code Control
DASF 16 S
 ecure model features to reduce the risk of malicious
Conclusion actors manipulating the features that feed into ML
training
DASF 42 D
 ata-centric MLOps and LLMOps promote models
Resources and as code
Further Reading

Applicable AI deployment model:

Acknowledgments Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

Appendix:
Glossary
DATA PREP 2.2

License Feature manipulation DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
In almost all cases, raw data requires
preprocessing and transformation before it DASF 2 S
 ync users and groups to inherit your organizational roles
is used to build a model. This process, known to access data
as feature engineering, involves converting DASF 3 R
 estrict access using IP access lists to limit IP addresses
raw data into structured features, the building that can authenticate to your data and AI platform
blocks of the model. Feature engineering is
critical to quality and effectiveness of the DASF 4 R
 estrict access using private link as a strong control that
model. However, how data are annotated into limits the source for inbound requests
features can introduce the risk of incorporating DASF 16 S
 ecure model features to prevent and track unauthorized
attacker biases into an AI/ML system. This can
updates to features and for lineage or traceability
compromise the integrity and accuracy of the
model and is a significant security concern for DASF 42 D
 ata-centric MLOps and LLMOps promote models
models used in critical decision-making (e.g., as code
financial forecasting, fraud detection).

Data operations → Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 17
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

DATA PREP 2.3

Raw data criteria DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
An attacker who understands raw data
selection criteria may be able to introduce
DASF 2 S
 ync users and groups to inherit your organizational roles
Executive malicious input that compromises system to access data
Summary integrity or functionality later in the model DASF 3 R
 estrict access using IP access lists to restrict the IP
lifecycle. Exploitation of this knowledge allows addresses that can authenticate to Databricks
the attacker to bypass established security
DASF 4 R
 estrict access using private link as strong controls that
Introduction measures and manipulate the system’s
limit the source for inbound requests
output or behavior. Implementing stringent
security measures to safeguard against such DASF 43 U
 se access control lists to control access to data, data
manipulations is essential for maintaining the streams and notebooks
⟶ Risks in AI System
integrity and reliability of ML systems.
Components DASF 42 D
 ata-centric MLOps and LLMOps for unit and integration
Data operations → testing

Understanding Applicable AI deployment model:

Databricks Data
Intelligence Platform
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

AI Risk Mitigation
Pre-trained LLMs: | Foundational models: | External models:

Controls

Conclusion DATA PREP 2.4

Adversarial partitions DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Resources and If an attacker can influence the partitioning of
Further Reading datasets used in training and evaluation, they
DASF 2 S
 ync users and groups to inherit your organizational roles
can effectively exercise indirect control over to access data
the ML system by making them vulnerable to DASF 3 R
 estrict access using IP access lists to restrict the IP
Acknowledgments adversarial attacks, where carefully crafted addresses that can authenticate to Databricks
inputs lead to incorrect outputs. These attacks
DASF 4 R
 estrict access using private link as strong controls that
can exploit the space partitioning capabilities
limit the source for inbound requests
of machine learning models, such as tree
Appendix:
ensembles and neural networks, leading to DASF 17 T
 rack and reproduce the training data used for ML
Glossary
misclassifications even in high-confidence model training to track and reproduce the training data
scenarios. This form of “model control” can lead partitions and the human owner accountable for ML
to biased or compromised outcomes. Therefore, model training, as well as identify ML models and runs
License it is crucial that datasets accurately reflect the derived from a particular dataset
intended operational reality of the ML system. DASF 42 D
 ata-centric MLOps and LLMOps for unit and integration
Implementing stringent security measures
testing
to safeguard against such manipulations is
essential for maintaining the integrity and
reliability of ML systems. Applicable AI deployment model:

Data operations → Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

The DASF is a very important, foundational document. I think it will go far in helping to
bridge the knowledge gap between ML and security experts.
DATABRICKS
AI SECURITY
FRAMEWORK Diana Kelley
Protect AI
(DASF) CISO
VERSION 1.0 18
 2.3 Datasets

Prepared data must be grouped into different datasets: a training set, a validation set and a
testing set. The training set is used as input to the machine learning algorithm. The validation
set is used to tune hyperparameters and to monitor the machine learning algorithm for
overfitting. The test set is used after learning is complete to evaluate performance.

When creating these groupings, special care must be taken to avoid predisposing the
Executive ML algorithm to future attacks, such as adversarial partitions. In particular, the training
Summary
set deeply influences an ML system’s future behavior. Manipulating the training data

Introduction
represents a direct and potent means of compromising ML systems. By injecting malicious or
adversarial samples into the training set, attackers can subtly influence the model’s behavior,
⟶ Risks in AI System potentially leading to misclassification, performance degradation or even security breaches.
Components

These approaches, often called “data poisoning” or “backdoor attacks,” pose a significant
Understanding
Databricks Data threat to the robustness and reliability of ML systems deployed in various critical domains.
Intelligence Platform
AI Risk Mitigation Dataset security concerns with foundation models include the potential for leaks of sensitive
Controls
information. Fine-tuning and pretraining of LLMs further increases these risks as it allows

Conclusion
customizations with sensitive data.

Resources and
Further Reading RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Acknowledgments DATASETS 3.1

Data poisoning DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Appendix:
Attackers can compromise an ML system by
Glossary DASF 2 S
 ync users and groups to inherit your organizational roles
contaminating its training data to manipulate
its output at the inference stage. All three initial to access data
components of a typical ML system — raw data, DASF 3 R
 estrict access using IP access lists to restrict the IP
License data preparation and datasets — are susceptible addresses that can authenticate to your data and AI
to poisoning attacks. Intentionally manipulated platform
data, possibly coordinated across these
components, derail the ML training process and DASF 4 R
 estrict access using private link as strong controls that
create an unreliable model. Practitioners must limit the source for inbound requests
assess the potential extent of training data an DASF 5 C
 ontrol access to data and other objects for permissions
attacker might control internally and externally
model across all data assets to protect data and sources
and the resultant risks.
DASF 7 E
 nforce data quality checks on batch and streaming
Data operations →
datasets for data sanity checks, and automatically detect
anomalies before they make it to the datasets
DASF 11  apture and view data lineage to capture the lineage all
C
the way to the original raw data sources
DASF 16 S
 ecure model features
DASF 17  rack and reproduce the training data used for ML model
T
training and identify ML models and runs derived from a
particular dataset
DASF 51 S
 hare data and AI assets securely
DASF 14 A
 udit actions performed on datasets

Applicable AI deployment model:

DATABRICKS
AI SECURITY Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
FRAMEWORK Pre-trained LLMs: | Foundational models: | External models:
(DASF)
VERSION 1.0 19
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

DATASETS 3.2

Ineffective storage and encryption DASF 8 E

 ncrypt data at rest

Data stored and managed insecurely pose DASF 9 E

 ncrypt data in transit
significant risks, especially for ML systems. It’s DASF 5 C
 ontrol access to data and other objects for metadata
Executive crucial to consider who has access to training encryption across all data assets
Summary datasets and the reasons behind this access.
While access controls are a vital mitigation
strategy, their effectiveness is limited with Applicable AI deployment model:
public data sources, where traditional security
Introduction Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
measures may not apply. Therefore, it’s essential
to ask: What are the implications if an attacker
Pre-trained LLMs: | Foundational models: | External models:

gains access and control over your data

⟶ Risks in AI System
sources? Understanding and preparing for this
Components
scenario is critical for safeguarding the integrity
of ML systems.

Understanding Data operations →

Databricks Data
Intelligence Platform
AI Risk Mitigation
Controls DATASETS 3.3

Label flipping DASF 8 E

 ncrypt data at rest
Conclusion DASF 9 E
 ncrypt data in transit
Label-flipping attacks are a distinctive type of
data poisoning where the attacker manipulates DASF 5 C
 ontrol access to data and other objects for metadata
the labels of a fraction of the training data. In encryption across all data assets
Resources and these attacks, the attacker changes the labels of
Further Reading specific training points, which can mislead the
ML model during training. Even with constrained Applicable AI deployment model:
capabilities, these attacks have been shown to Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Acknowledgments significantly degrade the system’s performance,
demonstrating their potential to compromise the
Pre-trained LLMs: | Foundational models: | External models:

accuracy and reliability of ML models.

Appendix: Data operations →
Glossary

License

The DASF is a great example of Databricks’ leadership in AI and is a valuable contribution

to the industry at a critical time. We know the greatest risk associated with artificial
intelligence for the foreseeable future is bad people, and this framework offers an
effective counterbalance to those cybercriminals. The DASF is a pragmatic, operational
and efficient way to secure your organization.
DATABRICKS
AI SECURITY
FRAMEWORK Chris “Tito” Sestito
(DASF) CEO and Co-founder
VERSION 1.0 20
 2.4 Data Catalog Governance

Data catalog and governance is a comprehensive approach that comprises the principles,
practices and tools to manage an organization’s data assets throughout their lifecycle.
Managing governance for data and AI assets enables centralized access control, auditing,
lineage, data, and model discovery capabilities, and allows organizations to limit the risk of
data or model duplication, improper use of classified data for training, loss of provenance,
and model theft.
Executive
Summary
Additionally, if sensitive information in datasets is inadequately secured, breaches and leaks
can expose personally identifiable information (PII), financial data and even trade secrets,
Introduction
and cause potential legal repercussions, reputational damage and financial losses.
⟶ Risks in AI System
Components Proper data catalog governance allows for audit trails and tracing the origin and
transformations of data used to train AI models. This transparency encourages trust and
Understanding
Databricks Data
accountability, reduces risk of biases, and improves AI outcomes.
Intelligence Platform
AI Risk Mitigation
Controls
RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Conclusion
GOVERNANCE 4.1

Resources and Lack of traceability and DASF 5 C

 ontrol access to data and other objects for permissions
Further Reading model across all data assets to protect data and sources
transparency of model assets
DASF 7 E
 nforce data quality checks on batch and streaming
The absence of traceability in data, model assets
datasets for data sanity checks, and automatically detect
Acknowledgments and models and the lack of accountable human
anomalies before they make it to the datasets
oversight pose significant risks in machine
learning systems. This lack of traceability can: DASF 11 C
 apture and view data lineage to capture the lineage all
the way to the original raw data sources
Appendix:  ndermine the supportability and adoption
U
Glossary of these systems, as it hampers the ability to DASF 16 S
 ecure model features
maintain and update them effectively
DASF 17 T
 rack and reproduce the training data used for ML
Impact trust and transparency, which are model training and identify ML models and runs derived
License essential for users to understand and rely on from a particular dataset
the system’s decisions
DASF 18 G
 overn model assets for traceability
Limit the organization’s ability to meet
regulatory, compliance and legal obligations,
as these often require clear documentation Applicable AI deployment model:

and tracking of data and model-related Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
processes Pre-trained LLMs: | Foundational models: | External models:

Data operations →

GOVERNANCE 4.2

Lack of end-to-end ML lifecycle DASF 19 M

 anage end-to-end machine learning lifecycle for
measuring, versioning, tracking model artifacts, metrics
Continuously measure, track and analyze key and results
metrics, such as performance, accuracy and user
engagement, to ensure the AI system’s reliability. DASF 42 D
 ata-centric MLOps and LLMOps unit and integration
Demonstrating consistent performance builds testing
trustworthiness among users, customers and DASF 21 M
 onitor data and AI system from a single pane of glass
regulators.

Data operations →
Applicable AI deployment model:
DATABRICKS
AI SECURITY Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
FRAMEWORK Pre-trained LLMs: | Foundational models: | External models:
(DASF)
VERSION 1.0 21
 2.5 Machine Learning Algorithms

A machine learning algorithm is a method that operates on a dataset to produce an ML

model that optimizes a model task on the data. While the machine learning algorithm forms
the technical core of any ML system, attacks against it generally present significantly less
security risk compared to the data used for training, testing and eventual operation. However,
it is crucial to recognize and mitigate certain security risks associated with the choice of
Executive
Summary algorithm and its operational mode.

Machine learning algorithms primarily fall into two broad categories: offline and online. Offline
Introduction
systems are trained on a fixed dataset, “frozen” and subsequently used for predictions with
⟶ Risks in AI System new data. This approach is particularly common for classification tasks. Conversely, online
Components
systems continuously learn and adapt through iterative training with new data.

Understanding
Databricks Data From a security perspective, offline systems possess certain advantages. Their fixed, static
Intelligence Platform
AI Risk Mitigation nature reduces the attack surface and minimizes exposure to data-borne vulnerabilities
Controls
over time. In contrast, online systems are constantly exposed to new data, potentially increasing
their susceptibility to poisoning attacks, adversarial inputs and manipulation of learning
Conclusion
processes. Therefore, the choice between offline and online learning algorithms should be made
Resources and carefully, considering the ML system’s specific security requirements and operating environment.
Further Reading

Acknowledgments RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Appendix: ALGORITHMS 5.1

Glossary

Lack of tracking and reproducibility DASF 20 T

 rack ML training runs for documenting, measuring,
versioning, tracking model artifacts including algorithms,
License of experiments training environment, hyperparameters, metrics and
ML development is often poorly documented results
and tracked, and results that cannot be DASF 42 D
 ata-centric MLOps and LLMOps promote models as
reproduced may lead to overconfidence in an ML code and automate ML tasks for cross-environment
system’s performance. Common issues include: reproducibility
Critical details missing from a model’s
description
Applicable AI deployment model:

 esults that are fragile, producing dramatically

R Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
different results on a different GPU (even one Pre-trained LLMs: | Foundational models: | External models:
that is supposed to be spec-identical)

E xtensive tweaks to the authors’ system until

it outperforms the untweaked “baseline,”
resulting in asserted improvements that
aren’t borne out in practice (particularly
common in academic work)

Additionally, adversaries may gain initial access

to a system by compromising the unique portions
of the ML supply chain. This could include the
model itself, training data or its annotations, parts
of the ML software stack, or even GPU hardware.
In some instances, the attacker will need
secondary access to fully carry out an attack
DATABRICKS
using compromised supply chain components.
AI SECURITY
FRAMEWORK Model operations →
(DASF)
VERSION 1.0 22
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

ALGORITHMS 5.2

Model drift DASF 17 T

 rack training data with MLflow and Delta Lake to track
upstream data changes
Model drift in machine learning systems can
occur due to changes in feature data or target DASF 16 S
 ecure model features to track changes to features
Executive dependencies. This drift can be broadly DASF 21 M
 onitor data and AI system from a single pane of glass
Summary classified into three scenarios: for changes and take action when changes occur. Have
C
 oncept drift: where the statistical properties a feedback loop from a monitoring system and refresh
of the target variable change over time models over time to help avoid model staleness.
Introduction
D
 ata drift: involving changes in the
distribution of input data Applicable AI deployment model:

⟶ Risks in AI System U
 pstream data changes: occur due to Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Components alterations in data collection or processing Pre-trained LLMs: | Foundational models: | External models:

methods before the data reaches the model

Clever attackers can exploit these scenarios to

Understanding
evade an ML system for adversarial purposes.
Databricks Data
Intelligence Platform Model operations →
AI Risk Mitigation
Controls

ALGORITHMS 5.3
Conclusion
Hyperparameters stealing DASF 20 T
 rack ML training runs in the model development
process, including parameter settings, securely
Hyperparameters in machine learning are
Resources and DASF 43 U
 se access control lists via workspace access controls
often deemed confidential due to their
Further Reading
commercial value and role in proprietary DASF 42 D
 ata-centric MLOps and LLMOps employing separate
learning processes. If attackers gain access model lifecycle stages by UC schema
to these hyperparameters, they may steal or
Acknowledgments manipulate them — altering, concealing or even
adding hyperparameters. Such unauthorized Applicable AI deployment model:
interventions can harm the ML system, Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Appendix: compromising performance and reliability or
revealing sensitive algorithmic strategies.
Pre-trained LLMs: | Foundational models: | External models:
Glossary
Model operations →

License

ALGORITHMS 5.4

Malicious libraries DASF 53 T

 hird-party library control to limit the potential for
malicious third-party libraries and code to be used on
Attackers can upload malicious libraries to mission-critical workloads
public repositories that have the potential
to compromise systems, data and models.
Administrators should manage and restrict the Applicable AI deployment model:
installation and usage of third-party libraries, Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
safeguarding systems, pipelines and data. Pre-trained LLMs: | Foundational models: | External models:
This risk may also manifest in 2.2 Data Prep in
exploratory data analysis (EDA).

Model operations →

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 23
 2.6 Evaluation

Assessing the effectiveness of a machine learning system in achieving its intended

functionalities is a critical step in its development cycle. Post-learning evaluation utilizes
dedicated datasets to systematically analyze the performance of a trained model on its
specific task.

RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S
Executive
Summary

EVALUATION 6.1

Introduction S
 SO with IdP and MFA to limit who can access your data
Evaluation data poisoning DASF 1

and AI platform
Upstream attacks against data, where the data
is tampered with before it is used for machine DASF 2 S
 ync users and groups to inherit your organizational roles
⟶ Risks in AI System
learning, significantly complicate the training to access data
Components
and evaluation of ML models. Poisoning of the DASF 3 R
 estrict access using IP access lists to restrict the IP
evaluation data impacts the model validation addresses that can authenticate to your data and AI
and testing process. These attacks can corrupt platform
Understanding
or alter the data in a way that skews the training
Databricks Data R
 estrict access using private link as strong controls that
process, leading to unreliable models. DASF 4
Intelligence Platform limit the source for inbound requests
AI Risk Mitigation Model operations →
Controls DASF 5 C
 ontrol access to data and other objects for permissions
model across all data assets to protect data and sources
DASF 7 E
 nforce data quality checks on batch and streaming
Conclusion datasets for data sanity checks, and automatically detect
anomalies before they make it to the datasets
DASF 11 C
 apture and view data lineage to capture the lineage all
Resources and the way to the original raw data sources
Further Reading
DASF 45 E
 valuate models to capture performance insights for
language models
Acknowledgments DASF 44 T
 rigger actions in response to a specific event via
automated jobs to notify human-in-the-loop (HITL)
DASF 49 A
 utomate LLM evaluation
Appendix:
Glossary DASF 42 D
 ata-centric MLOps and LLMOps unit and integration
testing

License Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DASF
EVALUATION 6.2

Insufficient evaluation data DASF 22 B

 uild models with all representative, accurate and
relevant data sources to evaluate on clean and sufficient
Evaluation datasets can also be too small or too data
similar to the training data to be useful. Poor
evaluation data can lead to biases, hallucinations DASF 25 U
 se retrieval augmented generation (RAG) with large
and toxic output. It is difficult to effectively language models (LLMs)
evaluate large language models (LLMs), as these DASF 47 C
 ompare LLM outputs on set prompts to assess LLM
models rarely have an objective ground truth project with an interactive prompt interface
labeled. Consequently, organizations frequently
struggle to determine the trustworthiness of DASF 45 E
 valuate models to capture performance insights for
these models in critical, unsupervised use cases, language models
given the uncertainties in their evaluation.

Model operations → Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

DATABRICKS Pre-trained LLMs: | Foundational models: | External models:

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 24
 2.7 Machine Learning Models

A machine learning model is a program that can find patterns or make decisions from a
previously unseen dataset. During training, the machine learning algorithm is optimized to
find certain patterns or outputs from the dataset, depending on the task. The output of this
process — often a computer program with specific rules and data structures — is called a
Executive
Summary machine learning model.

Introduction Deploying a fully trained machine learning model to production introduces several critical
risks to address. Notably, some risks discussed in the previous section on evaluation risks,
⟶ Risks in AI System
such as overfitting, directly apply here. Open source or commercial models, not trained
Components
within your organization, carry the same risks with the added challenge that your organization
Understanding lacks control over the model’s development and training.
Databricks Data
Intelligence Platform
AI Risk Mitigation Additionally, external models may be Trojan horse backdoors or harboring other uncontrolled
Controls
risks, depriving you of the competitive advantage of leveraging your own data and potentially

Conclusion exposing your data to unauthorized access. Therefore, it is crucial to carefully consider and
mitigate these potential risks before deploying any pretrained model to production.
Resources and
Further Reading

RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Acknowledgments

MODEL 7.1
Appendix:
Glossary Backdoor machine learning/ DASF 1 S
 SO with IdP and MFA to limit who can access your data
and AI platform
Trojaned model
DASF 43 U
 se access control lists to limit who can bring models
There are inherent risks when using public ML/
License and limit the use of public models
LLM models or outsourcing their training, akin to
the dangers associated with executable (.exe) DASF 42 D
 ata-centric MLOps and LLMOps promote models as
files. A malicious third party handling the training code using CI/CD. Scan third-party models continuously
process could tamper with the data or deliver to identify hidden cybersecurity risks and threats such
a “Trojan model” that intentionally misclassifies as malware, vulnerabilities and integrity issues to detect
specific inputs. Additionally, open source models possible signs of malicious activity, including malware,
may contain hidden malicious code that can tampering and backdoors. See resources section for
exfiltrate sensitive data upon deployment. These third-party tools.
risks are pertinent in both external models and
DASF 23 R
 egister, version, approve, promote and deploy models
outsourced model development scenarios,
and scan models for malicious code when using third-
necessitating scrutiny and verification of models
party models or libraries
before use.
DASF 19 M
 anage end-to-end machine learning lifecycle
Model operations →
DASF 5 C
 ontrol access to data and other objects
DASF 34 R
 un models in multiple layers of isolation. Models are
considered untrusted code: deploy models and custom
LLMs with multiple layers of isolation.

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 25
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL 7.2

Model assets leak DASF 24 C

 ontrol access to models and model assets

Adversaries may target ML artifacts for DASF 1 S

 SO with IdP and MFA to limit who can access your data
exfiltration or as a basis for staging ML attacks. and AI platform
These artifacts encompass models, datasets and DASF 2 S
 ync users and groups to inherit your organizational roles
metadata generated during interactions with a to access data
model. Additionally, insiders risk leaking critical
model assets like notebooks, features, model DASF 3 R
 estrict access using IP access lists that can
Executive files, plots and metrics. Such leaks can expose authenticate to your data and AI platform
Summary trade secrets and sensitive organizational DASF 4 R
 estrict access using private link as strong controls that
information, underlining the need for stringent
limit the source for inbound requests
security measures to protect these valuable
assets. DASF 5 C
 ontrol access to data and other objects for permissions
Introduction
model across all data assets to protect data and sources
Model operations →
DASF 42 D
 ata-centric MLOps and LLMOps to maintain separate
⟶ Risks in AI System model lifecycle stages
Components DASF 33 M
 anage credentials securely to prevent credentials of
data sources used for model training from leaking
through models
Understanding
Databricks Data
Intelligence Platform Applicable AI deployment model:

AI Risk Mitigation Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Controls Pre-trained LLMs: | Foundational models: | External models:

Conclusion
MODEL 7.3

Resources and ML Supply chain vulnerabilities DASF 22 B

 uild models with all representative, accurate
Further Reading and relevant data sources to minimize third-party
Due to the extensive data, skills and dependencies for models and data where possible
computational resources required to train
machine learning algorithms, it’s common DASF 47 P
 retrain a large language model (LLM) on your own IP
Acknowledgments practice to reuse and slightly modify models DASF 48 U
 se hardened runtime for machine learning
developed by large corporations. For example,
ResNet, a popular image recognition model DASF 53 T
 hird-party library control
Appendix: from Microsoft, is often adapted for customer- DASF 42 D
 ata-centric MLOps and LLMOps promote models as
Glossary specific tasks. These models are curated
code using CI/CD. Scan third-party models continuously
in a Model Zoo (Caffe hosts popular image
to identify hidden cybersecurity risks and threats such
recognition models) or hosted by third-party
as malware, vulnerabilities and integrity issues to detect
License ML SaaS (OpenAI LLMs are an example). In this
possible signs of malicious activity, including malware,
attack, the adversary attacks the models hosted
tampering and backdoors. See resources section for
in Caffe, thereby poisoning the well for anyone
third-party tools.
else. Adversaries can also host specialized
models that will receive less scrutiny, akin to DASF 45 E
 valuate models and validate (aka, stress testing) to
watering hole attacks. verify reported function and disclosed weaknesses in
the models
Model operations →

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

MODEL 7.4

Source code control attack DASF 52 S

 ource code control to control and audit your knowledge
object integrity
The attacker might modify the source code used
in the ML algorithm, such as the random number DASF 53 T
 hird-party library control for third-party library
generator or any third-party libraries, which are integrity
often open source.

DATABRICKS Model operations → Applicable AI deployment model:

AI SECURITY Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

FRAMEWORK Pre-trained LLMs: | Foundational models: | External models:
(DASF)
VERSION 1.0 26
 2.8 Model Management

Responsible AI depends upon accountability. Accountability presupposes transparency. AI

transparency reflects the extent to which information about an AI system and its outputs is
available to individuals interacting with it — regardless of whether they are even aware that
Executive they are doing so.
Summary

Organizations can increase trust by creating a centralized place for model management:
Introduction
development, tracking, discovering, governing, encrypting and accessing models with proper

⟶ Risks in AI System
security controls. Doing so reduces the risk of model theft, improper reuse and model
Components
inversion. Transparency is also added by appropriate levels of information based on the
stage of the AI lifecycle and tailored to the role or knowledge of practitioners or individuals
Understanding
Databricks Data interacting with the AI system. By promoting higher levels of understanding, transparency
Intelligence Platform
AI Risk Mitigation increases confidence in the AI system.
Controls

Conclusion RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Resources and
MODEL MANAGEMENT 8.1
Further Reading

Model attribution DASF 5 C

 ontrol access to data and other objects for permissions
model across all data assets to protect data and sources
Acknowledgments Inadequate governance in machine learning,
including a lack of robust access controls, DASF 28 C
 reate model aliases, tags and annotations for
unclear model classification and insufficient documenting and discovering models

Appendix: documentation, can lead to the improper use or DASF 29 B

 uild MLOps workflows with human-in-the-loop (HITL) ,
Glossary sharing of models. This risk is particularly acute model stage management and approvals
when transferring models outside their designed
purpose. To mitigate these risks, groups that DASF 51 S
 hare data and AI assets securely
post models must provide precise descriptions
License of their intended use and document how they
Applicable AI deployment model:
address potential risks.
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Model operations →
Pre-trained LLMs: | Foundational models: | External models:

Companies need not sacrifice security for AI innovation. The Databricks AI Security
Framework is a comprehensive tool supporting the adoption of secure AI. We are grateful
for Databricks’ partnership in the journey to trustworthy AI and this tool makes AI security
practical and actionable for Databricks customers.
DATABRICKS
AI SECURITY
FRAMEWORK Robert Booker
(DASF) Chief Strategy Officer
VERSION 1.0 27
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL MANAGEMENT 8.2

Model theft DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Training machine learning systems, particularly
large language models, involves considerable DASF 2 S
 ync users and groups to inherit your organizational roles
Executive investment. A significant risk is the potential to access data
Summary theft of a system’s knowledge through DASF 3 R
 estrict access using IP access lists that can
direct observation of their input and output authenticate to your data and AI platform
observations, akin to reverse engineering. This
Introduction can lead to unauthorized access, copying or DASF 4 R
 estrict access using private link as strong controls that
exfiltration of proprietary models, resulting in limit the source for inbound requests
economic losses, eroded competitive advantage DASF 5 C
 ontrol access to data and other objects for permissions
and exposure of sensitive information.
⟶ Risks in AI System model across all data assets to protect data and sources
Components This attack can be as simple as attackers making DASF 24 C
 ontrol access to models and model assets
legitimate queries and analyzing the responses
to recreate a model. Once replicated, the model DASF 30 E
 ncrypt models
Understanding can be inverted, enabling the attackers to
DASF 31 S
 ecure model serving endpoints to prevent access and
Databricks Data extract feature information or infer details about
compute theft
Intelligence Platform the training data.
AI Risk Mitigation DASF 51 S
 hare data and AI assets securely
Model operations →
Controls DASF 32 S
 treamline the usage and management of various large
language model (LLM) providers and rate-limit APIs
DASF 33 M
 anage credentials securely to prevent credentials of
Conclusion
data sources used for model training from leaking through
models

Resources and
Further Reading Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

Acknowledgments

Appendix: MODEL MANAGEMENT 8.3

Glossary
Model lifecycle without HITL DASF 5 C
 ontrol access to data and other objects for permissions
model across all data assets to protect data and sources
(human-in-the-loop)
License DASF 24 C
 ontrol access to models and model assets
Lack of sufficient controls in a machine learning
and systems development lifecycle can result DASF 28 C
 reate model aliases, tags and annotations
in the unintended deployment of incorrect or
DASF 29 B
 uild MLOps workflows with human-in-the-loop (HILP)
unapproved models to production. Implementing
with permissions, versions and approvals to promote
model lifecycle tracking within an MLOps
models to production
framework is advisable to mitigate this risk.
This approach should include human oversight, DASF 42 D
 ata-centric MLOps and LLMOps promote models as
ensuring permissions, version control and code using CI/CD
proper approvals are in place before models
are promoted to production. Such measures
are crucial for maintaining ML system integrity, Applicable AI deployment model:

reliability and security. Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Model operations → Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 28
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL MANAGEMENT 8.4

Model inversion DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
In machine learning models, private assets like
training data, features and hyperparameters, DASF 2 S
 ync users and groups to inherit your organizational roles
Executive which are typically confidential, can potentially to access data
Summary be recovered by attackers through a process DASF 3 R
 estrict access using IP access lists that can
known as model inversion. This technique authenticate to your data and AI platform
involves reconstructing private elements
Introduction without direct access, compromising the DASF 4 R
 estrict access using private link as strong controls that
model’s security. Model inversion falls under the limit the source for inbound requests
“Functional Extraction” category in the MITRE DASF 5 C
 ontrol access to data and other objects for permissions
ATLAS framework, highlighting its relevance as a
⟶ Risks in AI System model across all data assets to protect data and sources
significant security threat.
Components
DASF 24 C
 ontrol access to models and model assets
Model operations →
DASF 30 E
 ncrypt models
Understanding DASF 31 S
 ecure model serving endpoints
Databricks Data
Intelligence Platform DASF 32 S
 treamline the usage and management of various large
AI Risk Mitigation language model (LLM) providers and rate-limit APIs
Controls

Applicable AI deployment model:

Conclusion
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

Resources and
Further Reading

Acknowledgments
2.9 Model Serving and Inference Requests

Appendix: Model Serving exposes your machine learning models as scalable REST API endpoints for
Glossary
inference and provides a highly available and low-latency service for deploying models.

License
Deploying a fully trained machine learning model introduces significant risks, including
adversarial inputs, data poisoning, privacy concerns, access control issues, model
vulnerabilities and versioning challenges. Using third-party or SaaS models amplifies
these risks and introduces further limitations like lack of customization, model mismatch,
ownership concerns and data privacy risks. Careful evaluation and mitigation strategies are
necessary to securely and responsibly deploy fully trained models in production.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 29
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.1

Prompt inject DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
A direct prompt injection occurs when a user
injects text that is intended to alter the behavior DASF 2 S
 ync users and groups to inherit your organizational roles
of the LLM. Malicious input, known as model to access data
Executive
evasion in the MITRE ATLAS framework, is a DASF 3 R
 estrict access using IP access lists that can
Summary
significant threat to machine learning systems. authenticate to your data and AI platform
These risks manifest as “adversarial examples”:
inputs deliberately designed to deceive models. DASF 4 R
 estrict access using private link as strong controls that
Introduction Attackers use direct prompt injections to bypass limit the source for inbound requests
safeguards in order to create misinformation DASF 5 C
 ontrol access to data and other objects for permissions
and cause reputational damage. Attackers may
model across all data assets to protect data and sources
⟶ Risks in AI System wish to extract the system prompt or reveal
Components private information provided to the model in the DASF 24 C
 ontrol access to models and model assets
context but not intended for unfiltered access
DASF 46 S
 tore and retrieve embeddings securely to integrate
by the user. Large language model (LLM) plug-ins
data objects for security-sensitive data that goes into
are particularly vulnerable, as they are typically
Understanding LLMs as RAG inputs
required to handle untrusted input and it is
Databricks Data
difficult to apply adequate application control. DASF 30 E
 ncrypt models
Intelligence Platform
Attackers can exploit such vulnerabilities, with
AI Risk Mitigation DASF 31 S
 ecure model serving endpoints
severe potential outcomes including remote
Controls
code execution. DASF 32 S
 treamline the usage and management of various large
Model deployment and serving → language model (LLM) providers and rate-limit inference
queries allowed by the model.
Conclusion
Designing robust prompts can help mitigate attacks such
as jailbreaking.
Resources and Implement gates between users/callers and the actual
Further Reading model by performing input validation post-processing on
all proposed queries, rejecting anything not meeting the
model’s definition of input correctness, and returning only
Acknowledgments the minimum amount of information needed to be useful.
DASF 37 S
 et up inference tables for monitoring and debugging
prompts
Appendix:
Glossary
Additional controls to consider:
Robust Intelligence AI Firewall Prompt Injection rule: Flags
License malicious user input that might direct the LLM to perform an
action unintended by the model creator.

HiddenLayer AISec SafeLLM Proxy.

Please see the resources section for a collection of

third-party tools.

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 30
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.2

Model inversion DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Malicious actors can recover the private
assets used in machine learning models, DASF 2 S
 ync users and groups to inherit your organizational roles
known as functional extraction in the MITRE to access data
Executive
ATLAS framework. This process includes DASF 3 R
 estrict access using IP access lists that can
Summary
reconstructing private training data, features authenticate to your data and AI platform
and hyperparameters the attacker cannot
otherwise access. The attacker can also recover DASF 4 R
 estrict access using private link as strong controls that
Introduction a functionally equivalent model by iteratively limit the source for inbound requests
querying the model. DASF 5 C
 ontrol access to data and other objects for permissions
Model deployment and serving → model across all data assets to protect data and sources
⟶ Risks in AI System
Components DASF 24 C
 ontrol access to models and model assets
DASF 46 S
 tore and retrieve embeddings securely to integrate
data objects for security-sensitive data that goes into
Understanding LLMs as RAG inputs
Databricks Data
DASF 30 E
 ncrypt models
Intelligence Platform
AI Risk Mitigation DASF 31 S
 ecure model serving endpoints
Controls
DASF 32 S
 treamline the usage and management of various large
language model (LLM) providers and rate-limit inference
queries allowed by the model.
Conclusion
Designing robust prompts can help mitigate attacks such
as jailbreaking.
Resources and Implement gates between users/callers and the actual
Further Reading model by performing input validation post-processing on
all proposed queries, rejecting anything not meeting the
model’s definition of input correctness, and returning only
Acknowledgments the minimum amount of information needed to be useful.

Open source and commercial solutions provide a variety

of modules including prompt and output scanners for
Appendix: various responsible AI or jailbreaking attacks.
Glossary
DASF 37 S
 et up inference tables for monitoring and debugging
model prompts

License
Additional controls to consider:
Robust Intelligence AI Firewall Prompt Injection rule: Flags
malicious user input that might direct the LLM to perform an
action unintended by the model creator.

Please see the resources section for a collection of

third-party tools.

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 31
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.3

Model breakout DASF 34 R

 un models in multiple layers of isolation with
unprivileged VMs and network segregation. Protects
Malicious users can exploit adversarial examples back-end internal systems from LLM access. The most
to mislead machine learning systems, including reliable mitigation is to always treat all LLM output as
large language models (LLMs). These specially potentially malicious and remember that an untrusted
Executive
crafted inputs aim to disrupt the normal entity has been able to inject text as user input. All LLM
Summary
functioning of these systems, leading to several output should be inspected and sanitized before being
potential hazards. An attacker might use these further parsed to extract information related to the plug-
examples to force the system to deviate from its in. Plug-in templates should be parameterized wherever
Introduction intended environment, exfiltrate sensitive data possible, and any calls to external services must be
or interact inappropriately with other systems. strictly parameterized at all times and made in a least-
Additionally, adversarial inputs can cause false privileged context.
⟶ Risks in AI System predictions, leak sensitive information from the
training data, or manipulate the system into DASF 37 S
 et up inference tables for monitoring and debugging
Components
executing unintended actions on internal and model prompts
external systems.

Understanding Model deployment and serving →

Applicable AI deployment model:
Databricks Data
Intelligence Platform Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

AI Risk Mitigation Pre-trained LLMs: | Foundational models: | External models:

Controls

Conclusion MODEL SERVING — INFERENCE REQUESTS 9.4

Looped input DASF 37 S

 et up inference tables for monitoring and debugging
models to capture incoming requests and outgoing
Resources and There is a notable risk in machine learning responses to your model serving endpoint and
Further Reading systems when the output produced by the automatically log them in tables. Afterward, you can use
system is reintroduced into the real world and the data in this table to monitor, debug and improve ML
subsequently cycles back as input, creating models and decide if these inferences are of sufficient
Acknowledgments a harmful feedback loop. This can reinforce quality for input to model training.
removing security filters, biases or errors,
potentially leading to increasingly skewed or
Appendix: inaccurate model performance and unintended
Applicable AI deployment model:
Glossary system behaviors.
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Model deployment and serving → Pre-trained LLMs: | Foundational models: | External models:

License

As organizations strive to incorporate machine learning and generative AI capabilities, a

meticulous approach to security and governance throughout the AI lifecycle is essential.
The Databricks AI Security Framework stands as a guiding light, providing actionable
control recommendations and fostering collaboration among diverse AI teams. In the
dynamic landscape of AI, this framework serves as a comprehensive guide, addressing
security risks at every stage of the AI/ML lifecycle, ensuring responsible, secure and
compliant integration for the organization.

DATABRICKS
AI SECURITY Hasan Yasar
Carnegie Mellon
FRAMEWORK Technical Director, Teaching Professor
(DASF) University Continuous Deployment of Capability | Software Engineering Institute
VERSION 1.0 32
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.5

Infer training data membership DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Adversaries may pose a significant privacy threat
to machine learning systems by simulating or DASF 2 S
 ync users and groups to inherit your organizational roles
Executive inferring whether specific data samples were to access data
Summary part of a model’s training set. Such inferences DASF 3 R
 estrict access using IP access lists that can
can be made by: authenticate to your data and AI platform
Using techniques like Train Proxy via DASF 4 R
 estrict access using private link as strong controls that
Introduction
Replication to create and host shadow limit the source for inbound requests
models replicating the target model’s
behavior DASF 5 C
 ontrol access to data and other objects for permissions
⟶ Risks in AI System model across all data assets to protect data and sources
Components Analyzing the statistical patterns in the
model’s prediction scores to conclude the DASF 24 C
 ontrol access to models and model assets
training data DASF 28 C
 reate model aliases, tags and annotations
Understanding These methods can lead to the unintended DASF 46 S
 tore and retrieve embeddings securely to integrate
Databricks Data leakage of sensitive information, such as
data objects for security-sensitive data that goes into
Intelligence Platform individuals’ personally identifiable information
LLMs as RAG inputs
AI Risk Mitigation (PII) in the training dataset or other forms of
Controls protected intellectual property. DASF 30 E
 ncrypt models

Model deployment and serving → DASF 31 S

 ecure model serving endpoints
DASF 32 S
 treamline the usage and management of various large
Conclusion
language model (LLM) providers and rate-limit inference
queries allowed by the model.

Resources and Designing robust prompts can help mitigate attacks such
Further Reading as jailbreaking.

Implement gates between users/callers and the actual

model by performing input validation post-processing on
Acknowledgments all proposed queries, rejecting anything not meeting the
model’s definition of input correctness, and returning only
the minimum amount of information needed to be useful.
Appendix: DASF 37 S
 et up inference tables for monitoring and debugging
Glossary prompts

DASF 45 E
 valuate models for custom evaluation metrics
License

Additional controls to consider:

Robust Intelligence AI Firewall Prompt Injection rule: Flags
malicious user input that might direct the LLM to perform an
action unintended by the model creator.

Robust Intelligence AI Firewall PII Detection rule: Flags user input

and model output suspected of containing PII.

The HiddenLayer AISec Platform, specifically MLDR, monitors

inputs and related outputs to ML models to determine if an
adversary is attempting an inference
with a malicious intent.

Please see the resources section for a collection of

third-party tools.

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 33
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.6

Discover ML model ontology DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Adversaries may aim to uncover the ontology of
a machine learning model’s output space, such DASF 2 S
 ync users and groups to inherit your organizational roles
as identifying the range of objects or responses to access data
Executive
the model is designed to detect. This can be DASF 3 R
 estrict access using IP access lists that can
Summary
achieved through repeated queries to the model, authenticate to your data and AI platform
which may force it to reveal its classification
system or by accessing its configuration files DASF 4 R
 estrict access using private link as strong controls that
Introduction or documentation. Understanding a model’s limit the source for inbound requests
ontology allows adversaries to gain insights in DASF 5 C
 ontrol access to data and other objects for permissions
designing targeted attacks that exploit specific
model across all data assets to protect data and sources
⟶ Risks in AI System vulnerabilities or characteristics.
Components DASF 24 C
 ontrol access to models and model assets
Model deployment and serving →
DASF 28 C
 reate model aliases, tags and annotations
DASF 46 S
 tore and retrieve embeddings securely to integrate
Understanding
data objects for security-sensitive data that goes into
Databricks Data
LLMs as RAG inputs
Intelligence Platform
AI Risk Mitigation DASF 30 E
 ncrypt models
Controls
DASF 31 S
 ecure model serving endpoints
DASF 32 S
 treamline the usage and management of various large
Conclusion language model (LLM) providers and rate-limit inference
queries allowed by the model.

Designing robust prompts can help mitigate attacks such

Resources and as jailbreaking.
Further Reading
Implement gates between users/callers and the actual
model by performing input validation post-processing on
all proposed queries, rejecting anything not meeting the
Acknowledgments model’s definition of input correctness, and returning only
the minimum amount of information needed to be useful.

Open source and commercial solutions provide a variety

Appendix:
of modules including prompt and output scanners for
Glossary
various responsible AI or jailbreaking attacks.
DASF 37 S
 et up inference tables for monitoring and debugging
License model prompts

DASF 45 E
 valuate models for custom evaluation metrics

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 34
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.7

Denial of service (DoS) DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Adversaries may target machine learning
systems with a flood of requests to degrade DASF 2 S
 ync users and groups to inherit your organizational roles
or shut down the service. Since many machine to access data
Executive
learning systems require significant amounts of DASF 3 R
 estrict access using IP access lists that can
Summary
specialized compute, they are often expensive authenticate to your data and AI platform
bottlenecks that can become overloaded.
Adversaries can intentionally craft inputs that DASF 4 R
 estrict access using private link as strong controls that
Introduction require heavy amounts of useless compute from limit the source for inbound requests
the machine learning system. DASF 5 C
 ontrol access to data and other objects for permissions
Model deployment and serving → model across all data assets to protect data and sources
⟶ Risks in AI System
Components DASF 24 C
 ontrol access to models and model assets
DASF 46 S
 tore and retrieve embeddings securely to integrate
data objects for security-sensitive data that goes into
Understanding LLMs as RAG inputs
Databricks Data
DASF 30 E
 ncrypt models
Intelligence Platform
AI Risk Mitigation DASF 31 S
 ecure model serving endpoints
Controls
DASF 32 S
 treamline the usage and management of various large
language model (LLM) providers and rate-limit inference
queries allowed by the model.
Conclusion
Designing robust prompts can help mitigate attacks such
as jailbreaking.
Resources and Implement gates between users/callers and the actual
Further Reading model by performing input validation post-processing on
all proposed queries, rejecting anything not meeting the
model’s definition of input correctness, and returning only
Acknowledgments the minimum amount of information needed to be useful.
DASF 37 S
 et up inference tables for monitoring and debugging
prompts
Appendix:
Glossary
Additional controls to consider:
Robust Intelligence AI Firewall Prompt Injection rule: Flags
License malicious user input that might direct the LLM to perform an
action unintended by the model creator.

Please see the resources section for a collection of

third-party tools.

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 35
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.8

LLM hallucinations DASF 25 U

 se retrieval augmented generation (RAG) with large
language models (LLMs)
Large language models (LLMs) are known to
inadvertently generate incorrect, misleading and/or
or factually false outputs, or leak sensitive DASF 26 F
 ine-tune large language models (LLMs) on highly
data. This situation may arise when training relevant, contextual data to reduce the risks of LLMs by
models on datasets containing potential biases grounding with the domain-specific data
in their training data, limitations in contextual
understanding or confidential information. DASF 27 P
 retrain a large language model (LLM) on highly relevant,
Executive Model deployment and serving → contextual data to reduce the risks of LLMs by grounding
Summary with the domain-specific data. The LLMs will investigate
that data for giving the responses.

Introduction DASF 46 C
 reate embeddings to securely integrate data objects
with sensitive data that goes into LLMs

⟶ Risks in AI System
DASF 49 A
 utomate LLM evaluation to evaluate RAG applications
with LLM-as-a-judge and get out-of-the-box metrics
Components
like toxicity, latency, tokens and more to quickly and
efficiently compare and contrast various LLMs to navigate
your RAG application requirements
Understanding
Databricks Data
Intelligence Platform
Additional controls to consider:
AI Risk Mitigation
Use guardrails to define and enforce assurance for LLM
Controls
applications. Please see the resources section for a collection of
third-party tools.

Conclusion
Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Resources and Pre-trained LLMs: | Foundational models: | External models:

Further Reading

Acknowledgments MODEL SERVING — INFERENCE REQUESTS 9.9

Input resource control DASF 1 S

 SO with IdP and MFA to limit who can access your data
Appendix: and AI platform
The attacker might modify or exfiltrate resources
Glossary DASF 2 S
 ync users and groups to inherit your organizational roles
(e.g., documents, web pages) that will be
ingested by the GenAI model at runtime via the to access data
RAG process. This capability is used for indirect DASF 3 R
 estrict access using IP access lists that can
License prompt injection attacks. For example, rows authenticate to your data and AI platform
from a database or text from a PDF document
that are intended to be summarized generically DASF 4 R
 estrict access using private link as strong controls that
by the LLM can be extracted by simply asking for limit the source for inbound requests
them via direct prompt injection. DASF 5 C
 ontrol access to data and other objects for permissions
Model deployment and serving → model across all data assets to protect data and sources
that are used for RAG
DASF 46 S
 tore and retrieve embeddings securely to integrate
data objects for security-sensitive data that goes into
LLMs as RAG inputs

Additional controls to consider:

Robust Intelligence AI Firewall Prompt Injection rule:
Flags malicious user input that might direct the LLM to perform an
action unintended by the model creator.

Robust Intelligence AI Firewall PII Detection rule: Flags user input

and model output suspected of containing PII.

Please see the resources section for a collection of

third-party tools.

DATABRICKS Applicable AI deployment model:

AI SECURITY Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

FRAMEWORK Pre-trained LLMs: | Foundational models: | External models:
(DASF)
VERSION 1.0 36
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE REQUESTS 9.10

Accidental exposure of DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
unauthorized data to models
DASF 2 S
 ync users and groups to inherit your organizational roles
In GenAI, large language models (LLMs) are also
to access data
becoming an integral part of the infrastructure
and software applications. LLMs are being used DASF 3 R
 estrict access using IP access lists that can
Executive to create more powerful online search, help authenticate to your data and AI platform
Summary software developers write code, and even power
DASF 4 R
 estrict access using private link as strong controls that
chatbots that help with customer service. LLMs
limit the source for inbound requests
are being integrated with corporate databases
Introduction and documents to enable powerful retrieval DASF 5 C
 ontrol access to data and other objects for permissions
augmented generation (RAG) scenarios when model across all data assets to protect data and sources
LLMs are adapted to specific domains and use that are used for RAG
⟶ Risks in AI System cases. For example: rows from a database or
text from a PDF document that are intended to
DASF 16 S
 ecure model features to reduce the risk of malicious
Components actors manipulating the features that feed into ML
be summarized generically by the LLM. These
scenarios in effect expose a new attack surface training
to potentially confidential and proprietary DASF 46 S
 tore and retrieve embeddings securely to integrate
Understanding enterprise data that is not sufficiently secured data objects for security-sensitive data that goes into
Databricks Data or overprivileged, which can lead to use of LLMs as RAG inputs
Intelligence Platform unauthorized data as an input source to models.
AI Risk Mitigation A similar risk exists for tabular data models that
Controls rely upon lookups to feature store tables at Applicable AI deployment model:
inference time. Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Model deployment and serving → Pre-trained LLMs: | Foundational models: | External models:
Conclusion

Resources and
Further Reading
2.10 Model Serving and Inference Response
Acknowledgments

While the technical intricacies of the algorithm may seem like the most vulnerable point
Appendix:
Glossary for malicious actors seeking to compromise the integrity and reliability of the ML system,
an equally effective, and often overlooked, attack vector lies in how it generates output
License (inference response). The inference response represents the real-world manifestation of
the model’s learned knowledge and forms the basis for its decision-making capabilities.
Consequently, compromising the inference response directly can have devastating
consequences, undermining the system’s integrity and reliability.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 37
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE RESPONSE 10.1

Lack of audit and monitoring DASF 35 T

 rack model performance to evaluate quality

inference quality DASF 36 S

 et up monitoring alerts

Effectively audit, track and assess the DASF 37 S

 et up inference tables for monitoring and debugging
Executive
performance of machine learning models by models to capture incoming requests and outgoing
Summary
monitoring inference tables to gain valuable responses to your model serving endpoint and log them
insights into the model’s decision-making in a table. Afterward, you can use the data in this table
process and identify any discrepancies or to monitor, debug and improve ML models and decide if
Introduction anomalies. these inferences are of quality to use as input to model
training.
These tables should include the model’s user
or system making the request, inputs, and
⟶ Risks in AI System
the corresponding predictions or outputs. Applicable AI deployment model:
Components
Monitoring the model serving endpoints provides
real-time audit in operational settings.
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

Understanding Model deployment and serving →

Databricks Data
Intelligence Platform
AI Risk Mitigation
Controls MODEL SERVING — INFERENCE RESPONSE 10.2

Output manipulation DASF 30 E

 ncrypt models for model endpoints with encryption
Conclusion in transit
An attacker can compromise a machine learning
system by tweaking its output stream, also DASF 31 S
 ecure model serving endpoints
known as a man-in-the-middle attack. This is DASF 32 S
 treamline the usage and management of various large
Resources and achieved by intercepting the data transmission language model (LLM) providers to rate-limit inference
Further Reading between the model’s endpoint, which generates queries allowed by the model. Then audit, reproduce and
its predictions or outputs, and the intended make your models more compliant.
receiver of this information. Such an attack
Acknowledgments poses a severe security threat, allowing the
attacker to read or alter the communicated Applicable AI deployment model:
results, potentially leading to data leakage, Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
misinformation or misguided actions based on
Appendix:
manipulated data.
Pre-trained LLMs: | Foundational models: | External models:
Glossary
Model deployment and serving →

License

The DASF is the first-ever framework that would allow businesses to mitigate AI/ML
risks at scale versus approaches that operate in silos — collectivism at best for
responsible AI/ML.
DATABRICKS
AI SECURITY
FRAMEWORK Ebrima N. Ceesay, PhD, CISSP
Capital One Financial
(DASF) Senior Distinguished Engineer
VERSION 1.0 38
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE RESPONSE 10.3

Discover ML model ontology DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Adversaries may aim to uncover the ontology of
a machine learning model’s output space, such DASF 2 S
 ync users and groups to inherit your organizational roles
Executive as identifying the range of objects or responses to access data to restrict IP addresses
Summary the model is designed to detect. This can be DASF 3 IP access lists to restrict the IP addresses that can
achieved through repeated queries to the model, authenticate to Databricks
which may force it to reveal its classification
Introduction system or by accessing its configuration files DASF 4 R
 estrict access using private link as strong controls that
or documentation. Understanding a model’s limit the source for inbound requests
ontology allows adversaries to gain insights in DASF 5 U
 nity Catalog privileges and securable objects for
designing targeted attacks that exploit specific
⟶ Risks in AI System permissions model across all data assets to protect data
vulnerabilities or characteristics.
Components and sources
Model deployment and serving → DASF 24 P
 rotect model assets, lifecycle and security with UC in
MLflow Model Registry
Understanding
Databricks Data DASF 28 C
 reate and model aliases, tags and annotations in Unity
Intelligence Platform Catalog for documenting and discovering models
AI Risk Mitigation DASF 30 E
 ncrypt models
Controls
DASF 31 S
 ecure serving endpoint with Model Serving
DASF 32 S
 treamline the usage and management of various large
Conclusion language model (LLM) providers and rate-limit inference
queries allowed by the model.

The most reliable mitigation is to always treat all LLM

Resources and productions as potentially malicious and under the
Further Reading control of any entity that has been able to inject text into
the LLM user’s input.

Implement gates between users/callers and the actual

Acknowledgments
model by performing input validation on all proposed
queries, rejecting anything not meeting the model’s
definition of input correctness, and returning only the
Appendix: minimum amount of information needed to be useful.
Glossary
DASF 37 S
 et up inference tables for monitoring and debugging
models

License

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

The Databricks AI Security Framework provides a comprehensive set of actionable

guidelines to help secure our data and AI ecosystem end to end.
DATABRICKS
AI SECURITY
FRAMEWORK Grizel Lopez
(DASF) Senior Director of Engineering
VERSION 1.0 39
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

MODEL SERVING — INFERENCE RESPONSE 10.4

Discover ML model family DASF 1 S

 SO with IdP and MFA to limit who can access your data
and AI platform
Adversaries targeting machine learning systems
may strive to identify the general family or type DASF 2 S
 ync users and groups to inherit your organizational roles
of the model in use. Attackers can obtain this to access data
information from documentation that describes DASF 3 R
 estrict access using IP access lists that can
the model or through analyzing responses from authenticate to your data and AI platform
Executive carefully constructed inputs. Knowledge of the
Summary model’s family is crucial for crafting attacks DASF 4 R
 estrict access using private link as strong controls that
tailored to exploit the identified weaknesses of limit the source for inbound requests
the model. DASF 5 C
 ontrol access to data and other objects for permissions
Introduction Model deployment and serving → model across all data assets to protect data and sources
DASF 24 C
 ontrol access to models and model assets

⟶ Risks in AI System DASF 28 C

 reate model aliases, tags and annotations
Components DASF 46 S
 tore and retrieve embeddings securely to integrate
data objects to integrate data data that goes into LLMs as
RAG inputs
Understanding
DASF 30 E
 ncrypt models
Databricks Data
Intelligence Platform DASF 31 S
 ecure model serving endpoints
AI Risk Mitigation
DASF 32 S
 treamline the usage and management of various large
Controls
language model (LLM) providers and rate-limit inference
queries allowed by the model.

Conclusion Designing robust prompts can help mitigate attacks such

as jailbreaking.

Implement gates between users/callers and the actual

Resources and model by performing input validation post-processing on
Further Reading all proposed queries, rejecting anything not meeting the
model’s definition of input correctness, and returning only
the minimum amount of information needed to be useful.
Acknowledgments Open source and commercial solutions provide a variety
of modules including prompt and output scanners for
various responsible AI or jailbreaking attacks.
Appendix:
DASF 37 S
 et up inference tables for monitoring and debugging
Glossary
models

DASF 45 Evaluate models for custom evaluation metrics

License

Applicable AI deployment model:

Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

Pre-trained LLMs: | Foundational models: | External models:

MODEL SERVING — INFERENCE RESPONSE 10.5

Black-box attacks DASF 30 E

 ncrypt models for model endpoints with encryption
in transit
Public or compromised private model serving
connectors (e.g., API interfaces) are vulnerable DASF 31 S
 ecure model serving endpoints
to black-box attacks. Although black-box DASF 32 S
 treamline the usage and management of various large
attacks generally require more trial-and-error language model (LLM) providers to rate-limit inference
attempts (inferences), they are notable for queries allowed by the model. Then audit, reproduce and
requiring significantly less access to the target make your models more compliant.
system. Successful black-box attacks quickly
erode trust in enterprises serving the model
connectors. Applicable AI deployment model:

Model deployment and serving → Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

DATABRICKS Pre-trained LLMs: | Foundational models: | External models:

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 40
 2.11 Machine Learning Operations (MLOps)

MLOps is a useful approach for creating quality AI solutions. It is a core function of machine
learning engineering, focused on streamlining the process of taking machine learning
models to production and then maintaining and monitoring them. By adopting an MLOps
Executive approach, data scientists and machine learning engineers can collaborate and increase the
Summary
pace of model development and production by implementing continuous integration and

Introduction continuous deployment (CI/CD) practices with proper monitoring, validation and governance
of ML models with a “security in the process” mindset. Organizations without MLOps will risk
⟶ Risks in AI System
missing some of the controls we discussed above or not applying them consistently at scale
Components
to manage thousands of models.
Understanding
Databricks Data
Intelligence Platform
RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S
AI Risk Mitigation
Controls

OPERATIONS 11.1
Conclusion
Lack of MLOps — repeatable DASF 45 E
 valuate models to capture performance insights for
language models
enforced standards
Resources and DASF 44 T
 rigger actions in response to a specific event to trigger
Further Reading Operationalizing an ML solution requires joining
automated jobs to keep human-in-the-loop (HITL)
data from predictions, monitoring and feature
tables with other relevant data. DASF 42 D
 ata-centric MLOps and LLMOps. MLOps best practices:
separate environments by workspace and schema,
Acknowledgments Duplicating data, moving AI assets, and driving
promote models with code, MLOps Stacks for repeatable
governance and tracking across these stages
ML infra across environments.
may represent roadblocks to practitioners
who would rather shortcut security controls to
Appendix:
deliver their solution. Many organizations will Applicable AI deployment model:
Glossary
find that the simplest way to securely combine
ML solutions, input data and feature tables is to
Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

leverage the same platform that manages other

Pre-trained LLMs: | Foundational models: | External models:

License production data.

An ML solution comprises data, code and

models. These assets must be developed,
tested (staging) and deployed (production). For
each of these stages, we also need to operate
within an execution environment. Security is
an essential component of all MLOps lifecycle
stages. It ensures the complete lifecycle meets
the required standards by keeping the distinct
execution environments — development, staging
and production.

Operations and platform →

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 41
 2.12 Data and AI Platform Security

Abundant real-world evidence suggests that actual attackers use simple tactics to subvert
ML-driven systems. The choice of platform used for building and deploying AI models can
have inherent risks and rewards.
Executive
Summary

RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

Introduction

PLATFORM 12.1
⟶ Risks in AI System
Lack of vulnerability management DASF 38 P
 latform security — vulnerability management to build,
Components
deploy and monitor AI/ML models on a platform that
Detecting and promptly addressing software takes responsibility seriously and shares remediation
vulnerabilities in systems that support data and timeline commitments
Understanding AI/ML operations is a critical responsibility for
Databricks Data software and service providers. Attackers do
Intelligence Platform not necessarily need to target AI/ML algorithms Applicable AI deployment model:
AI Risk Mitigation directly; compromising the layers underlying AI/ Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Controls ML systems is often easier. Therefore, adhering Pre-trained LLMs: | Foundational models: | External models:
to traditional security threat mitigation practices,
such as a secure software development lifecycle,
Conclusion is essential across all software layers.

Operations and platform →

Resources and
Further Reading
PLATFORM 12.2

Acknowledgments Lack of penetration testing DASF 39 P

 latform security — penetration testing and bug bounty
to build, deploy and monitor AI/ML models on a platform
and bug bounty that takes responsibility seriously and shares remediation
Penetration testing and bug bounty programs timeline commitments. A bug bounty program removes a
Appendix:
are vital in securing software that supports data barrier researchers face in working with Databricks.
Glossary
and AI/ML operations. Unlike in direct attacks
on AI/ML algorithms, adversaries often target
Applicable AI deployment model:
underlying software risks, such as the OWASP
License | |
Top 10. These foundational software layers are Predictive ML models: RAG-LLMs: Fine-tuned LLMs:

generally more prone to attacks than the AI/ML Pre-trained LLMs: | Foundational models: | External models:
components.

Penetration testing involves skilled experts

actively seeking and exploiting weaknesses,
mimicking real attack scenarios. Bug bounty
programs encourage external ethical hackers to
find and report vulnerabilities, rewarding them
for their discoveries. This combination of internal
and external security testing enhances overall
system protection, safeguarding the integrity of
AI/ML infrastructures against cyberthreats.

Operations and platform →

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 42
 RISK/DESCRIPTION M I T I G AT I O N C O N T R O L S

PLATFORM 12.3

Lack of incident response DASF 39 P

 latform security — Incident Response Team

AI/ML applications are mission-critical for business.

Your chosen platform vendor must address security Applicable AI deployment model:
issues in machine learning operations quickly and Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
effectively. The program should combine automated Pre-trained LLMs: | Foundational models: | External models:
monitoring with manual analysis to address general
and ML-specific threats.

Operations and platform →

Executive
Summary
PLATFORM 12.4

Unauthorized privileged access DASF 40 P

 latform security — Internal access
Introduction
A significant security threat in machine learning
platforms arises from malicious internal actors, such Applicable AI deployment model:
as employees or contractors. These individuals might Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
⟶ Risks in AI System
gain unauthorized access to private training data or
Components Pre-trained LLMs: | Foundational models: | External models:
ML models, posing a grave risk to the integrity and
confidentiality of the assets. Such unauthorized access
can lead to data breaches, leakage of sensitive or
Understanding proprietary information, business process abuses, and
Databricks Data potential sabotage of the ML systems. Implementing
Intelligence Platform stringent internal security measures and monitoring
AI Risk Mitigation protocols is critical to mitigate insider risks from the
Controls platform vendor.

Operations and platform →

Conclusion

PLATFORM 12.5
Resources and
Further Reading
Poor security in the software DASF 41 P
 latform security — secure SDLC

development lifecycle
Acknowledgments Software platform security is an important part of Applicable AI deployment model:

any progressive security program. ML hackers have Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:

shown that they don’t need to know sophisticated AI/ Pre-trained LLMs: | Foundational models: | External models:
Appendix: ML concepts to compromise a system. Hackers have
Glossary busied themselves with exposing and exploiting bugs
in a platform where AI is built, as those systems are
well known to them. The security of AI depends on the
License platform’s security.

Operations and platform →

PLATFORM 12.6

Lack of compliance DASF 50 P

 latform compliance to build on a
compliant platform
As AI applications become prevalent, they are
increasingly subject to scrutiny and regulations, such
as the General Data Protection Regulation (GDPR) Applicable AI deployment model:
in the European Union and the California Consumer Predictive ML models: | RAG-LLMs: | Fine-tuned LLMs:
Privacy Act (CCPA) in the United States. Navigating Pre-trained LLMs: | Foundational models: | External models:
these regulations can be complex, particularly
regarding data privacy and user rights. Utilizing a
compliance-certified platform can be a significant
advantage for organizations. These platforms are
specifically designed to meet regulatory standards,
providing essential tools and resources to help
organizations build and deploy AI applications that
are compliant with these laws. By leveraging such
platforms, organizations can more effectively address
regulatory compliance challenges, ensuring their
DATABRICKS
AI initiatives align with legal requirements and best
AI SECURITY
practices for data protection.
FRAMEWORK
(DASF) Operations and platform →
VERSION 1.0 43
 03
Understanding Databricks Data Intelligence Platform
AI Risk Mitigation Controls

In this section, we delve into the comprehensive risk mitigation controls available in the
Databricks Data Intelligence Platform for Artificial Intelligence (AI) and Machine Learning
Executive
Summary (ML). As organizations increasingly harness the power of AI, a nuanced understanding of
these robust controls becomes imperative to ensure data integrity, security and regulatory
Introduction compliance throughout the data lifecycle.

Risks in AI System
Components
3.1 The Databricks Data Intelligence Platform

⟶ Understanding
Databricks Data Databricks is the data and AI company with origins in academia and the open source
Intelligence Platform
AI Risk Mitigation community. Databricks was founded in 2013 by the original creators of Apache Spark™, Delta
Controls
Lake and MLflow. We pioneered the concept of the lakehouse to combine and unify the best of

Conclusion
data warehouses and data lakes. Databricks made this vision a reality in 2020; since then, it has
seen tremendous adoption as a category. Today, 74% of global CIOs report having a lakehouse
Resources and in their estate, and almost all of the remainder intend to have one within the next three years.
Further Reading

In November 2023, we announced the Databricks Data Intelligence Platform. It’s built on a
Acknowledgments
lakehouse to provide an open, unified foundation for all data and governance. We built the

Appendix:
Data Intelligence Platform to allow every employee in every organization to find success
Glossary
with data and AI. The Data Intelligence Engine, at the heart of the platform, understands
the semantics of your data and how it flows across all of your workloads. This allows for
License
new methods of optimization, as well as for technical and nontechnical users to use natural
language to discover and use data and AI in the context of your business.

MOSAIC AI DATABRICKS DATABRICKS DATABRICKS

UNITY CATALOG PLATFORM ARCHITECTURE PLATFORM SECURITY

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 44
 Executive
Summary

Introduction
Data Intelligence Platform
Risks in AI System
Mlsaic AI Delta Liše ŠaŽlew Wlrkflow DataŽricks ­§ª
Create, tune and Automate data Optimize job cost Build with
Components serve custom LLMs quality based on past runs text-to-SQL

⟶ Understanding
Data ntellience Xnine
Databricks Data Use nenerative Ad to understand the semantics o your data
Intelligence Platform
AI Risk Mitigation
Controls
Unity Catallg
Unifed security, novernance and cataloninn

Delta Lake
Conclusion

Unifed data storane or reliability and sharinn

Resources and
Further Reading
Open Data Lake
All raw data álons, texts, audio, video, imanesò
Acknowledgments

Appendix:
Glossary

License

The Databricks Data Intelligence Platform combines AI assets — from data and features to
models — into one catalog, ensuring full visibility and fine-grained control throughout the
AI workflow. We provide automatic lineage tracking, centralized governance, and seamless
cross-workspace collaboration for simplified MLOps and enhanced productivity. Furthermore,
we give customers complete control and ownership of their data and models with privacy
controls to maintain compliance as well as efficiency and granular models on their data, fine-
tuned at lower costs.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 45
 Executive
Summary Databricks Mosaic AI

Introduction Databricks provides a scalable, collaborative platform that empowers ML teams to prepare
and process data, streamline cross-team collaboration, and standardize the full ML lifecycle
Risks in AI System
Components from experimentation to production, including generative AI and large language models
(LLMs). You can both build models from scratch and tune existing models on your data
⟶ Understanding
Databricks Data
to maintain privacy and control. However, it’s not just about building and serving models.
Intelligence Platform
AI Risk Mitigation
Databricks Mosaic AI covers the end-to-end AI workflow to help you deploy and manage
Controls
models all the way through production. Our AI offerings include:

Conclusion

Resources and
Further Reading
1 | End-to-end RAG (retrieval augmented generation) to build high-quality
conversational agents on your data, leveraging the Mosaic AI Vector Search

Acknowledgments
(Public Preview) for increased relevance and accuracy.

2 | Integrating data-centric applications with leading AI APIs like OpenAI.

Appendix:
Glossary
3 | Training of predictive ML models either from scratch on an organization’s
tabular data or by fine-tuning existing models such as MPT and Llama 2, to
License
further enhance AI applications with a deep understanding of a target domain.

4 | Efficient and secure serverless inference on your enterprise data and

connected to Unity Catalog’s governance and quality monitoring functionality.

5 | End-to-end MLOps based on the popular MLflow open source project, with
all data produced automatically actionable, tracked and monitorable in the
lakehouse.

6 | Improve visibility and proactively detect anomalies in your entire data and
AI workflow, reducing risks, time to value, and high operational costs with
Databricks Lakehouse Monitoring (Public Preview).

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 46
 Executive
Summary Databricks Unity Catalog

Introduction Databricks Unity Catalog is the industry’s first unified governance solution for data and AI
on the lakehouse. With Unity Catalog, organizations can seamlessly govern their structured
Risks in AI System
Components and unstructured data, machine learning models, notebooks, dashboards, and files on
any cloud or platform. Data scientists, analysts and engineers can use Unity Catalog to
⟶ Understanding
Databricks Data
securely discover, access and collaborate on trusted data and AI assets, leveraging AI to
Intelligence Platform
AI Risk Mitigation
boost productivity and unlock the full potential of the lakehouse environment. This unified
Controls
approach to governance accelerates data and AI initiatives while ensuring regulatory
compliance in a simplified manner. Unity Catalog provides:
Conclusion

Resources and
Further Reading
1 | Access control for data and AI: Unity Catalog is the only governance
Acknowledgments solution for data and AI. The foundational capabilities of Unity Catalog are in
governance and access control of all your data and AI assets. This simplified
Appendix: governance experience works across workspaces and clouds helps you
Glossary
manage your entire data estates. Discover and classify structured and
unstructured data, ML models, notebooks, dashboards and arbitrary files on
License
any cloud. Consolidate, map and query data from various platforms, including
MySQL, PostgreSQL, Amazon Redshift, Snowflake, Azure SQL, Azure Synapse
and Google’s BigQuery in one place. Accelerate your data and AI initiatives
with a single point of access for data exploration. Boost productivity by
securely searching, understanding and extracting insights from your data and
AI using natural language.

2 | Open data sharing and collaboration: Easily share data and AI assets
across clouds, regions and platforms with open source Delta Sharing, natively
integrated within Unity Catalog. Securely collaborate with anyone, anywhere
to unlock new revenue streams and drive business value without relying on
proprietary formats, complex ETL processes or costly data replication.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 47
 3 | Centralized data search and discovery: Quickly find, understand and
reference data from across your data estate, boosting productivity. Data
search in Unity Catalog is secure by default, limiting search results based on
access privileges of the users and adding an additional layer of security for
privacy considerations.

4 | A
 utomated lineage for all workloads: Build better understanding of
your data estate with automated lineage, tags and auto-generated data
insights. Create a unified, transparent view of your entire data ecosystem
Executive
Summary with automated and granular lineage for all workloads in SQL, R, Python and
Scala, and across all asset types — tables, files, notebooks, workflows and
Introduction dashboards. Lineage can be retrieved via REST APIs to support integrations
with our catalog partners.
Risks in AI System
Components 5 | Security and compliance: Ability to define access policies at scale for
all data and AI assets such as files, tables, ML models, notebooks and
⟶ Understanding
Databricks Data
dashboards and to audit the access patterns.
Intelligence Platform
AI Risk Mitigation
Controls

Conclusion

Resources and
Further Reading

Acknowledgments

Appendix:
Glossary

License Databricks Platform Architecture

Databricks is a hybrid platform as a service (PaaS) general-purpose data-agnostic

compute platform.

We use the phrase “hybrid PaaS” because our lakehouse architecture is split into two
separate planes to simplify your permissions, avoid data duplication and reduce risk. The
control plane is the management plane where Databricks runs the workspace application
and manages notebooks, configuration and clusters. The compute plane handles your
data processing. Customers deploy a compute plane (virtual network and compute) in a
cloud service provider account (such as AWS, Azure or GCP) that the customer owns. With
serverless deployments, the compute plane exists in the customer’s Databricks account
rather than their cloud service provider account. Customers get the benefits of PaaS with
DATABRICKS
AI SECURITY the option to keep their data processing clusters locally within their environment.
FRAMEWORK
(DASF)
VERSION 1.0 48
 The phrase “general-purpose data-agnostic” means that, unlike a pure SaaS, Databricks
doesn’t know what data your teams process with the Databricks Platform. The actual code,
business logic, model artifacts, SaaS, open source models, choice of LLMs, and datasets are
provided by your teams. You won’t find recommendations like “truncate user IDs” or “hash
feature names” because we don’t know what data you’re analyzing and what models you are
deploying.

If you’re new to Databricks or the lakehouse architecture, start with an overview of the
architecture and a review of common security questions before you hop into specific
Executive
Summary
recommendations. You’ll see those in our Security and Trust Center and the Security and
Trust Overview Whitepaper.
Introduction

Risks in AI System
Components

⟶ Understanding
Databricks Data
Intelligence Platform
AI Risk Mitigation
Controls

Conclusion

Resources and Databricks Platform Security

Further Reading

Data and AI are your most valuable assets and always have to be protected — that’s why
Acknowledgments
security is built into every layer of the Databricks Data Intelligence Platform. Databricks
Appendix: Security is based on three core principles: Trust, Technology and Transparency.
Glossary

License

1 | Trust: Third-party audit firms regularly audit Databricks systems and

processes. Databricks customers can trust independent validation of internal
security processes.

2 | Technology: Databricks deploys modern technology solutions combined

with secure processes across the enterprise to maximize security. Security
design and tools are applied throughout. Databricks considers security in
the platform architecture design, network security processes, automated
penetration testing on the production systems, and vulnerability scanning
tools during development.

3 | Transparency: Databricks provides customers with full attestation reports

(for example, SOC 2 Type 2), certifications (for example, ISO 27001) and
detailed architecture overviews. Our transparency enables you to meet your
DATABRICKS
AI SECURITY regulatory needs while taking advantage of our platform.
FRAMEWORK
(DASF)
VERSION 1.0 49
 Our Databricks Security team regularly works with customers to securely deploy AI systems
on our platform with the appropriate security and governance features. We understand how
ML systems are designed for security, teasing out possible security engineering risks and
making such risks explicit. Databricks is committed to providing a data intelligence platform
where business stakeholders, data engineers, data scientists, ML engineers, data governance
Executive officers and data analysts can trust that their data and AI models are secure.
Summary

Introduction 3.2 Databricks AI Risk Mitigation Controls

Risks in AI System
Components At Databricks, we strive to continuously innovate and advance our product offerings to
simplify the ability to build AI-powered solutions on the Databricks Data Intelligence Platform
⟶ Understanding
safely. We believe there is no greater accelerant to delivering ML to production than building
Databricks Data
Intelligence Platform on a unified, data-centric AI platform. On Databricks, data and models can be managed
AI Risk Mitigation
Controls and governed in a single governance solution with Unity Catalog. With Mosaic AI Model
Serving, we streamlined the complexities associated with infrastructure for real-time model
Conclusion
deployment, providing a scalable and user-friendly solution. For long-term efficiency and

Resources and performance stability in ML production, Databricks Lakehouse Monitoring plays a pivotal role.
Further Reading
This tool ensures continuous performance monitoring, contributing to sustained excellence
in machine learning operations. These components collectively form the data pipelines of an
Acknowledgments
ML solution, all of which can be orchestrated using Databricks Workflows.
Appendix:
Glossary Perhaps the most significant recent change in the machine learning landscape has been
the rapid advancement of generative AI. Generative models such as large language models
License
(LLMs) and image generation models have revolutionized the field, unlocking previously
unattainable levels of natural language and image generation. However, their arrival also
introduces new challenges and decisions to be made in the context of MLOps.

With all these developments in mind, below is a list of the necessary mitigation controls
for organizations to address AI security risks. This mitigation guidance incorporates new
Databricks features such as Models in Unity Catalog, Model Serving, and Lakehouse
Monitoring into our MLOps architecture recommendations.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 50
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 1 SSO with IdP and MFA

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.1 DATA PREP 2.1 Implementing single sign-on with an identity
DATA PREP 2.2 DATA PREP 2.3 provider’s (IdP) multi-factor authentication is
Configuration
DATA PREP 2.4 DATASETS 3.1 critical for secure authentication. It adds an extra
EVALUATION 6.1 MODEL 7.1 layer of security, ensuring that only authorized
users access the Databricks Platform. PRODUCT REFERENCE
MODEL 7.2 MODEL MANAGEMENT 8.2

MODEL MANAGEMENT 8.4

MODEL SERVING — INFERENCE REQUESTS 9.1

AWS | Azure | GCP

Executive
MODEL SERVING — INFERENCE REQUESTS 9.2
Summary
MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

Introduction MODEL SERVING — INFERENCE REQUESTS 9.7

MODEL SERVING — INFERENCE REQUESTS 9.9

MODEL SERVING — INFERENCE REQUESTS 9.10

Risks in AI System MODEL SERVING — INFERENCE RESPONSE 10.3

Components MODEL SERVING — INFERENCE RESPONSE 10.4

⟶ Understanding
Databricks Data
Intelligence Platform
DASF 2 Sync users and groups
AI Risk Mitigation
Controls RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.1 DATA PREP 2.1 Synchronizing users and groups from your
Conclusion DATA PREP 2.2 DATA PREP 2.3 identity provider (IdP) with Databricks using
Configuration
DATA PREP 2.4 DATASETS 3.1 the SCIM standard facilitates consistent and
EVALUATION 6.1 MODEL 7.2 automated user provisioning for enhancing
security. PRODUCT REFERENCE
Resources and MODEL MANAGEMENT 8.2
Further Reading MODEL MANAGEMENT 8.4

MODEL SERVING — INFERENCE REQUESTS 9.1

AWS | Azure | GCP

MODEL SERVING — INFERENCE REQUESTS 9.2

Acknowledgments
MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

Appendix:
Glossary MODEL SERVING — INFERENCE REQUESTS 9.9

MODEL SERVING — INFERENCE REQUESTS 9.10

MODEL SERVING — INFERENCE RESPONSE 10.3

License MODEL SERVING — INFERENCE RESPONSE 10.4

DASF 3 Restrict access using IP access lists

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.1 DATA PREP 2.1 Configure IP access lists to restrict authentication
DATA PREP 2.2 DATA PREP 2.3 to Databricks from specific IP ranges, such as
Configuration
DATA PREP 2.4 DATASETS 3.1 VPNs or office networks, and strengthen network
EVALUATION 6.1 MODEL 7.2 security by preventing unauthorized access from
untrusted locations. PRODUCT REFERENCE
MODEL MANAGEMENT 8.2

MODEL MANAGEMENT 8.4

MODEL SERVING — INFERENCE REQUESTS 9.1

AWS | Azure | GCP

MODEL SERVING — INFERENCE REQUESTS 9.2

MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

MODEL SERVING — INFERENCE REQUESTS 9.9

DATABRICKS MODEL SERVING — INFERENCE REQUESTS 9.10

AI SECURITY MODEL SERVING — INFERENCE RESPONSE 10.3

FRAMEWORK MODEL SERVING — INFERENCE RESPONSE 10.4
(DASF)
VERSION 1.0 51
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 4 Restrict access using private link

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.1 DATA PREP 2.1 Use AWS PrivateLink, Azure Private Link or GCP
DATA PREP 2.2 DATA PREP 2.3 Private Service Connect to create a private
Configuration
DATA PREP 2.4 DATASETS 3.1 network route between the customer and the
Executive EVALUATION 6.1 MODEL 7.2 Databricks control plane or the control plane and
Summary the customer’s compute plane environments to PRODUCT REFERENCE
MODEL MANAGEMENT 8.2
enhance data security by avoiding public internet
MODEL MANAGEMENT 8.4

MODEL SERVING — INFERENCE REQUESTS 9.1

exposure. AWS | Azure | GCP
Introduction
MODEL SERVING — INFERENCE REQUESTS 9.2

MODEL SERVING — INFERENCE REQUESTS 9.5

Risks in AI System MODEL SERVING — INFERENCE REQUESTS 9.6

Components MODEL SERVING — INFERENCE REQUESTS 9.7

MODEL SERVING — INFERENCE REQUESTS 9.9

MODEL SERVING — INFERENCE REQUESTS 9.10

⟶ Understanding
MODEL SERVING — INFERENCE RESPONSE 10.3
Databricks Data
Intelligence Platform MODEL SERVING — INFERENCE RESPONSE 10.4

AI Risk Mitigation
Controls

DASF 5 Control access to data and other objects

Conclusion
RISKS DESCRIPTION CONTROL CATEGORY

Resources and RAW DATA 1.1 RAW DATA 1.4 Implementing Unity Catalog for unified
Further Reading DATA PREP 2.1 DATASETS 3.1 permissions management and assets simplifies
Implementation
DATASETS 3.2 DATASETS 3.3 access control and enhances security.
GOVERNANCE 4.1 EVALUATION 6.1
Acknowledgments PRODUCT REFERENCE
MODEL 7.1 MODEL 7.2

MODEL MANAGEMENT 8.1

MODEL MANAGEMENT 8.2

AWS | Azure | GCP
Appendix:
Glossary MODEL MANAGEMENT 8.3

MODEL MANAGEMENT 8.4

MODEL SERVING — INFERENCE REQUESTS 9.1

License MODEL SERVING — INFERENCE REQUESTS 9.2

MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

MODEL SERVING — INFERENCE REQUESTS 9.9

MODEL SERVING — INFERENCE REQUESTS 9.10

MODEL SERVING — INFERENCE RESPONSE 10.3

MODEL SERVING — INFERENCE RESPONSE 10.4

DASF 6 Classify data

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.2 Tags are attributes containing keys and optional
values that you can apply to different securable
Implementation
objects in Unity Catalog. Organizing securable
objects with tags in Unity Catalog aids in
efficient data management, data discovery PRODUCT REFERENCE

and classification, essential for handling large

DATABRICKS datasets. AWS | Azure | GCP
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 52
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 7 Enforce data quality checks on batch and streaming datasets

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.3 RAW DATA 1.9 Databricks Delta Live Tables (DLT) simplifies ETL
DATA PREP 2.1 DATASETS 3.1 development with declarative pipelines that
Implementation
GOVERNANCE 4.1 EVALUATION 6.1 integrate quality control checks and performance
Executive monitoring.
Summary PRODUCT REFERENCE

AWS | Azure | GCP

Introduction

Risks in AI System DASF 8 Encrypt data at rest

Components
RISKS DESCRIPTION CONTROL CATEGORY

⟶ Understanding RAW DATA 1.4 DATASETS 3.2 Databricks supports customer-managed

Databricks Data DATASETS 3.3 encryption keys to strengthen data at rest
Configuration
Intelligence Platform protection and greater access control.
AI Risk Mitigation
Controls PRODUCT REFERENCE

AWS | Azure | GCP

Conclusion

Resources and DASF 9 Encrypt data in transit

Further Reading
RISKS DESCRIPTION CONTROL CATEGORY

Acknowledgments RAW DATA 1.4 DATASETS 3.2 Databricks supports TLS 1.2+ encryption to
DATASETS 3.3 protect customer data during transit. This applies
Out-of-the-box
to data transfer between the customer and the
Appendix: Databricks control plane and within the compute
Glossary plane. Customers can also secure inter-cluster PRODUCT REFERENCE

communications within the compute plane per

their security requirements. AWS | Azure | GCP
License

DASF 10 Version data

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.5 RAW DATA 1.7 Store data in a lakehouse architecture using Delta
tables. Delta tables can be versioned to revert
Implementation
any user’s or malicious actor’s poisoning of data.
Data can be stored in a lakehouse architecture
in the customer’s cloud account. Both raw data PRODUCT REFERENCE

and feature tables are stored as Delta tables with

access controls to determine who can read and AWS | Azure | GCP
modify them. Data lineage with UC helps track
and audit changes and the origin of ML data
sources. Each operation that modifies a Delta
Lake table creates a new table version. User
actions are tracked and audited, and lineage
of transformations is available all in the same
platform. You can use history information to audit
operations, roll back a table or query a table at a
DATABRICKS specific point in time using time travel.
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 53
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 11 Capture and view data lineage

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.6 DATA PREP 2.1 Unity Catalog tracks and visualizes real-time data
Executive DATASETS 3.1 GOVERNANCE 4.1 lineage across all languages to the column level,
Out-of-the-box
Summary EVALUATION 6.1 providing a traceable history of an object from
notebooks, workflows, models and dashboards.
This enhances transparency and compliance, PRODUCT REFERENCE
Introduction with accessibility provided through the Catalog
Explorer. AWS | Azure | GCP

Risks in AI System
Components
DASF 12 Delete records from datasets
⟶ Understanding
RISKS DESCRIPTION CONTROL CATEGORY
Databricks Data
Intelligence Platform RAW DATA 1.8 Data governance in Delta Lake, the lakehouse
AI Risk Mitigation
storage layer, utilizes its atomicity, consistency,
Controls Implementation
isolation, durability (ACID) properties for effective
data management. This includes the capability to
remove data based on specific predicates from PRODUCT REFERENCE
Conclusion
a Delta Table, including the complete removal
of data’s history, supporting compliance with AWS | Azure | GCP
regulations like GDPR and CCPA.
Resources and
Further Reading

DASF 13 Use near real-time data

Acknowledgments

RISKS DESCRIPTION CONTROL CATEGORY

Appendix: RAW DATA 1.9 Use Databricks for near real-time data ingestion,
Glossary
processing, machine learning, and AI for
Implementation
streaming data.

License PRODUCT REFERENCE

AWS | Azure | GCP

DASF 14 Audit actions performed on datasets

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.10 DATASETS 3.1 Databricks auditing, enhanced by Unity Catalog’s
events, delivers fine-grained visibility into data
Implementation
access and user activities. This is vital for robust
data governance and security, especially in
regulated industries. It enables organizations to PRODUCT REFERENCE

proactively identify and manage overentitled

users, enhancing data security and ensuring AWS | AWS | GCP
compliance.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 54
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 15 Explore datasets and identify problems

RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.1 Iteratively explore, share and prep data for
the machine learning lifecycle by creating
Implementation
reproducible, editable and shareable datasets,
Executive tables and visualizations. Within Databricks this
Summary EDA process can be accelerated with Mosaic PRODUCT REFERENCE

AI AutoML. AutoML not only generates baseline

models given a dataset, but also provides the AWS | Azure | GCP
Introduction underlying model training code in the form of
a Python notebook. Notably for EDA, AutoML
calculates summary statistics on the provided
Risks in AI System dataset, creating a notebook for the data
Components scientist to review and adapt.

⟶ Understanding
Databricks Data DASF 16 Secure model features
Intelligence Platform
AI Risk Mitigation
Controls RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.1 DATA PREP 2.2 Databricks Feature Store is a centralized
DATASETS 3.1 GOVERNANCE 4.1 repository that enables data scientists to find and
Conclusion Implementation
ALGORITHMS 5.2 share features and also ensures that the same
MODEL SERVING — INFERENCE REQUESTS 9.10 code used to compute the feature values is used
for model training and inference. Unity Catalog’s PRODUCT REFERENCE
Resources and
capabilities, such as security, lineage, table
Further Reading
history, tagging and cross-workspace access, AWS | Azure | GCP
are automatically available to the feature table to
reduce the risk of malicious actors manipulating
Acknowledgments
the features that feed into ML training.

Appendix:
Glossary
DASF 17 Track and reproduce the training data used for ML model training

License RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.4 DATASETS 3.1 MLflow with Delta Lake tracks the training data
GOVERNANCE 4.1 ALGORITHMS 5.2 used for ML model training. It also enables the
Configuration
identification of specific ML models and runs
derived from particular datasets for regulatory
and auditable attribution. PRODUCT REFERENCE

AWS | Azure | GCP

DASF 18 Govern model assets

RISKS DESCRIPTION CONTROL CATEGORY

GOVERNANCE 4.1 With Unity Catalog, organizations can implement

a unified governance framework for their
Configuration
structured and unstructured data, machine
learning models, notebooks, features, functions,
and files, enhancing security and compliance PRODUCT REFERENCE

across clouds and platforms.

DATABRICKS
AWS | Azure | GCP

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 55
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 19 Manage end-to-end machine learning lifecycle

RISKS DESCRIPTION CONTROL CATEGORY

GOVERNANCE 4.2 MODEL 7.1 Databricks includes a managed version of MLflow

featuring enterprise security controls and
Implementation
high availability. It supports functionalities like
experiments, run management and notebook
revision capture. MLflow on Databricks allows PRODUCT REFERENCE

tracking and measuring machine learning model

Executive training runs, logging model training artifacts and AWS | Azure | GCP
Summary securing machine learning projects.

Introduction
DASF 20 Track ML training runs

Risks in AI System RISKS DESCRIPTION CONTROL CATEGORY

Components
ALGORITHMS 5.1 ALGORITHMS 5.3 MLflow tracking facilitates the automated
recording and retrieval of experiment details,
Implementation
⟶ Understanding including algorithms, code, datasets, parameters,
Databricks Data configurations, signatures and artifacts.
Intelligence Platform PRODUCT REFERENCE
AI Risk Mitigation
Controls AWS | Azure | GCP

Conclusion
DASF 21 Monitor data and AI system from a single pane of glass

Resources and RISKS DESCRIPTION CONTROL CATEGORY

Further Reading
RAW DATA 1.3 GOVERNANCE 4.2 Databricks Lakehouse Monitoring offers a
ALGORITHMS 5.2 single pane of glass to centrally track tables’
Implementation
Acknowledgments data quality and statistical properties and
automatically classifies data. It can also track the
performance of machine learning models and PRODUCT REFERENCE
Appendix: model serving endpoints by monitoring inference
Glossary tables containing model inputs and predictions AWS | Azure | N/A
through a single pane of glass.

License

DASF 22 Build models with all representative, accurate and relevant data sources

RISKS DESCRIPTION CONTROL CATEGORY

EVALUATION 6.2 MODEL 7.3 Harnessing internal data and intellectual

property to customize large AI models can offer
Implementation
a significant competitive edge. However, this
process can be complex, involving coordination
across various parts of the organization. The Data PRODUCT REFERENCE

Intelligence Platform addresses this challenge

by integrating data across traditionally isolated AWS | Azure | GCP
departments and systems. This integration
facilitates a more cohesive data and AI strategy,
enabling the effective training, testing and
evaluation of models using a comprehensive
dataset. Use caution when preparing data for
traditional models and GenAI training to ensure
that you are not unintentionally including data
that causes legal conflicts, such as copyright
violations, privacy violations or HIPAA violations.
DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 56
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 23 Register, version, approve, promote and deploy model

RISKS DESCRIPTION CONTROL CATEGORY

MODEL 7.1 MLflow Model Registry supports managing

the machine learning model lifecycle with
Implementation
capabilities for lineage tracking, versioning,
staging and model serving.
PRODUCT REFERENCE
Executive
Summary AWS | Azure | GCP

Introduction
DASF 24 Control access to models and model assets

Risks in AI System RISKS DESCRIPTION CONTROL CATEGORY

Components
MODEL 7.2 MODEL MANAGEMENT 8.2 Organizations commonly encounter challenges
MODEL MANAGEMENT 8.3 in tracking and controlling access to ML models,
Implementation
⟶ Understanding MODEL MANAGEMENT 8.4 auditing their usage, and understanding
Databricks Data MODEL SERVING — INFERENCE REQUESTS 9.1 their evolution in complex machine learning
Intelligence Platform workflows. Unity Catalog integrates with the PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.2
AI Risk Mitigation MLflow Model Registry across model lifecycles.
Controls
MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

This approach simplifies the management and AWS | Azure | GCP
oversight of ML models, proving particularly
MODEL SERVING — INFERENCE REQUESTS 9.7
valuable in environments with multiple teams
Conclusion MODEL SERVING — INFERENCE RESPONSE 10.3
and diverse projects.
MODEL SERVING — INFERENCE RESPONSE 10.4

Resources and
Further Reading
DASF 25 Use retrieval augmented generation (RAG) with large language models (LLMs)
Acknowledgments
RISKS DESCRIPTION CONTROL CATEGORY

EVALUATION 6.2 Generating relevant and accurate responses

Appendix: MODEL SERVING — INFERENCE REQUESTS 9.8 in large language models (LLMs) while avoiding
Implementation
Glossary hallucinations requires grounding them
in domain-specific knowledge. Retrieval
augmented generation (RAG) addresses this PRODUCT REFERENCE
License by breaking down extensive datasets into
manageable segments (“chunks”) that are AWS | Azure | GCP
“vector embedded.” These vector embeddings
are mathematical representations that help the
model understand and quantify different data
segments. As a result, LLMs produce responses
that are contextually relevant and deeply rooted
in the specific domain knowledge.

DASF 26 Fine-tune large language models (LLMs)

RISKS DESCRIPTION CONTROL CATEGORY

MODEL SERVING — INFERENCE REQUESTS 9.8 Data is your competitive advantage. Use it
to customize large AI models to beat your
Implementation
competition. Produce new model variants with
tailored LLM response style and structure via
fine-tuning. PRODUCT REFERENCE

DATABRICKS
Fine-tune your own LLM with open models to
own your IP.
AWS | Azure | N/A

AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 57
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 27 Pretrain a large language model (LLM)

RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.8 MODEL 7.3 Data is your competitive advantage. Use it
MODEL SERVING — INFERENCE REQUESTS 9.8 to customize large AI models to beat your
Implementation
competition by pretraining models with your
data, imbuing the model with domain-specific
knowledge, vocabulary and semantics. Pretrain PRODUCT REFERENCE

your own LLM with MosaicML to own your IP.

AWS | Azure | N/A
Executive
Summary

DASF 28 Create model aliases, tags and annotations

Introduction
RISKS DESCRIPTION CONTROL CATEGORY

Risks in AI System MODEL MANAGEMENT 8.1 Model aliases in machine learning workflows
Components MODEL MANAGEMENT 8.3 allow you to assign a mutable, named reference
Implementation
MODEL SERVING — INFERENCE REQUESTS 9.5 to a specific version of a registered model.
MODEL SERVING — INFERENCE REQUESTS 9.6 This functionality is beneficial for tracking and
⟶ Understanding managing different stages of a model’s lifecycle, PRODUCT REFERENCE
MODEL SERVING — INFERENCE RESPONSE 10.3
Databricks Data indicating the current deployment status of any
Intelligence Platform
MODEL SERVING — INFERENCE RESPONSE 10.4
given model version. AWS | Azure | GCP
AI Risk Mitigation
Controls

DASF 29 Build MLOps workflows

Conclusion

RISKS DESCRIPTION CONTROL CATEGORY

Resources and RAW DATA 1.8 MODEL MANAGEMENT 8.1 The lakehouse forms the foundation of a data-
Further Reading MODEL MANAGEMENT 8.3 centric AI platform. Key to this is the ability to
Implementation
manage both data and AI assets from a unified
governance solution on the lakehouse. Databricks
Acknowledgments Unity Catalog enables this by providing PRODUCT REFERENCE

centralized access control, auditing, approvals,

model workflow, lineage, and data discovery AWS | Azure | GCP
Appendix: capabilities across Databricks workspaces.
Glossary
These benefits are now extended to MLflow
Models with the introduction of Models in Unity
License Catalog. Through providing a hosted version of
the MLflow Model Registry in Unity Catalog, the
full lifecycle of an ML model can be managed
while leveraging Unity Catalog’s capability to
share assets across Databricks workspaces and
trace lineage across both data and models.

DASF 30 Encrypt models

RISKS DESCRIPTION CONTROL CATEGORY

MODEL MANAGEMENT 8.2 Databricks Platform secures model assets and

MODEL MANAGEMENT 8.4 their transfer with TLS 1.2+ in-transit encryption.
Out-of-the-box
MODEL SERVING — INFERENCE REQUESTS 9.1 Additionally, Unity Catalog’s managed model
MODEL SERVING — INFERENCE REQUESTS 9.2 registry provides encryption at rest for persisting
models, further enhancing security. PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.5

MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

AWS | Azure | GCP

MODEL SERVING — INFERENCE RESPONSE 10.2

DATABRICKS MODEL SERVING — INFERENCE RESPONSE 10.3

AI SECURITY MODEL SERVING — INFERENCE RESPONSE 10.4

FRAMEWORK MODEL SERVING — INFERENCE RESPONSE 10.5
(DASF)
VERSION 1.0 58
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 31 Secure model serving endpoints

RISKS DESCRIPTION CONTROL CATEGORY

MODEL MANAGEMENT 8.2 Model serving involves risks of unauthorized

MODEL MANAGEMENT 8.4 data access and model tampering, which can
Out-of-the-box
MODEL SERVING — INFERENCE REQUESTS 9.1 compromise the integrity and reliability of
MODEL SERVING — INFERENCE REQUESTS 9.2 machine learning deployments. Mosaic AI Model
Serving addresses these concerns by providing PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.5
secure-by-default REST API endpoints for MLflow
MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

machine learning models, featuring autoscaling, AWS | Azure | N/A
high availability and low latency.
MODEL SERVING — INFERENCE RESPONSE 10.2
Executive
Summary MODEL SERVING — INFERENCE RESPONSE 10.3

MODEL SERVING — INFERENCE RESPONSE 10.4

MODEL SERVING — INFERENCE RESPONSE 10.5

Introduction

Risks in AI System DASF 32 Streamline the usage and management of various

Components
large language model (LLM) providers

⟶ Understanding RISKS DESCRIPTION CONTROL CATEGORY

Databricks Data
MODEL MANAGEMENT 8.2 External models are third-party models hosted
Intelligence Platform
AI Risk Mitigation MODEL MANAGEMENT 8.4 outside of Databricks. Supported by Model
Out-of-the-box
Controls MODEL SERVING — INFERENCE REQUESTS 9.1 Serving AI Gateway, Databricks external models
MODEL SERVING — INFERENCE REQUESTS 9.2 via the AI Gateway allow you to streamline the
usage and management of various large language PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.5
Conclusion model (LLM) providers, such as OpenAI and
MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

Anthropic, within an organization. You can also AWS | Azure | N/A
use Mosaic AI Model Serving as a provider to
MODEL SERVING — INFERENCE RESPONSE 10.2
Resources and serve predictive ML models, which offers rate
MODEL SERVING — INFERENCE RESPONSE 10.3
Further Reading limits for those endpoints. As part of this support,
MODEL SERVING — INFERENCE RESPONSE 10.4
Model Serving offers a high-level interface that
MODEL SERVING — INFERENCE RESPONSE 10.5 simplifies the interaction with these services by
Acknowledgments providing a unified endpoint to handle specific
LLM-related requests. In addition, Databricks
support for external models provides centralized
Appendix: credential management. By storing API keys in
Glossary one secure location, organizations can enhance
their security posture by minimizing the exposure
of sensitive API keys throughout the system. It
License also helps to prevent exposing these keys within
code or requiring end users to manage keys safely.

DASF 33 Manage credentials securely

RISKS DESCRIPTION CONTROL CATEGORY

MODEL 7.2 MODEL MANAGEMENT 8.2 Databricks Secrets stores your credentials
and references them in notebooks, scripts,
Implementation
configuration properties and jobs.

Integrating with heterogeneous systems PRODUCT REFERENCE

requires managing a potentially large set
of credentials and safely distributing them
across an organization. Instead of directly
AWS | Azure | GCP

entering your credentials into a notebook, use

Databricks Secrets to store your credentials and
reference them in notebooks and jobs to prevent
credential leaks through models. Databricks
secret management allows users to use and
DATABRICKS share credentials within Databricks securely.
AI SECURITY You can also choose to use a third-party secret
FRAMEWORK management service, such as AWS Secrets
(DASF) Manager or a third-party secret manager.
VERSION 1.0 59
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 34 Run models in multiple layers of isolation

RISKS DESCRIPTION CONTROL CATEGORY

MODEL 7.1 Databricks Serverless Compute provides

Executive MODEL SERVING — INFERENCE REQUESTS 9.3 a secure-by-design model serving service
Out-of-the-box
Summary featuring defense-in-depth controls like
dedicated VMs, network segmentation, and
encryption for data in transit and at rest. It PRODUCT REFERENCE
Introduction adheres to the principle of least privilege for
enhanced security. AWS | Azure | N/A

Risks in AI System
Components
DASF 35 Track model performance
⟶ Understanding
RISKS DESCRIPTION CONTROL CATEGORY
Databricks Data
Intelligence Platform MODEL SERVING — INFERENCE RESPONSE 10.1 Databricks Lakehouse Monitoring provides
AI Risk Mitigation
performance metrics and data quality
Controls Implementation
statistics across all account tables. It tracks the
performance of machine learning models and
model serving endpoints by observing inference PRODUCT REFERENCE
Conclusion
tables with model inputs and predictions.
AWS | Azure | N/A

Resources and
Further Reading
DASF 36 Set up monitoring alerts
Acknowledgments
RISKS DESCRIPTION CONTROL CATEGORY

RAW DATA 1.3 Databricks SQL alerts can monitor the metrics
Appendix: MODEL SERVING — INFERENCE RESPONSE 10.1 table for security-based conditions, ensuring
Implementation
Glossary data integrity and timely response to potential
issues:
PRODUCT REFERENCE
License S
 tatistic range Alert: Triggers when a specific
statistic, such as the fraction of missing
values, exceeds a predetermined threshold
AWS | Azure | N/A

D
 ata distribution shift alert: Activates
upon shifts in data distribution, as indicated
by the drift metrics table

B
 aseline divergence alert: Alerts if data
significantly diverges from a baseline,
suggesting potential needs for data analysis
or model retraining, particularly
in InferenceLog analysis

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 60
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 37 Set up inference tables for monitoring and debugging models

RISKS DESCRIPTION CONTROL CATEGORY

MODEL SERVING — INFERENCE REQUESTS 9.1 Databricks inference tables automatically record
MODEL SERVING — INFERENCE REQUESTS 9.2 incoming requests and outgoing responses
Implementation
MODEL SERVING — INFERENCE REQUESTS 9.3 to model serving endpoints, storing them as
MODEL SERVING — INFERENCE REQUESTS 9.4 a Unity Catalog Delta table. This table can be
used to monitor, debug and enhance ML models. PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.5
By coupling inference tables with Lakehouse
MODEL SERVING — INFERENCE REQUESTS 9.6

MODEL SERVING — INFERENCE REQUESTS 9.7

Monitoring, customers can also set up automated AWS | Azure | N/A
Executive monitoring jobs and alerts on inference tables,
Summary MODEL SERVING — INFERENCE RESPONSE 10.1
such as monitoring text quality or toxicity from
MODEL SERVING — INFERENCE RESPONSE 10.3
endpoints serving LLMs, etc.
MODEL SERVING — INFERENCE RESPONSE 10.4

Introduction Critical applications of an inference table include:

R
 etraining dataset creation: Building
datasets for the next iteration of your models
Risks in AI System
Components Q
 uality monitoring: Keeping track of
production data and model performance

D
 iagnostics and debugging: Investigating and
⟶ Understanding
resolving issues with suspicious inferences
Databricks Data
Intelligence Platform M
 islabeled dataidentification: Compiling
AI Risk Mitigation data that needs relabeling
Controls

Conclusion DASF 38 Platform security — vulnerability management

RISKS DESCRIPTION CONTROL CATEGORY

Resources and
Further Reading PLATFORM 12.1 Managing vulnerabilities entails addressing
complex security challenges with performance
Out-of-the-box
impact considerations. Databricks’ formal
Acknowledgments and documented vulnerability management
program, overseen by the chief security officer PRODUCT REFERENCE

(CSO), is approved by management, undergoes

Appendix: annual reviews and is communicated to all AWS | Azure | GCP
Glossary relevant internal parties. The policy requires that
vulnerabilities be addressed based on severity:
critical vulnerabilities within 14 days, high
License severity within 30 days and medium severity
within 60 days.

DASF 39 Platform security — Incident Response Team

RISKS DESCRIPTION CONTROL CATEGORY

PLATFORM 12.2 PLATFORM 12.3 Databricks has established a formal incident

response plan that outlines key elements such
Out-of-the-box
as roles, responsibilities, escalation paths
and external communication protocols. The
platform handles over 9TB of audit logs daily, PRODUCT REFERENCE

aiding customer and Databricks security

investigations. A dedicated security incident AWS | Azure | GCP
response team operates an internal Databricks
instance, consolidating essential log sources for
thorough security analysis. Databricks ensures
continual operational readiness with a 24/7/365
on-call rotation. Additionally, a proactive hunting
program and a specialized detection team
DATABRICKS
support the incident response program.
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 61
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 40 Platform security — internal access

RISKS DESCRIPTION CONTROL CATEGORY

PLATFORM 12.4 Databricks personnel, by default, do not have

access to customer workspaces or production
Out-of-the-box
environments. Access may be temporarily
requested by Databricks staff for purposes
such as investigating outages, security events PRODUCT REFERENCE
or supporting deployments. Customers have

Executive
the option to disable this access. Additionally, AWS | Azure | GCP
staff activity within these environments is
Summary recorded in customer audit logs. Accessing these
areas requires multi-factor authentication, and
employees must connect to the Databricks VPN.
Introduction

Risks in AI System DASF 41 Platform security — secure SDLC

Components
RISKS DESCRIPTION CONTROL CATEGORY

⟶ Understanding PLATFORM 12.5 Databricks engineering integrates security

Databricks Data throughout the software development lifecycle
Intelligence Platform Out-of-the-box
(SDLC), encompassing both technical and
AI Risk Mitigation process-level controls under the oversight of our
Controls chief security officer (CSO). Activities within our PRODUCT REFERENCE
SDLC include:

Conclusion
C
 ode peer reviews
AWS | Azure | GCP

S
 tatic and dynamic scans for code and
containers, including dependencies
Resources and F
 eature-level security reviews
Further Reading
A
 nnual software engineering security training
C
 ross-organizational collaborations between
Acknowledgments security, product management, product
security and security champions

These development controls are augmented

Appendix:
by internal and external penetration testing
Glossary
programs, with findings tracked for resolution
and reported to our executive team. Databricks’
processes undergo an independent annual
License review, the results of which are published in our
SOC 2 Type 2 report, available upon request.

DASF 42 Employ data-centric MLOps and LLMOps

RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.2 DATA PREP 2.3 MLOps enhances efficiency, scalability, security
DATA PREP 2.4 GOVERNANCE 4.2 and risk reduction in machine learning projects.
Implementation
ALGORITHMS 5.1 ALGORITHMS 5.3 Databricks integrates with MLflow, focusing on
enterprise reliability, security and scalability for
EVALUATION 6.1 MODEL 7.1
managing the machine learning lifecycle. The PRODUCT REFERENCE
MODEL 7.2 MODEL 7.3
latest update to MLflow introduces new LLMOps
MODEL MANAGEMENT 8.3
features for better management and deployment AWS | Azure | GCP
OPERATIONS 11.1 of large language models (LLMs). This includes
integrations with Hugging Face Transformers,
OpenAI and the external models in Mosaic AI
Model Serving.

MLflow also integrates with LangChain and a

prompt engineering UI, facilitating generative
DATABRICKS AI application development for use cases such
AI SECURITY
as chatbots, document summarization and text
FRAMEWORK
classification.
(DASF)
VERSION 1.0 62
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 43 Use access control lists

RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.3 ALGORITHMS 5.3 Databricks access control lists (ACLs) enable
Executive MODEL 7.1 you to configure permissions for accessing and
Implementation
Summary interacting with workspace objects, including
folders, notebooks, experiments, models,
clusters, pools, jobs, Delta Live Tables pipelines, PRODUCT REFERENCE
Introduction alerts, dashboards, queries and SQL warehouses.
AWS | Azure | GCP

Risks in AI System
Components
DASF 44 Triggering actions in response to a specific event

⟶ Understanding RISKS DESCRIPTION CONTROL CATEGORY

Databricks Data
Intelligence Platform EVALUATION 6.1 OPERATIONS 11.1 Webhooks in the MLflow Model Registry enable
AI Risk Mitigation you to automate machine learning workflow by
Implementation
Controls triggering actions in response to specific events.
These webhooks facilitate seamless integrations,
allowing for the automatic execution of various PRODUCT REFERENCE
Conclusion processes. For example, webhooks are used for:

C
 I workflow trigger: Validate your model
AWS | Azure | GCP

Resources and automatically when creating a new version

Further Reading T
 eam notifications: Send alerts through a
messaging app when a model stage transition
request is received
Acknowledgments
M
 odel fairness evaluation: Invoke a workflow
to assess model fairness and bias upon a
Appendix: production transition request
Glossary
A
 utomated deployment: Trigger a
deployment pipeline when a new tag is
created on a model
License

DASF 45 Evaluate models

RISKS DESCRIPTION CONTROL CATEGORY

EVALUATION 6.1 Model evaluation is a critical component of

EVALUATION 6.2 MODEL 7.3 the machine learning lifecycle. It provides data
Implementation
MODEL SERVING — INFERENCE REQUESTS 9.5 scientists with the tools to measure, interpret
MODEL SERVING — INFERENCE REQUESTS 9.6 and explain the performance of their models.
MLflow plays a critical role in accelerating PRODUCT REFERENCE
MODEL SERVING — INFERENCE RESPONSE 10.4
model development by offering insights into
OPERATIONS 11.1
the reasons behind a model’s performance and AWS | Azure | GCP
guiding improvements and iterations. MLflow
offers many industry-standard native evaluation
metrics for classical machine learning algorithms
and LLMs, and also facilitates the use of custom
evaluation metrics.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 63
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 46 Store and retrieve embeddings securely

RISKS DESCRIPTION CONTROL CATEGORY

MODEL SERVING — INFERENCE REQUESTS 9.1 Mosaic AI Vector Search is a vector database
MODEL SERVING — INFERENCE REQUESTS 9.2 that is built into the Databricks Data Intelligence
Implementation
MODEL SERVING — INFERENCE REQUESTS 9.5 Platform and integrated with its governance
MODEL SERVING — INFERENCE REQUESTS 9.6 and productivity tools. A vector database is a
Executive database that is optimized to store and retrieve PRODUCT REFERENCE
MODEL SERVING — INFERENCE REQUESTS 9.7
Summary embeddings. Embeddings are mathematical
MODEL SERVING — INFERENCE REQUESTS 9.8

MODEL SERVING — INFERENCE REQUESTS 9.9

representations of the semantic content of data, AWS | Azure | N/A
typically text or image data. Embeddings are
Introduction MODEL SERVING — INFERENCE REQUESTS 9.10
usually generated by feature extraction models
MODEL SERVING — INFERENCE RESPONSE 10.4
for text, image, audio or multi-modal data, and
are a key component of many GenAI applications
Risks in AI System that depend on finding documents or images
Components that are similar to each other. Examples are RAG
systems, recommender systems, and image and
video recognition.
⟶ Understanding
Databricks Data Databricks implements the following security
Intelligence Platform controls to protect your data:
AI Risk Mitigation
Controls E
 very customer request to Vector Search
is logically isolated, authenticated and
authorized
Conclusion M
 osaic AI Vector Search encrypts all data at
rest (AES-256) and in transit (TLS 1.2+)

Resources and
Further Reading
DASF 47 Compare LLM outputs on set prompts
Acknowledgments
RISKS DESCRIPTION CONTROL CATEGORY

EVALUATION 6.2 New, no-code visual tools allow users to

Appendix: compare models’ output based on set prompts,
Implementation
Glossary which are automatically tracked within MLflow.
With integration into Mosaic AI Model Serving,
customers can deploy the best model to PRODUCT REFERENCE
License production. The AI Playground is a chat-like
environment where you can test, prompt and AWS | Azure | N/A
compare LLMs.

DASF 48 Use hardened Runtime for Machine Learning

RISKS DESCRIPTION CONTROL CATEGORY

MODEL 7.3 Databricks Runtime for Machine Learning

(Databricks Runtime ML) now automates
Out-of-the-box
cluster creation with versatile infrastructure,
encompassing pre-built ML/DL libraries
and custom library integration. Enhanced PRODUCT REFERENCE

scalability and cost management tools optimize

performance and expenditure. The refined user AWS | Azure | GCP
interface caters to various expertise levels, while
new collaboration features support team-based
projects. Comprehensive training resources and
DATABRICKS detailed documentation complement
AI SECURITY these improvements.
FRAMEWORK
(DASF)
VERSION 1.0 64
 D E S C R I P T I O N O F C O N T R O L I M P L E M E N TAT I O N
CONTROL/RISK
O N DATA B R I C K S P L AT F O R M

DASF 49 Automate LLM evaluation

RISKS DESCRIPTION CONTROL CATEGORY

EVALUATION 6.1 The “LLM-as-a-judge” feature in MLflow 2.8

MODEL SERVING — INFERENCE REQUESTS 9.8 automates LLM evaluation, offering a practical
Implementation
alternative to human judgment. It’s designed
to be efficient and cost-effective, maintaining
consistency with human scores. This tool PRODUCT REFERENCE

supports various metrics, including standard and

customizable GenAI metrics, and allows users AWS | Azure | GCP
to select an LLM as a judge and define specific
Executive grading criteria.
Summary

Introduction DASF 50 Platform compliance

RISKS DESCRIPTION CONTROL CATEGORY

Risks in AI System
Components PLATFORM 12.6 Develop your solutions on a platform created
using some of the most rigorous security
Out-of-the-box
and compliance standards in the world. Get
⟶ Understanding independent audit reports verifying that
Databricks Data Databricks adheres to security controls for PRODUCT REFERENCE
Intelligence Platform ISO 27001, ISO 27018, SOC 1, SOC 2, FedRAMP,
AI Risk Mitigation
Controls
HITRUST, IRAP, etc. AWS | Azure | GCP

Conclusion DASF 51 Share data and AI assets securely

RISKS DESCRIPTION CONTROL CATEGORY

Resources and
Further Reading RAW DATA 1.1 RAW DATA 1.6 Databricks Delta Sharing lets you share data
RAW DATA 1.7 DATASETS 3.1 and AI assets securely in Databricks with users
Out-of-the-box
MODEL MANAGEMENT 8.1 outside your organization, whether those users
Acknowledgments
MODEL MANAGEMENT 8.2 use Databricks or not.
PRODUCT REFERENCE

Appendix:
Glossary
AWS | Azure | GCP

License DASF 52 Source code control

RISKS DESCRIPTION CONTROL CATEGORY

DATA PREP 2.1 MODEL 7.4 Databricks’ Git Repository integration supports
effective code and third-party libraries
Out-of-the-box
management, enhancing customer control over
their development environment.
PRODUCT REFERENCE

AWS | Azure | GCP

DASF 53 Third-party library control

RISKS DESCRIPTION CONTROL CATEGORY

ALGORITHMS 5.4 MODEL 7.3 Databricks’ library management system allows

MODEL 7.4 administrators to manage the installation and
Out-of-the-box
usage of third-party libraries effectively. This
DATABRICKS feature enhances the security and efficiency
AI SECURITY of systems, pipelines and data by giving PRODUCT REFERENCE
FRAMEWORK
administrators precise control over their
(DASF)
VERSION 1.0
development environment. AWS | Azure | GCP
65
 04
Conclusion

In an era defined by data-driven decision-making and intelligent automation, the importance

of AI security cannot be overstated. The Databricks AI Security Framework provides essential
guidance for securely developing, deploying and maintaining AI models at scale — and
Executive
Summary
ensuring they remain secure and continue to deliver business value. The emergence of AI
highlights the rapid advancement and specialized needs of its security. However, at its
Introduction heart, AI security is still rooted in the foundational principles of cybersecurity. Data teams
and security teams must actively collaborate to pursue their common goal of improving the
Risks in AI System
Components security of AI systems. Whether you are implementing traditional machine learning solutions
or LLM-driven applications, the core tenets of machine learning adoption remain constant:
Understanding
Databricks Data
Intelligence Platform
AI Risk Mitigation Databricks AI Security Framework (DASF)
Controls

⟶ Conclusion AI Business Risks Controls Data Platform

Use Case

Datasets
Resources and
Further Reading Stakeholders 1
Compliance Use case
identified
Applications
Acknowledgments

Appendix: AI Deploµment Select subset Select subset of Implement

Models 3 4
Glossary of DASF risks DASF controls controls on Data
(55 risks ⟶) ApplicaÍle (5W controls ]) ApplicaÍle Platform
risks controls
Predictive ML models identified identified

License Foundational APIs

Finetuned LLMs 2
Pretrained LLMs Deploèmenï
model
identified
Rag with LLMs

External Models

Figure 2: Implementation guidance of DASF controls on the Databricks Data Intelligence Platform..

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 66
 1 | Identify the AI business use case: Always keep your business goals in mind.
Make sure there is a well-defined use case with stakeholders you are trying to
secure adequately, whether already implemented or in planning phases. This
will help inform which AI system components are of greatest business value
for any given business use case

Executive
2 | Determine the AI deployment model: Choose an appropriate model (e.g.,
Summary predictive ML models, Foundation Model APIs, RAG LLMs, fine-tuned LLMs and
pretrained LLMs, as described in Section: 1.2 How to use this document) to
Introduction
determine how shared responsibilities (especially for securing each component)

Risks in AI System are split across the 12 ML/GenAI components between your organization, the
Components
Databricks Data Intelligence Platform and any partners involved.

Understanding
Databricks Data
3 | Select the most pertinent risks: From our documented list of 55 security
Intelligence Platform risks, pinpoint the ones most relevant to your organization based on the
AI Risk Mitigation
Controls outcome of step #2. Identify the specific threats linked to each risk and the
targeted ML/GenAI component for every threat.
⟶ Conclusion
4 | Choose and implement controls: Select controls that align with your
Resources and organization’s risk appetite. These controls are defined generically for
Further Reading
compatibility with any data platform. Our framework also provides guidelines on

Acknowledgments tailoring these controls specifically for the Databricks Data Intelligence Platform
with specific Databricks product references by cloud. You use these controls
Appendix: alongside your organization’s policies and have the right assurance in place.
Glossary

License

Databricks stands uniquely positioned as a secure, unified, data-centric platform for both
MLOps and LLMOps by taking a defense-in-depth approach to helping organizations
implement security across all AI system components. Red teaming and testing can help
iteratively improve and mitigate discovered weaknesses of models. As we embrace the
ongoing wave of AI advancements, it’s clear that employing a robust, secure MLOps
strategy will remain central to unlocking AI’s full potential. With firm, secure MLOps
foundations in place, organizations will be able to maximize their AI investments to drive
innovation and deliver business value.

A lot of care has been taken to make this whitepaper accurate; however, as AI is an evolving
field, please reach out to us if you have any feedback. If you’re interested in participating
in one of our AI Security workshops, please contact [email protected].
DATABRICKS
AI SECURITY If you are curious about how Databricks approaches security, please visit our
FRAMEWORK
(DASF) Security and Trust Center.
VERSION 1.0 67
 05
Resources and Further Reading

We have discussed many different capabilities in this document, with documentation links
where possible. Organizations that prioritize high security can learn more than what is in this
document. Here are additional resources to dive deeper:
Executive
Summary

AI and Machine Learning on Databricks

Introduction

Training Course: Generative AI Fundamentals ⟶

Risks in AI System
Components Webpage: AI and Machine Learning on Databricks ⟶
Industry Solutions: Solution Accelerators ⟶
Understanding
Databricks Data Blogs: Responsible AI ⟶ | AI/ML Blogs ⟶
Intelligence Platform
AI Risk Mitigation eBooks: Data, Analytics and AI Governance ⟶ | Big Book of MLOps 2nd Edition ⟶
Controls
Learning Library: Generative AI Engineering With Databricks ⟶

Conclusion

Databricks Unity Catalog

⟶ Resources and
Further Reading
Webpages: Databricks Unity Catalog ⟶ | AI Governance ⟶
Acknowledgments eBook: Data and AI Governance ⟶

Appendix: Databricks Platform Security

Glossary

License
Review the security features in the Security and Trust Center, along with the overall
documentation about the Databricks security and compliance programs.

The Security and Trust Overview Whitepaper provides an outline of the Databricks
architecture and platform security practices.

Databricks Platform Security Best Practices | AWS | Azure | GCP

Data Sharing and Collaboration

Webpage: Delta Sharing ⟶

eBook: Data Sharing and Collaboration With Delta Sharing ⟶
Blogs: What’s New in Data Sharing and Collaboration ⟶ | AI Model Sharing ⟶
DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 68
 Industry Resource

An Architectural Risk Analysis of Machine Learning Systems ⟶

NIST AI Risk Management Framework ⟶

MITRE ATLAS Adversarial ML ⟶

Executive OWASP Top 10 for LLMs ⟶
Summary
Guidelines for Secure AI System Development ⟶
Introduction Executive Order on the Safe, Secure, and Trustworthy Development and Use
of Artificial Intelligence ⟶
Risks in AI System
Components Generative AI Framework for HMG ⟶

NIST Adversarial Machine Learning:

Understanding A Taxonomy and Terminology of Attacks and Mitigations ⟶
Databricks Data
Intelligence Platform
AI Risk Mitigation
Secure by Design — Shifting the Balance of Cybersecurity Risk:
Controls Principles and Approaches for Secure by Design Software ⟶

Multilayer Framework for Good Cybersecurity Practices for AI ⟶

Conclusion

⟶ Resources and Third-Party Tools

Further Reading

Acknowledgments
Model scanners: HiddenLayer Model Scanner ⟶ | fickling ⟶ | ModelScan ⟶
AI Risk Database ⟶ | NB Defense ⟶
Appendix:
Glossary Model validation tools: Robust Intelligence continuous validation ⟶
Vigil LLM security scanner ⟶ | Garak automated scanning ⟶ | HiddenLayer MLDR ⟶
License
Citadel Lens ⟶

Guardrails for LLMs: NeMo Guardrails ⟶ | Guradrails AI ⟶ | Lakera Guard ⟶

Robust Intelligence AI Firewall ⟶ | Protect AI Guardian ⟶ | Arthur Shield ⟶
Laiyer LLM Guard ⟶ | Amazon Guardrails ⟶ | Meta Llama Guard ⟶
HiddenLayer AISec Platform ⟶

The information in this document does not constitute or imply endorsement or recommendation of any third-party organization,
product or service by Databricks. Links and references to websites and third-party materials are provided for informational
purposes only and do not represent endorsement or recommendation of such resources over others.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 69
 06
Acknowledgments

This whitepaper would not be possible without the insight and guidance provided by our
reviewers and contributors at Databricks and externally. Additionally, we extend our appreciation
to the frameworks that inspired our research (MITRE, OWASP, NIST, BIML, etc.), as they have
Executive
Summary
played a pivotal role in shaping the foundation of the Databricks AI Security Framework.

We would like to thank the following reviewers and contributors:

Introduction

DATABRICKS
Risks in AI System
Components

Understanding
Databricks Data
Intelligence Platform Matei Zaharia Fermín Serna Omar Khawaja Arun Pamulapati David Wells
Chief Technology Chief Security Vice President, Senior Staff Security Staff Security
AI Risk Mitigation
Officer and Co-Founder Officer Field CISO Field Engineer Field Engineer
Controls

Conclusion

Resources and Kelly Albano Erika Ehrli Abhi Arikapudi David Veuve Tim Lortz
Further Reading Product Marketing Senior Director Senior Director Head of Security Lead Specialist
Manager Product Marketing Security Engineering Field Engineering Solutions Architect

⟶ Acknowledgments

Appendix:
Glossary Joseph Bradley Arthur Dooner Veronica Gomes Jeffrey Hirschey Aliaksandra Nita
Principal ML Specialist Solutions Architect Senior Product Counsel Senior Technical
Product Specialist Solutions Architect Program Manager

License

NAVY FEDERAL
ROBUST INTELLIGENCE CREDIT UNION

Neil Archibald Hyrum Anderson Alie Fordyce Adam Swanda Riyaz Poonawala
Senior Staff Chief Technology Product Policy AI Security Researcher Vice President
Security Engineer Officer — Threat Intelligence Information Security

CARNEGIE MELLON UNIVERSITY PROTECT AI BARRACUDA META

Hasan Yasar Diana Kelley Grizel Lopez Brandon Sloane

Technical Director, Teaching Professor CISO Senior Director Risk Lead
Continuous Deployment of Capability Software of Engineering
Engineering Institute

CAPITAL ONE FINANCIAL HIDDENLAYER

Ebrima N. Ceesay, PhD, CISSP Christopher Sestito Abigail Maines Hiep Dang
Senior Distinguished Engineer Co-founder & CEO CRO VP of Strategic
Tech Alliances

HITRUST
DATABRICKS
AI SECURITY Robert Booker Jeremy Huval
FRAMEWORK EVP Strategy Chief Innovation
(DASF) Research and Innovation Center of Excellence Officer
and Chief Strategy Officer
VERSION 1.0 70
 07
Appendix: Glossary

A
Adversarial examples: Modified testing samples that induce misclassification of a machine
Executive
Summary learning model at deployment time.

AI governance: The actions to ensure stakeholder needs, conditions and options are
Introduction
evaluated to determine balanced, agreed-upon enterprise objectives; setting direction
Risks in AI System through prioritization and decision-making; and monitoring performance and compliance
Components
against agreed-upon directions and objectives. AI governance may include policies on the

Understanding nature of AI applications developed and deployed versus those limited or withheld.
Databricks Data
Intelligence Platform
AI Risk Mitigation Artificial intelligence (AI): A multidisciplinary field of computer science that aims to create
Controls
systems capable of emulating and surpassing human-level intelligence.

Conclusion

B
Resources and
Further Reading
Bug bounty program: A program that offers monetary rewards to ethical hackers for

Acknowledgments
successfully discovering and reporting a vulnerability or bug to the application’s developer.
Bug bounty programs allow companies to leverage the hacker community to improve their
⟶ Appendix: systems’ security posture over time.
Glossary

License
C
Compute plane: Where your data is processed in Databricks Platform architecture.

Concept drift: A situation where statistical properties of the target variable change and the
very concept of what you are trying to predict changes as well. For example, the definition
of what is considered a fraudulent transaction could change over time as new ways are
developed to conduct such illegal transactions. This type of change will result in concept drift.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 71
 Continuous integration and continuous delivery (or continuous deployment) (CI/CD):
CI is a modern software development practice in which incremental code changes are
made frequently and reliably. CI/CD is common to software development, but it is becoming
increasingly necessary to data engineering and data science. By automating the building,
testing and deployment of code, development teams are able to deliver releases more
frequently and reliably than with the manual processes still common to data engineering and
data science teams.

Executive
Control plane: The back-end services that Databricks manages in your Databricks account.
Summary
Notebook commands and many other workspace configurations are stored in the control
plane and encrypted at rest.
Introduction

Risks in AI System
Components
D
Data classification: A crucial part of data governance that involves organizing and
Understanding
Databricks Data categorizing data based on its sensitivity, value and criticality.
Intelligence Platform
AI Risk Mitigation
Controls Data drift: The features used to train a model are selected from the input data. When
statistical properties of this input data change, it will have a downstream impact on the
Conclusion
model’s quality. For example, data changes due to seasonality, personal preference changes,

Resources and trends, etc., will lead to incoming data drift.

Further Reading

Data governance: Data governance is a comprehensive approach that comprises the

Acknowledgments principles, practices and tools to manage an organization’s data assets throughout their
lifecycle. By aligning data-related requirements with business strategy, data governance
⟶ Appendix:
Glossary provides superior data management, quality, visibility, security and compliance capabilities
across the organization. Implementing an effective data governance strategy allows
License
companies to make data easily available for data-driven decision-making while safeguarding
their data from unauthorized access and ensuring compliance with regulatory requirements.

Data Intelligence Platform: A new era of data platform that employs AI models to deeply
understand the semantics of enterprise data. It builds the foundation of the data lakehouse —
a unified system to query and manage all data across the enterprise — but automatically
analyzes both the data (contents and metadata) and how it is used (queries, reports, lineage,
etc.) to add new capabilities.

Data lake: A central location that holds a large amount of data in its native, raw format.
Compared to a hierarchical data warehouse, which stores data in files or folders, a data lake
uses a flat architecture and object storage to store the data. With object storage, data is
stored with metadata tags and a unique identifier, which makes it easier to locate and retrieve
DATABRICKS
AI SECURITY data across regions and improves performance. By leveraging inexpensive object storage
FRAMEWORK
(DASF) and open formats, data lakes enable many applications to take advantage of the data.
VERSION 1.0 72
 Data lakehouse: A new, open data management architecture that combines the flexibility,
cost-efficiency and scale of data lakes with the data management and ACID transactions of
data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data.

Data lineage: A powerful tool that helps organizations ensure data quality and
trustworthiness by providing a better understanding of data sources and consumption. It
captures relevant metadata and events throughout the data’s lifecycle, providing an end-to-
end view of how data flows across an organization’s data estate.

Executive
Summary
Data partitioning: A partition is composed of a subset of rows in a table that share the same
value for a predefined subset of columns called the partitioning columns. Data partitioning
Introduction can speed up queries against the table as well as data manipulation.

Risks in AI System Data pipeline: A data pipeline implements the steps required to move data from source
Components
systems, transform that data based on requirements, and store the data in a target system. A

Understanding
data pipeline includes all the processes necessary to turn raw data into prepared data that
Databricks Data
Intelligence Platform
users can consume. For example, a data pipeline might prepare data so data analysts and
AI Risk Mitigation
Controls
data scientists can extract value from the data through analysis and reporting. An extract,
transform and load (ETL) workflow is a common example of a data pipeline.
Conclusion
Data poisoning: Attacks in which a part of the training data is under the control of the
Resources and adversary.
Further Reading

Data preparation (data prep): The set of preprocessing operations performed in the
Acknowledgments
early stages of a data processing pipeline, i.e., data transformations at the structural and
syntactical levels.
⟶ Appendix:
Glossary
Data privacy: Attacks against machine learning models to extract sensitive information
License about training data.

Data streaming: Data that is continuously and/or incrementally flowing from a variety of
sources to a destination to be processed and analyzed in near real-time. This unlocks a new
world of use cases around real-time ETL, real-time analytics, real-time ML and real-time
operational applications that in turn enable faster decision-making.

Databricks Delta Live Tables: A declarative framework for building reliable, maintainable and
testable data processing pipelines. You define the transformations to perform on your data
and Delta Live Tables manages task orchestration, cluster management, monitoring, data
quality and error handling.

Databricks Feature Store: A centralized repository that enables data scientists to find and
share features and also ensures that the same code used to compute the feature values is
DATABRICKS
AI SECURITY used for model training and inference.
FRAMEWORK
(DASF)
VERSION 1.0 73
 Databricks Secrets: Sometimes accessing data requires that you authenticate to external
data sources through Java Database Connectivity (JDBC). Databricks Secrets stores your
credentials so you can reference them in notebooks and jobs instead of directly entering
your credentials into a notebook.

Databricks SQL: The collection of services that bring data warehousing capabilities and
performance to your existing data lakes. Databricks SQL supports open formats and
standard ANSI SQL. An in-platform SQL editor and dashboarding tools allow team members

Executive
to collaborate with other Databricks users directly in the workspace. Databricks SQL also
Summary
integrates with a variety of tools so that analysts can author queries and dashboards in their
favorite environments without adjusting to a new platform.
Introduction

Databricks Workflows: Orchestrates data processing, machine learning and analytics

Risks in AI System
Components pipelines on the Databricks Data Intelligence Platform. Workflows has fully managed
orchestration services integrated with the Databricks Platform, including Databricks Jobs to
Understanding
Databricks Data
run non-interactive code in your Databricks workspace and Delta Live Tables to build reliable
Intelligence Platform
AI Risk Mitigation
and maintainable ETL pipelines.
Controls

Datasets: A dataset in machine learning and artificial intelligence refers to a collection of

Conclusion data that is used to train and test algorithms and models.

Resources and Delta Lake: The optimized storage layer that provides the foundation for storing data and
Further Reading
tables in the Databricks lakehouse. Delta Lake is open source software that extends Parquet

Acknowledgments
data files with a file-based transaction log for ACID transactions and scalable metadata
handling. Delta Lake is fully compatible with Apache Spark™ APIs, and was developed for
⟶ Appendix: tight integration with Structured Streaming, allowing you to easily use a single copy of data
Glossary
for both batch and streaming operations and providing incremental processing at scale.
License
Denial of service (DoS): An attack meant to shut down access to information systems,
devices or other network resources, making them inaccessible to their intended users. DoS
attacks accomplish this by flooding the target with traffic, or sending it information that
triggers a crash. In both instances, the DoS attack deprives legitimate users (i.e., employees,
members or account holders) of the service or resource they expected due to the actions of
a malicious cyberthreat actor.

DevSecOps: Stands for development, security and operations. It’s an approach to

culture, automation and platform design that integrates security as a shared responsibility
throughout the entire IT lifecycle.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 74
 E
Embeddings: Mathematical representations of the semantic content of data, typically
text or image data. Embeddings are generated by a large language model and are a key
component of many GenAI applications that depend on finding documents or images that
are similar to each other. Examples are RAG systems, recommender systems, and image and
video recognition.

Executive Exploratory data analysis (EDA): Methods for exploring datasets to summarize their
Summary
main characteristics and identify any problems with the data. Using statistical methods
and visualizations, you can learn about a dataset to determine its readiness for analysis
Introduction
and inform what techniques to apply for data preparation. EDA can also influence which
Risks in AI System algorithms you choose to apply for training ML models.
Components

External models: Third-party models hosted outside of Databricks. Supported by Model

Understanding
Databricks Data Serving, external models allow you to streamline the usage and management of various large
Intelligence Platform
AI Risk Mitigation language model (LLM) providers, such as OpenAI and Anthropic, within an organization.
Controls

Extract, transform and load (ETL): The foundational process in data engineering of
Conclusion combining data from multiple sources into a large, central repository called a data
warehouse. ETL uses a set of business rules to clean and organize raw data and prepare it for
Resources and
Further Reading storage, data analytics and machine learning (ML).

Acknowledgments
F
⟶ Appendix:
Glossary Feature engineering: The process of extracting features (characteristics, properties,
attributes) from raw data to develop machine learning models.
License

Fine-tuned LLM: Adapting a pretrained LLM to specific datasets or domains.

Foundation Model: A general purpose machine learning model trained on vast quantities of
data and fine-tuned for more specific language understanding and generation tasks.

G
Generative: Type of machine learning methods that learn the data distribution and can
generate new examples from distribution.

Generative AI: Also known as GenAI, this is a form of machine learning that uses large
quantities of data to train models to produce content.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 75
 H
Hardened runtime: Databricks handles the actual base system image (e.g., AMI) by
leveraging Ubuntu with a hardening configuration based on CIS. As a part of the Databricks
Threat and Vulnerability Management program, we perform weekly scanning of the AMIs as
they are making their way from dev to production.

Human-in-the-loop (HITL): The process of machine learning that allows people to validate

Executive a machine learning model’s predictions as right or wrong at the time of training and inference
Summary
with intervention.

Introduction Hyperparameter: A parameter whose value is set before the machine learning process
begins. In contrast, the values of other parameters are derived via training.
Risks in AI System
Components

Understanding
I
Databricks Data
Intelligence Platform
Identity provider (IdP): A service that stores and manages digital identities. Companies use
AI Risk Mitigation
Controls
these services to allow their employees or users to connect with the resources they need.
They provide a way to manage access, adding or removing privileges, while security remains
Conclusion
tight.

Resources and
Further Reading Inference: The stage of ML in which a model is applied to a task by running data points into a
machine learning model to calculate an output such as a single numerical score. For example,
Acknowledgments
a classifier model produces the classification of a test sample.

⟶ Appendix: Inference tables: A table that automatically captures incoming requests and outgoing
Glossary
responses for a model serving endpoint and logs them as a table.

License
Insider risk: An insider is any person who has or had authorized access to or knowledge of
an organization’s resources, including personnel, facilities, information, equipment, networks
and systems. Should an individual choose to act against the organization, with their privileged
access and their extensive knowledge, they are well positioned to cause serious damage.

IP access list (IP ACL): Enables you to restrict access to your AI system based on a user’s
IP address. For example, you can configure IP access lists to allow users to connect only
through existing corporate networks with a secure perimeter. If the internal VPN network is
authorized, users who are remote or traveling can use the VPN to connect to the corporate
network. If a user attempts to connect to the AI system from an insecure network, like from a
coffee shop, access is blocked.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 76
 J
Jailbreaking: An attack that employs prompt injection to specifically circumvent the safety
and moderation features placed on LLMs by their creators.

L
Label-flipping (LF) attacks: A targeted poisoning attack where the attackers poison their
Executive
training data by flipping the labels of some examples from one class (e.g., the source class)
Summary
to another (e.g., the target class).
Introduction
Lakehouse Monitoring: Databricks Lakehouse Monitoring lets you monitor the statistical

Risks in AI System properties and quality of the data in all of the tables in your account. You can also use
Components
it to track the performance of machine learning models and model serving endpoints by
monitoring inference tables that contain model inputs and predictions.
Understanding
Databricks Data
Intelligence Platform
Large language model (LLM): A model trained on massive datasets to achieve advanced
AI Risk Mitigation
Controls
language processing capabilities based on deep learning neural networks.

Conclusion LLM-as-a-judge: A scalable and explainable way to approximate human preferences, which
are otherwise very expensive to obtain. Evaluating large language model (LLM) based chat
Resources and
Further Reading assistants is challenging due to their broad capabilities and the inadequacy of existing
benchmarks in measuring human preferences. Use LLMs as judges to evaluate these models
Acknowledgments
on more open-ended questions.

⟶ Appendix: LLM hallucination: A phenomenon wherein a large language model (LLM) — often a
Glossary
generative AI chatbot or computer vision tool — perceives patterns or objects that are

License nonexistent or imperceptible to human observers, creating outputs that are nonsensical or
altogether inaccurate.

M
Machine learning (ML): A form of AI that learns from existing data and makes predictions
without being explicitly programmed.

Machine learning algorithms: Pieces of code that help people explore, analyze and find
meaning in complex datasets. Each algorithm is a finite set of unambiguous step-by-step
instructions that a machine can follow to achieve a certain goal. In a machine learning model,
the goal is to establish or discover patterns that people can use to make predictions or
categorize information.
DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 77
 Machine learning models: Process of using mathematical models of data to help a
computer learn without direct instruction. Machine learning uses algorithms to identify
patterns within data, and those patterns are then used to create a data model that can make
predictions. For example, in natural language processing, machine learning models can parse
and correctly recognize the intent behind previously unheard sentences or combinations of
words. In image recognition, a machine learning model can be taught to recognize objects —
such as cars or dogs. A machine learning model can perform such tasks by having it “trained”
with a large dataset. During training, the machine learning algorithm is optimized to find
Executive
Summary
certain patterns or outputs from the dataset, depending on the task. The output of this
process — often a computer program with specific rules and data structures — is called a
Introduction machine learning model.

Risks in AI System Machine learning operations (MLOps): The practice of creating new machine learning (ML)
Components
models and running them through a repeatable, automated workflow that deploys them to

Understanding
production. An MLOps pipeline provides a variety of services to data science processes,
Databricks Data
Intelligence Platform
including model version control, continuous integration and continuous delivery (CI/CD),
AI Risk Mitigation
Controls
model catalogs for models in production, infrastructure management, monitoring of live
model performance, security, and governance. MLOps is a collaborative function, often
Conclusion comprising data scientists, devops engineers, security teams and IT.

Resources and Malicious libraries: Software components that were intentionally designed to cause harm
Further Reading
to computer systems or the data they process. Such packages can be distributed through

Acknowledgments
various means, including phishing emails, compromised websites or even legitimate
software repositories.
⟶ Appendix:
Glossary Metadata: Data that annotates other data and AI assets. It generally includes the
permissions that govern access to them with descriptive information, possibly including their
License
data descriptions, data about data ownership, access paths, access rights and data volatility.

MLflow Model Registry: A centralized model store, set of APIs, and UI to collaboratively
manage the full lifecycle of an MLflow model. It provides model lineage (which MLflow
experiment and run produced the model), model versioning, model aliasing, model tagging
and annotations.

MLSecOps: The integration of security practices and considerations into the ML

development and deployment process. This includes ensuring the security and privacy
of data used to train and test models, as well as protecting deployed models and the
infrastructure they run on from malicious attacks.

Model drift: The decay of models’ predictive power as a result of the changes in real-
DATABRICKS world environments.
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 78
 Model inference: The use of a trained model on new data to create a result.

Model inversion: In machine learning models, private assets like training data, features and
hyperparameters, which are typically confidential, can potentially be recovered by attackers
through a process known as model inversion. This technique involves reconstructing private
elements without direct access, compromising the model’s security.

Model management: A single place for development, tracking, discovering, governing,

encrypting and accessing models with proper security controls.
Executive
Summary
Model operations: The building of predictive ML models, the acquisition of models from a
model marketplace, or the use of LLMs like OpenAI or Foundation Model APIs. Developing a
Introduction
model requires a series of experiments and a way to track and compare the conditions and
Risks in AI System results of those experiments.
Components

Model Zoo: A repository or library that contains pretrained models for various machine
Understanding
Databricks Data
learning tasks. These models are trained on large datasets and are ready to be deployed or
Intelligence Platform
AI Risk Mitigation
fine-tuned for specific tasks.
Controls

Mosaic AI AutoML: Helps you automatically apply machine learning to a dataset. You
Conclusion provide the dataset and identify the prediction target, while AutoML prepares the dataset
for model training. AutoML then performs and records a set of trials that creates, tunes and
Resources and
Further Reading evaluates multiple models. After model evaluation, AutoML displays the results and provides
a Python notebook with the source code for each trial run so you can review, reproduce and
Acknowledgments
modify the code. AutoML also calculates summary statistics on your dataset and saves this
information in a notebook that you can review later.
⟶ Appendix:
Glossary
Mosaic AI Model Serving: A unified service for deploying, governing, querying and
License monitoring models fine-tuned or pre-deployed by Databricks like Llama 2, MosaicML MPT or
BGE, or from any other model provider like Azure OpenAI, AWS Bedrock, AWS SageMaker and
Anthropic. Model Serving provides a highly available and low-latency service for deploying
models. The service automatically scales up or down to meet demand changes, saving
infrastructure costs while optimizing latency performance.

Mosaic AI Vector Search: A vector database that is built into the Databricks Data
Intelligence Platform and integrated with its governance and productivity tools. A vector
database is a database that is optimized to store and retrieve embeddings. Embeddings are
mathematical representations of the semantic content of data, typically text or image data.
Embeddings are generated by a large language model and are a key component of many
GenAI applications that depend on finding documents or images that are similar to each
other. Examples are RAG systems, recommender systems, and image and video recognition.
DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 79
 Model theft: Theft of a system’s knowledge through direct observation of its input and
output observations, akin to reverse engineering. This can lead to unauthorized access,
copying or exfiltration of proprietary models, resulting in economic losses, eroded
competitive advantage and exposure of sensitive information.

N
Notebook: A common tool in data science and machine learning for developing code and
Executive presenting results.
Summary

Introduction
O
Risks in AI System
Components
Offline system: ML systems that are trained up, “frozen,” and then operated using new data
on the frozen trained system.
Understanding
Databricks Data Online system: An ML system is said to be “online” when it continues to learn during
Intelligence Platform
AI Risk Mitigation operational use, modifying its behavior over time.
Controls

Ontology: A formally defined vocabulary for a particular domain of interest used to capture
Conclusion
knowledge about that (restricted) domain of interest. Adversaries may discover the ontology

Resources and
of a machine learning model’s output space — for example, the types of objects a model can
Further Reading
detect. The adversary may discover the ontology by repeated queries to the model, forcing
it to enumerate its output space. Or the ontology may be discovered in a configuration file or
Acknowledgments
in documentation about the model.
⟶ Appendix:
Glossary
P
License
Penetration testing (pen testing): A security exercise where a cybersecurity expert
attempts to find and exploit vulnerabilities in a computer system through a combination of
an in-house offensive security team, qualified third-party penetration testers and a year-
round public bug bounty program. The purpose of this simulated attack is to identify any
weak spots in a system’s defenses that attackers could take advantage of.

Pretrained LLM: Training an LLM from scratch using your own data for better domain
performance.

Private link: Enables private connectivity between users and their Databricks workspaces
and between clusters on the compute plane and core services on the control plane within
the Databricks workspace infrastructure.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 80
 Prompt injection
Direct: A direct prompt injection occurs when a user injects text that is intended to alter
the behavior of the LLM
Indirect: When a user might modify or exfiltrate resources (e.g., documents, web pages)
that will be ingested by the GenAI model at runtime via the RAG process.

R
Executive Red teaming: NIST defines cybersecurity red teaming as “a group of people authorized
Summary
and organized to emulate a potential adversary’s attack or exploitation capabilities
Introduction against an enterprise’s security posture. The Red Team’s objective is to improve enterprise
cybersecurity by demonstrating the impacts of successful attacks and by demonstrating
Risks in AI System
Components what works for the defenders (i.e., the Blue Team) in an operational environment.” (CNSS
2015 [80]) Traditional red teaming might combine physical and cyberattack elements,
Understanding attack multiple systems, and aim to evaluate the overall security posture of an organization.
Databricks Data
Intelligence Platform Penetration testing (pen testing), in contrast, tests the security of a specific application
AI Risk Mitigation
Controls or system. In AI discourse, red teaming has come to mean something closer to pen testing,
where the model may be rapidly or continuously tested by a set of evaluators and under
Conclusion
conditions other than normal operation.

Resources and
Further Reading Reinforcement learning from human feedback (RLHF): A method of training AI models
where human feedback is used as a source of reinforcement signals. Instead of relying solely
Acknowledgments on predefined reward functions, RLHF incorporates feedback from humans to guide the
learning process.
⟶ Appendix:
Glossary
Resource control: A capability in which the attacker has control over the resources

License consumed by an ML model, particularly for LLMs and RAG applications.

Responsible AI: Responsible Artificial Intelligence (Responsible AI) is an approach to

developing, assessing and deploying AI systems in a safe, trustworthy and ethical way.
Characteristics of trustworthy AI systems include: valid and reliable, safe, secure and resilient,
accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with
harmful bias managed.

Retrieval augmented generation (RAG): An architectural approach that can improve the
efficacy of large language model (LLM) applications by leveraging custom data. This is done
by retrieving data/documents relevant to a question or task and providing them as context
for the LLM.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 81
 S
Serverless compute: An architectural design that follows infrastructure as a service (IaaS)
and platform as a service (PaaS), and which primarily requires the customer to provide
the necessary business logic for execution. Meanwhile, the service provider takes care of
infrastructure management. Compared to other platform architectures like PaaS, serverless
provides a considerably quicker path to realizing value and typically offers better cost
efficiency and performance.
Executive
Summary
Single-sign on (SSO): A user authentication tool that enables users to securely access
multiple applications and services using just one set of credentials.
Introduction

Software development lifecycle (SDLC): A structured process that enables the production
Risks in AI System
Components of high-quality, low-cost software, in the shortest possible production time. The goal of the
SDLC is to produce superior software that meets and exceeds all customer expectations
Understanding
Databricks Data and demands. The SDLC defines and outlines a detailed plan with stages, or phases, that
Intelligence Platform
AI Risk Mitigation each encompasses their own process and deliverables. Adherence to the SDLC enhances
Controls
development speed and minimizes project risks and costs associated with alternative
methods of production.
Conclusion

Source code control: A capability in which the attacker has control over the source code of
Resources and
Further Reading the machine learning algorithm.

Acknowledgments System for Cross-domain Identity Management (SCIM): An open standard designed to
manage user identity information. SCIM provides a defined schema for representing users
⟶ Appendix:
and groups, and a RESTful API to run CRUD operations on those user and group resources.
Glossary
The goal of SCIM is to securely automate the exchange of user identity data between your
License company’s cloud applications and any service providers, such as enterprise SaaS applications.

T
Train proxy: The ability of an attacker to extract training data of a generative model by
prompting the model on specific inputs.

Train proxy via replication: Adversaries may replicate a private model. By repeatedly
querying the victim’s ML Model Inference API Access, the adversary can collect the target
model’s inferences into a dataset. The inferences are used as labels for training a separate
model offline that will mimic the behavior and performance of the target model.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 82
 Trojan: A malicious code/logic inserted into the code of a software or hardware system,
typically without the knowledge and consent of the organization that owns/develops the
system, and which is difficult to detect and may appear harmless, but can alter the intended
function of the system upon a signal from an attacker to cause a malicious behavior desired
by the attacker. For Trojan attacks to be effective, the trigger must be rare in the normal
operating environment so that it does not affect the normal effectiveness of the AI and raise
the suspicions of human users.

Executive
Trojan horse backdoor: In the context of adversarial machine learning, the term “backdoor”
Summary
describes a malicious module injected into the ML model that introduces some secret and
unwanted behavior. This behavior can then be triggered by specific inputs, as defined by
Introduction
the attacker.
Risks in AI System
Components
U
Understanding
Databricks Data Unity Catalog (UC): A unified governance solution for data and AI assets on the Databricks
Intelligence Platform
AI Risk Mitigation Data Intelligence Platform. It provides centralized access control, auditing, lineage and data
Controls
discovery capabilities across Databricks workspaces.
Conclusion

Resources and
V
Further Reading
Vulnerability management: An information security continuous monitoring (ISCM) process
Acknowledgments of identifying, evaluating, treating and reporting on security vulnerabilities in systems and the
software that runs on them. This, implemented alongside other security tactics, is vital for
⟶ Appendix:
Glossary organizations to prioritize possible threats and minimizing their “attack surface.”

License
W
Watering hole attacks: A form of cyberattack that targets groups of users by infecting
websites that they commonly visit to gain access to the victim’s computer and network.

Webhooks: Enable you to listen for Model Registry events so your integrations can
automatically trigger actions. You can use webhooks to automate and integrate your machine
learning pipeline with existing CI/CD tools and workflows. For example, you can trigger CI
builds when a new model version is created or notify your team members through Slack each
time a model transition to production is requested.

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF)
VERSION 1.0 83
 08
License

This work is licensed under the Creative Commons Attribution-Share Alike 4.0 License.

Click here to view a copy of this license or send a letter to:

Executive
Summary
Creative Commons
171 Second Street, Suite 300
Introduction San Francisco, California, 94105
USA
Risks in AI System
Components

Understanding
Databricks Data
Intelligence Platform
AI Risk Mitigation
Controls

Conclusion

Resources and
Further Reading

Acknowledgments
About Databricks Security & Trust Center
Databricks is the data and AI company Your data security is our priority
Appendix:
Glossary
Learn more ⟶ Learn more ⟶

⟶ License

Databricks is the data and AI company. More than 10,000 organizations worldwide —
including Comcast, Condé Nast, Grammarly and over 50% of the Fortune 500 — rely on the
Databricks Data Intelligence Platform to unify and democratize data, analytics and AI.

Databricks is headquartered in San Francisco, with offices around the globe, and was
founded by the original creators of Lakehouse, Apache Spark™, Delta Lake and MLflow.

To learn more, follow Databricks on LinkedIn, X and Facebook.

Evaluate Databricks for yourself. Visit us at databricks.com and try Databricks free!

DATABRICKS
AI SECURITY
FRAMEWORK
(DASF) © Databricks 2024. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.
VERSION 1.0 Privacy Policy | Terms of Use 84