2

E
D

N
D
IT
IO
N
eBook

The Big Book

of MLOps
NOW INCLUDING A
SECTION ON LLMOPS

M O D E LO P S DATAO P S DEVOPS

J O S E P H B R A D L E Y | R A F I K U R L A N S I K | M AT T T H O M S O N | N I A L L T U R B I T T
 2

Contents Introduction ................................................................................................................................................................................................. 5

Big Book of MLOps V1 Recap ............................................................................................................................................................. 6

Why should I care about MLOps? ............................................................................................................................................................................................................................................ 6

Guiding principles ............................................................................................................................................................................................................................................................................... 6

Semantics of development, staging and production ............................................................................................................................................................................................... 7

ML deployment patterns ............................................................................................................................................................................................................................................................... 8

What’s New? ............................................................................................................................................................................................. 10

Unity Catalog ........................................................................................................................................................................................................................................................................................ 10
............................................................................................................................................................................................................................. 11

Model Serving ........................................................................................................................................................................................................................................................................................ 13

.................................................................................................................................................................................................................. 13

Lakehouse Monitoring ..................................................................................................................................................................................................................................................................... 15

................................................................................................................................................................................................................. 15

Design Decisions ...................................................................................................................................................................................... 17

Unity Catalog .......................................................................................................................................................................................................................................................................................... 17
..................................................................................................................................................................................................................................... 17
................................................................................................................................................................................................................................................................................... 18
....................................................................................................................................................................................................................................................................... 21
....................................................................................................................................................................................................................................... 23

Model Serving........................................................................................................................................................................................................................................................................................ 27
................................................................................................................................................................................................................................................ 28
R .................................................................................................................................................................................................................................... 29
........................................................................................................................................................................................................................................ 30
 3

Contents Reference Architecture ..................................................................................................................................................................... 31

Multi-environment view .............................................................................................................................................................................................................................................................. 32

Development ....................................................................................................................................................................................................................................................................................... 34
.............................................................................................................................................................................................................................................................................................. 35
............................................................................................................................................................................................................................... 35
........................................................................................................................................................................................................................................................................... 36
...................................................................................................................................................................................................................................... 36
............................................................................................................................................................................................ 37
........................................................................................................................................................................................................................................................................ 38

Staging ....................................................................................................................................................................................................................................................................................................... 39
.............................................................................................................................................................................................................................................................................................. 40
............................................................................................................................................................................................................................................................................. 40
........................................................................................................................................................................................................................................................ 40
............................................................................................................................................................................................................................................................................................. 41
............................................................................................................................................................................................................................................................... 41

Production .............................................................................................................................................................................................................................................................................................. 42
....................................................................................................................................................................................................................................................................... 44
................................................................................................................................................................................................................................................................... 45
............................................................................................................................................................................................................................................................ 46
....................................................................................................................................................................................................................................................................... 48
................................................................................................................................................................................................................................... 48
..................................................................................................................................................................................................................................................... 49
................................................................................................................................................................................................................................................................................ 49

Implementing MLOps on Databricks ................................................................................................................................................................................................................................. 50

 4

Contents LLMOps ........................................................................................................................................................................................................ 51

What changes with LLMs?........................................................................................................................................................................................................................................... 51

Key components of LLM-powered applications....................................................................................................................................................................................................... 54

............................................................................................................................................................................................................................................................ 54
.............................................................................................................................................................................................................................................. 56
................................................................................................................................................................................................................ 58
.............................................................................................................................................................................................................................................. 59
................................................................................................................................................................................................................................................................... 60
.................................................................................................................................................................................... 61
................................................................................................................................................................................................................................................................. 62
..................................................................................................................................................................................................................................... 63
........................................................................................................................................................................................................................................... 63
.............................................................................................................................................................................................................................................................................. 64
................................................................................................................................................................................................................................... 64
.......................................................................................................................................................................................................................................... 65
.............................................................................................................................................................................................................. 66
....................................................................................................................................................................................................................................................................67
..................................................................................................................................................................................................................................................... 69
.......................................................................................................................................................................................................................... 69
...............................................................................................................................................................................................70
............................................................................................................................................................................................................................................................................ 71
..................................................................................................................................................................................................................................................... 71
.............................................................................................................................................................................................................................................................. 71
................................................................................................................................................................................................................................. 72
...................................................................................................................................................................................................... 72
........................................................................................................................................................................................... 73

Reference architecture................................................................................................................................................................................................................................................................... 74
................................................................................................................................................................................................................................... 74
.............................................................................................................................................................................................................................75

Conclusion ................................................................................................................................................................................................................ 78
 5

CH A PT E R 1

Introduction

+ +

.
 6

CH A PT E R 2

Big Book of MLOps

V1 Recap

Why should I care about MLOps?

Guiding principles
 7

Semantics of development, staging and production

EXECUTION ENVIRONMENT

Note:

D EVELOPED T EST ED D EPLOY ED

Each of these stages has distinct access controls and quality guarantees
 8

ML deployment patterns
Code and models often progress asynchronously

D E P LOY C O D E
 9

D E P LOY M O D E LS

recommend a deploy code approach for the majority of use cases

 10

CH A PT E R 3

What’s New?

Unity Catalog

UN I TY CATALO G
 11

BE NE FITS A ND A RCH IT ECT UR E I M P L I CAT I O NS

Read access to production assets

Models in Unity Catalog

M
 12

Lineage

Discoverability
 13

Model Serving

B E N E FITS AN D A RC H ITECTU R E I M PLI CATI O N S

Lakehouse native
 14

High availability and scalability

Online evaluation

Secure and cost effective

 15

Lakehouse Monitoring

B E N E FITS AN D A RC H ITECTU R E I M PLI CATI O N S

Lakehouse native

Monitoring with Model Serving

 16

Customization

Dashboards and alerts

 17

CH A PT E R 4

Design Decisions

Unity Catalog

O RG A N IZ I N G DATA A N D AI AS S E TS

Scalability
 18

Governance

Collaboration

CO N C E P TS

U N I T Y CATA LO G

CATA LO G

SCHEMA
 19

Catalog

Schema

Data tables

Volume

.
 20

Functions

Registered model

Model version

Model alias

.
 21

CO N S I D E R ATI O N S

Team size

Complexity
 of projects
 22

Access
 levels and permissions

Models,
 functions and features in Unity Catalog

Discoverability
 23

R ECO M M E N D E D O RG AN IZ ATI O N
 24

Catalog level






















 25

Schema level

TA B LE S



MODELS

VO LU M E S
 26

TA B LE S

F U N CT I O N S
 27

Model Serving

1 Pre-deployment testing

2 Real-time model deployment

 28

PRE -DE PLOYME NT TE ST I NG

Deployment readiness checks

Load testing

L AT E N CY

THROUGHPUT

S TA N DA R D LOA D E VA LUAT I O N

S T R E S S AS S E S S M E N T
 29

RE A L-TIME MODE L DEP LOY M ENT

A/B
 testing

R E L AT I O N TO OT H E R PAT T E R N S :

Gradual
 rollout

R E L AT I O N TO OT H E R PAT T E R N S :

Shadow
 deployment

R E L AT I O N TO OT H E R PAT T E R N S :
 30

Implementing in Databricks

 O D E L A L I AS E S
M

C O N T R O L E N D P O I N T T R A F F I C

L A K E H O U S E M O N I TO R I N G W I T H M O D E L S E RV I N G
 31

CH A PT E R 5

Reference
Architecture
Multi-environment view