0% found this document useful (0 votes)
42 views

DynamoDB Data Modelling

This document contains a presentation on advanced data modeling with Amazon DynamoDB. The presentation covers DynamoDB basics like tables, items, primary keys and attributes. It discusses different types of primary keys and how they impact data access and querying capabilities. The presentation also covers unique aspects of DynamoDB like how it partitions data and the use of secondary indexes to enable additional query patterns beyond the primary key.

Uploaded by

Anupam Raj
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
42 views

DynamoDB Data Modelling

This document contains a presentation on advanced data modeling with Amazon DynamoDB. The presentation covers DynamoDB basics like tables, items, primary keys and attributes. It discusses different types of primary keys and how they impact data access and querying capabilities. The presentation also covers unique aspects of DynamoDB like how it partitions data and the use of secondary indexes to enable additional query patterns beyond the primary key.

Uploaded by

Anupam Raj
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 223

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

DAT410

Advanced data modeling with


Amazon DynamoDB

Alex DeBrie
Principal
DeBrie Advisory

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

• Amazon DynamoDB basics

• Unique aspects of DynamoDB

• Data modeling walkthrough

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related talks

• DAT329: Data modeling core concepts for Amazon DynamoDB

• DAT330: Deep dive into Amazon DynamoDB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Alex DeBrie

• AWS Data Hero

• Author, The DynamoDB Book

• Independent consultant

dynamodbbook.com

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB basics

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB basics

• Table
• Item
• Primary key
• Attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Item

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Primary keys

• Simple primary key (partition key)


• Composite primary key (partition key + sort key)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

Item

Item

Item

Item

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key
Combination must be unique

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

Item collection

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Primary key design is crucial
to DynamoDB success

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition Partition Partition
1 4 7

PutItem:
CustomerEmail:“alexdebrie1..” fx(CustomerEmail):
BookingId: “01FFAB…” Partition Partition Partition
This item belongs to 2 5 8
Partition 1

Partition Partition Partition


Constant time 3 6 9

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Primary keys

• Simple primary key (partition key)


• Composite primary key (partition key + sort key)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple primary key

• Only key-value operations


• Operations on individual items
▪ GetItem
▪ PutItem, UpdateItem, DeleteItem
• Must provide entire primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition Partition Partition
1 4 7

PutItem:
CustomerEmail:“alexdebrie1..” fx(CustomerEmail):
BookingId: “01FFAB…” Partition Partition Partition
This item belongs to 2 5 8
Partition 1

Partition Partition Partition


3 6 9

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

Sorted by sort key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key Query

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key Query

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

Item collection

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

• Key-value operations + Query operation


• Query: read a range of items within an item collection
▪ Must include partition key
▪ May include conditions on sort key
▪ Good for “List” operations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Secondary index

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Secondary indexes

• Data from your main table is copied into a secondary index


▪ New partition key + sort key
▪ Allows for additional, read-based access patterns
• Choose which attributes are projected into secondary index
▪ ALL
▪ KEYS_ONLY
▪ INCLUDE

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB basics – review

• Key terminology
• Types of primary keys and their implications
• Secondary indexes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition Partition Partition
1 4 7

PutItem:
CustomerEmail:“alexdebrie1..” fx(CustomerEmail):
BookingId: “01FFAB…” Partition Partition Partition
This item belongs to 2 5 8
Partition 1

Partition Partition Partition


3 6 9

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lots of databases partition!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions X 3 replicas

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions X 3 replicas == 3,000 storage nodes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions X 3 replicas == 3,000 storage nodes
▪ 2TB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions X 3 replicas == 3,000 storage nodes
▪ 2TB / 10GB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in DynamoDB

• DynamoDB infrastructure shared across an entire Region**


• Math:
▪ 10TB / 10GB == 1,000 partitions X 3 replicas == 3,000 storage nodes
▪ 2TB / 10GB == 200 partitions per storage node

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Everything
fails, all
the time

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Photo by Guido van Nispen / CC BY 2.0
Lots of databases partition!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition Partition Partition
1 4 7

PutItem:
CustomerEmail:“alexdebrie1..” fx(CustomerEmail):
BookingId: “01FFAB…” Partition Partition Partition
This item belongs to 2 5 8
Partition 1

Partition Partition Partition


3 6 9

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple primary key

• Only key-value mode


• Operations on individual items
▪ GetItem
▪ PutItem, UpdateItem, DeleteItem
• Must provide entire primary key

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Composite primary key

• Key-value or wide-column mode


• Query: read a range of items within an item collection
▪ Must include partition key
▪ May include sort key to limit range
▪ Good for “List” operations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://twitter.com/floydophone/status/1708567162280112392

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition Partition Partition
1 4 7

PutItem:
CustomerEmail:“alexdebrie1..” fx(CustomerEmail):
BookingId: “01FFAB…” Partition Partition Partition
This item belongs to 2 5 8
Partition 1

Partition Partition Partition


3 6 9

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partition 1 Replica Group

Leader

Replica Replica

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is a partition?

• Replica group with three storage nodes


▪ 1 leader, two replicas
• All writes originate at leader
• Reads can go to any node
▪ Use “ConsistentRead=True” to read from leader node

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

• Operation-based pricing

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB billing

• Based on operations, not resources


• Write operations: 1 WCU per 1KB written
• Read operations: 1 RCU per 4KB read

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

• Operation-based pricing

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB
hides operational complexity

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

• Operation-based pricing

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB
hides operational complexity

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB
hides operational complexity
but
reveals performance costs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

• Operation-based pricing

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB
hides operational complexity
but
reveals performance costs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data modeling walkthrough

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?
▪ How big are your items?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?
▪ How big are your items?
• Know the DynamoDB basics

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?
▪ How big are your items?
• Know the DynamoDB basics
▪ Single-item actions for core operations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?
▪ How big are your items?
• Know the DynamoDB basics
▪ Single-item actions for core operations
▪ Query for ‘Find many’ needs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Before we start

• Know your access patterns


• Know your domain
▪ What are your constraints?
▪ What’s your data distribution?
▪ How big are your items?
• Know the DynamoDB basics
▪ Single-item actions for core operations
▪ Query for ‘Find many’ needs
▪ Batch + Transaction operations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 1: SearchFlightOptions

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin Destination

Date

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin Destination Date

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Knowing your domain

• Not always a direct flight


• Basic statistics: ~5000 flights / day to ~350 airports
• “Hub-and-spoke” model
▪ ~6-8 hubs that handle most traffic

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options
1. If origin is not a hub, find routes from origin to hub

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options
1. If origin is not a hub, find routes from origin to hub

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options
1. If origin is not a hub, find routes from origin to hub
2. If destination is not a hub, find routes from hub to destination

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options
1. If origin is not a hub, find routes from origin to hub
2. If destination is not a hub, find routes from hub to destination
3. If neither are hubs, find routes between hubs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Approaching the problem

1. Find any direct routes between the airports


2. Generate connecting options
1. If origin is not a hub, find routes from origin to hub
2. If destination is not a hub, find routes from hub to destination
3. If neither are hubs, find routes between hubs
3. Return candidates

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Doing the math

• Flight item record: ~1.5KB


• Fetching direct connections: ~2 flights == 3 KB == 0.5 RCUs (EC)
• Fetching origin to hubs: ~4 flights == 6KB == 1 RCU (EC) * 7 hubs == 7 RCUs
• Fetching hub to destination: ~4 flights == 6KB == 1 RCU (EC) * 7 hubs == 7
RCUs

Total: ~14.5 RCUs


($0.25 per 70,000 executions)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Is there a better way?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Doing the math

• Flight item record: ~1.5KB


• Fetching direct connections: ~2 flights == 3 KB == 0.5 RCUs (EC)
• Fetching origin to hubs: ~4 flights == 6KB == 1 RCU (EC) * 7 hubs == 7 RCUs
• Fetching hub to destination: ~4 flights == 6KB == 1 RCU (EC) * 7 hubs == 7
RCUs

Total: ~14.5 RCUs


($0.25 per 70,000 executions)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Flight record item

• Two pieces:
▪ Route data (Origin, destination, date, departure time)
▪ Current flight status (seats available, pricing)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Doing the math part 2

• Flight metadata record: 100 bytes


▪ 5000 flights per day == 50KB
▪ 180 days of history == 50KB * 180 == 9 MB

• DynamoDB math:
▪ 10 routes with 2 legs per route == 20 flights
▪ GetItem @ 1.5KB = 0.5KB (EC) * 20 flights == 10 RCUs

Total: 10 RCUs
($0.25 per 100,000 executions)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways

• Not all problems break down neatly into database queries


• Use your domain knowledge to find a solution that works
• Do the math!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 2: BookFlight

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight requirements

• Only confirm booking if seats are available


▪ Customer may choose seats when booking
• A flight may contain multiple legs
• After confirming flight, credit user’s frequent flyer points

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: Maintaining constraints

• All write operations allow ConditionExpressions


▪ If it evaluates to False, the write is rejected

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: Maintaining constraints

• All write operations allow ConditionExpressions


▪ If it evaluates to False, the write is rejected

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: Maintaining constraints

• All write operations allow ConditionExpressions


▪ If it evaluates to False, the write is rejected
• Structure your items to allow for ConditionExpressions to maintain conditions

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: Maintaining constraints

• All write operations allow ConditionExpressions


▪ If it evaluates to False, the write is rejected
• Structure your items to allow for ConditionExpressions to maintain conditions
▪ Avoid the read-modify-write cycle

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

Book with specific seats

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap


Book with specific seats
- Decrement AvailableSeats

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific


seats

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific - Increment HeldSeats


seats - Decrement AvailableSeats

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific - Increment HeldSeats


- AvailableSeats >= seats booked
seats - Decrement AvailableSeats

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific - Increment HeldSeats


- AvailableSeats >= seats booked
seats - Decrement AvailableSeats

Select specific seats


from prior purchase

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific - Increment HeldSeats


- AvailableSeats >= seats booked
seats - Decrement AvailableSeats

Select specific seats - Decrement HeldSeats


from prior purchase - Update seats in ClaimedSeatMap

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight workflows

Operation Updates ConditionExpression

- Update seats in ClaimedSeatMap - ClaimedSeatMap seats available


Book with specific seats
- Decrement AvailableSeats - AvailableSeats >= seats booked

Book without specific - Increment HeldSeats


- AvailableSeats >= seats booked
seats - Decrement AvailableSeats

Select specific seats - Decrement HeldSeats


- ClaimedSeatMap seats available
from prior purchase - Update seats in ClaimedSeatMap

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight requirements

• Only confirm booking if seats are available Condition


▪ Customer may choose seats when booking Expressions
• A flight may contain multiple legs
• After confirming flight, credit user’s frequent flyer points

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: full flow

1. Reserve seats for all legs of flight


1. If fails, reject booking
2. Process payment for flight
1. If fails, rollback seat reservations + reject booking
3. Create trip record for user
4. Credit frequent flyer points

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DynamoDB transactions

• Process multiple operations in single, atomic request


• Each operation can have a condition expression
• But: no long-running transactions!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BookFlight: full flow

1. Reserve seats for all legs of flight DynamoDB Transaction!


1. If fails, reject booking
2. Process payment for flight External system!
1. If fails, rollback seat reservations + reject booking
3. Create trip record for user
4. Credit frequent flyer points

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Write operation takeaways

• Use ConditionExpressions to maintain constraints


▪ Structure your items to allow for direct operations
• Use TransactWriteItem if necessary for multi-item operations
▪ But: cost, latency, ‘single-shot’
• For long-running transactions, use client-side transactions / Step Functions
▪ Limit/avoid where possible
• Use Amazon EventBridge or Amazon DynamoDB Streams for asynchronous
updates

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 3: Complex filtering

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering

• Filtering on 2+ attributes, each of which is optional


• This is a hard problem in DynamoDB!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unique aspects of DynamoDB

• Partitioning?

• Multi-tenant

• Every request is an index hit

• Eventually consistent reads by default

• Operation-based pricing

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
1. Left-to-right
2. No gaps/skipping
3. Up to and including the first range

Baron Schwartz, The Left-Prefix Index Rule


https://orangematter.solarwinds.com/2019/02/05/the-left-prefix-index-rule/

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin Destination Class

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering

• Filtering on 2+ attributes, each of which is optional


• This is a hard problem in DynamoDB!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering

• Filtering on 2+ attributes, each of which is optional


• This is a hard problem in DynamoDB!
• Know your domain and do the math!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB == 60KB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB == 60KB / 4 == 15 RCUs

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB == 60KB / 4 == 15 RCUs * 0.5 = 7.5 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB == 60KB / 4 == 15 RCUs * 0.5 = 7.5 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 1: User bookings search

• Goal: enable users to search bookings with many optional filters


• Math:
▪ Booking item: ~3KB
▪ >90% of customers have <20 bookings
▪ 20 bookings X 3KB == 60KB / 4 == 15 RCUs * 0.5 = 7.5 RCUs (EC)

Watch the tail!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GSISK GSIPK

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB / 4 == 7500 RCUs * 0.5 = 3750 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB / 4 == 7500 RCUs * 0.5 = 3750 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB / 4 == 7500 RCUs * 0.5 = 3750 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Secondary indexes

• Data from your main table is copied into a secondary index


▪ New partition key + sort key
▪ Allows for additional, read-based access patterns
• Choose which attributes are projected into secondary index
▪ ALL
▪ KEYS_ONLY
▪ INCLUDE

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Secondary index projections

• Benefits:
▪ Can reduce item size ➔ smaller RCU + WCU consumption
▪ Can avoid writes to index altogether → lowers WCU consumption
• Downsides:
▪ Less schema flexibility
▪ Might need a follow-up GetItem

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~3KB
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB / 4 == 7500 RCUs * 0.5 = 3750 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~200 bytes
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 3KB == 30000KB / 4 == 7500 RCUs * 0.5 = 3750 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• Goal: enable Organizations to search bookings with filters


• Math:
▪ Booking item: ~200 bytes
▪ Organizations have ~500 employees with ~20 bookings each
▪ 500 * 20 bookings X 200 bytes == 2000KB / 4 == 500 RCUs * 0.5 = 250 RCUs (EC)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 2: Organization bookings search

• How can we reduce the search space further?


▪ Require some attributes, but not all.
▪ E.g.: date range

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes
• Reduce search space where you can

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes
• Reduce search space where you can
▪ Require an attribute that is useful for filtering

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example 3: Large target dataset

• Examples:
▪ Organizational filtering of flights for large orgs (>5000 employees)
▪ Filtering of large product catalog (Amazon.com)
▪ Full-text search
• Look to external, purpose-built systems

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Integrating with external systems

• Remember: you’re adding another operational burden


• Tip: keep it as small as possible
• DynamoDB integration methods:
▪ DynamoDB Streams
▪ Export to S3
• Incremental export!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes
• Reduce search space where you can
▪ Require an attribute that is useful for filtering

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex filtering options

• Over-fetching / client-side filtering


▪ Good when: overall target dataset is small
• Reduced projection into secondary index
▪ Good when: larger items but small amount of filterable attributes
• Reduce search space where you can
▪ Require an attribute that is useful for filtering
• Integrate with an external system if you must

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key takeaways

• DynamoDB is unique from other databases


• Understand the core API operations
• Know your domain
• Do the math
• Handle your write operations appropriately
• DynamoDB integrates well with supporting systems

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you! Please complete the session
survey in the mobile app

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

You might also like