Azure storage deep dive
Download as PPTX, PDF4 likes1,429 views
The document provides a technical overview of Azure Storage Services architecture. It discusses: 1. The global namespace that routes requests to the appropriate partition based on account name, service type, and object name. 2. The front-end, partition, and stream layers that make up the storage architecture. The front-end routes requests to partitions, which contain object indexes and route to streams for data. 3. How streams provide an append-only distributed file system across extend nodes, with synchronous replication for high availability and consistency.
Azure storage deep dive
- 2. Still alive?
- 3. Yves Goeleven Founder of MessageHandler.net • Windows Azure MVP
- 4. Exhibition theater @ kinepolis
- 7. What you might have expected Azure Storage Services • VM (Disk persistence) • Azure Files • CDN • StoreSimple • Backup & replica’s • Application storage • Much much more...
- 8. I assume you know what storage services does!
- 9. The real agenda Deep Dive Architecture • Global Namespace • Front End Layer • Partition Layer • Stream Layer
- 10. WARNING!
- 11. Level 666
- 13. Which discusses how Storage defies the rules of the CAP theorem • Distributed & scalable system • Highly available • Strongly Consistent • Partition tolerant • All at the same time!
- 14. That can’t be done, right?
- 15. Let’s dive in
- 16. Global namespace
- 17. Global namespace Every request gets sent to • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName
- 18. Global namespace DNS based service location • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • DNS resolution • Datacenter (f.e. Western Europe) • Virtual IP of a Stamp
- 19. Stamp Scale unit • A windows azure deployment • 3 layers • Typically 20 racks each • Roughly 360 XL instances • With disks attached • 2 Petabyte (before june 2012) • 30 Petabyte Intra-stamp replication
- 20. Incoming read request Arrives at Front End • Routed to partition layer • http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • Reads usually from memory Partition Layer Front End Layer FE FE FE FE FE FE FE FE FE PS PS PS PS PS PS PS PS PS
- 21. Incoming read request 2 nodes per request • Blazingly fast • Flat “Quantum 10” network topology • 50 Tbps of bandwidth • 10 Gbps / node • Faster than typical ssd drive
- 22. Incoming Write request Written to stream layer • Synchronous replication • 1 Primary Extend Node • 2 Secondary Extend Nodes Front End Layer Stream Layer Partition Layer Primary Secondary Secondary PS PS PS PS PS PS PS PS PS EN EN EN EN EN EN EN
- 23. Geo replication Intra-stamp replicationIntra-stamp replication Inter-stamp replication
- 24. Location Service Stamp Allocation • Manages DNS records • Determines active/passive stamp configuration • Assigns accounts to stamps • Geo Replication • Disaster Recovery • Load Balancing • Account Migration
- 25. Front end layer
- 26. Front End Nodes Stateless role instances • Authentication & Authorization • Shared Key (storage key) • Shared Access Signatures • Routing to Partition Servers • Based on PartitionMap (in memory cache) • Versioning • x-ms-version: 2011-08-18 • Throttling Front End Layer FE FE FE FE FE FE FE FE FE
- 27. Partition Map Determine partition server • Used By Front End Nodes • Managed by Partition Manager (Partition Layer) • Ordered Dictionary • Key: http(s)://AccountName.Service.core.windows.net/PartitionName/ObjectName • Value: Partition Server • Split across partition servers • Range Partition
- 28. Practical examples Blob • Each blob is a partition • Blob uri determines range • Range only matters for listing • Same container: More likely same range • https://myaccount.blobs.core.windows.net/mycontainer/subdir/file1.jpg • https://myaccount.blobs.core.windows.net/mycontainer/subdir/file2.jpg • Other container: Less likely • https://myaccount.blobs.core.windows.net/othercontainer/subdir/file1.jpg PS PS PS
- 29. Practical examples Queue • Each queue is a partition • Queue name determines range • All messages same partition • Range only matters for listing queues PS PS PS
- 30. Practical examples Table • Each group of rows with same PartitionKey is a partition • Ordering within set by RowKey • Range matters a lot! • Expect continuation tokens! PS PS PS
- 31. Continuation tokens Requests across partitions • All on same partition server • 1 result • Across multiple ranges (or to many results) • Sequential queries required • 1 result + continuation token • Repeat query with token until done • Will almost never occur in development! • As dataset is typically small PS PS PS
- 32. Partition layer
- 33. Partition layer Dynamic Scalable Object Index • Object Tables • System defined data structures • Account, Entity, Blob, Message, Schema, PartitionMap • Dynamic Load Balancing • Monitor load to each part of the index to determine hot spots • Index is dynamically split into thousands of Index Range Partitions based on load • Index Range Partitions are automatically load balanced across servers to quickly adapt to changes in load • Updates every 15 seconds
- 34. Consists of • Partition Master • Distributes Object Tables • Partitions across Partition Servers • Dynamic Load Balancing • Partition Server • Serves data for exactly 1 Range Partition • Manages Streams • Lock Server • Master Election • Partition Lock Partition layer Partition Layer PS PS PS PM PM LS LS
- 35. How it works Dynamic Scalable Object Index Partition Layer PS PS PS PM PM LS LS Front End Layer FE FE FE FE Account Container Blob Aaaa Aaaa Aaaa Harry Pictures Sunrise Harry Pictures Sunset Richard Images Soccer Richard Images Tennis Zzzz Zzzz zzzz A-H H’-R R’-Z Map
- 36. Not in the critical path! Dynamic Load Balancing Stream Layer Partition Layer Primary Secondary Secondary PS PS PS PS PS PS PS PS PS EN EN EN EN EN EN EN PM PM LS LS
- 37. Architecture • Log Structured Merge-Tree • Easy replication • Append only persisted streams • Commit log • Metadata log • Row Data (Snapshots) • Blob Data (Actual blob data) • Caches in memory • Memory table (= Commit log) • Row Data • Index (snapshot location) • Bloom filter • Load Metrics Partition Server Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics
- 38. HOT Data • Completely in memory Incoming Read request Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics
- 39. COLD Data • Partially Memory Incoming Read request Memory Data Persistent Data Commit Log Stream Metadata Log Stream Row Data StreamRow Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics
- 40. Persistent First • Followed by memory Incoming Write Request Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics
- 41. Not in critical path! Checkpoint process Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Row Data StreamCommit Log Stream
- 42. Not in critical path! Geo replication process Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Memory Data Persistent Data Commit Log StreamCommit Log Stream Metadata Log Stream Row Data Stream Blob Data Stream Memory Table Row Cache Index Cache Bloom Filters Load Metrics Row Data Stream
- 43. Stream layer
- 44. Append-Only Distributed File System • Streams are very large files • File system like directory namespace • Stream Operations • Open, Close, Delete Streams • Rename Streams • Concatenate Streams together • Append for writing • Random reads Stream layer
- 45. Concepts • Block • Min unit of write/read • Checksum • Up to N bytes (e.g. 4MB) • Extent • Unit of replication • Sequence of blocks • Size limit (e.g. 1GB) • Sealed/unsealed • Stream • Hierarchical namespace • Ordered list of pointers to extents • Append/Concatenate Stream layer Extent Stream //foo/myfile.data Extent Extent Extent Pointer 1 Pointer 2 Pointer 3 Block Block Block Block Block Block Block Block Block Block Block
- 46. Consists of • Paxos System • Distributed consensus protocol • Stream Master • Extend Nodes • Attached Disks • Manages Extents • Optimization Routines Stream layer Stream Layer EN EN EN EN SM SM SM Paxos
- 47. Allocation process • Partition Server • Request extent / stream creation • Stream Master • Chooses 3 random extend nodes based on available capacity • Chooses Primary & Secondary's • Allocates replica set • Random allocation • To reduce Mean Time To Recovery Extent creation Stream Layer EN EN EN SM Partition Layer PS Primary Secondary Secondary
- 48. Bitwise equality on all nodes • Synchronous • Append request to primary • Written to disk, checksum • Offset and data forwarded to secondaries • Ack Replication Process Journaling • Dedicated disk (spindle/ssd) • For writes • Simultaneously copied to data disk (from memory or journal) Stream Layer EN EN EN SM Partition Layer PS Primary Secondary Secondary
- 49. Ack from primary lost going back to partition layer • Retry from partition layer • can cause multiple blocks to be appended (duplicate records) Dealing with failure Unresponsive/Unreachable Extent Node (EN) • Seal the failed extent & allocate a new extent and append immediately ExtentExtent Stream //foo/myfile.data Extent Extent Pointer 1 Pointer 2 Pointer 3 Block Block Block Block Block Block Block Block Block Block Block Block Extent Block Block Block Extent Block
- 50. Replication Failures (Scenario 1) Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Block Seal 120 120 Primary Block Block Block Secondary Block Block Block Block Block Secondary Block Block Block Block
- 51. Replication Failures (Scenario 2) Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Seal 120 100 Primary Block Block Block Secondary Block Block Block Secondary Block Block Block ?
- 52. Data Stream • Partition Layer only reads from offsets returned from successful appends • Committed on all replicas • Row and Blob Data Streams • Offset valid on any replica • Safe to read Network partitioning during read Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block Block
- 53. Log Stream • Logs are used on partition load • Check commit length first • Only read from • Unsealed replica if all replicas have the same commit length • A sealed replica Network partitioning during read Stream Layer EN EN EN SM Partition Layer PS Primary Block Block Block Secondary Block Block Block Secondary Block Block Block Block 1, 2 Seal Secondary Block Block Block Block Block Secondary Block Block Block Block
- 54. Downside to this architecture • Lot’s of wasted disk space • ‘Forgotten’ streams • ‘Forgotten’ extends • Unused blocks in extends after write errors • Garbage collection required • Partition servers mark used streams • Stream manager cleans up unreferenced streams • Stream manager cleans up unreferenced extends • Stream manager performs erasure coding • Reed Solomon algorithm • Allows recreation of cold streams after deleting extends Garbage Collection
- 55. Wrapup
- 56. Layering & co-design • Stream Layer • Availability with Partition/failure tolerance • For Consistency, replicas are bit-wise identical up to the commit length • Partition Layer • Consistency with Partition/failure tolerance • For Availability, RangePartitions can be served by any partition server • Are moved to available servers if a partition server fails • Designed for specific classes of partitioning/failures seen in practice • Process to Disk to Node to Rack failures/unresponsiveness • Node to Rack level network partitioning Beating the CAP theorem
- 57. Want to know more? • White paper: http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdf • Video: http://www.youtube.com/watch?v=QnYdbQO0yj4 • More slides: http://sigops.org/sosp/sosp11/current/2011-Cascais/11-calder.pptx • Erasure coding: http://www.snia.org/sites/default/files2/SDC2012/presentations/Cloud/ ChengHuang_Ensuring_Code_in_Windows_Azure.pv3.pdf Resources
- 58. Still alive?
- 59. And take home the Lumia 1320 Present your feedback form when you exit the last session & go for the drink Give Me Feedback
- 60. Follow Technet Belgium @technetbelux Subscribe to the TechNet newsletter aka.ms/benews Be the first to know
- 61. Belgiums’ biggest IT PRO Conference
Editor's Notes
- #6: Real time message processing as a service Think of it as IFTTT for internet of things Solves today’s integration issues Scalability, data volume, multitude protocols & platforms, multitude of integration points, saas & social integration, mobile platforms, business ecosystems, ownership & centralized management, …