Distributed systems and consistency

Editor's Notes

• #4: - What's a distributed system? - Short answer: "big data" - Lots of machines, geographically distributed - Actual answer: any system where events are not global - Can a read and write happen at the same time? == Distributed - Mostly things are queued - Or in database systems, it's fudged -- no lock, so no problem
• #5: - Why are they good? - Centralized systems scale poorly & expensively - More locks, more contention - Really fast hardware - Vertical scaling - Diminishing returns -- will always eventually fail - Distributed systems scale well & cheaply - Lots of cheap hardware - No locks, no contention - Linear scaling -- can theoretically scale indefinitely
• #6: - So what's the catch? - Consistency - In a centralized system consistency is simple: single source of truth - The problem is writing to it performantly - In a distributed system writes are really fast - But the definition of "truth" is much, much harder
• #7: - CAP theorem - Consistency (all nodes see the same data at the same time) - Availability (a guarantee that every request receives a response about whether it was successful or failed) - Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system) - Pick 2 - But actually it's usually a sliding scale
• #8: - P: No partition tolerance = centralized database - Can't connect to read or write? You're broken. - Replication log got corrupted? You're broken. <img: welcome to our ool>
• #9: - A: No availability guarantee = guessing - Read or write didn't work: try again - Cost/benefit calculation -- everything is unavailable *sometimes* - High-volume logs, statistics - Google BigTable locks data on write, will throw errors if you try to read it - Mars orbiters! Not all the data makes it back, and that's okay.
• #10: - C: Lower consistency = Amazon S3, Riak, other distributed systems - "Eventual" consistency - Write will work, or definitely fail - Reads will work, but might not be "true" - Keep retrying for the truth
• #11: - Why is this a big deal now? - The last 10 years have been about systems getting so big that P has become a bigger and bigger problem - Network was expensive, now it's cheap - And everything is networked - Storage was expensive, now it's cheap - Sacrificing A has been the accidental solution - Instead we can deliberately dial down C to get bigger
• #12: - Ways to get to eventual consistency - There are a ton! - App level: - Write locking - Last write wins - Infrastructure level: - Log structured storage, multiversion concurrency control - Vector clocks and siblings - New: language level! - Bloom
• #13: - Eventual consistency at write time: 1 - Write-time locking - Like a centralized database, except reads are okay with stale data - Slower writes, potential points of failure - Cheap, fast reads
• #14: - Eventual consistency at write time: 2 - Last write wins - This is Amazon S3. - Relies on accurate clocks - Cheap reads and writes - Can lose data! - Okay for image files, bad for payment processing
• #15: - Side note: twitter is eventually consistent - Your timeline doesn't always turn up exactly in order - Older tweets can slot themselves in - Tweets can disappear - Two new tweets can never collide - This is a form of eventual consistency, last write wins, but no conflicts
• #16: - A consistency approach: log-structured storage - Also called append-only databases - Eventual consistency where *consistency* is important, but *currency* is not <diagram>
• #17: - How LSS works - Each write is appended - Indexes are also appended - To get a value, consult the index - As the data grows, throw away older values - Index doesn't need to be updated as often - If you find operations before the index, rebuild an index from them - Relies of lots of really cheap storage - But it turns out we have that!
• #18: - Why is this good? - Don't have to care about the size or schema of the object - Deleting old objects is automatic - Can't corrupt the index - Reads and writes are cheap - Point-in-time consistency is automatic: just read values older than the one you started with - BUT: you still could be behind reality
• #19: - Another consistency approach: vector clocks - Eventual consistency where consistency and currency both matter - Vector, as in math - It means an array, but mathematicians are annoying <diagram> - Simultaneous writes produce siblings - never any data lost
• #22: - Not good enough! - Read consistency: quorum reads - N or more sources must return the same value - Write consistency: quorum writes - N or more nodes must receive the new value
• #23: - Pretty good - But man do siblings suck! http://3.bp.blogspot.com/-h60iS4_uwfg/T2B4rntiV4I/AAAAAAAAK9M/Wc_jaXLRowg/s400/istock_brothers-fighting-300x198.jpg
• #24: - Dealing with siblings - 1: Consistency at read time through clever resolution - Cheap, fast writes - Potentially slower reads, duplicated dispute resolution logic - Pay on every read - 2: Avoid creating them in the first place - Put a sharded lock in front of your writes - Potentially slower writes - Pay once on write - 3: CRDTs: Commutative Replicated Data Types - monotonic lattices of commutative operations - Don't panic
• #26: - Monotonicity - Means operations only affect the data in one way - Simplest example: setter vs. incrementer - Bad: http://en.wikipedia.org/wiki/File:Monotonicity_example3.png - Good: http://en.wikipedia.org/wiki/File:Monotonicity_example1.png - The setter can get it wrong, destroy information - The incrementer doesn't need to know the exact value, just that it goes up by one ( Also good: http://en.wikipedia.org/wiki/File:Monotonicity_example2.png ) - Instead of storing values, store operations
• #27: - Commutativity - Means the order of operations isn't important - 1 + 5 + 10 == 10 + 5 + 1 - Also: (1+5) + 10 == 1 + (5+10) - Means you don't need to know what order the operations happened in - Just that they happened
• #28: - Lattices - A data structure consisting of a set of operatios - Like vector clocks, a (partial) order of operations - Doesn't have to be exact - Just enough to able to avoid having to re-run every operation every time
• #29: - Put it all together: CRDTs - Commutative Replicated Data Types - Each node stores operations in a lattice - As data is distributed, lattices are merged - Because operations are commutative, collisions are okay - Because the exact order is irrelevant
• #30: - CRDTs are a monotonic data structure - Each new operation only adds information - It's never taken away or destroyed - This is really exciting! - It means we don't have to build application logic to handle it - Just get your data types right, and the database will sort it out - Enables radically distributed systems
• #32: - Crazy future shit: Bloom - A language where all the operations available are monotonic, commutative - Calls to non-monotonic operations are special - Allows for compiler-level analysis of distributed code - Flag in advance whether or not you are safe, where you need coordination, and what type - Crazy shit
• #33: - In summary: - Big data is easy - Distributed systems are the answer - Distribution makes consistency harder in exchange for better partition - The solution may be changing the way data is stored - Don't store a value, store a sequence of operations - Make the operations commutative, the structure monotonic - Pretty cool stuff
• #34: Log Structured Storage: http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-storage Lattice data structures and CALM theorem: http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf Bloom: http://www.bloom-lang.net/ Ops: Riak in the Cloud https://speakerdeck.com/u/randommood/p/getting-starte
• #35: Other sources: http://en.wikipedia.org/wiki/Multiversion_concurrency_control http://en.wikipedia.org/wiki/Monotonic_function http://en.wikipedia.org/wiki/Commutative_property http://en.wikipedia.org/wiki/CAP_theorem http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing http://pagesperso-systeme.lip6.fr/Marc.Shapiro/papers/RR-6956.pdf http://en.wikipedia.org/wiki/Vector_clock