Downloaded 37 times
Uploaded byDaniel Marcous
Distributed Databases - Concepts & Architectures
The document discusses distributed databases, highlighting their structure, terminology, types, advantages, and challenges compared to centralized databases. It outlines key concepts such as data distribution, node autonomy, and management challenges like reliability and availability. Additionally, it explores various architectures and historical development of distributed databases, including notable technologies and their applications in large organizations.
More Related Content
Similar to Distributed Databases - Concepts & Architectures
Distributed Databases - Concepts & Architectures
- 1.
- 2. What? Introduction A distributed databaseis a database in which storage devices are not all attached to a common processing unit such as the CPU, controlled by a distributed database management system.
- 3.
- 4. ● RDBMS -Relational Database Management System ● DDB - Distributed Database ● Node - a unit in a distributed system (mainly a single server) ● DDBMS - Distributed Database Management System ○ In charge of managing the different DDB nodes as one integrated system ● Centralized System - data is stored in one place ● Homogenous system - built of parts (nodes) that all act the same way / consist of the same hardware (Opposite of Heterogeneous). Understanding the vocabulary
- 5.
- 6. Distributed Database Concepts ●Number of processing elements (database nodes) ● Connection between nodes over a computer network ● Logical interrelation between different database nodes ● Absence of node homogeneity
- 7. Types of DistributedDatabases
- 8. Multiprocessing Systems ● ParallelSystems ○ Shared Memory (tightly coupled) - multiple processors share the same main memory ○ Shared Disk (loosely coupled) - multiple processors share the same secondary disk storage ● Truly Distributed Systems ○ Shared Nothing - each processor with its own memory and disk, interrelations are only through network (no SPOF)
- 9. ● Distribution -Data and software distributed over multiple nodes ● Autonomy - Provision DBMS as one whole VS multiple standalone DBMSs ● Heterogeneity - use of different software / hardware on different nodes Classification of Distributed Systems
- 10. Why? The power ofdistribution Reasons for choosing a distributed database over a “plain” centralized database.
- 11.
- 12. ● More computingpower ○ CPU ○ Memory ○ Storage ○ Network bandwidth ● Parallelism ○ Inter-query ○ Intra-query Performance
- 13. Ease of use/ development ● Transparency ● Geographically distributed sites ● Backups ● Elasticity ○ Growing ○ Shrinking
- 14.
- 15. ● Transparency -One software (Ring) to rule them all ○ Management - one command ○ Data - one query ● Autonomy - Degree of Independence ○ Different settings / configurations / Cache size ○ “Master” node / Master Election ● Keeping track of data distribution ○ which server has the table / partition I need? Management Challenges
- 16. ● Reliability -Probability of failures ○ Does one server failure affects the whole system? (“Freeze”) ● Availability - Percent of time when a data source is available ○ If a node goes down, does it’s data get lost? unavailable until its up again? ● Recovery ○ What is a single point of time? ○ Nodes clocks Synchronisation (NTP) ● Transaction Management - Server X must assure that the data is “safe” and no Complex Features Implementation
- 17.
- 18. CAP Theorem ● EricBrewer (Berkeley->Yahoo->Google) ○ C - a read see all previously completed writes ○ A - reads and writes always succeed ○ P - read and write while network is down ● Choose 2! (2000) ● Sorry, actually only C or A… (2012)
- 19. How? Internals How does adistributed database work? ● Advanced Concepts ● Architectures
- 20.
- 21. Replication ● Assumptions ○ Nodeswill fail ○ Commodity Hardware - prone to failure ● Settings ○ Replication Factor ○ Data / Actions /Apply logs ○ Synchronous / ASynchronous ○ Delay
- 22. Fragmentation ● Dividing asingle Data Object (Table/ File) into multiple parts ● Types ○ Horizontal - row wise ○ Vertical - column wise (Vertica/ Parquet) ○ Hybrid - both ● Advantages ○ Reports on part of the data - horizontal ○ Increased parallelism - multiple physical files
- 23. Distributed Processing ● Accessby key Only! ○ Using Hash Tables ■ keys are hashed and spread (=sharded) across nodes ■ result of hash tells you which node to access ■ Hash maps exist on every node / client ● Batch Processing ○ MapReduce ■ Map - partition by key
- 24. Data Locality ● Localstorage (VS centralised storage controller) ○ Bring the processing to the data ○ Free bandwidth ● Smart Load Balancing ○ Route users to the “closest” node with the data (replication duh..) ● Data sorted by Key /Hash Key ○ Same / Close enough key = Same node ○ “Process” all the rides in the TLV area
- 25. ACID BASE● Atomicity ○ Transactions ●Consistency ○ Locked until done ● Isolation ○ No interference ● Durability ○ Completed = Persistent ● Basic Availability ○ Response to every request ● Soft State ○ States change, results are not determinant ● Eventual Consistency ○ Consistent state may take time but is promised ○ (CAS - Compare & Swap Operations exist)
- 26.
- 27. Plain Old CentralizedDatabase ● Oracle ● SQLServer (MS) ● DB2 (IBM) ● MySQL ● PostgreSQL
- 28. Relational (ACID) “Distributed”Database ● Oracle RAC (Real Applications Cluster) ● DB2 Data Sharing ● PostgresXL
- 29. Federated Database System ●IBM IIDR
- 30. Data Warehouse ● OracleExadata ● Teradata ● SQL Data Warehouse (MS) ● Vertica (HP) ● Greenplum (EMC)
- 31. Interactive Multiple ParallelProcessing (MPP) ● Dremel (Big Query, Google) ● Redshift (Amazon) ● Presto (Facebook) ● Impala (Cloudera)
- 32. NoSQL (BASE) SharedNothing Database ● MongoDB ● CouchBase ● Cassandra ● HBase
- 33. When?/Where? History and Present Wheredid the ideas come from and what do we have present for use nowadays?
- 34.
- 35. Articles ● Old School ○Fundamentals of Database Systems (1989) ○ Principles of Distributed Database Systems (1991) ● Distributed File System ○ The Google file system (2003) ● Distributed Processing ○ MapReduce: simplified data processing on large clusters (2004) ● Interactive Querying on large scale
- 36.
- 37. ● Document DB(Mostly JSON) ○MongoDB ○CouchBase ● Key-Value DB ○Cassandra ○HBase ● Graph DB ○Neo4J NoSQL – Database Types
- 38.
- 39. Big Guys ● Google- Inside tools ○ MapReduce ○ Dremel -> Big Query ○ Flume -> DataFlow ● Facebook - Inside tools open-sourced and modified ○ Cassandra -> HBase ○ Presto ● Yahoo - Hadoop / HBase
- 40. ● IDF ● Waze ●Viber - Couchbase ● Liveperson - MongoDB, CouchBase ● SimilarWeb - HBase Israel
- 41. Distribution is awesome, but requirescomplex skills to do right. Don’t overkill it.