Download as PDF, PPTX
![Costs: AWS II
• Standard latest generation: db.m4.large (8 GB) at $1,533/yr [2017]
• Previous generation: Medium instances (3.75GB) useful for testing
($1,577/yr [2014] vs $2,411/yr [2013])
• Large instance (7.5GB) production ready ($3,241/yr vs $4,777/yr
[2013])
• Multi-AZ production ready: db.m4.2xlarge (32GB, 8vCPUs) at
$12,264/yr [2017]
• Previous generation: m3.2XL (30GB, 8vCPUs) ($12,964/yr)](https://image.slidesharecdn.com/databasesinthehostedcloudopensourcesummiteuropepragueoct2017-171025084316/75/Databases-in-the-Hosted-Cloud-26-2048.jpg)
Uploaded byColin Charles
Databases in the Hosted Cloud
The document discusses the concept of Database as a Service (DBaaS) with a focus on offering MySQL and PostgreSQL solutions across various cloud providers such as AWS, Google Cloud, and Rackspace. It covers aspects like deployment methods, service level agreements, version availability, and cost considerations, as well as monitoring and management options available for cloud databases. The author also highlights the importance of backup strategies, operational considerations, and upcoming technologies in the database landscape.
More Related Content
Similar to Databases in the Hosted Cloud
Databases in the Hosted Cloud
- 1. Databases in theHosted Cloud Colin Charles, Chief Evangelist, Percona Inc. [email protected] / [email protected] http://bytebot.net/blog/ | @bytebot on Twitter Open Source Summit Europe, Prague, Czech Republic 23 October 2017
- 2. whoami • Chief Evangelist,Percona Inc • Focusing on the MySQL ecosystem (MySQL, Percona Server, MariaDB Server), as well as the MongoDB ecosystem (Percona Server for MongoDB) + 100% open source tools from Percona like Percona Monitoring & Management, Percona xtrabackup, Percona Toolkit, etc. • Founding team of MariaDB Server (2009-2016), previously at Monty Program Ab, merged with SkySQL Ab, now MariaDB Corporation • Formerly MySQL AB (exit: Sun Microsystems) • Past lives include Fedora Project (FESCO), OpenOffice.org • MySQL Community Contributor of the Year Award winner 2014
- 3. Agenda • MySQL asa service offering (DBaaS) • Choices • Considerations • MySQL versions & access • Costs • Deeper into RDS • Should you run this on EC2 or an equivalent? • Conclusion
- 4. Database as aservice • Database as a Service (DBaaS) • MySQL/PostgreSQL available on-demand, without any installation/ configuration of hardware/software • Pay-per-usage based • Provider maintains database, you don’t maintain, upgrade, or administer the database
- 5. New way ofdeployment • Enter a credit card number • call API (or use the GUI) ec2-run-instances ami-xxx -k ${EC2_KEYPAIR} -t m1.large nova boot --image centos6-x86_64 --flavor m1.large db1 credit: http://www.flickr.com/photos/68751915@N05/6280507539/
- 6. Why DBaaS? • “Couldn’twe just have a few more servers to handle the traffic spike during the elections?” • Don’t have a lot of DBAs, optimise for operational ease • Rapid deployment & scale-out
- 7. Your choices today •Amazon Web Services Relational Database Service (RDS) • Rackspace Cloud Databases • Google Cloud SQL • Oracle Cloud MySQL Service • Microsoft Azure Database
- 8. There are more •Jelastic - PaaS offering MySQL, MariaDB Server • ClearDB - MySQL partnered with Heroku, Azure clouds • Joyent - Image offers Percona MySQL and a Percona SmartMachine
- 9. Some new entrants •Google Compute Engine offers Percona XtraDB Cluster as a “click- to-deploy” app • comes with Galera 3, Percona Toolkit, XtraBackup as well • Pivotal CloudFoundry - “MySQL” PaaS which is MariaDB Galera Cluster 10 • Red Hat OpenShift • MySQL 5.5/5.6/5.7, MariaDB 10.0/10.1 • PostgreSQL 9.2/9.4/9.5 • MongoDB 2.4/2.6/3.2
- 10. Beware • GenieDB -globally distributed MySQL as a service, master-master replication, works on EC2, Rackspace, Google Compute Engine, HP Cloud • Xeround - 2 weeks notice...
- 12. Thank you HPCloud •http://www.bytebot.net/blog/archives/2015/10/27/sunsetting- hpcloud-whom-contributed-to-making-mysql-better • Utility user • enforce_storage_engine • prevent LOAD DATA INFILE/SELECT INTO OUTFILE • restrict # of binlog files
- 14. What else doyou get? • AWS RDS has PostgreSQL • Most MongoDB users run within the cloud — but not hosted, it tends to be “roll your own” (and lately MongoDB Atlas based - reasonable free tier!) • Compose.io (now IBM) & the like (MongoLab) — hosted MongoDB, Redis, Enhanced PostgreSQL, ElasticSearch, RethinkDB, MySQL with group replication, ScyllaDB, etc.
- 15.
- 16. Regions & AvailabilityZones • Region: a data centre location, containing multiple Availability Zones • Availability Zone (AZ): isolated from failures from other AZs + low- latency network connectivity to other zones in same region
- 17. Location, location, location •AWS RDS: US East (N. Virginia, Ohio), US West (Oregon, Northern California), US West (California), EU (Ireland, Frankfurt, London), APAC (Singapore, Tokyo, Sydney, Seoul, Mumbai), South America (São Paulo), GovCloud, Canada (Central), China (Beijing) • Rackspace: USA (Dallas DFW, Chicago ORD, N. Virginia IAD), APAC (Sydney, Hong Kong), EU (London)* • Google Cloud SQL: US (Berkeley County, SC, Council Bluffs, IA), EU (Belgium), Asia (Taiwan, Japan)
- 18. Service Level Agreements(SLA) • AWS - 99.95% in a calendar month • Rackspace - 99.9% in a calendar month • Google - 99.95% in a calendar month • SLAs exclude “scheduled maintenance” which may storage I/O + elevate latency • AWS is 30 minutes/week, so really 99.65%
- 19. Support • AWS -forums; $49/mo gets email; $100+ phone # • Rackspace - live chat, phone #, forums • Google - forums; $150/mo gets support portal; $400+ for phone # • Microsoft - forums; $29/mo to start
- 20. Who manages this? •AWS: self-management, Enterprise ($15k+) • Rackspace: $100 + 0.04 cents/hr over regular pricing • Google: self-management • Microsoft: generally self, premier can give you advisory services
- 21. MySQL versions • AWS:MySQL Community 5.1, 5.5, 5.6, 5.7 / MariaDB Server 10.0, 10.1 • Rackspace: MariaDB Server 10, 10.1, 10.1-enc, MySQL 5.7/5.6/5.1, Percona Server 5.6 • Google: MySQL Community 5.5, 5.6, 5.7 • Microsoft: MySQL Community 5.6, 5.7
- 22. Access methods • AWS- within Amazon, externally via mysql client, API access. • Rackspace - private hostname within Rackspace network, API access. • Google - within AppEngine, a command line Java tool (gcutil), standard mysql client • Microsoft - within Azure, externally via mysql client
- 23. Can you configureMySQL? • You don’t access my.cnf naturally • In AWS you have parameter groups which allow configuration of MySQL source: http://www.mysqlperformanceblog.com/2013/08/21/amazon-rds-with-mysql-5-6-configuration-variables/
- 24. Cost • Subscribe torelevant newsletters of your services • Cost changes rapidly, plus you get new instance types and new features (IOPS) • Don’t forget network access costs • Monitor your costs daily, hourly if possible (EC2 instances can have spot pricing)
- 25. Costs: AWS • AWSprices vary between regions • http://aws.amazon.com/rds/pricing/
- 26. Costs: AWS II •Standard latest generation: db.m4.large (8 GB) at $1,533/yr [2017] • Previous generation: Medium instances (3.75GB) useful for testing ($1,577/yr [2014] vs $2,411/yr [2013]) • Large instance (7.5GB) production ready ($3,241/yr vs $4,777/yr [2013]) • Multi-AZ production ready: db.m4.2xlarge (32GB, 8vCPUs) at $12,264/yr [2017] • Previous generation: m3.2XL (30GB, 8vCPUs) ($12,964/yr)
- 27. Costs: Rackspace • Optionto have regular Cloud Database or Managed Instances • 4GB instance is $2,111.4 (vs. $2,102/yr in 2015 and $3,504/yr in 2013) • 8GB instance is $4,213.8 (vs. $4,205/yr in 2015 and $6,658/yr in 2013) • Consider looking at I/O priority, and the actual TPS you get • High Available instances with automatic failover available too
- 28. Costs: Google • Youmust enable billing before you create Cloud SQL instances • https://developers.google.com/cloud-sql/docs/billing • Previously (first generation): • Testing (D8 - 4GB RAM) - $3,204.7 vs $4,274.15 (in 2015) • XL equivalent (D16 - 8GB RAM) - $6413.05 vs $8,548.30 (in 2015) • Today: • db-n1-standard-8 (8 vCPUs, 30GB RAM) - $6,762.72 • Packages billing plans are cheaper than per-use billing plans
- 29. Where do youhost your application? • Typically within the compute clusters of the service you’re running the DBaaS in • Even Google now offers Compute Engine so there is no such language limitation, ala the early AppEngine days
- 30. RDS: Multi-AZ • Providesenhanced durability (synchronous data replication) • Increased availability (automatic failover) • Warning: can be slow (1-10 mins+) • Easy GUI administration • Doesn’t give you another usable “read-replica” though
- 31. External replication • MySQL5.6 you can do RDS -> Non-RDS • enable backup retention, you now have binlog access • target: exporting data out of RDS • Replicate into RDS with 5.5.33 or later • AWS provides stored procedures like mysql.rds_set_external_master nowadays
- 32. MySQL 5.6/5.7, MariaDB10/10.1 • MySQL 5.6 provides crash-safe slaves, the InnoDB memcached interface, online schema changes, full-text InnoDB indexes, optimizer improvements, INFORMATION_SCHEMA enhancements, scalability/ replication improvements, PERFORMANCE_SCHEMA enhancements • 5.7 - encryption, JSON • MariaDB 10 has some of that that, plus multi-source replication, GTIDs that don’t require full restarts, threadpool, audit plugin and more • 10.1 - encryption!
- 33. Getting started • Importingdata into the cloud? • mysqldump is a good choice today • Upgrading from RDS 5.5 to RDS 5.6? • mysqldump before, but nowadays you can do this via Read Replicas
- 34. Handling backups • Youdon’t get to use xtrabackup! (or mydumper) • Google Cloud SQL automates backups (has a backup window - 4h) • Amazon has automated backups (with point-in-time recovery), with full daily snapshots (has a backup window). • Rackspace allow instance backups too
- 35. Monitoring • Options arelimited, AWS has the best options currently available • Today you have CloudWatch • Google has improved on their read/write graphs • Rackspace has started with basic graphs, visuals for MySQL, have a Cloud Intelligence product • Percona Monitoring & Management - http:// pmmdemo.percona.com/ • DataDog, etc.
- 36. Storage Engines • MySQL(/MariaDB) have many • cool ones include TokuDB, SPIDER, CONNECT • You basically use InnoDB and MyISAM with cloud solutions • MyISAM on RDS won’t guarantee point-in-time recovery, snapshot restore
- 37. High Availability • Planfor node failures • Don’t assume node provisioning is quick • Backup, backup, backup! • “Bad” nodes exist • HA is not equal across options - RDS wins so far • Rackspace has High Availability Database instances built on their ObjectRocket platform
- 38. Unsupported features • AWSMySQL: GTIDs (but MariaDB Server GTIDs work!), InnoDB Cache Warming (intra-schema parallel replication in 5.7 works - this was an XtraDB 5.6 feature), InnoDB transportable tablespaces, authentication plugins, password strength plugin, replication filters, semi-sync replication • AWS MariaDB: Data at Rest Encryption, MariaDB Galera Cluster, HandlerSocket, Multi-source Replication, Password validation plugin, simple_password_check, and cracklib_password_check, Replication Filters, Storage engine-specific object attributes, table and Tablespace Encryption • Google: UDFs, PERFORMANCE_SCHEMA, LOAD DATA INFILE, INSTALL PLUGIN, SELECT ... INTO OUTFILE • mysqlsh?
- 39. Option groups • InnoDBmemcached • MariaDB Server Audit Plugin
- 40. Provisioned IOPS • Availableon Amazon, and recently Google • Faster, predictable, consistent I/O performance with low latencies • Good throughput, RAID on backed • EBS is more reliable
- 41. More on RDS •log access via API • no SUPER access to skip replication errors easily • sync_binlog=0 not available • no OS access (sar, ps, tcpdump) • https://github.com/boto/boto
- 42. Warning: automatic upgrades •Regressions happen even with a minor version upgrade in the MySQL world (though this is happening a lot less as quality has increased tremendously) • InnoDB update that modifies rows PK triggers recursive behaviour until all disk space is exceeded? 5.5.24->5.5.25 (fixed: 5.5.25a) • Using query cache for partitioned tables? Disabled since 5.5.22- >5.5.23!
- 43. Benchmarking for use •sysbench • OLTP test, use tables with 20M rows and 20M transactions, check 1-128 threads/run (run this on RDS, Rackspace) • June 2013, tps, performance per dollar, Rackspace delivers more performance across all flavours except 512MB instance • Yahoo! Cloud Serving Benchmark • https://github.com/brianfrankcooper/YCSB • Google’s PerfKit Benchmarker • https://github.com/GoogleCloudPlatform/PerfKitBenchmarker
- 44. Roadmaps? • There don’tseem to be public roadmaps. You find out when there’s a change! • Watch out for those events…
- 45.
- 46. PostgreSQL in RDS •loading data? backup_retention=0 • disable multi-AZ when loading • disable autovacuum • dump compressed, restore in parallel • don’t disable fsync (really!) • sync replication using multi-AZ • you can control the upgrade time though — this is a bonus • Use PIOPS • SSL should be on
- 47. Running MySQL inEC2 • Can do multiple geographic regions via replication • Run just one Percona Server/ MariaDB server/instance • Use additional EBS volumes for InnoDB tablespaces • RAID EBS volumes (RAID1) • Warm up data partitions, mount partitions with noatime, nodirtime • Vertical scaling with SSD-backed storage • Monitoring with Icinga/nagios • Snapshot backups and save to S3 • Can use Elastic Load Balancer • Can use spot instances • Can use tools like MHA to provide automatic failover • Can use MariaDB Galera Cluster/ Percona XtraDB Cluster
- 48. AWS Aurora • Biggerinstances work better • Zero-downtime migration from RDS • Metrics via CloudWatch, Connectors via MariaDB • 99.99% uptime • MySQL 5.6 “fork”, no optimiser, not traditional replication (but Aurora <->MySQL works of course) • Auto scaling - compute, memory, storage • Replicas (15) for reads • Automated backups in S3, DB snapshots • Encryption with key server being Amazon KMS • Spatial data support - like InnoDB 5.7!
- 49. Looking ahead • OtterTune:automatically find good settings for a database configuration - https://github.com/cmu-db/ottertune • Peloton: self-driving database management system - http:// pelotondb.io/ 60% reduction in latency, 22-35% better throughput https://aws.amazon.com/blogs/ai/tuning-your-dbms-automatically-with-machine-learning/
- 50. Some closing thoughts •Hardware varies per region • Sometimes, software manageability varies per region • Beware cost on your credit card!
- 51. Come see Perconaat the Expo Hall • We’ll help solve all your MySQL or MongoDB database problems! • Lets discuss database strategies and more! • We’re all about championing open source databases
- 52. Thank you! Colin Charles [email protected]/ [email protected] http://bytebot.net/blog | @bytebot on twitter slides: slideshare.net/bytebot