Downloaded 50 times
Uploaded byPythian
Percona Live 2014 - Scaling MySQL in AWS
The document presents a comprehensive overview of scaling MySQL on AWS, detailing options such as RDS and EC2/MySQL, along with their advantages and challenges. It discusses various scaling patterns, performance/availability aspects, and implementation strategies for both RDS and EC2, highlighting the importance of multi-availability zones and disaster recovery. Additionally, it addresses potential failure scenarios and emphasizes the need for effective monitoring and management strategies to mitigate risks.
More Related Content
Similar to Percona Live 2014 - Scaling MySQL in AWS
Percona Live 2014 - Scaling MySQL in AWS
- 1. Scaling MySQL inAWS Presented by: Laine Campbell April 3rd, 2014
- 2. Agenda 1. Overview ofoptions: RDS and EC2/MySQL 2. MySQL scaling patterns 3. Performance/Availability 4. Implementation choices 5. Common failure patterns
- 3. Who the *&%^#$am I? Laine Campbell Co-Founder and CEO, Blackbird (formerly PalominoDB) 9 years building the DB team/infrastructure at Travelocity. 7 years at PalominoDB/Blackbird, supporting 50+ companies, 1000s of databases and way too much coffee.
- 4. AWS Options forMySQL: RDS and EC2/MySQL A love story...
- 5. AWS Relational DatabaseService (RDS) Basic Operations Managed Ease of Deployment Supports Scaling via Replication Reliable via Replication, EBS RAID, Multi-AZ
- 6. Managed Operations Backups andRecovery Provisioning Patching Auto Failover Replication
- 7. RDS Backup andRecovery Storage is done via EBS Snapshot and binlog based (point in time) A Non Multi-AZ implementation creates spikes in latency during backups Avoided in Multi-AZ via backups on the secondary Snapshots only
- 8. Advanced Backup andRecovery Creating non-RDS backups done via mysqldump, mydumper, custom extraction You can create non-RDS replicas using a logical backup in 5.6 only non-RDS replicas will break during AZ failovers - thus not useful for production or for large datasets
- 9. Disaster Recovery Cross regionreplication is supported in 5.6 Cross region replication incurs cross-region data transfer costs Relay replicas recommended if you wish to minimize expenses
- 10. Provisioning Initial creation ofsingle or multi-AZ masters Single command replica creation (serialized) via snapshots, multi-AZ avoids a one minute IO suspension.
- 11. Patching Automatically managed in maintenancewindows Alerts sent for the coming week, so you can determine impact, reschedule, etc… Multi-AZ mitigates impact of invasive maintenance
- 12. RDS Challenges (Opportunities?) Abstractionfrom kernel, OS processlist, OS commands etc... No SUPER access, changes to management via Stored Procedure (minimal but annoying) Log access becomes more challenging (but manageable) The more experienced of an operator you are, the grumpier you will be!
- 13. RDS Challenges (Opportunities?) Snapshotbackups not portable/accessible outside of RDS Multi-AZ failover can strand replicas when relaxing binlog consistency for performance. (sync_binlog=0). Without the ability to manually CHANGE MASTER, one must rebuild all replicas after a failover.
- 14. RDS Visibility Impacts Agentbased instrumentation that requires localhost installation won’t work No access to TCPDUMP/Port listening SAR, processlist for swapping, vmstat, iostat etc... Log forensics become harder but manageable (must download first)
- 15. EC2 and MySQL Allthe MySQL you’ve come to love and hate Any topologies you can dream Access to many more types of instances and storage
- 16. Why RDS orEC2? You can’t run 5.6, and you can’t tolerate the risk of single region? (~99.65% SLA per month) Use EC2 You don’t have operational expertise to manage backups, provisioning and replication? Use RDS pro-tip, if you can’t manage a system, how can you troubleshoot advanced performance issues with the visibility issues in RDS?
- 17. Why RDS orEC2? Want MariaDB, XtraDB? Use EC2 Large data-sets generally require file level backups and portability? Use EC2 pro-tip, if you can’t get a mysqldump or a parallel dump to load/export in a timely fashion, you probably don’t want RDS
- 18. Scaling Patterns forMySQL in AWS
- 19. Scaling in RDS- Vertical RAM up to 244 GB per instance, creating excellent ability to put large datasets in RAM Network performance up to 10 GB CPU up to 32 cores Provisioned IOPs are game changers, and mandatory for production, performance sensitive applications.
- 20. Scaling in RDS- Provisioned IOPs 1,000 - 30,000 IOPS 100 GB to 3 TB Stable, predictable IO Realizing Max IOPS - 20,000 ● cr1.8xlarge Instance Type ● MySQL 16 KB Page Size ● Full Duplex IO Channel ● 50% reads, 50% writes
- 21. Scaling in RDS- Provisioned IOPs Overprovisioning from realized, can create latency reductions ● In an unbalanced workload, for instance reads consuming channel limits ● Write channel bandwidth remains unsaturated ● By doubling IOPS, you increase concurrency, thus reducing latency. Transaction rates increase ● Consumption of IOPS can reduce as transaction rates increase, and manifest as: ○ Improved use of group commit ○ larger log writes
- 22. Scaling in RDS- Reads Native replication allows for scale out of reads, just as in EC2 or your own datacenter RAM up to 244 GB per instance, creating much better ability to put large datasets in RAM 5.6 allows for the memcache plugin
- 23. Scaling in RDS- Writes Like any system, you must split workloads if writes consume max capacity of PIOPS. ● Functional Partitioning ● Sharding
- 24. Scaling in RDS- Concerns Sharding: ● Management of RDS instances to roll shards up and down can be a new paradigm. ● Overall, this can be done, but does require a logical shift. Resource Constraints: ● No access to SSDs (up to 91,250 read or 78,750 write IOPS of 14KB size) Data Movements: ● No access to data copies outside of replica builds can dramatically increase data movement time
- 25. Scaling in EC2- Vertical Higher variety of instances. Similar top level constraints of: ● RAM ● CPU ● PIOPS ● Network Ephemeral storage SSD create a whole new class of IO performance: (up to 91,250 read or 78,750 write IOPS of 14KB size)
- 26. Scaling in EC2- Reads In addition to standard MySQL replication, you have new options ● Galera, MariaDB/Galera and XtraDB Cluster ● Tungsten Replicator and Cluster
- 27. Scaling in EC2- Writes Sharding still becomes necessary, but in EC2 over RDS, one has access to snapshots: ● Management of large datasets becomes much easier ● Shard management functions in more typical paradigms
- 28. Scaling in EC2- Concerns SSD and Ephemeral Storage ● Instances become even more volatile ● Backups via EBS snapshot are impossible, requiring LVMs or similar ● One might consider keeping writes to PIOPs max (20,000) for writes and leverage SSD for reads
- 29. Availability for MySQLin AWS
- 30. AWS Availability: Regionsand Zones
- 31. AWS Availability: Regionsand Zones Amazon Regions equate to data-centers in different geographical regions. Availability zones are isolated from one another in the same region to minimize impact of failures.
- 32. AWS Availability: Regionsand Zones Amazon states AZs do not share : •Cooling •Network •Security •Generators •Facilities
- 33. AWS Availability: Regionsand Zones Apr, 2011 - US East Region EBS Failed ● Incorrect network failover. ● Saturated intra-node communications. ● Cascading failures impacted EBS in all AZs. Jul, 2012 - US East Partial Impact ● Electrical storms impacted multiple sites. ● Failover of metadata DB took too long. ● EBS I/O was frozen to minimize corruption.
- 34. AWS Availability: Regionsand Zones 99.95% Monthly SLA for a region (multiple AZs) ● Implies multiple AZ is mandatory ● Implies multi-region is necessary for 99.99% or higher
- 35. Availability in RDS- Multi-AZ The core of an HA solution Block level replication, active/passive Saves you from most master crashes Reduces impact of backups, upgrades, locks for provisioning replicas When not in 5.6, and using log_sync != 1, you often lose replicas during failover
- 36. Availability in RDS- Multi-AZ IO impact from replication You do not get to choose the failover AZ, meaning you must be ready to move app servers
- 37. Availability in RDS- Replicas Redundant replicas make total sense. N+1 meets most needs with the ease of provisioning You must have replicas in every AZ you have app servers in (if using replicas for reads) AWS states cross-AZ latency impact of low single digit millisecond impact. Real world indicates occasional much larger spikes
- 39. Availability in RDS- Replicas Redundant replicas make total sense. N+1 meets most needs with the ease of provisioning You must have replicas in every AZ you have app servers in (if using replicas for reads) AWS states cross-AZ latency impact of low single digit millisecond impact. Real world indicates occasional much larger spikes
- 40. Availability in EC2- Options You can use Galera, XtraDB Cluster, or similar for a read/write anywhere solution MySQL MHA can be used to do failovers Continuent’s Tungsten product can also manage failovers
- 41.
- 42. AWS Availability: Regionsand Zones Type of Change EC2 RDS Master (Non Multi-AZ) RDS Master (Multi-AZ) RDS Replica Instance resize up/down Rolling Migrations Moderate Downtime Minimal Downtime Moderate Downtime (take out of service) EBS <-> PIOPS Severe Performance impact. Severe Performance impact. Minor Performance impact. Severe Performance Impact (take out of service) PIOPS Amount Change Minor Performance impact. Minor Performance impact. Minor Performance impact. Performance Impact (take out of service) Disk Space Change (add) Performance impact. Performance impact. Minor Performance impact. Performance Impact (take out of service) Disk Space Change (reduce) Rolling Migrations Moderate Downtime Moderate Downtime Moderate Downtime (take out of service)
- 43.
- 44. Predicting and ManagingFailure Operations is about managing change and mitigating risk
- 45. Predicting and ManagingFailure Local Failures • Database crashes • Human error o Misconfigure o Write to a replica o Drop a table/database/career • Localized EBS hangs and corruption • Unacceptable/unpredictable performance
- 46. Predicting and ManagingFailure Local Failures ● When it goes bad, don’t waste time diagnosing. o Shoot it in the head! ● Plan! ○ Simulate availability and region level failures ○ Wipe storage, reduce IOPS, shut down ○ Chaos monkey is your friend ● Observe! ○ Monitor for early failures, predict
- 47. Predicting and ManagingFailure Mitigation In RDS: Use Multi-AZ Use replicas in multiple AZs Replicate to multiple regions, and out of AWS In EC2: Use a failover (Galera, Tungsten, MHA/HAProxy) Use multiple AZs and regions Frequent Backups (practicing restores)