Amazon Web Services Building Blocks for Drupal Applications and Hosting

Amazon Web Services Building Blocks for Drupal Applications and Hosting

• 1.
  BUILDING
THE
CLOUD
 Jeff
Barr,
Senior
Web
Services
Evangelist
 [email protected]
 @jeEarr

• 2.
  G OALS 
FOR 
T ODAY 
 •  Introduce
you
to
AWS
 •  Show
you
how
the
pieces
fit
together
 •  Provide
background
for
second
half
of
this
webinar
 •  Answer
your
AWS
and
cloud
compuRng
quesRons

• 3.
  I NTRODUCING 
AWS
 • AWS
=
“Amazon
Web
Services”
 •  One
of
our
three
businesses:
 –  Online
Retail
 –  Seller
Business
(Marketplace)
 –  Web
Service
Infrastructure
 Amazon Web Services •  Pay
As
You
Go
Pricing
Model
 Or “AWS” for short… •  Two
Levels:
 –  Infrastructure
as
a
Service
(IaaS)
 –  Pla_orm
as
a
Service
(PaaS)

• 4.
  B UILDING 
THE 
C LOUD 
 •  Business
 •  Security
 •  Presence
 •  Technical

• 5.
  B UILDING 
THE 
C LOUD 
‐
B USINESS 
 •  Amazon’s
Brand
Name
 •  Self‐serve
Signup
 •  User
Accounts
 •  Pay
As
You
Go
 •  Credit
Card
Billing

• 6.
  B UILDING 
THE 
C LOUD 
‐
S ECURITY 
 •  Security
is
Job
#1
for
Us
 •  Designed
for
MulR‐Tenancy
 •  SAS
70
CerRficaRon
 •  Security
White
Paper
 –  Physical
Security
 –  Logical
Security
 •  Published
Best
PracRces
 •  hfp://aws.amazon.com/security

• 7.
  B UILDING 
THE 
C LOUD 
‐
P RESENCE 
 AWS
Regions:
 US
East
(Northern
Virginia)
 US
West
(Northern
California)
 Europe
(Dublin)
 Asia
Pacific
(Singapore)
 More
to
come…
 AWS
CloudFront
LocaRons
 Ashburn,
VA
/
Dallas,
TX
/
Los
Angeles,
CA
/
Miami,
FL
/
Newark,
NJ
/
 Palo
Alto,
CA
/
Seafle,
WA
/
St.
Louis,
MO
/
Amsterdam
/
Dublin
/
 Frankfurt
/
London
/
Hong
Kong
/
Tokyo
/
Singapore
/
New
York

• 8.
  BUILDING
THE
CLOUD
‐
TECHNICAL
 


• 9.
  A MAZON 
ELASTIC 
C OMPUTE 
C LOUD 
 •  Amazon
EC2:
on‐demand
compute
power
 –  Obtain
and
boot
new
server
instances
in
minutes
 –  Scale
capacity
up
or
down
as
needed
 –  $0.085
per
hour
and
up
 •  Key
features:
 –  Support
for
Windows,
Linux,
and
OpenSolaris
 –  Complete
flexibility
to
install
any
open
source
or
commercial
soqware
 –  Supports
all
major
web
and
applicaRon
pla_orms
 –  MulRple
Availability
Zones
 –  ElasRc
IP
Addresses
 –  ElasRc
Block
Storage
 –  ElasRc
Load
Balancing


 –  Auto
Scaling
 –  Amazon
CloudWatch

• 10.
  A MAZON 
RELATIONAL 
D ATABASE 
S ERVICE 
(RDS)
 •  Create
a
new
relaRonal
database
with
one
API
call
 •  Offload
common
administraRve
tasks
to
AWS
 •  Use
exisRng
code
and
tools
 •  Scale
up
processing
or

 storage
with
one
API
call
 •  Recover
database
using

 snapshot
or
log
file
 •  MySQL
5.1

• 11.
  AWS
P ACE 
OF 
I NNOVATION 
–
2010
 »
Amazon
EC2
Reserved
Instances

 »
RDS
SSL
Support
 



with
Windows,
Extra
Large
High

 »
CloudWatch
EBS
Metrics
 



Memory
Instances
 »
S3
Console
Support
 »
Amazon
S3
Versioning
Feature
 »
Import/Export
API
 »
Consolidated
Billing
for
AWS
 »
CloudFront
HTTPS
Support
 »
Lower
pricing
for
Outbound
Data

 »
CloudFront
POP
in
NY
 



Transfer
 »
Singapore
Region
 »
CloudFront
Price
ReducRon
 »
RDS
in
Europe
 »
ElasRc
MapReduce
updates
 »
ELB
sessions
 2010
 Jan
 Feb
 Mar
 Apr
 May
 Jun
 Jul
 »
??
 »
Amazon
SNS
 »
??
 »
??
 »
RDS
Console
Support
 » S3
Reduced
Redundancy
Storage
 »
AWS
SoluRon
Pages
 » RDS
MulR‐AZ
 » CloudFront
Streaming
Logs
 » VPC
in
Europe

• 12.
  F INAL 
THOUGHTS 
 •  AWS
Trends:
 –  ConRnuous
innovaRon
 –  AddiRonal
services
 –  AddiRonal
features
 –  AddiRonal
locaRons
 •  This
is
the
Rme
to
learn
and
start
to
use
the
cloud!

• 13.
  G ETTING 
STARTED 
W ITH 
AWS
 •  Create
developer
account
at
hfp://aws.amazon.com
 •  Download
access
keys,
SDKs,
developer
tools
 •  Build
and
deploy
applicaRon
 •  AWS
Blog:
hfp://aws.typepad.com


• 14.
  Hosting Drupal onAWS http://acquia.com/resources/ acquia-tv Barry Jaspan Senior Architect, Acquia [email protected] @bjaspan Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 15.
  Acquia Hosting Overview •  Managed web hosting for Drupal •  High performance, scalability, availability •  Combined infrastructure- and application-level support from one vendor •  Single or multi-tier •  Load balancers, web nodes, database clusters, memcached, network filesystems, ... •  One instance, separate instances, or any combination •  Automated •  We provisioned for a “20M uniques on day one” customer ... in a few minutes. Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 16.
  Load balancing Amazon WebServices Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 17.
  Load balancing onAWS •  No hardware LB is available •  Round-robin DNS to web nodes is bad for high-availability •  Elastic Load Balancer (ELB) •  CNAME requires http://www.TLD •  Limited flexibility (but now with stickiness!) •  Elastic IP address (EIP) + software LB •  Allows http://TLD •  Only one EIP per server •  We use nginx for LB and Varnish for caching Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 18.
  Software load balancer AmazonWeb Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 19.
  Shared les Amazon WebServices Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 20.
  File sharing onAWS •  Drupal needs a POSIX filesystem for user- uploaded files •  Elastic Block Store (EBS) is single-instance •  rsync does not scale past N=2 web nodes •  S3FS is worse than useless for this •  Too slow, non-POSIX, unreliable, unsupported •  A network filesystem is required •  NFS? •  More later... Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 21.
  Basic web stack AmazonWeb Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 22.
  HA load balancer •  Load balancer might fail •  Elastic IP •  Requires a hot spare •  EIP must be reassigned to a hot spare •  Elastic Load Balancer •  For us, can’t talk directly to web nodes •  nginx provides virtual host flexibility, concurrency protection, etc •  Varnish provides caching •  Also requires a hot spare (or second active instance) Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 23.
  HA load balancerwith EIP Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 24.
  HA lesystem •  NFS •  Cannot tolerate server failure •  Not replicated •  DRBD with virtual IP reassignment won’t work •  AFS •  Tolerates server failure •  Not replicated •  GlusterFS •  Replicated and distributed •  GPL, commercially supported, under active dev •  Easy to use, tricky to optimize just right Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 25.
  HA lesystem with GlusterFS Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 26.
  HA database •  Multi-AZ Relational DB Service (RDS) •  Failover takes 3 minutes •  Active/passive master MySQL replication •  Must avoid “split-brain” operation •  All web nodes must fail over simultaneously •  Heartbeat with virtual IP reassignment not possible •  Failover mechanism must itself be HA •  Using MySQL proxy just leads to the same problem •  Requires synchronous app-level failover •  We store db election in GlusterFS •  Any other HA replicated data store would do Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 27.
  High-availability web stack AmazonWeb Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 28.
  Machine con g •  All these instances need to be built •  Incremental custom AMIs •  Time consuming during development/debugging •  Poor fit for source-code control •  Leads to mystery configurations •  Automated configuration scripts/language •  Incremental updates are simple •  Good fit for source-code control •  Known process from a clean, base AMI •  Combining the two may be best Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 29.
  Static con gwith puppet Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 30.
  Elasticity •  Elastic scalability means dynamic configuration •  Balancers: info on all web nodes •  Web nodes (Drupal): info on all db servers •  FS servers: info on all EBS volumes to use •  FS clients: info on all FS servers to use •  All (re-)started instances get a new IP •  EIPs are limited and cost more to use •  All servers need current internal IPs •  Admins need current external IPs •  Requires frequent manual updates, or Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc. automation
• 31.
  Dynamic data-driven cong Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 32.
  Email from AWS •  AWS IPs are on many spam blacklists •  Can’t reliably be used to send mail •  Requires a non-cloud mail server or service provider •  Note: Jeff tells me this situation has improved Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 33.
  Colo mail server AmazonWeb Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 34.
  Everything else •  I’ve only talked about cloud-specific issues •  To self-host (anywhere), you also need: •  Backups •  24x7 monitoring •  Ops staff •  Caffeine •  Ibuprofen Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.
• 35.
  Management and opsstack Amazon Web Services Building Blocks for Drupal Applications and Hosting © 2010 Acquia, Inc.