Erik’s Blog

raghu ramakrishnan
- a triumphant preso
- “key chalengeds in cloud comoputing .. and the y! approach”

this is a watershed time.  we’ve spent lots of time building packabged software now wer’re moving to the cloud

key challenges
- elastic scaling
- availabiolity
– if the cloud goes down, everyone is hosed.  consistency or performance myst be traded for availoability.
- handliong failures
– if things go wrong, what can the developer count on when things come up?
- operational efficiency
– cloud managers are db admins for 1000s of clients
- the right abstractions

yahoo’s cloud
- the cloud is an ecosystem.  it’s bigger than a single componenet.  all the pueces must work together seamlessly.

data management in the cloud
- how to make sense of the many options
- what are you trying todo?
- oltp vs olap
- oltp
– random access to a few records
– read-heavy vs write-heavy
- olap
– scan access to a large number of records
– by rows vs columns vs unstructired
- storage
– common features
— managed service. rest apis
— replication
— global footprint
– sherpa
– mopbstor

y! storage problem
- small records, 100kb or less
- structured records, lots of fields
- extreme data scale

typical applications
- user logins and profiles
– single=-record transactions suffice
- events
– alerts, social network activity
– ad clicks
app-specific data
- postings to messsage boards
- uploaded photos and tags

vsld data serving stores
- scale based on partitioning data accross machines
- range selections
– requests span machines
- availability
- replication
- durability
– is it required?
- how is data stored on a single machine?

the cap theorem
- consistency vs availability vs partition tolerance
- consistency => serializability

approaches to cap
- use a single version of a db w/ defered reconciliation
- defer transaction commit
- eventual consistency eg dynamo
- restrict transatctions eg sharded mysql
- object timelines, eg sherpa
- ref: julianbrowne.cim/artice/viewer/brewers-cap-theorem

single slide hadoop primer
- hadoop is wrte optimized, not ideal for serving

out there in the world
- oltp
– oracle, mysql,
– write optimized: cassandra
– main-mem; memchached

ways of using hadoop
- data workloads -> olap -> pig for row ops, zebra for column ops, map reduce for others

hadoop based apps
- we own the terasort benchmark

sherpa`
- parallel db
- geo replication
- structured, flexible schemas
- hashed and ordered tables
- components
– req -> routers -> (record looked up, if necessary) -> lookup cached -> individual machine
- raghu is awesome (“And then!”, sprinting through dense slides)
- write-ahead
- asynch replication
– why? we’re doing geo replication due to the physics involved
– supposing an eearthquake hits and ca falls in th ocean, two users can continue to update their profiles
- consistency model
– acid requiores synch updates
– eventual consistency works
– is there any middle ground?
– sherpa follows a timeline of changes achieved through a standard per-record primary copy protocol

operability
- cloud allows us to apperate at scale
- tablet splitting and balancing
- automatic transfer of mastership

comparing systems
- main point: all of this needs to be thought through and handled automatically

example
- sherpa, oracle, mysql work well for oltp

banchmark tiers
- cluster performance
- replication
- scale out
- availability
- we’d like to do this a group effort, in keeping w/ our philosophy

the integrated cloud
- big idea: declrative lang for specifying structure of service
- key insight: multi-env
- central mechanism: the integrated cloud
- surrendra will talk about htis

foundation componenets
- how to describe app
- desc for resources, entrypoijts, bindings, etc

yst hadled 16.4 million uniques for mj death news

acm socc
- acm symposium on cloud computing