Simpleton pattern

Problem

We don’t want multiple objects of the same class, but we also don’t want to clutter our code with the kind of checks required to implement the Singleton pattern

Solution

Kill program execution if a second attempt to instantiate an object occurs

Example

class Foo {
    function constructor(){
        if(objectOfClassAlreadyExists('Foo')){
            stopExecution();
        }

        ...

    }
}

...

variable foo = new Foo //all good

...

variable bar = new Foo  //poof!

- 280k image requests/sec
- 300k req/sec

demo
- sports, travel, mail, news

internal expectations
- global

common challenges
- spped of light
- spikes
- cost
– space, paower, bandwidth, replication bandwidth
- partitioned network failures
– data center failures
– cap theorem
- consumer/user intuition
– replication is not bcp
–

mobstor & sherpa
- mobstor
– storage and delivery cloud
– cdns make sense when we have 90% cache hit rate
– features
— global
—- caching
—- protocol termination
—- authentication
—- content routing
— local
—- auto expiration
—- de-dup
—- object placement
—- re-replication
– layeruing
— dns gslb –> hardware vip –> scalable session mgr –> geo replication –> internal dns w/ loop feedback –> hardware vip –> local replication –> separate metadata replication
- sherpa
– simply put, it’s sharded mysql w/ replication
– stack: dns –> hadware vip –> scalable router & session mgr –> geo repl –> …

physics & econ for a global cloud
- what’s your target sla?
– distance + speed of light + network degredation = latency
– selective replication
- lessons learned
– intuition is usually wrong: let data drive data
– provide hoks, experimental fedback, and mobility
– n-way global repl is really expensive
– customers don’t understand 95/5 billing
– customers don’t understand cap theorem
– verify all provisioning
— there are a lot of non-hardware issues that can affect hardware performance
– strive for quality, but plan for failure

why care about y! cloud?
- commitment to open source
– componenet approach
– traffic server
— a handful of ppl for more than a year worked on open sourcing
— it’s a huge benefit to the community
— 400tb/day on 150 commodity boxs!
— would you want to build your company oin a web server that no one else uses?
– zookeeper

q/a
- why mobstor?
– mobstor solved the problem of brittl urls
- why dora?
– mobstor was built on filers
– dora was developed to replace filers
- commodity box costs
– < $4k / box
– highest density sata: 12 drives x 1tb/drive
- k/v store vs. relational db?
– everyone would love acid transactions, but they also need consistency & geo replication
- uses of k/v stores?
– mail uses it to assoc abuse records w/ ip addresses

about 150 ppl, 3/4+ full

falicies
- machines are reliable
- machines are unique or identifiable
- a data set should fit on one machine

hadoop
- it’s not a database
– it doesn’t serve data in real-time
– it augments existing DBs
– it does enable deeper analysis that would normally slow a relational DB
- leverages commodity hardware for big data & analytics
- cloudera does for hadoop what redhat does for linux

examples
- fox
– hat ppl are watching on set-top obxes
- autodesk
- D.E.Shaw
– analyze financial data
- mailtrust
– use hadoop to process mail logs and generate indexes that suport staff can use to make adhoc queries

data
- scientific and experimental data
- storage
– multiple machines are req’d to store the amount of data we’re interested in
– replication protects data from failure
— data is also 3 times as available

map-reduce
- allows for processing data locally
- allows for jobs to fail and be restarted

hadoop’s fault tolerance
- handled at software level

using hadoop
- map-reduce
– natively written in java
– map-reduce can be written in any language
- hive
– provides sql interface
- pig
– high level lang for ad-hoc analysis
– imperative lang
– great for researchers and techinical prod. managers

high performance DB and analytics.  when is it time for hadoop
- in general
– generation rate exceeds load capacity
– performance/cost considerations
– workloads that impede performance
- db
– 1000s of transactions per second
– many concurrent queries
– read/write
– many tables
– structured data
– high-end machines
– annual fees
- hadoop
– append only update pattern
– arbitrary keys
– unstructured or structured data
– commodity hardware
– free, open source

arch
- traditional: web server –> db –> oracle –> biz analytics
- hadoop: web server –> db –> hadop –> oracle –> biz analytics

cost
- data storage costs drops every year
- hadoop removes bottlenecks; use the right tool for the job
- makes biz intel apps smarter

tools
- cloudera’s distro for hadoop
- cloudera desktop

open cloud access protocol (opencap)
- definition, deployment, and life cycle mgmt of cloiud resources
- aloc, provisioning, and metering of clourd resources
- metasdat,/registry for cloud resources
- virtual infrastructure
- why ietf?  they are brilliant folks.  we’re not attaching any vendor-specific details
- open source implentation planned
- structure
– resource model
— all infastructure
— nodes, networks, etc
– resource properties
— modeled as json objects
— standard catalog of attributes, extensible
– resource operations
— operation, control, etc.
– management ntification services
…
- smtp is simple.  protocols must be simple
- traffic server has bindings built in (or vice versa?)
- open cloud consortium
– a national testbed to bring clouds together

raghu ramakrishnan
- a triumphant preso
- “key chalengeds in cloud comoputing .. and the y! approach”

this is a watershed time.  we’ve spent lots of time building packabged software now wer’re moving to the cloud

key challenges
- elastic scaling
- availabiolity
– if the cloud goes down, everyone is hosed.  consistency or performance myst be traded for availoability.
- handliong failures
– if things go wrong, what can the developer count on when things come up?
- operational efficiency
– cloud managers are db admins for 1000s of clients
- the right abstractions

yahoo’s cloud
- the cloud is an ecosystem.  it’s bigger than a single componenet.  all the pueces must work together seamlessly.

data management in the cloud
- how to make sense of the many options
- what are you trying todo?
- oltp vs olap
- oltp
– random access to a few records
– read-heavy vs write-heavy
- olap
– scan access to a large number of records
– by rows vs columns vs unstructired
- storage
– common features
— managed service. rest apis
— replication
— global footprint
– sherpa
– mopbstor

y! storage problem
- small records, 100kb or less
- structured records, lots of fields
- extreme data scale

typical applications
- user logins and profiles
– single=-record transactions suffice
- events
– alerts, social network activity
– ad clicks
app-specific data
- postings to messsage boards
- uploaded photos and tags

vsld data serving stores
- scale based on partitioning data accross machines
- range selections
– requests span machines
- availability
- replication
- durability
– is it required?
- how is data stored on a single machine?

the cap theorem
- consistency vs availability vs partition tolerance
- consistency => serializability

approaches to cap
- use a single version of a db w/ defered reconciliation
- defer transaction commit
- eventual consistency eg dynamo
- restrict transatctions eg sharded mysql
- object timelines, eg sherpa
- ref: julianbrowne.cim/artice/viewer/brewers-cap-theorem

single slide hadoop primer
- hadoop is wrte optimized, not ideal for serving

out there in the world
- oltp
– oracle, mysql,
– write optimized: cassandra
– main-mem; memchached

ways of using hadoop
- data workloads -> olap -> pig for row ops, zebra for column ops, map reduce for others

hadoop based apps
- we own the terasort benchmark

sherpa`
- parallel db
- geo replication
- structured, flexible schemas
- hashed and ordered tables
- components
– req -> routers -> (record looked up, if necessary) -> lookup cached -> individual machine
- raghu is awesome (“And then!”, sprinting through dense slides)
- write-ahead
- asynch replication
– why? we’re doing geo replication due to the physics involved
– supposing an eearthquake hits and ca falls in th ocean, two users can continue to update their profiles
- consistency model
– acid requiores synch updates
– eventual consistency works
– is there any middle ground?
– sherpa follows a timeline of changes achieved through a standard per-record primary copy protocol

operability
- cloud allows us to apperate at scale
- tablet splitting and balancing
- automatic transfer of mastership

comparing systems
- main point: all of this needs to be thought through and handled automatically

example
- sherpa, oracle, mysql work well for oltp

banchmark tiers
- cluster performance
- replication
- scale out
- availability
- we’d like to do this a group effort, in keeping w/ our philosophy

the integrated cloud
- big idea: declrative lang for specifying structure of service
- key insight: multi-env
- central mechanism: the integrated cloud
- surrendra will talk about htis

foundation componenets
- how to describe app
- desc for resources, entrypoijts, bindings, etc

yst hadled 16.4 million uniques for mj death news

acm socc
- acm symposium on cloud computing

Shelton Shugar just delivered an excelllent keynote address “Accelerating Innovation with Cloud Computing” at the 4th “Cloud Conference and Expo”:http://cloudcomputingexpo.com/ in Santa Clara.  The subtitle of the expo is “”.  This is also the 7th annual virtualization summit.

notes
Yahoo is not here to sell you anything; we’re not into consulting or selling software.  At Yahooo! cloud computing is not about saving money.  Our motivation arises from the fact that cloud computing drives innovation.  Cloud computing is the “engine of innovation”.  Yahoo! has hundreds of products and platforms all over the world.  Many of these products were the result of acquisition, so they came onboard w/ their own infrastructure, down tot he metal.  Cloud computing at y! is about streamlining the services these products and platforms require.  We store hundreds of petabytes of data all ove the world, and petabytes of internet traffic daily.  We think about scale foremost and features second.

cloud strategy
we are building a private cloud, deployed in data centers all over the world.  focusing in two areas: data processing and serving.  data processing refers to data minigna nd analysis.  serving refers to app environments for our products, edge capabilities for fast delivery, and a channel for data to flow into storage.  This is a multi-year effort.  “Open source plays a central role”.  We both consume and produce open source.

inside the y! cloud
5 buckets: edge services, cloud serving where we host apps w/in y!, online storage for serving content to consumers, a batch rocessing data warehouse, data collection services to clean, de-dup, and filter incoming data.

Serving is based on the Yahoo! Traffic Server.  Over half of all y! traffic flows through YST.

The app serving layer is based on a tiered architecture.  Apps can be cloned.  Traffic can be split natively, which allows for bucket testing.  THis frees developer from having to worry about versions of the platform, location of machines, etc.  Capacity can be moved via point and click.

Storage
Uses Restful apis.  Deployed worldwide.  Global replication is supported natively.  Multiple consistency models are provided.  Mobstore (mass object store) is used to store large objects (1mb-2gb) such as images and video.  Objects are immutable.  Structured content is provided via a product called Sherpa, a key-value store.  Content can be replicated easily.  Sherpa is intended to support enough of the capabilities properties used to build.mainatin relational dbs for.

Batch processing is oriented around Hadoop.  This has been running for a few years.  It now runs on 10s of thousands of machines.  80PB worth.  We use it to optimize our sadvertising, process weblogs.  1000s of yahoos are trained to run jobs on it.  hdfs allows thousands of computers to be treated as a single machine.  Pig is a higher-level procedural lang that generates map-reduce code.  It’s almost as efficient a well-written map-reduce code.  the internal joke is that most people don’t write well-written map-reduce code.  We’re building columnar storage.

An example: the y! homepage
When a user visits the homepage, the user is usung y! cloud services.  Content is optimized using a feedback loop to provide relevant stories in the news offered.  Hadoop is used to optimize ad matching.  Hadoop is used to build the search index.  edge services are used to cahce and load-balance the page content, normalize the news feeds.

Another example of useafge: y! mail.  Hadoop is used to identify and filter spam.  before hadoop, mail engineers had to spend lots of time maintaining storage and machines to process a huge amount of data.  hadooop abstracts scale for processing enormous data, handles failures, and manages multiple users.  this allows the scientists to focus on their jobs.  mail uses cloud storage’s replication services to help detect abuse.

Y! soprts usage of cloud services.  Edges services provides a proxy service to route requests for dynamic content.  this allows y! sports to provide the most up-to-date content.  People want scores as fast as possible.  the cunsumers are happy due to faster access to content.

y! finance.  y! is #1 for finance.  finance uses hadoop to spped advertisinf optimaization by importing resource utilization.

yql is an sql-like language.  it allows developers to qu ery, filter, join etc data.  yql uses sherpa instead of mamnaging its oawn storage.

open source @ y!
hadoop.  we contribute our code for hadoop to open source.  external developers benefit and contribute back.  pig is open source.  zookeeper is a system used to coordinate mutliple systems.  open cissur is a consortium was designed to facilitate to cloud computing.  it has 9 members.  y! contribution is m45, w/ 1000 cores. we work w/ some of the leading universities in the world.   We’ve built an enormous community around hadoop.  we can hire people straight out of university.  open source attracts the best and the brightest.

About 500 people were in attendance.

The highlight of the talk was his announcement of the newly open-sourced Yahoo! Traffic Server, now an Apache Incubator project.  A “recent post”:http://ostatic.com/blog/guest-post-yahoos-cloud-team-open-sources-traffic-server on OStatic gives more information about the project.  trafic server can process up to 34k trasnsactions/sec on commodity hardware.  it’s modular.  it’s how we implement our cahcing, proxy, load balancing, etc.  we push 400tb daily through it.  we use it in online storage to help direct traffic.  we’re hoping to create a vibrnt community around traffic server like we did w/ hadoop.

Back in june, we announced the y! distribution of hadoop.  we select the code we need and test it well.  it’s a solid collection of code that’s been proven to work.  shelton annouced that we’re now updating our releas.

change
we’re fully committed to cloud computing.  “moving to the cloud requires change”.  if you’re like us, w/ lots of legacy systems, you need to make a big organization commitment.  it’s more like  amarriage than a transaction.  it takes invesment to create these services and migrate to them.  it takes time.  ours is a multi-year effort.  cloud computing is worth it for us.  it’s changing our cutlure.  we’re able to deploy so much faster than before.

Notes from Christian Heilmann’s Developer Evangelism handbook

• The Developer Evangelist Handbook
  • “If you are a clever company you also open your products to third party developers and release interfaces to the world” – http://developer-evangelism.com/handbook.php#defining
  • “If something new comes out of your company that should get out to developers take it and access it like an outside developer would. Develop something with it, then document what you have developed and then start writing how you build the thing – and voilà – you’ve got half an article or presentation already finished.” – http://developer-evangelism.com/handbook.php#mindset
  • common niches of evangelism: “Writing code tutorials, Blogging, Public speaking, Training, Social web coverage” – http://developer-evangelism.com/handbook.php#role

• remove the brand
  • “As a developer evangelist you have to keep your independence.”
  • “Your independence and your integrity is your main weapon. If you lost it you are not effective any longer. People should get excited about what you do because they trust your judgment – not because you work for a certain company.”
  • ways to work w/ the competition: “Remain an independent voice”, “Become a specialist in a certain underlying technology”, “Keep your finger on the pulse” – http://developer-evangelism.com/brand-and-competition.php#competiton
  • “You can’t be a professional evangelist and bad-mouth the competition at the same time. We all are professionals and work on projects to make the web a better place.” – http://developer-evangelism.com/brand-and-competition.php#respect
  • “Acknowledge when the competition is better”
  • “Know about the competition”

• Work with your own company
  • “Your job as a developer evangelist is to listen to developers, understand their problems and communicate with management to try to sort the issues out.” – http://developer-evangelism.com/internal.php
  • “There is no “off the record”” – http://developer-evangelism.com/internal.php#companychanges
  • “if people ask you what is going on don’t say “no comment” as that implies you know something but are not allowed to say it. Simply state that you are not in a position to know yet but that you are investigating.”

what is a module in yui 3?
- modules are not plugins, but there is a plugin module
- module names are passed into a sandbox w/ the ‘use’ method
- prefer YUI().use instead of var Y = new YUI(); Y.use …
- you can have multiple use() calls to defer loading
- community modules vs basic yui core team modules

custom modules
- registration
– by seed YUI().use
— seed will import
– by inclusion
— manually add script include and then YUI.use
– YUI(config)
— most performant
— this takes advantage of onload handling
— reduces number of http requests req’d in ‘by inclusion’
— organization
—- use YUI_config global var to manage registration
—- you can have multiple config options

building custom modules
- YUI.add(‘foo’, fn(Y){mod code}, version, requirement list);
- naming convention: utilities are all lowercase, classes are camelcase w/ uppercase leading char
- plugins extend host modules
- stack: utilities –> classes –> plugins –> mashups

how to use and build plugins
- plugins allow us to extend an existing class at runtime
- the def of a plugin looks much like that for a module class
- instead of extending y.base, we extend y.plugin.base

mashups and legacy code
- using multiple modules, including external dependencies, enhancing dom, defining event listeners
- use case: using a pre-existing yui2-based object in yui3
- check out zakas’ talkon scalable app arch
- cool: organize app as module repo
- conclusions
– define apps at a granular level
– modular apps are easier to test
– share code thru yui3 gallery
– use yui custom modules to integrate pre-existing code

q/a
- differences btwn yui2 and yui3 lazy loading?
– yui3 will load everything as a single item, if module requirements are defined using config option
– yui3 will load items in the order they are specified
- reusing modules across multiple sandbox
– yes, if defined as such in config

github/caridy
twitter/caridy
caridy.name

I’ve got yaws (git hash 5f35f5b7451ea4388c53df9f4e00caad0caa6b45) running on CentOS 5.3.  I just added a virtual host entry in yaws.conf and tried to restart yaws, but the restart failed:

[me@mymachine /]$ sudo etc/init.d/yaws restart
Stopping yaws:                                             [FAILED]

After hunting around a bit, it seems yaws will fail to restart (and stop) if the docroot doesn’t point to an actual directory.  In my case, I added the virtual host entry before actually creating the docroot directory.  Making the directory fixed the problem.

• ref http://www.scribd.com/doc/17914883/US-Federal-Cloud-Computing-Initiative-RFQ-GSA
• “cloud computing is a major feature of the president’s initiative to modernize information technology” (1)
• “cloud computing is a model for enabling …” (1)
• General Cloud Computing Requirements
  • “The Contractor shall support internet bandwidth of at least 1Gb/s ” (4)
  • “The Contractor shall have a minimum of two data center facilities at two different geographic locations in the Continental United States (CONUS) and all services acquired under the BPA will be guaranteed to reside in CONUS” (4)
  • “The Contractor shall support provisioning of practically unlimited storage, computing capacity, memory (e.g. at 1000 times our minimum resource unit metrics), independently from the physical location of the facilities.” (4)
  • “The Contractor shall support service provisioning and de-provisioning times (scale up/down), making the service available within near real-time of ordering.” (4)

• IaaS Common Technical Requirements
  • “The Contractor shall provide the ability to provision virtual machines, storage and bandwidth dynamically, as requested and as required. ” (6)
  • “Contractor shall support secure provisioning, de-provisioning and administering [such as Secure Sockets Layer (SSL)/Transport Layer Security (TLS) or Secure Shell (SSH)]in its service offerings.” (6)
  • “The Contractor shall support the terms of service requirement of terminating the service at any time (on-demand). ” (6)
  • “The Contractor shall provide a robust, fault tolerant infrastructure that allows for high availability of 99.95%.”
  • “Within a month of a major outage occurrence resulting in greater than 1-hour of unscheduled downtime. The Contractor shall describe the outage including description of root-cause and fix. ” (6)
  • “Service provisioning and de-provisioning times (scale up and down) in near real-time” (6)
  • “The Contractor shall provide a secure, dual factor method of remote access which allows Government designated personnel the ability to perform duties on the hosted infrastructure.” (6)
  • “The Contractor shall manage data isolation in a multi-tenant environment.” (6)
  • “The Contractor shall manage data remanence throughout the data life cycle.” (6)
  • “The Contractor shall enable Order Management via Application Programming Interface (API).” (7)
  • ”…80% threshold for the order. ” (8)
  • “The Contractor shall provide Trouble Ticketing via API.” (8)
  • “The Contractor shall maintain user profiles and present the user with his/her profile at the time of login.” (8)
  • “The Contractor shall identify Tier 1 Internet providers it is peered with, and where this peering occurs. The Contractor shall provide its Autonomous Number System” (8)
  • “IP Addressing: 1) The Contractor shall provide IP address assignment, and if capable, include Dynamic Host Configuration Protocol (DHCP). 2) The Contractor shall provide IP address and IP port assignment on external network interfaces. 3) The Contractor should provide dedicated virtual private network (VPN) connectivity between customer and the vendor. 4) The Contractor should map IP addresses to domains owned by the Government, allowing websites or other applications operating in the cloud to be viewed externally as Government URLs and services. 5) The Contractor shall provide an infrastructure that is IPv6 capable.” (9)
  • “Cloud Storage Services shall consist of the following REQUIRED Services, Service Options, Service Attributes and Service Units. ” and API request definitions (”…TPUT operations performed against Container/Bucket are used to create that container …”) (10)
  • “Service Units … Provides the requirements for the minimum purchasable units of the Service Attributes. These Service Units may be purchased the minimum or in multiples of the minimum. The customer shall be billed for the actual service units used.” (10)

• “storage for files / objects supporting a single file/object sizes of up to 5GB” (11)
• “storage tiers” (11)
• “A minimum equivalent CPU processor speed of 1.1GHz shall be provided. Additional options for CPU Processor Speed may be provided, however it is not required. … The CPU shall support 32-bit and 64-bit operations” (13)
• “Windows and LINUX OS’s at a minimum. Additional OS options may be provide or supported; however, this is not required.”(13)
• “Physical memory (RAM) reserved for virtual machine instance or Computing supporting a minimum of 1GB of RAM.”(13)
• “Disk Space allocated for virtual machine supporting a minimum of 40GB.”(13)
• “Table 11: Virtual Machine Bundles” (13)
• “Provide the capability to dynamically reallocate virtual machines based on load, with no service interruption.” (14)
• “Perform Live migrations (ability to move running VM’s) from one host to another.” (13)
• Table 13: Cloud Web Hosting Requirements (15)
• Table 14: Cloud Web Hosting Bundling (18)
• “The Government and the Contractor will work in good faith to establish an Interconnection Security Agreement (ISA) and/or a Memorandum of Understanding (MOU) as provided in the National Institute of Standards and Technology (NIST) Special Publication 800-47, Security Guide for Interconnecting Information Technology Systems, Appendix A – Security Requirements and Appendix B – Personnel Security. ” (18)
• “6.3 Management Reporting Deliverables” (20)