notes from cloudera basic training > hadoop mapreduce deep dive
//we moved quickly through this, so the notes are sparse
- job
– a full program
- task
– by default, hadoop creates the same amount of tasks as there are input blocks
– task attempts
— tasks are attempted at least once
— multiple attempts in parellel are performed w/ speculative execution turned on
- tasktracker
– forks jvm process for each task
- job distribution
– mapreduce programs = jar + xml config
– running a job puts jar and xml in hdfs
- data distribution
– data locality decreases when multiple tasks are running
- mapreduce flow
– client creates joconf
— identify map and reducer classes
— specify inputs/outputs
— set optional settings
– job launches jobclient
— runjob blcks until the job completes
— submitjob is non-blocking
– …
– tasttracker
— perioducally query jobtracker for work
– …
– write for cache coherency (re-use objects in loops(?))
— reusing memory locations => 2x speed-up
— all k/v pairs given by hadoop use this model
//is avro comparable to thrift?
- getting data to mapper
– data sets are specified
– input sets contain at least 1 record and are composed of full blocks
- file input format
– most people use SequenceFileInputFormat
– usually we store all our data in hdfs and then ignore what we don’t need, rather than spending time formatting the data when it’s input
– …
- shuffling
– what happens btwn map and reduce
- write the output
– OutputFormat is analagous to InputFormat
