This is an old revision of the document!
Table of Contents
Use Hadoop (test) Cluster
Word count, vanilla
Ross writes ….
I did the classic map-reduce example: a word count of a flat text file, in this case James Joyce's Ulysses.
- First, I downloaded the data.
- Second, I copied the data from my home folder to the Hadoop Distributed File System (HDFS).
- Third, I ran the word count example bundled with the Cloudera Hadoop distribution.
- Fourth, I copied and merged the results from the HDFS back to the local filesystem.
- Finally, I inspected the output (reformatted below).
curl http://www.gutenberg.org/cache/epub/4300/pg4300.txt > ulysses.txt hadoop dfs -copyFromLocal ~/ulysses.txt /tmp/ulysses.txt hadoop jar /usr/lib/hadoop/hadoop-examples.jar wordcount /tmp/ulysses.txt /tmp/results hadoop dfs -getmerge /tmp/results ~/results.tsv hadoop dfs -rmr /tmp/results sort --numeric-sort --key=2 --reverse results.tsv | head --lines=5 the 13600 of 8127 and 6542 a 5842 to 4787
I use the Hadoop Streaming API with R (via RHadoop) and Python (via mrjob). I highly recommend these solutions because you can write map-reduce jobs in just a few lines of code.
Python Wordcount
This is the IBM article on the “Building Hadoop” page.
Assumption: Long sentences and large word lengths mean complexity. Told you, example.
Read up on the IBM article about the benefits of this example. I just wanted to know how python front ended the Hadoop cluster. First we download some books, rename the files, and note one is compressed.
wget -U firefox http://www.gutenberg.org/cache/epub/76/pg76.txt wget -U firefox http://www.gutenberg.org/cache/epub/3285/pg3285.txt mv pg3285.txt DS.txt mv pg76.txt HF.txt gzip DS.txt
Then we stage them up in HDFS.
hadoop fs -put DS.txt.gz /tmp hadoop fs -put HF.txt /tmp
And look at what's up there.
hadoop fs -ls /tmp -rw-r--r-- 3 hmeij07 supergroup 459378 2013-05-23 14:24 /tmp/DS.txt.gz -rw-r--r-- 3 hmeij07 supergroup 610155 2013-05-23 14:24 /tmp/HF.txt drwxrwxrwt - mapred supergroup 0 2013-05-16 13:58 /tmp/hadoop-mapred drwxr-xr-x - qactweet1 supergroup 0 2013-05-17 12:52 /tmp/ngrams -rw-r--r-- 3 hmeij07 supergroup 1573150 2013-05-20 14:26 /tmp/ulysses.txt
Next we build two files in our local home directory; a mapper and a reduce file
vi mapper.py # with the following contents ...(python=indentation matters!)
#!/usr/bin/env python
import sys
# read stdin
for linein in sys.stdin:
# strip blanks
linein = linein.strip()
# split into words
mywords = linein.split()
# loop on mywords, output the length of each word
for word in mywords:
# the reducer just cares about the first column,
# normally there is a key - value pair
print '%s %s' % (len(word), 0)
vi statsreducer.awk # with the following contents ...
awk '{delta = $1 - avg; avg += delta / NR; \
mean2 += delta * ($1 - avg); sum=$1+sum } \
END { print NR, sum/NR, sqrt(mean2 / NR); }'
# finally, make them executable
chmod u+x mapper.py
chmod u+x statsreducer.awk
Let us see what these programs do on command line:
# basic word length and occurrence count
# mapper file
zcat DS.txt.gz | ./mapper.py
9 0
2 0
3 0
5 0
10 0
2 0
4 0
5 0
3 0
7 0
# reducer filer, L2R:
# NR - the number of words in total
# sum/NR - the average word length
# sqrt(mean2/NR) - the standard deviation
zcat DS.txt.gz | ./mapper.py | sort | ./statsreducer.awk
10 5 2.68328
# consult IBM article on how to interpret this
Next we run these commands in the Hadoop environment. The -output naming is unconventional, the *.txt output is actually a directory, not a file. Please note that everything prefixed by a tilde (~) is in our home directory not on Hadoop's filesystem.
hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar \ -input /tmp/HF.txt -output /tmp/HFstats.txt \ -file ~/mapper.py -file ~/statsreducer.awk \ -mapper ~/mapper.py -reducer ~/statsreducer.awk hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar \ -input /tmp/DS.txt.gz -output /tmp/DSstats.txt \ -file ~/mapper.py -file ~/statsreducer.awk \ -mapper ~/mapper.py -reducer ~/statsreducer.awk
Lets look at the output. One is wordier than the other.
hadoop fs -cat /tmp/HFstats.txt/part-00000 14 4 1.77281 hadoop fs -cat /tmp/DSstats.txt/part-00000 10 5 2.68328
Clean up.
hadoop fs -rmr /tmp/DFstats.txt/* # for some reason -rmr does not imply remove recursively, bug? hadoop fs -rmr /tmp/HFstats.txt Deleted hdfs://qactweet1:54310/tmp/HFstats.txt
Ok, so I get this. We should be able to this with other languages and databases.
Rhadoop
Perl Hadoop's native Streaming
First we create a script for the map step, note that this only print individual words back
vi wc_mapper.pl # make user executable and add contents below
#!/usr/bin/perl -w
# read in a file of text, print word by word
while(<>) {
@line = split;
foreach $word (@line) {
print "$word\n";
}
}
Next we create a script for the reduce step, note that the use of a hash avoids the sort
vi wc_reducer.pl # make user executable and add contents below
#!/usr/bin/perl -w
# store words in hash and print key-value results
while (<>) {
chomp;
$seen{$_}++;
}
foreach $key (keys %seen) {
print "$seen{$key} $key\n";
}
Next test this on the command line
perl wc_mapper.pl HF.txt | perl wc_reducer.pl | sort -rn | head 6050 and 4708 the 2935 a 2903 to 2475 I 1942 was 1733 of 1427 it 1372 he 1367 in
Load the text input file up to Hadoop's HDFS and submit the job
hadoop fs -put HF.txt /tmp hadoop dfs -ls /tmp # results Found 2 items -rw-r--r-- 3 hmeij07 supergroup 459378 2013-05-23 14:24 /tmp/DS.txt.gz -rw-r--r-- 3 hmeij07 supergroup 610155 2013-05-23 14:24 /tmp/HF.txt # submit, note that -mapper and -reducer options are paired with a -file option # pointing to files in our home dirrectory (not HDFS). these files will be copied # to each datanode for execution, and then the results will be tabulated hadoop jar \ /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar \ -input /tmp/HF.txt -output /tmp/HF.out \ -file ~/wc_mapper.pl -mapper ~/wc_mapper.pl \ -file ~/wc_reducer.pl -reducer ~/wc_reducer.pl hadoop fs -ls /tmp/HF.out # results Found 3 items -rw-r--r-- 3 hmeij07 supergroup 0 2013-05-24 15:10 /tmp/HF.out/_SUCCESS drwxrwxrwt - hmeij07 supergroup 0 2013-05-24 15:10 /tmp/HF.out/_logs -rw-r--r-- 3 hmeij07 supergroup 161788 2013-05-24 15:10 /tmp/HF.out/part-00000 hadoop fs -cat /tmp/HF.out/part-00000 | sort -rn | head # results 6050 and 4708 the 2935 a 2903 to 2475 I 1942 was 1733 of 1427 it 1372 he 1367 in # clean up space
Perl Hadoop::Streaming
<code>
yum install cpan cpan> install Hadoop::Streaming
[root@qactweet1 hmeij07]# cat hadoop_streaming.txt Installing /usr/local/share/perl5/Hadoop/Streaming.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Mapper.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Reducer.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Combiner.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Role/Iterator.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Role/Emitter.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Reducer/Input.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Reducer/Input/Iterator.pm Installing /usr/local/share/perl5/Hadoop/Streaming/Reducer/Input/ValuesIterator.pm
<code>
- How to use this?
MySQL
- to come
SAS
- I think we'll wait for the next version, to beta'ish
“ Where Hadoop is concerned the latest release already runs on the platform, technically, but it's limited to a SAS-customized version of the open source software based on Apache Hadoop v1.0 (also known as version 0.20.20x). SAS says HPA will run on mainstream distributions of Hadoop from the likes of Cloudera, with an upcoming December release of HPA that will based on Apache Hadoop v2.0 (also known as version 0.23).
Whether you're using SAS's current Hadoop software or plan to embrace the v2.0 release, HPA provides a graphical user interface that lets you tap HDFS, MapReduce, Pig, and Hive to apply SAS analyses to the vast data sets residing on Hadoop. MapReduce is the primary model for processing data on Hadoop. Pig is an open source Apache programming tool and language for writing MapReduce jobs. Hive is data warehousing infrastructure built on top of Hadoop that supports data summarization, query, and analysis. HPA also supports Pig and MapReduce code generation, visual editing and syntax checking. Finally SAS Data Integration Studio data transformations and SAS DataFlux data quality routines have also been adapted to Hadoop. ”
