This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
cluster:114 [2013/05/17 14:20] hmeij [Hadoop Cluster] |
cluster:114 [2013/09/10 18:59] hmeij |
||
---|---|---|---|
Line 2: | Line 2: | ||
**[[cluster: | **[[cluster: | ||
- | ==== Hadoop Cluster ==== | + | ==== Build Hadoop |
- | These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added. | + | [[cluster: |
+ | |||
+ | These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added. | ||
* CTOvision [[http:// | * CTOvision [[http:// | ||
Line 15: | Line 17: | ||
* Noll [[http:// | * Noll [[http:// | ||
* IBM article [[http:// | * IBM article [[http:// | ||
+ | |||
+ | And | ||
+ | |||
+ | * White [[http:// | ||
==== Building ==== | ==== Building ==== | ||
Line 32: | Line 38: | ||
alternatives --install / | alternatives --install / | ||
alternatives --auto java | alternatives --auto java | ||
- | # fix this as some Haddop | + | # fix this as some Hadoop |
cd /usr/java | cd /usr/java | ||
ln -s ./ | ln -s ./ | ||
Line 63: | Line 69: | ||
vi / | vi / | ||
# hadoop | # hadoop | ||
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50070 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50070 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50075 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50075 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50090 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50090 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50105 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50105 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50030 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50030 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50060 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50060 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8020 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8020 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50010 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50010 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50020 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50020 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50100 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50100 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8021 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8021 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 9001 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 9001 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8012 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8012 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54310 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54310 -j ACCEPT |
- | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54311 -j ACCEPT | + | -A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54311 -j ACCEPT |
# plus 127.0.0.1:0 and maybe 9000 | # plus 127.0.0.1:0 and maybe 9000 | ||
# hadoop admin status | # hadoop admin status | ||
Line 104: | Line 110: | ||
* Edit the conf files, consult Dakini site for content | * Edit the conf files, consult Dakini site for content | ||
* Copy those 3 files to all work nodes | * Copy those 3 files to all work nodes | ||
- | * The display command should point to the MyClsuster | + | * The display command should point to the MyCluster |
< | < | ||
Line 163: | Line 169: | ||
</ | </ | ||
- | * Allright, lets some filesystem entries | + | * Alright, lets some filesystem entries |
< | < | ||
Line 182: | Line 188: | ||
* And from a remote machine access your head node | * And from a remote machine access your head node | ||
* Hadoop Map/Reduce Administration | * Hadoop Map/Reduce Administration | ||
- | * [[http://qactweet1.wesleyan.edu: | + | * [[http://headnode.wesleyan.edu: |
* The Namenode | * The Namenode | ||
- | * [[http://qactweet1.wesleyan.edu: | + | * [[http://headnode.wesleyan.edu: |
TODO | TODO | ||
- | * Install Streaming if not installed already | ||
* Run some jobs | * Run some jobs | ||
+ | * Find a MOOC course | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: |