DokuWiki

User Tools

Site Tools

Trace:
cluster:114

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:114 [2013/05/17 14:20]
hmeij [Hadoop Cluster] 		cluster:114 [2013/09/10 18:59] (current)
hmeij
Line 2: Line 2:
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
  
-==== Hadoop Cluster ====+==== Build Hadoop (test) Cluster ====
  
-These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added.  Please review theese sites so this page makes sense to you.+[[cluster:115|Use Hadoop (test) Cluster]] 
 + 
 +These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added.  Please review these sites so this page makes sense to you.
  
   * CTOvision [[http://ctovision.com/2012/01/cloudera-hadoop-quickstart/]]    * CTOvision [[http://ctovision.com/2012/01/cloudera-hadoop-quickstart/]] 
Line 15: Line 17:
   * Noll [[http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/]]   * Noll [[http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/]]
   * IBM article [[http://www.ibm.com/developerworks/data/library/techarticle/dm-1209hadoopbigdata/]]   * IBM article [[http://www.ibm.com/developerworks/data/library/techarticle/dm-1209hadoopbigdata/]]
 +
 +And
 +
 +  * White [[http://hadoopbook.com/]]
  
 ==== Building ==== ==== Building ====
Line 32: Line 38:
 alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1600  alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1600 
 alternatives --auto java alternatives --auto java
-# fix this as some Haddop scripts look at this location+# fix this as some Hadoop scripts look at this location
 cd /usr/java cd /usr/java
 ln -s ./latest/bin ln -s ./latest/bin
Line 63: Line 69:
 vi /etc/sysconfig/iptables vi /etc/sysconfig/iptables
 # hadoop # hadoop
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50070 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50070 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50075 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50075 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50090 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50090 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50105 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50105 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50030 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50030 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50060 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50060 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8020 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8020 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50010 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50010 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50020 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50020 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50100 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50100 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8021 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8021 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 9001 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 9001 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8012 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8012 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54310 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54310 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54311 -j ACCEPT+-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54311 -j ACCEPT
 # plus 127.0.0.1:0 and maybe 9000 # plus 127.0.0.1:0 and maybe 9000
 # hadoop admin status # hadoop admin status
Line 104: Line 110:
   * Edit the conf files, consult Dakini site for content   * Edit the conf files, consult Dakini site for content
   * Copy those 3 files to all work nodes   * Copy those 3 files to all work nodes
-  * The display command should point to the MyClsuster files+  * The display command should point to the MyCluster files
  
 <code> <code>
Line 163: Line 169:
 </code> </code>
  
-  * Allright, lets some filesystem entries+  * Alright, lets some filesystem entries
  
 <code> <code>
Line 182: Line 188:
   * And from a remote machine access your head node   * And from a remote machine access your head node
     * Hadoop Map/Reduce Administration     * Hadoop Map/Reduce Administration
-      * [[http://qactweet1.wesleyan.edu:50030]] +      * [[http://headnode.wesleyan.edu:50030]] 
     * The Namenode     * The Namenode
-      * [[http://qactweet1.wesleyan.edu:50070/dfshealth.jsp]]+      * [[http://headnode.wesleyan.edu:50070]]
  
 TODO TODO
  
-  * Install Streaming if not installed already 
   * Run some jobs   * Run some jobs
 +  * Find a MOOC course
  
 \\ \\
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
cluster/114.1368800429.txt.gz · Last modified: 2013/05/17 14:20 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki