DokuWiki

User Tools

Site Tools

Trace:
cluster:114

Warning: Undefined array key -1 in /usr/share/dokuwiki/inc/html.php on line 1458

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:114 [2013/05/17 10:20]
hmeij [Hadoop Cluster] 		cluster:114 [2013/09/10 14:59] (current)
hmeij
Line 2: Line 2:
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
  
-==== Hadoop Cluster ====+==== Build Hadoop (test) Cluster ====
  
-These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added.  Please review theese sites so this page makes sense to you.+[[cluster:115|Use Hadoop (test) Cluster]] 
 + 
 +These are my notes building a test Hadoop cluster on virtual machines in VMware. They consists of a blending of instructions posted by others with my commentary added.  Please review these sites so this page makes sense to you.
  
   * CTOvision [[http://ctovision.com/2012/01/cloudera-hadoop-quickstart/]]    * CTOvision [[http://ctovision.com/2012/01/cloudera-hadoop-quickstart/]] 
Line 15: Line 17:
   * Noll [[http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/]]   * Noll [[http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/]]
   * IBM article [[http://www.ibm.com/developerworks/data/library/techarticle/dm-1209hadoopbigdata/]]   * IBM article [[http://www.ibm.com/developerworks/data/library/techarticle/dm-1209hadoopbigdata/]]
 +
 +And
 +
 +  * White [[http://hadoopbook.com/]]
  
 ==== Building ==== ==== Building ====
Line 32: Line 38:
 alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1600  alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1600 
 alternatives --auto java alternatives --auto java
-# fix this as some Haddop scripts look at this location+# fix this as some Hadoop scripts look at this location
 cd /usr/java cd /usr/java
 ln -s ./latest/bin ln -s ./latest/bin
Line 63: Line 69:
 vi /etc/sysconfig/iptables vi /etc/sysconfig/iptables
 # hadoop # hadoop
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50070 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50070 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50075 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50075 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50090 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50090 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50105 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50105 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50030 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50030 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50060 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50060 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8020 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8020 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50010 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50010 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50020 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50020 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 50100 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 50100 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8021 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8021 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 9001 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 9001 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 8012 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 8012 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54310 -j ACCEPT +-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54310 -j ACCEPT 
--A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.1.81-129.133.1.88 --dport 54311 -j ACCEPT+-A INPUT -m state --state NEW -m tcp -p tcp -m iprange --src-range 129.133.x.xxx-129.133.x.xxx --dport 54311 -j ACCEPT
 # plus 127.0.0.1:0 and maybe 9000 # plus 127.0.0.1:0 and maybe 9000
 # hadoop admin status # hadoop admin status
Line 104: Line 110:
   * Edit the conf files, consult Dakini site for content   * Edit the conf files, consult Dakini site for content
   * Copy those 3 files to all work nodes   * Copy those 3 files to all work nodes
-  * The display command should point to the MyClsuster files+  * The display command should point to the MyCluster files
  
 <code> <code>
Line 163: Line 169:
 </code> </code>
  
-  * Allright, lets some filesystem entries+  * Alright, lets some filesystem entries
  
 <code> <code>
Line 182: Line 188:
   * And from a remote machine access your head node   * And from a remote machine access your head node
     * Hadoop Map/Reduce Administration     * Hadoop Map/Reduce Administration
-      * [[http://qactweet1.wesleyan.edu:50030]] +      * [[http://headnode.wesleyan.edu:50030]] 
     * The Namenode     * The Namenode
-      * [[http://qactweet1.wesleyan.edu:50070/dfshealth.jsp]]+      * [[http://headnode.wesleyan.edu:50070]]
  
 TODO TODO
  
-  * Install Streaming if not installed already 
   * Run some jobs   * Run some jobs
 +  * Find a MOOC course
  
 \\ \\
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
cluster/114.1368800429.txt.gz · Last modified: 2013/05/17 10:20 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki