User Tools

Site Tools


cluster:115

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
cluster:115 [2013/09/10 15:00]
hmeij [Use Hadoop (test) Cluster]
cluster:115 [2013/09/10 15:04] (current)
hmeij [Rhadoop]
Line 369: Line 369:
   * permissions reset on /hdfs, not sure why   * permissions reset on /hdfs, not sure why
   * also use /​sanscratch/​zookeepers   * also use /​sanscratch/​zookeepers
-  * +  * some more notes below
  
 +
 +<​code>​
 +
 +
 +install.packages('​rJava'​)
 +install.packages("​int64"​)
 +install.packages(c("​Rcpp",​ "​RJSONIO",​ "​bitops",​ "​digest",​ "​functional",​ "​stringr",​ "​plyr",​ "​reshape2"​))
 +
 +wget http://​cran.r-project.org/​src/​contrib/​Archive/​Rcpp/​Rcpp_0.9.8.tar.gz
 +wget -O rmr-2.2.0.tar.gz http://​goo.gl/​bhCU6
 +wget -O rhdfs_1.0.5.tar.gz https://​github.com/​RevolutionAnalytics/​rhdfs/​blob/​master/​build/​rhdfs_1.0.5.tar.gz?​raw=true
 +
 +R CMD INSTALL Rcpp_0.9.8.tar.gz
 +R CMD INSTALL rmr-2.2.0.tar.gz
 +R CMD INSTALL rhdfs_1.0.5.tar.gz
 +R CMD INSTALL rhbase_1.2.0.tar.gz
 +
 +yum install openssl098e openssl openssl-devel flex boost ruby ruby-libs ruby-devel php php-libs php-devel \
 +automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel
 +
 +b2 install --prefix=/​usr/​local
 +
 +thrift: ./configure --prefix=/​usr/​local --with-boost=/​usr/​local;​ make
 +make install
 +
 +cp -p /​usr/​local/​lib/​libthrift-0.9.0.so /usr/lib/
 +cd /usr/lib; ln -s libthrift-0.9.0.so libthrift.so
 +
 +SKIP (nasty replaced with straight copy, could go to nodes)
 +http://​www.cpan.org
 +'o conf commit'​
 +cpan> install Hadoop::​Streaming ​
 +
 +whitetail only, unpack hbase, edit conf/​hbase-site.xml,​ add to /​etc/​rc.local
 +also edit conf/​regionservers
 +copy /​usr/​local/​hbase-version-dir to nodes:/​usr/​local
 +
 +  <​property>​
 +    <​name>​hbase.zookeeper.quorum</​name>​
 +    <​value>​example1,​example2,​example3</​value>​
 +    <​description>​The directory shared by RegionServers.
 +    </​description>​
 +  </​property>​
 +  <​property>​
 +    <​name>​hbase.zookeeper.property.dataDir</​name>​
 +    <​value>/​export/​zookeeper</​value>​
 +    <​description>​Property from ZooKeeper'​s config zoo.cfg.
 +    The directory where the snapshot is stored.
 +    </​description>​
 +  </​property>​
 +
 +
 +</​code>​
  
  
cluster/115.txt ยท Last modified: 2013/09/10 15:04 by hmeij