Differences

This shows you the differences between two versions of the page.

--- cluster:115 [2013/09/03 20:00]
hmeij
+++ cluster:115 [2013/09/10 19:04] (current)
hmeij [Rhadoop]
@@ Line 2: / Line 2: @@
 **[[cluster:0|Back]]**
-===== Use Hadoop (test) Cluster =====
+===== Use Hadoop Cluster =====
 [[cluster:114|Build Hadoop Cluster]]
@@ Line 356: / Line 356: @@
 pkg-config --cflags thrift
 cp -p /usr/local/lib/libthrift-0.9.0.so /usr/lib/
+HBASE_ROOT/bin/hbase thrift start &
+lsof -i:9090 that is server, port 9095 is monitor
 </code>
 Configure for distributed environment: [[http://hbase.apache.org/book/standalone_dist.html#standalone]]
+  * used 3 zookeepers with quorum, see config example online
+  * start with rolling_restart, the start & stop have a timing issue
+  * /hbase owened by root:root
+  * permissions reset on /hdfs, not sure why
+  * also use /sanscratch/zookeepers
+  * some more notes below
+<code>
+install.packages('rJava')
+install.packages("int64")
+install.packages(c("Rcpp", "RJSONIO", "bitops", "digest", "functional", "stringr", "plyr", "reshape2"))
+wget http://cran.r-project.org/src/contrib/Archive/Rcpp/Rcpp_0.9.8.tar.gz
+wget -O rmr-2.2.0.tar.gz http://goo.gl/bhCU6
+wget -O rhdfs_1.0.5.tar.gz https://github.com/RevolutionAnalytics/rhdfs/blob/master/build/rhdfs_1.0.5.tar.gz?raw=true
+R CMD INSTALL Rcpp_0.9.8.tar.gz
+R CMD INSTALL rmr-2.2.0.tar.gz
+R CMD INSTALL rhdfs_1.0.5.tar.gz
+R CMD INSTALL rhbase_1.2.0.tar.gz
+yum install openssl098e openssl openssl-devel flex boost ruby ruby-libs ruby-devel php php-libs php-devel \
+automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel
+b2 install --prefix=/usr/local
+thrift: ./configure --prefix=/usr/local --with-boost=/usr/local; make
+make install
+cp -p /usr/local/lib/libthrift-0.9.0.so /usr/lib/
+cd /usr/lib; ln -s libthrift-0.9.0.so libthrift.so
+SKIP (nasty replaced with straight copy, could go to nodes)
+http://www.cpan.org
+'o conf commit'
+cpan> install Hadoop::Streaming
+whitetail only, unpack hbase, edit conf/hbase-site.xml, add to /etc/rc.local
+also edit conf/regionservers
+copy /usr/local/hbase-version-dir to nodes:/usr/local
+  <property>
+    <name>hbase.zookeeper.quorum</name>
+    <value>example1,example2,example3</value>
+    <description>The directory shared by RegionServers.
+    </description>
+  </property>
+  <property>
+    <name>hbase.zookeeper.property.dataDir</name>
+    <value>/export/zookeeper</value>
+    <description>Property from ZooKeeper's config zoo.cfg.
+    The directory where the snapshot is stored.
+    </description>
+  </property>
+</code>
@@ Line 551: / Line 615: @@
 ==== Perl Hadoop::Streaming ====
+  * All nodes
   * [[http://search.cpan.org/~spazm/Hadoop-Streaming-0.122420/lib/Hadoop/Streaming.pm]]

DokuWiki

User Tools

Site Tools

Differences

Page Tools