Install Hbase 2.x with Hadoop 3.x

Hbase - Pseudo Distributed mode of installation:

This is a method for Hbase Installation, known as Pseudo Distributed mode of Installation. Below are the steps to install HBase through this method. Prior to the installation steps below, please run Hadoop 3.x installation.

1. Download hbase-2.1.0-bin.tar.gz

2. Unzip the binary files:

tar -zxvf hbase-2.1.0-bin.tar.gz in /usr/local
ln -s hbase-2.1.0 hbase
chmod -R hadoop:hadoop hbase-2.1.0

3. Edit hbase environment variables:
Open hbase-env.sh as following below and mention JAVA_HOME path and Region servers' path in the location and export the command as shown:

cd /usr/local/hbase/conf/
vi hbase-env.sh

export JAVA_HOME=/etc/alternatives/java_sdk_1.8.0
export HBASE_SSH_OPTS="-p 22 -l hadoop"
export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
export HBASE_MANAGES_ZK=true

4. Add OS environment variables into ~/.bashrc. Final list of variables including hadoop is as below:

export JAVA_HOME=/etc/alternatives/java_sdk_1.8.0
export HDFS_NAMENODE_USER="hadoop"
export HDFS_DATANODE_USER="hadoop"
export HDFS_SECONDARYNAMENODE_USER="hadoop"
export YARN_RESOURCEMANAGER_USER="hadoop"
export YARN_NODEMANAGER_USER="hadoop"

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export YARN_HOME=/usr/local/hadoop
export HADOOP_SSH_OPTS="-p 1338 -l hadoop"
export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:/usr/local/hadoop/bin:$HBASE_HOME/bin:/etc/alternatives/java_sdk_1.8.0/bin

5. Edit HBase-site.xml

vi /usr/local/hbase/conf/hbase-site.xml
< ?xml version="1.0"?>
< ?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hadoop/hbase/zookeeper</value>
</property>
<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>
</configuration>

6. Start Hadoop daemons first and after that start HBase daemons

/usr/local/hadoop/bin/start-all.sh
/usr/local/hbase/bin/start-hbase.sh

7. Check jps, you should see below:

prompt> jps
62272 NodeManager
61680 SecondaryNameNode
51761 HMaster
61333 DataNode
61125 NameNode
34533 ResourceManager
60726 Elasticsearch
51976 HRegionServer
52602 Jps
51660 HQuorumPeer

8. Hbase comes with a web UI to check master status. The URL is as below:

http://localhost:16010

Note:
At the time of installation, Hbase is not fully integrated with Hadoop 3.0. There's 2 extra steps are required:

1. The HMaster refuse to start due to the error below:

Java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster Caused by: java.lang.ClassNotFoundException: org.apache.htrace.SamplerBuilder

This is because in hbase 2.0, we have 2 different version of htrace-core.x.x.x.incubating.jar

cd /usr/local/hbase/lib/client-facing-thirdparty/:
htrace-core-3.1.0-incubating.jar
htrace-core-4.2.0-incubating.jar

Currently, only version 3.1.0 has the required class SamplerBuilder. We need to remove version 4.2.0:
mv htrace-core-4.2.0-incubating.jar htrace-core-4.2.0-incubating.jar.bak

2. Hbase is not compatible with hadoop 3.x binaries, you will see error below:

ERROR [master/MasterNode/MasterNode:16000] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter

The default compilation of HBase is done with default hadoop profile i.e 2.0. HBase lib folder contains hadoop dependencies with hadoop-2.7 versions which doesn't support hsync capabilities. So try building HBase from source by passing -Dhadoop.profile=3.0. This builds Hbase distribution with hadoop-3.0 dependencies. Use this distribution to run HBase.

Alternately, you can bypass the dependency by editing hbase-site.xml and add the following:

<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>

Be the first to comment

Leave a Reply

Your email address will not be published.


*