In the world of NoSQL


I have been getting the following exception in my HBase master logs:

14/08/15 16:51:53 WARN util.FSTableDescriptors: The following folder is in HBase's root
directory and doesn't contain a table descriptor, do consider deleting it: .logs2

The reason was because I a long time ago had some issues, where I had moved /hbase/.logs to /hbase/.logs2 solve them. Don’t remember what the issue was, but since then I had been getting the above error message.

Apparently HBase WARNs about every directory in /hbase/ that shouldn’t be there, the solution was, obviously enough, to remove /hbase/.logs2.

root@hbase1:~$ sudo -u hdfs hdfs dfs -rm -r -skipTrash /hbase/.logs2

I don’t recommend deleting stuff from /hbase/ unless you know exactly what it is and why you are doing so.

Read more...

§618 · August 17, 2014 · Uncategorized · (No comments) ·



Like the subject says, recently when decommissioning a node I needed to restart the NameNode.

After restarting it, it instantly crashed with the following error message:

2014-08-16 11:52:52,528 WARN org.apache.hadoop.ipc.Server: IPC Server handler 3176 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.blockReport from 1.2.3.4:38596: error: java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(FSNamesystem.java:4286)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(FSNamesystem.java:3960)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.checkMode(FSNamesystem.java:4091)
... cut ...
2014-08-16 11:52:52,749 FATAL org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception.
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(FSNamesystem.java:4286)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(FSNamesystem.java:3960)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$1400(FSNamesystem.java:3852)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isInSafeMode(FSNamesystem.java:4380)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3060)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3032)
        at java.lang.Thread.run(Thread.java:722)
2014-08-16 11:52:52,752 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

I think the most interesting part of the above exception is:
Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088

blockSafe and BlockManager active are both higher than blockTotal, which looks suspicious.

My next step was to stop all DataNodes, and then try to start the cluster again. Same thing happened again every time I retried it.

After a few times I noticed that one DataNode was also crashing together with the NameNode.

2014-08-16 11:50:38,779 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
    Two block files with the same block id exist on disk:  /data/1/.../blk_1234 and /data/2/.../blk_1234

The solution was to start the NameNode and all DataNodes except the one that was crashing.

After waiting about 10-15 minutes I started the last DataNode, and it came back up without any problems. It seems that the NameNode required all the DataNodes that didn’t have duplicate blocks to settle before it was possible to start the DataNode with duplicate blocks.

Read more...

§604 · August 16, 2014 · Uncategorized · (No comments) ·



If you have limited connectivity between nodes, i.e. if half your datanodes are connected to one switch, and the other half to another, it’s wise to configure it so that the NameNode is aware of this.

To configure this, you set property topology.script.file.name to point to a script that reads the node names as arguments, and prints out the location of the node to standard out (space separated). It needs to be able to handle both hostname and IP address.

To configure the NameNode, add the following to /etc/hadoop/conf/hdfs-site.xml:

  <property>
    <name>topology.script.file.name</name>
    <value>/etc/hadoop/conf/topology.sh</value>
  </property>

My topology script looks like this:

#!/bin/bash
 
HADOOP_CONF=/etc/hadoop/conf
 
declare -A topology
while read host ip rack; do
  topology[$host]=$rack
  topology[$ip]=$rack
done < ${HADOOP_CONF}/topology.data
 
while [ -n "$1" ]; do
  echo -n "${topology[$1]:=/default/rack} "
  shift
done

And my topology.data contains:

node1.domain.com 1.2.3.1 /dc-se/rack-1
node2.domain.com 1.2.3.2 /dc-se/rack-2

Read more...

§590 · August 16, 2014 · Hadoop · (No comments) · Tags: , , ,



Got the following exception when starting the datanode after it had terminated due to a disk failure (without rebooting the server):

2013-10-11 11:24:02,122 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.BindException: Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:718)
	at org.apache.hadoop.ipc.Server.bind(Server.java:403)
	at org.apache.hadoop.ipc.Server.bind(Server.java:375)
	at org.apache.hadoop.hdfs.net.TcpPeerServer.<init>(TcpPeerServer.java:106)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:555)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:741)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:344)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1795)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)
2013-10-11 11:24:02,126 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-10-11 11:24:02,128 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at hbase10.network.com/1.2.3.4
************************************************************/

After an application crashes it might leave a lingering socket, so to reuse that socket early you need to set the socket flag SO_REUSEADDR when attempting to bind to it to be allowed to reuse it. The HDFS datanode doesn’t do that, and I didn’t want to restart the HBase regionserver (which was locking the socket with a connection it hadn’t realized was dead).
The solution was to bind to the port with an application that sets SO_REUSEADDR and then stop that application, I used netcat for that:

[root@hbase10 ~]#  nc -l 50010

Read more...

§577 · October 11, 2013 · Hadoop · (No comments) · Tags: , , , ,



My HBase cluster refused to start after upgrading from CDH3 to CDH4. This is a known issue according to the cloudera documentation, and the workaround is to delete the /hbase ZNode.

— During an upgrade from CDH3 to CDH4, regions in transition may cause HBase startup failures.

Bug: None
Severity: Medium
Anticipated Resolution: To be fixed in a future release.
Workaround: Delete the /hbase ZNode in ZooKeeper before starting up CDH4.

So to delete the ZNode I did the following:

[root@hbase1 ~]# /usr/lib/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
... log entries
[zk: localhost:2181(CONNECTED) 0] rmr /hbase

After doing this the cluster started as it should.
Ref: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.3.0/CDH4-Release-Notes/cdh4ki_topic_1_11.html

Read more...

§567 · October 2, 2013 · Hadoop, HBase, ZooKeeper · (No comments) · Tags: , , , ,



When upgrading from CDH3 to CDH4 I came along the following problem when attempting to start the NameNode again:

2013-09-23 22:53:42,859 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /dfs/nn/in_use.lock acquired by nodename 32133@hbase1.network.com
2013-09-23 22:53:42,903 INFO org.apache.hadoop.hdfs.server.namenode.NNStorage: Using clusterid: CID-9ab09a80-a367-42d4-8693-6905b9c5a605
2013-09-23 22:53:42,913 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /dfs/nn/current
2013-09-23 22:53:42,928 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /dfs/nn/current/fsimage using no compression
2013-09-23 22:53:42,928 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 183625
2013-09-23 22:53:44,280 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-09-23 22:53:44,282 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.AssertionError: Should have reached the end of image file /dfs/nn/current/fsimage
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:235)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:786)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:647)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:349)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:261)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:639)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:476)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:613)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:598)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
2013-09-23 22:53:44,286 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-09-23 22:53:44,288 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hbase1.network.com/1.2.3.4
************************************************************/

Luckily I came across a post to the cdh-user mailing list by Bob Copeland containing:

I instrumented the code around the exception and found that the loader had
read
all but 16 bytes of the file, and the remaining 16 bytes are all zeroes. So
chopping off the last 16 bytes of padding was a suitable workaround, i.e.:

fsimage=/var/lib/hadoop/dfs/name/current/fsimage
cp $fsimage{,~}
size=$(stat -c %s $fsimage)
dd if=$fsimage~ of=$fsimage bs=$[size-16] count=1

Is this a known issue? I did all these tests in a scratch cdh3u5 VM and can
replicate at will if needed.

-Bob

Which solved my problems.

Ref: http://grokbase.com/p/cloudera/cdh-user/12ckdj9m47/cdh4-fsimage-upgrade-failure-workaround

Read more...

§558 · October 1, 2013 · Hadoop · (No comments) · Tags: , , , , ,



I ran out of space on the server running namenode, hbase master, hbase regionserver and a datanode and during the subsequent restarts hbase master wouldn’t start.
During log splitting it died with the following error:

2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.lang.AssertionError
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:112)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1491)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:648)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:834)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:750)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:283)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:202)
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:275)
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:205)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:408)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

I found two ways to get it to start up again, the first one I tried was to move away the log splitting directory in hdfs with the following command (strongly discouraged to do this):

$ hadoop fs -mv /hbase/.logs/hbase1.domain.com,60020,1367325077343-splitting /user/hdfs

After some help from #hbase on irc.freenode.net I moved it back and tried starting hbase master with java assertions disabled, and that solved the issue.

To disable assertions in the JVM you make sure that the parameter -da (or -disableassertions) is passed to java when invoked.

I did this by editing /etc/hbase/conf/hbase-site.sh and adding -da to the HBASE_MASTER_OPTS environment variable.

Read more...




Recently got the following error when trying to restart a newly installed VPS (Linux KVM virtualization, error message came from libvirt-bin):

error: Timed out during operation: cannot acquire state change lock

To solve this I undefined the VPS and then defined it again and started it.

# virsh dumpxml vps1.customer.com > vps1.customer.com.xml
# virsh undefine vps1.customer.com
# virsh define vps1.customer.com.xml
# virsh start vps1.customer.com

All other blogs I found said to restart libvirt-bin to fix it, but I preferred not to.

It should be safe restarting libvirt-bin since version 0.6.0 without affecting the running VPS instances, but based on previous experiencies with restarting the xen hypervisor I wanted to avoid it if possible.

Read more...

§219 · May 16, 2013 · Linux, Virtualization · Comments Off on libvirt-bin: error: Timed out during operation: cannot acquire state change lock · Tags: , , , ,



Example code to create a simple push/pull based ventilator and sink in Perl with ZMQ::LibZMQ3 bindings.

Ventilator code:

#!/usr/bin/perl -w
 
$|++;
 
use ZMQ::LibZMQ3;
use ZMQ::Constants qw/ZMQ_PUSH/;
 
my $context = zmq_init(1);
my $socket = zmq_socket($context, ZMQ_PUSH);
zmq_bind($socket, "tcp://*:2222");
 
while (1) {
  print "Sent one\n";
  zmq_sendmsg($socket, zmq_msg_init_data("Woop"));
}

Sink code:

#!/usr/bin/perl -w
 
$|++;
 
use ZMQ::LibZMQ3;
use ZMQ::Constants qw/ZMQ_PULL/;
 
my $context = zmq_init(1);
my $socket = zmq_socket($context, ZMQ_PULL);
zmq_connect($socket, "tcp://localhost:2222");
 
while (1) {
  my $msg = zmq_recvmsg($socket);
  my $data = zmq_msg_data($msg);
  print scalar(localtime).": received message: $data\n";
}

Read more...

§198 · January 16, 2013 · Message Queues, ZeroMQ · Comments Off on ZeroMQ p5-ZMQ (ZMQ::LibZMQ3) sink/ventilator example (push/pull) · Tags: , , , , ,



First of all, install some dependencies:

# apt-get install libmodule-build-perl libb-hooks-op-annotation-perl libtest-requires-perl  libtest-tcp-perl libtask-weaken-perl libtool automake autoconf build-essential libmodule-install-perl

Download and install zeromq:

# wget -qO - http://download.zeromq.org/zeromq-3.2.2.tar.gz|tar -zxv
# cd zeromq-3.2.2
# ./autogen.sh
# ./configure
# make
# make install

Download p5-ZMQ:

# git clone git://github.com/lestrrat/p5-ZMQ.git

Then install ZMQ::Constants:

# cd p5-ZMQ/ZMQ-Constants
# perl Makefile.PL
# make
# make install

Continue with ZMQ::LibZMQ3, you need a newer version of Module::Install::XSUtil than Debian ships with, together with Module::Install::AuthorTests that’s not availalbe as a Debian package:

# cd ../ZMQ-LibZMQ3
# apt-get remove libmodule-install-xsutil-perl
# perl -MCPAN -e 'install Module::Install::XSUtil'
# perl -MCPAN -e 'install Module::Install::AuthorTests'
# perl -MCPAN -e 'install Module::Install::CheckLib'
# perl -MCPAN -e 'install Devel::CheckLib'
# perl Makefile.PL
# make
# make install

Problems

root@server:~/p5-ZMQ/ZMQ-LibZMQ3# perl Makefile.PL
include /root/p5-ZMQ/ZMQ-LibZMQ3/inc/Module/Install.pm
Variable "@libs" is not imported at Makefile.PL line 66.
	(Did you mean &libs instead?)
syntax error at Makefile.PL line 63, near "assertlibs
        lib"
Global symbol "@libs" requires explicit package name at Makefile.PL line 66.
Execution of Makefile.PL aborted due to compilation errors.

This happens if you don’t have Module::Install::CheckLib installed.

If you don’t want to install Module::Install::CheckLib it also works by removing that part of the Makefile.PL by running:

# sed -i -re '60,68s/^/#/' Makefile.PL

After that it worked.

Read more...