In the world of NoSQL

I have been getting the following exception in my HBase master logs:

14/08/15 16:51:53 WARN util.FSTableDescriptors: The following folder is in HBase's root
directory and doesn't contain a table descriptor, do consider deleting it: .logs2

The reason was because I a long time ago had some issues, where I had moved /hbase/.logs to /hbase/.logs2 solve them. Don’t remember what the issue was, but since then I had been getting the above error message.

Apparently HBase WARNs about every directory in /hbase/ that shouldn’t be there, the solution was, obviously enough, to remove /hbase/.logs2.

root@hbase1:~$ sudo -u hdfs hdfs dfs -rm -r -skipTrash /hbase/.logs2

I don’t recommend deleting stuff from /hbase/ unless you know exactly what it is and why you are doing so.


§618 · August 17, 2014 · Uncategorized · (No comments) ·

Like the subject says, recently when decommissioning a node I needed to restart the NameNode.

After restarting it, it instantly crashed with the following error message:

2014-08-16 11:52:52,528 WARN org.apache.hadoop.ipc.Server: IPC Server handler 3176 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.blockReport from error: java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.checkMode(
... cut ...
2014-08-16 11:52:52,749 FATAL org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception.
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$1400(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isInSafeMode(
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$
2014-08-16 11:52:52,752 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

I think the most interesting part of the above exception is:
Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088

blockSafe and BlockManager active are both higher than blockTotal, which looks suspicious.

My next step was to stop all DataNodes, and then try to start the cluster again. Same thing happened again every time I retried it.

After a few times I noticed that one DataNode was also crashing together with the NameNode.

2014-08-16 11:50:38,779 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
    Two block files with the same block id exist on disk:  /data/1/.../blk_1234 and /data/2/.../blk_1234

The solution was to start the NameNode and all DataNodes except the one that was crashing.

After waiting about 10-15 minutes I started the last DataNode, and it came back up without any problems. It seems that the NameNode required all the DataNodes that didn’t have duplicate blocks to settle before it was possible to start the DataNode with duplicate blocks.


§604 · August 16, 2014 · Uncategorized · (No comments) ·

If you have limited connectivity between nodes, i.e. if half your datanodes are connected to one switch, and the other half to another, it’s wise to configure it so that the NameNode is aware of this.

To configure this, you set property to point to a script that reads the node names as arguments, and prints out the location of the node to standard out (space separated). It needs to be able to handle both hostname and IP address.

To configure the NameNode, add the following to /etc/hadoop/conf/hdfs-site.xml:


My topology script looks like this:

declare -A topology
while read host ip rack; do
done < ${HADOOP_CONF}/
while [ -n "$1" ]; do
  echo -n "${topology[$1]:=/default/rack} "

And my contains: /dc-se/rack-1 /dc-se/rack-2


§590 · August 16, 2014 · Hadoop · (No comments) · Tags: , , ,

Got the following exception when starting the datanode after it had terminated due to a disk failure (without rebooting the server):

2013-10-11 11:24:02,122 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain Problem binding to [] Address already in use; For more details see:
	at org.apache.hadoop.ipc.Server.bind(
	at org.apache.hadoop.ipc.Server.bind(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(
2013-10-11 11:24:02,126 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-10-11 11:24:02,128 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
SHUTDOWN_MSG: Shutting down DataNode at

After an application crashes it might leave a lingering socket, so to reuse that socket early you need to set the socket flag SO_REUSEADDR when attempting to bind to it to be allowed to reuse it. The HDFS datanode doesn’t do that, and I didn’t want to restart the HBase regionserver (which was locking the socket with a connection it hadn’t realized was dead).
The solution was to bind to the port with an application that sets SO_REUSEADDR and then stop that application, I used netcat for that:

[root@hbase10 ~]#  nc -l 50010


§577 · October 11, 2013 · Hadoop · (No comments) · Tags: , , , ,

My HBase cluster refused to start after upgrading from CDH3 to CDH4. This is a known issue according to the cloudera documentation, and the workaround is to delete the /hbase ZNode.

— During an upgrade from CDH3 to CDH4, regions in transition may cause HBase startup failures.

Bug: None
Severity: Medium
Anticipated Resolution: To be fixed in a future release.
Workaround: Delete the /hbase ZNode in ZooKeeper before starting up CDH4.

So to delete the ZNode I did the following:

[root@hbase1 ~]# /usr/lib/zookeeper/bin/
Connecting to localhost:2181
... log entries
[zk: localhost:2181(CONNECTED) 0] rmr /hbase

After doing this the cluster started as it should.


§567 · October 2, 2013 · Hadoop, HBase, ZooKeeper · (No comments) · Tags: , , , ,

When upgrading from CDH3 to CDH4 I came along the following problem when attempting to start the NameNode again:

2013-09-23 22:53:42,859 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /dfs/nn/in_use.lock acquired by nodename
2013-09-23 22:53:42,903 INFO org.apache.hadoop.hdfs.server.namenode.NNStorage: Using clusterid: CID-9ab09a80-a367-42d4-8693-6905b9c5a605
2013-09-23 22:53:42,913 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /dfs/nn/current
2013-09-23 22:53:42,928 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /dfs/nn/current/fsimage using no compression
2013-09-23 22:53:42,928 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 183625
2013-09-23 22:53:44,280 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-09-23 22:53:44,282 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.AssertionError: Should have reached the end of image file /dfs/nn/current/fsimage
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(
        at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
2013-09-23 22:53:44,286 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-09-23 22:53:44,288 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at

Luckily I came across a post to the cdh-user mailing list by Bob Copeland containing:

I instrumented the code around the exception and found that the loader had
all but 16 bytes of the file, and the remaining 16 bytes are all zeroes. So
chopping off the last 16 bytes of padding was a suitable workaround, i.e.:

cp $fsimage{,~}
size=$(stat -c %s $fsimage)
dd if=$fsimage~ of=$fsimage bs=$[size-16] count=1

Is this a known issue? I did all these tests in a scratch cdh3u5 VM and can
replicate at will if needed.


Which solved my problems.



§558 · October 1, 2013 · Hadoop · (No comments) · Tags: , , , , ,

I ran out of space on the server running namenode, hbase master, hbase regionserver and a datanode and during the subsequent restarts hbase master wouldn’t start.
During log splitting it died with the following error:

2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(
        at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(
2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

I found two ways to get it to start up again, the first one I tried was to move away the log splitting directory in hdfs with the following command (strongly discouraged to do this):

$ hadoop fs -mv /hbase/.logs/,60020,1367325077343-splitting /user/hdfs

After some help from #hbase on I moved it back and tried starting hbase master with java assertions disabled, and that solved the issue.

To disable assertions in the JVM you make sure that the parameter -da (or -disableassertions) is passed to java when invoked.

I did this by editing /etc/hbase/conf/ and adding -da to the HBASE_MASTER_OPTS environment variable.


Recently got the following error when trying to restart a newly installed VPS (Linux KVM virtualization, error message came from libvirt-bin):

error: Timed out during operation: cannot acquire state change lock

To solve this I undefined the VPS and then defined it again and started it.

# virsh dumpxml >
# virsh undefine
# virsh define
# virsh start

All other blogs I found said to restart libvirt-bin to fix it, but I preferred not to.

It should be safe restarting libvirt-bin since version 0.6.0 without affecting the running VPS instances, but based on previous experiencies with restarting the xen hypervisor I wanted to avoid it if possible.


§219 · May 16, 2013 · Linux, Virtualization · Comments Off on libvirt-bin: error: Timed out during operation: cannot acquire state change lock · Tags: , , , ,

Example code to create a simple push/pull based ventilator and sink in Perl with ZMQ::LibZMQ3 bindings.

Ventilator code:

#!/usr/bin/perl -w
use ZMQ::LibZMQ3;
use ZMQ::Constants qw/ZMQ_PUSH/;
my $context = zmq_init(1);
my $socket = zmq_socket($context, ZMQ_PUSH);
zmq_bind($socket, "tcp://*:2222");
while (1) {
  print "Sent one\n";
  zmq_sendmsg($socket, zmq_msg_init_data("Woop"));

Sink code:

#!/usr/bin/perl -w
use ZMQ::LibZMQ3;
use ZMQ::Constants qw/ZMQ_PULL/;
my $context = zmq_init(1);
my $socket = zmq_socket($context, ZMQ_PULL);
zmq_connect($socket, "tcp://localhost:2222");
while (1) {
  my $msg = zmq_recvmsg($socket);
  my $data = zmq_msg_data($msg);
  print scalar(localtime).": received message: $data\n";


§198 · January 16, 2013 · Message Queues, ZeroMQ · Comments Off on ZeroMQ p5-ZMQ (ZMQ::LibZMQ3) sink/ventilator example (push/pull) · Tags: , , , , ,

First of all, install some dependencies:

# apt-get install libmodule-build-perl libb-hooks-op-annotation-perl libtest-requires-perl  libtest-tcp-perl libtask-weaken-perl libtool automake autoconf build-essential libmodule-install-perl

Download and install zeromq:

# wget -qO -|tar -zxv
# cd zeromq-3.2.2
# ./
# ./configure
# make
# make install

Download p5-ZMQ:

# git clone git://

Then install ZMQ::Constants:

# cd p5-ZMQ/ZMQ-Constants
# perl Makefile.PL
# make
# make install

Continue with ZMQ::LibZMQ3, you need a newer version of Module::Install::XSUtil than Debian ships with, together with Module::Install::AuthorTests that’s not availalbe as a Debian package:

# cd ../ZMQ-LibZMQ3
# apt-get remove libmodule-install-xsutil-perl
# perl -MCPAN -e 'install Module::Install::XSUtil'
# perl -MCPAN -e 'install Module::Install::AuthorTests'
# perl -MCPAN -e 'install Module::Install::CheckLib'
# perl -MCPAN -e 'install Devel::CheckLib'
# perl Makefile.PL
# make
# make install


root@server:~/p5-ZMQ/ZMQ-LibZMQ3# perl Makefile.PL
include /root/p5-ZMQ/ZMQ-LibZMQ3/inc/Module/
Variable "@libs" is not imported at Makefile.PL line 66.
	(Did you mean &libs instead?)
syntax error at Makefile.PL line 63, near "assertlibs
Global symbol "@libs" requires explicit package name at Makefile.PL line 66.
Execution of Makefile.PL aborted due to compilation errors.

This happens if you don’t have Module::Install::CheckLib installed.

If you don’t want to install Module::Install::CheckLib it also works by removing that part of the Makefile.PL by running:

# sed -i -re '60,68s/^/#/' Makefile.PL

After that it worked.