In the world of NoSQL


Not really NoSQL related, but kind of.  A lot of people uses Scribe for processing large amount of events. Since I had some problems compiling it and since it’s not prepackaged I figured I’d post here how to do it.

It’s extremely important that you install the dependencies before starting!

First you need to install thrift(depends on package: libevent-dev libglib2.0-dev automake libboost-dev libssl-dev) and fb303 (needed to communicate with Scribe):

$ wget https://dist.apache.org/repos/dist/release/thrift/0.9.0/thrift-0.9.0.tar.gz
$ tar -zxf thrift-0.9.0.tar.gz; cd thrift-0.9.0
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
$ make 
$ sudo make install
$ cd contrib/fb303
$ ./bootstrap.sh
$ make
$ sudo make install

Then install scribe(depends on package: gawk libboost-system-dev libboost-filesystem-dev libtool):

$ git clone git://github.com/facebook/scribe.git
$ cd scribe
$ ./bootstrap.sh
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -DBOOST_FILESYSTEM_VERSION=2"
$ make

Make failed while trying to link scribed, complaining about:

file.cpp:(.text.startup+0x23): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x2f): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x3b): undefined reference to `boost::system::system_category()'

The solution to this was to take the g++-row from just before the error:

g++  -Wall -O3 -L/usr/lib  -lboost_system-mt -lboost_filesystem-mt -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a

and change it a bit (move “-lboost_system-mt -lboost_filesystem-mt” to the back of the line), and run it manually in the src directory:

$ cd src
$ g++  -Wall -O3 -L/usr/lib  -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a -lboost_system-mt -lboost_filesystem-mt

The “-DBOOST_FILESYSTEM_VERSION=2” is only needed if you have libboost 1.46 or newer.

Now you have a compiled scribed binary in that directory, either run it directly from there or run make install to install it globally.

Here’s an example config file that just logs to ./logs:

port=1463
max_msg_per_second=2000000
check_interval=3
 
<store>
  category=default
 
  type=file
  fs_type=std
  file_path=./logs/
  base_filename=tmpdata
  max_size=1000000000
</store>

Put it in the scribe directory and name it to local.conf and run ./src/scribed local.conf to start scribe with it.

§177 · October 18, 2012 · Uncategorized · Tags: , , , · [Print]

Comments are closed.