Hadoop on Logan's Nest

Hue with LDAP doesn't start up after Cloudera Manager upgrade

Tue, 03 Feb 2015 00:00:00 +0000

Latest Cloudera Manager omit ldap_username_pattern entry from hue.ini build process.

Paste below into safety valve as a workaround.

[desktop] [[ldap]] ldap_username_pattern="username=,ou=user,dc=7979,dc=us"

P.S: Had some weird experience without ""

Merging multiple avro files into single file using python

Tue, 02 Dec 2014 00:00:00 +0000

Couldn’t find any good example from Web so made one by myself.

It has bits that handling header and footer(it’s using bogus footer as it was really FYI only thing for me but you may need to recreate it if it does matter) as well.

CDH 5.2.0 spark history server issue

Fri, 17 Oct 2014 00:00:00 +0000

5.2.0-1.cdh5.2.0.p0.36 has been released couple of days ago.

It has a small bug with its spark bit, which make unable to start spark history server.

The problem is the default value of Spark History Location (=spark.eventLog.dir) is /user/spark/applicationHistory and it doesn’t append hdfs header by default, hence get recognised as local drive rather than HDFS.

Log compression for Cloudera Manager Hadoop

Fri, 04 Jul 2014 00:00:00 +0000

Cloudera manager uses standard log4j which doesn’t support compression by itself.

It requires Apache Extras™ for Apache log4j™ (http://logging.apache.org/log4j/extras/) or maybe log4j2.

Load log4j-(latest version).jar by placing it under /lib of CDH parcel or custom import.

Fix Hadoop leap second bug without rebooting

Tue, 03 Jul 2012 00:00:00 +0000

Source: http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/

Applied our hadoop node having CPU usage issue. (CentOS)

1
2
3


service ntpd stop
date `date +”%m%d%H%M%C%y.%S”`
service ntpd start

Works like a charm!

Python - write compressed log file into HDFS for hadoop hive mapreduce

Thu, 01 Mar 2012 00:00:00 +0000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


import pyhdfs
from cStringIO import StringIO
import binascii

-snip-

#Set hdfs connection info
hdfsaddress = “namenode”
hdfsport = 12345
hdfsfn = “filename”

#gzip compression level
clevel = 1

-snip-

 logger.info(“Writing compressed data into ” + hdfsfn + “.gz”)
 #open hdfs file
 fout = pyhdfs.open(hdfs, hdfsfn + “.gz”, “w”)
 #compress the data and store it in compressed_data
 buf = StringIO()
 f = gzip.GzipFile(mode=’wb’, compresslevel=clevel,fileobj=buf)
 try:
 f.write(concatlog)
 finally:
 f.close()
 compressed_data = buf.getvalue()
 #write compressed data into hdfs
 pyhdfs.write(hdfs,fout,compressed_data)
 #close hdfs file
 logger.info(“Writing task finished”)
 pyhdfs.close(hdfs,fout)
-snip-

Facebook scribe with hdfs

Fri, 17 Feb 2012 00:00:00 +0000

packages:
libevent
hadoop-0.20-libhdfs
JDK for hdfs support

Boost
http://sourceforge.net/projects/boost/
./bootstrap.sh
./bjam
./bjam install

thrift
download form http://thrift.apache.org/download/
./configure
make
make install

fb303
it’s in /contrib/fb303

./bootstrap.sh
./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
(To avoid compile error, recompile thrift with this option if it doesn’t work)
make
make install

Flume DFO local storage usage check

Mon, 30 Jan 2012 00:00:00 +0000

It might be useful when flume driver failed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


!/usr/bin/perl
#Check flume DFO directory size

sub trim($);
use strict;
use warnings;

my $exit=0;
my $backlog_size = `du -s flume | awk {‘print \$1’}`;
$backlog_size = trim($backlog_size);

if ( !$ARGV[0] || !$ARGV[1])
{
########################### Usage of the plugin
print “check_flume_backlog critical_size warning_size \n”;
exit 0;
}

######################### Case 1 if State is Critical
if ($backlog_size > $ARGV[0])
{
print “Critical: “.$backlog_size.”b\n”;
exit 2;
}

######################## Case 2 if State is Warning
if($backlog_size > $ARGV[1] || $backlog_size == 0)
{
print “Warning: “.$backlog_size.”b\n”;
exit 1;
}

######################## Case 3 if State is OK
if($backlog_size < $ARGV[0] && $backlog_size < $ARGV[1])
{
print “OK: “.$backlog_size.”b\n”;
exit 0;
}

sub trim($)
{
 my $string = shift;
 $string =~ s/^\s+//;
 $string =~ s/\s+$//;
 return $string;
}

And for centralised monitoring..