Hadoop

Hadoop, Cloudera Manager, Spark, Flume and Scribe field notes.

Hue with LDAP doesn't start up after Cloudera Manager upgrade

3 February, 2015 1 min read 28 words
A Cloudera Manager upgrade drops ldap_username_pattern from hue.ini, so Hue with LDAP will not start until you put it back via safety valve.

Merging multiple avro files into single file using python

2 December, 2014 2 min read 387 words
Python script that merges hundreds of Avro files into one, handling the header and footer, with notes on how slow the append gets.

CDH 5.2.0 spark history server issue

17 October, 2014 1 min read 59 words
CDH 5.2.0 ships a Spark history location with no hdfs:// prefix, so the history server reads it as a local path and will not start.

Log compression for Cloudera Manager Hadoop

4 July, 2014 1 min read 159 words
Cloudera Manager's log4j will not compress rotated logs. The Apache Extras rolling appender does, pasted into the logging safety valve.

Fix Hadoop leap second bug without rebooting

3 July, 2012 1 min read 27 words
The 2012 leap second pinned Hadoop nodes at 100% CPU. Resetting the clock with date and restarting ntpd fixes it without a reboot.

Python - write compressed log file into HDFS for hadoop hive mapreduce

1 March, 2012 1 min read 117 words
Python that gzips log data in memory with StringIO and writes it straight into HDFS through pyhdfs, ready for Hive or MapReduce.

Facebook scribe with hdfs

17 February, 2012 1 min read 82 words
Build order and configure flags for compiling Facebook Scribe with HDFS support: boost, thrift, fb303, then scribe itself.

Flume DFO local storage usage check

30 January, 2012 2 min read 330 words
A Perl Nagios check for the Flume DFO backlog directory, so a failed driver does not quietly fill the local disk.