Python

Python scripts for Avro, HDFS and ZeroMQ.

Merging multiple avro files into single file using python

2 December, 2014 2 min read 387 words
Python script that merges hundreds of Avro files into one, handling the header and footer, with notes on how slow the append gets.

ZeroMQ - PHP client & Python server with multi part message example

2 May, 2012 1 min read 65 words
Multi-part ZeroMQ messaging between a PHP client and a Python server, without relying on either language's own serialisation.

Python - write compressed log file into HDFS for hadoop hive mapreduce

1 March, 2012 1 min read 117 words
Python that gzips log data in memory with StringIO and writes it straight into HDFS through pyhdfs, ready for Hive or MapReduce.