Python on Logan's Nest

Merging multiple avro files into single file using python

Tue, 02 Dec 2014 00:00:00 +0000

Couldn’t find any good example from Web so made one by myself.

It has bits that handling header and footer(it’s using bogus footer as it was really FYI only thing for me but you may need to recreate it if it does matter) as well.

ZeroMQ - PHP client & Python server with multi part message example

Wed, 02 May 2012 00:00:00 +0000

Using pyobj or php data serialisation is not an option in this case..

Client:

1
2
3
4
5
6
7


<?php 
$context = new ZMQContext(); 
$requester = new ZMQSocket($context, ZMQ::SOCKET\_REQ); 
$requester->connect("tcp://localhost:9999"); 
$requester->send("Filename",ZMQ::MODE\_SNDMORE); 
$requester->send("Log",ZMQ::MODE\_NOBLOCK); 
?> 

Server:

Python - write compressed log file into HDFS for hadoop hive mapreduce

Thu, 01 Mar 2012 00:00:00 +0000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


import pyhdfs
from cStringIO import StringIO
import binascii

-snip-

#Set hdfs connection info
hdfsaddress = “namenode”
hdfsport = 12345
hdfsfn = “filename”

#gzip compression level
clevel = 1

-snip-

 logger.info(“Writing compressed data into ” + hdfsfn + “.gz”)
 #open hdfs file
 fout = pyhdfs.open(hdfs, hdfsfn + “.gz”, “w”)
 #compress the data and store it in compressed_data
 buf = StringIO()
 f = gzip.GzipFile(mode=’wb’, compresslevel=clevel,fileobj=buf)
 try:
 f.write(concatlog)
 finally:
 f.close()
 compressed_data = buf.getvalue()
 #write compressed data into hdfs
 pyhdfs.write(hdfs,fout,compressed_data)
 #close hdfs file
 logger.info(“Writing task finished”)
 pyhdfs.close(hdfs,fout)
-snip-