Logging on Logan's Nest

Add HTTP auth username into envoy proxy logging

Tue, 01 Jun 2021 00:00:00 +0000

Oddly couldn’t find any useful copy & paste solution for this.

As always, base64 decoding and split function copy & pasted from somewhere else.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


http_filters:
- name: envoy.filters.http.lua
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
 inline_code: |
 b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
 function base64decode(data)
 data = string.gsub(data, '[^'..b..'=]', '')
 return (data:gsub('.', function(x)
 if (x == '=') then return '' end
 local r,f='',(b:find(x)-1)
 for i=6,1,-1 do r=r..(f%2^i-f%2^(i-1)>0 and '1' or '0') end
 return r;
 end):gsub('%d%d%d?%d?%d?%d?%d?%d?', function(x)
 if (#x ~= 8) then return '' end
 local c=0
 for i=1,8 do c=c+(x:sub(i,i)=='1' and 2^(8-i) or 0) end
 return string.char(c)
 end))
 end
 function split(s, delimiter)
 result = {};
 for match in (s..delimiter):gmatch("(.-)"..delimiter) do
 table.insert(result, match);
 end
 return result;
 end
 function envoy_on_request(request_handle)
 local auth_header = request_handle:headers():get("Authorization")
 if auth_header ~= nil then
 local auth_token = split(auth_header," ")
 if auth_token[1] == "Basic" then
 local auth_decoded = split(base64decode(auth_token[2]),":")
 request_handle:headers():replace("x-username", auth_decoded[1])
 auth_header = null
 auth_token = null
 autn_decoded = null
 end
 end
 end
- name: envoy.filters.http.router
 typed_config: {}

Then it can be added into the log by using %REQ(x-username)% from envoy.extensions.access_loggers.file.v3.FileAccessLog.format

CloudFlare Log Search

Fri, 19 Jun 2020 00:00:00 +0000

Get the Zone ID
Select domain name from the dashboard then zone ID will show on bottom right corner.
Get the API Key
Select log search template to give minimum privilege
(Optional) Look up the fields available

1

curl -s -H "X-Auth-Email: <EMAIL>" -H "Authorization: Bearer <API KEY>" "https://api.cloudflare.com/client/v4/zones/<Zone ID>/logs/received/fields" | jq .

4. Run Log Search (modify date & time bit as you need)

Log compression for Cloudera Manager Hadoop

Fri, 04 Jul 2014 00:00:00 +0000

Cloudera manager uses standard log4j which doesn’t support compression by itself.

It requires Apache Extras™ for Apache log4j™ (http://logging.apache.org/log4j/extras/) or maybe log4j2.

Load log4j-(latest version).jar by placing it under /lib of CDH parcel or custom import.

Python - write compressed log file into HDFS for hadoop hive mapreduce

Thu, 01 Mar 2012 00:00:00 +0000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


import pyhdfs
from cStringIO import StringIO
import binascii

-snip-

#Set hdfs connection info
hdfsaddress = “namenode”
hdfsport = 12345
hdfsfn = “filename”

#gzip compression level
clevel = 1

-snip-

 logger.info(“Writing compressed data into ” + hdfsfn + “.gz”)
 #open hdfs file
 fout = pyhdfs.open(hdfs, hdfsfn + “.gz”, “w”)
 #compress the data and store it in compressed_data
 buf = StringIO()
 f = gzip.GzipFile(mode=’wb’, compresslevel=clevel,fileobj=buf)
 try:
 f.write(concatlog)
 finally:
 f.close()
 compressed_data = buf.getvalue()
 #write compressed data into hdfs
 pyhdfs.write(hdfs,fout,compressed_data)
 #close hdfs file
 logger.info(“Writing task finished”)
 pyhdfs.close(hdfs,fout)
-snip-

Facebook scribe with hdfs

Fri, 17 Feb 2012 00:00:00 +0000

packages:
libevent
hadoop-0.20-libhdfs
JDK for hdfs support

Boost
http://sourceforge.net/projects/boost/
./bootstrap.sh
./bjam
./bjam install

thrift
download form http://thrift.apache.org/download/
./configure
make
make install

fb303
it’s in /contrib/fb303

./bootstrap.sh
./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
(To avoid compile error, recompile thrift with this option if it doesn’t work)
make
make install

Flume DFO local storage usage check

Mon, 30 Jan 2012 00:00:00 +0000

It might be useful when flume driver failed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


!/usr/bin/perl
#Check flume DFO directory size

sub trim($);
use strict;
use warnings;

my $exit=0;
my $backlog_size = `du -s flume | awk {‘print \$1’}`;
$backlog_size = trim($backlog_size);

if ( !$ARGV[0] || !$ARGV[1])
{
########################### Usage of the plugin
print “check_flume_backlog critical_size warning_size \n”;
exit 0;
}

######################### Case 1 if State is Critical
if ($backlog_size > $ARGV[0])
{
print “Critical: “.$backlog_size.”b\n”;
exit 2;
}

######################## Case 2 if State is Warning
if($backlog_size > $ARGV[1] || $backlog_size == 0)
{
print “Warning: “.$backlog_size.”b\n”;
exit 1;
}

######################## Case 3 if State is OK
if($backlog_size < $ARGV[0] && $backlog_size < $ARGV[1])
{
print “OK: “.$backlog_size.”b\n”;
exit 0;
}

sub trim($)
{
 my $string = shift;
 $string =~ s/^\s+//;
 $string =~ s/\s+$//;
 return $string;
}

And for centralised monitoring..