Tools for everyday use – www.py – HTML-enabled echo replacement

If you often have to analyze log files or issue commands that produce lots of plain-text output, consider taking a look at my small utility http://www.py (which lives in htmecho GitHub project).

Its purpose is to capture the input, transform it to an HTML with some additional JavaScript payload and open the result in the default browser. This enriched output provides filtering ability and allows assigning colors to lines that match a particular regular expression.

I use it so often, so that if it didn’t exist I would write one today :) For convenience, I usually place it into ~/bin/www (followed by chmod +x ~/bin/www). On Windows I just ensure that ‘py’ file extension is mapped onto Python 3.2 executable.

For example the following console command:
grep -s -r -e "Error:" -e "Warning:" *.log | www
would open a browser (or add a new tab into an already opened browser window) with the output of the grep.
Here is a screen of
dmesg | www
command with BIOS pattern marked as yellow:

Here is a bit of history that might be of interest to developers.

Few years ago, as I was studying I/O area of the C++ standard library, I wrote a small tool that transformed its STDIN input into a temporary HTML file. Though by that time I had only the educational goal, I found this tool to be surprisingly useful in my working environment – as I often have to analyze logs (typically after some grep operations on them) and due to my dual-monitor setup I find it very convenient to issue some command on one screen and see the results on the other.

The only major change before I stopped working on it was adding a ShellExecute call in the end – so that the newly generated HTML file was opened automatically in the default browser (which must be chrome which is chrome in my case). I just put it into a local SCM and kept it for few years without changes.

Recently, as I often rely on LoadRunner logs analysis while handling support cases, few more ideas came to mind as we discussed this tool with my colleague – what if we enrich the generated HTML file with a bit of JavaScript to allow some grep-like functionality right in the browser window? What if we colorize particular filters, so that it will be easy to visually find the interesting spots in the logs? That gave me enough motivation to invest more time in the project and the colleague helped me with the JavaScript part. I also added an imaginary TODO item to ensure that the tool compiles and works on Linux, as I use it on my laptop. Now, a simple one-file educational application became cross-platform with OS-transparent and OS-agnostic layers. I also made a bit of interesting research on how to embed binary resources into Linux executable modules (as I wanted to simulate Windows Portable Executable Resources behavior which allows trivial embedding of whatever you want into the executable).

Next battle was for Unicode support. Or, to put it correctly, total lack of such support in the language. I tried to avoid WinAPIs for that and to be able to handle windows-1251, UTF-16 BE/LE and UTF-8 encodings – I had to spent few nights reading quite a number of materials on iwfstream and codecvt (not the best reading, imo), which is neither simple to use nor beautiful conceptually. And still, removing BOM-checks in C++ classic iwfstream-based code was a great pain. And, OMG!, after I ensured it works on Windows – it just didn’t work on Linux… though it was a personal free-time development, instead of bringing satisfaction and joy – it was just nasty time waste. Finally, though I’ve got to a stable point with support of cp1251 and UTF8, I just didn’t feel I want to proceed any further.

And here comes Python!

Just for sport interest, I tried to reach the same functionality by using Python 3.2 and total time spent until I got it working on two platforms I’m interested in was around 2 hours of coding and debugging.

Benefits brought by Python:
- single-file executable with no dependencies. Even though Python version had platform-agnostic parts too, they were typically one-liners and there was no reason to move them into separate modules. In C++ I had to implement platform-agnostic (and even linker-agnostic for Linux) resource embedding, cross-platform code to open browser and it naturally resulted in multiple headers/source files.
- ability to modify HTML template with JavaScript payload – again, in Python it is native to have multi-line string literals – so, go on and just edit the script. In C++, it was simple too, but only because HTML template had been intentionally kept as an external resource linked into an executable. On windows, it is possible to modify an embedded resource externally with a tool like Resource Hacker. I don’t have a clue on whether anything as simple is available on Linux.
- Unicode. Python 3.x rulez here: standard distribution of the language has plenty of encodings supported out of the box, BOMs are removed automagically granted you’ve specified the correct encoding. It was pity to conclude that pure C++ with its standard library totally suck at this part.
- Performance – for a tiny tool like this I won’t even bother with runtime performance. Productivity is what matters here and Python is an obvious winner here. I admit it wasn’t clever for me to write such a program in C++ in the first place, but as I said it was an educational project and … I like to code in C++ anyway, at least when it’s not a unicode related code.

Dealing with several Python versions in Sublime Text 2

Sublime Text 2 contains the following build system definition for Python (refer to ~/.config/sublime-text-2/Packages/Python/Python.sublime-build):

{
    "cmd": ["python", "-u", "$file"],
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python"
}

Thus default Python interpreter is used (which is typically a symlink to concrete version). In case if you have several versions of Python installed – you’d want to control which version of Python to invoke on Sublime Text’s “Build” command.

It can be easily done by adding custom build system in Tools > New Build System… menu with a slightly changed definition, e.g:

{
    "cmd": ["python3.2", "-u", "$file"],
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python"
}

After saving this file, e.g. as Python3.sublime-build, and restarting Sublime Text editor it would be possible to specify the version of interpreter you need in Tools > Build System menu.

If you’re interested in Russian version of this entry – take a look here on my impulse9 blog

Python ctypes as a handy tool for C/C++ developers

I have recently learned how useful Python ctypes library might be when it comes to cross-platform testing of dynamic libraries. ctypes handles library loading and allows easy function invocation in case if library exposes C-like interfaces. In addition, library provides a way of using non-trivial data types as function call arguments (such as struct, union, C-style arrays and their combinations).

Solution is cross-platform, it significantly simplifies development and testing of new functions addition to existing dynamic libraries at early stages, so that there are no other clients for those functions ready yet. Also, it can be used as a good sanity measure for automated build regression checks.

As an example, here is a small use case: suppose I have to check the following function from a dynamic library (.dll for Windows, .so on Linux) written in C/C++:

int get_ips(const unsigned short af, struct ipaddress_info ** ppaddrinfo);

Where ipaddress_info is defined like this:

typedef struct ipaddress_info {
    unsigned short family;
    unsigned char addr[16];
    unsigned int scope_id;
} IPADDRESS_INFO;

So given the definition above – you’d have a DLL on Windows (or a shared library on Linux) that exports get_ips function. And here is the sample Python script that is capable to load that library, perform a call to get_ips and display the contents of a returned array of structures:

import os
import sys
from ctypes import *
from socket import AF_INET, AF_INET6, AF_UNSPEC

# Python representation of a C struct defined
# in a shared library being tested
class ipaddress_info(Structure):
    _fields_ = [
        ("family",      c_ushort),
        ("addr",        c_ubyte * 16),
        ("scope_id",    c_uint),
    ]

def bytes2str(v):
    s = ""
    for x in v:
        s += "%.02X " % (x)
    return s

class dynlib_bridge:
    def __init__(self, dll_path, working_dir="."):
        tmp = os.getcwd()
        os.chdir(working_dir)
        self.invoke = cdll.LoadLibrary(dll_path)
        os.chdir(tmp)

class test_case_base:
    def __init__(self, real_values):
        self.__inout__ = dict()
        self.__result__ = None
        for arg_name, arg_value in real_values.items():
            self.__inout__[arg_name] = arg_value
    def actual(self, name):
        return self.__inout__[name]
    def result(self):
        return self.__result__
    def setResult(self, v):
        self.__result__ = v
        
# performs a positive test case
def test_pos(testobj):
    result = testobj.run()
    if result:
        print ("[tc+] OK  :", testobj.describe())
    else:
        print ("[tc+] FAIL:", testobj.describe())    

# performs a negative test case
def test_neg(testobj):
    result = testobj.run()
    if not result:
        print ("[tc-] OK  :", testobj.describe())
    else:
        print ("[tc-] FAIL:", testobj.describe())    

# wraps invokation of a shared library function
class get_ips(test_case_base):
    def __init__(self, dll, af):
        super(get_ips, self).__init__({
            'af': c_ushort(af),
            'ppaddrinfo': POINTER(ipaddress_info)(),
            })
        self.dll = dll
    def run(self):
        invoke_result = self.dll.invoke.get_ips(
            self.actual('af'), 
            byref(self.actual('ppaddrinfo')))
        self.setResult(invoke_result)
        return (invoke_result > 0)
    def describe(self):
        p = self.actual('ppaddrinfo')
        ips = []
        if self.result() > 0:
            for i in range(self.result()):
                ips.append("#%d: family = %d, scope = %d, %s" % (
                    i, 
                    p[i].family, 
                    p[i].scope_id, 
                    bytes2str(p[i].addr)))
        s = "result: %.02d\n%s" % (self.result(), "\n".join(ips))
        return s
        
if __name__ == "__main__":
    # you'd probably have to use LD_LIBRARY_PATH to make this work on Linux
    path_to_lib = sys.argv[1]
    path_to_env = sys.argv[2]
    print ("lib path :", path_to_lib)
    print ("env path :", path_to_env)
    dll = dynlib_bridge(path_to_lib, working_dir=path_to_env)
    
    test_pos(get_ips(dll, AF_UNSPEC))
    test_pos(get_ips(dll, AF_INET))
    test_pos(get_ips(dll, AF_INET6))
    test_neg(get_ips(dll, 99))

The first argument is a library file name; second – is the location from which library load should be performed (it is sufficient on Windows, on Linux you might need to deal with LD_LIBRARY_PATH if your library has its own dependencies).

For simplicity, I removed most of error handling code and some not very interesting things like calling another library function to perform memory cleanup. Also, code assumes cdecl calling convention (which actually was my case for both tested platforms)

If you’re interested in Russian version of this entry – take a look here on my impulse9 blog