Steam summer sale 2012

A short note for gamers – I was very surprised to see the details of Steam Summer Sale offering. Don’t miss the opportunity to get some amazing titles for completely ridiculous prices – here are some examples – check  out:

• “Square Enix Hit Collection” bundle with all series of DeusEx, Hitman, Tomb Raider, Just Cause et al. for as little as $25.
• “Valve Complete Pack” with all parts of Counter-Strike, Half-Life, Portal, Left 4 Dead, etc for about $27
• “Call of Duty®: Modern Warfare® Collection” providing you with 3 parts of Modern Warfare franchise for about $50

There are many others great offers – just explore the links above to find more items. Hurry up as the proposal is very time limited. Steam rocks!

Capturing pieces of the web by ukeeper

How often do you add a bookmark to what seems to be an interesting article and never really return to it just because there was no natural way for it to pop-up in your day-to-day activities? Do you often regret that software incompatibilities disallow you adding an article found in twitter on one device to be be read comfortably on another?

If you feel like these questions are relevant for you – you would probably want to take a look at the service http://www.ukeeper.com/. Idea behind this service (created by a well-known podcaster Umputun, http://www.umputun.com/) is elegant and simple – given a page link it captures the page content and sends it via email. That’s it. Once you get something in your mail – it will always be there ready to be read anytime by whatever email-capable gadget available.

The ukeeper.com service provides its clients with an email which is used for “dropping” the web page capture requests. Such a request is simply an email which has a web link in its body. The service replies with an email holding the content of the referenced page. It’s simple and it works from any device.

I’m using this service for deferred reading of the stuff found on the web and I feel like, finally, this is a really  working solution. I’ve added a GMail filter for messages coming from ukeeper to automatically skip inbox and go directly into “Reading” folder. And whenever I have some free time – I visit this folder and ensure it has no unread items.

GMail filter for web page snapshots captured via ukeeper

Update from July, 2012:

Now ukeeper provides a Safari and Chrome extensions for simplifying the process of sending a link into just a single mouse click – learn more at http://www.ukeeper.com/extras/

Direct links for Windows Applications troubleshooting

Here is a set of WGET-friendly links to some popular troubleshooting software tools (created for convenience):

Debugging Tools for Windows v. 6.10.3.233 (use it if you need !locks command, refer to this page for details):

• http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.10.3.233.msi
• http://msdl.microsoft.com/download/symbols/debuggers/dbg_amd64_6.10.3.233.msi

Debugging Tools for Windows v. 6.11.1.404:

• http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.11.1.404.msi
• http://msdl.microsoft.com/download/symbols/debuggers/dbg_amd64_6.11.1.404.msi

Latest SysInternals Suite:

• http://download.sysinternals.com/files/SysinternalsSuite.zip

Enabling automatic crash dump generation in case of an application fault

I composed this topic as I reminder for me and probably others who always forget all those registry paths.. See references at the bottom of the page.

So if you are using Windows Vista (or above) and have an application (let’s call it crash_me.exe) that crashes often and you want to generate crash dump in an unintended manner so that whenever it crashes – a dump is generated without user involvement – you’ll need to perform these two steps:

Step 1: Specify that a dump should be generated for your application by adding the following to the registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\crash_me.exe]

• DumpFolder REG_EXPAND_SZ – directory to hold the dump files
• DumpType REG_DWORD – Value: 2 (type of dump – full dump)

Step 2: Ensure that Windows Error Reporting UI is not shown by setting the following value (otherwise, crash dump generation won’t be performed until user selects anything in the “has stopped working” pop-up dialog):

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting]

• DontShowUI REG_DWORD – Value: 1

UPDATE: If you’re on Windows 7 then Fault Tolerant Heap might create obstacles for your troubleshooting process. Next steps were added specifically to deal with FTH (taken from MSDN article on Fault Tolerant Heap):

Step 3: Disable FTH (requires reboot): set HKLM\Software\Microsoft\FTH\Enabled to 0

Step 4: Reset FTH mitigation configuration: Rundll32.exe fthsvc.dll,FthSysprepSpecialize

References:
- MSDN articles: Collecting User-Mode Dumps, Fault Tolerant Heap
- Codehead’s article How To Disable “X has stopped working” Popup In Windows Vista

Tools for everyday use – www.py – HTML-enabled echo replacement

If you often have to analyze log files or issue commands that produce lots of plain-text output, consider taking a look at my small utility http://www.py (which lives in htmecho GitHub project).

Its purpose is to capture the input, transform it to an HTML with some additional JavaScript payload and open the result in the default browser. This enriched output provides filtering ability and allows assigning colors to lines that match a particular regular expression.

I use it so often, so that if it didn’t exist I would write one today :) For convenience, I usually place it into ~/bin/www (followed by chmod +x ~/bin/www). On Windows I just ensure that ‘py’ file extension is mapped onto Python 3.2 executable.

For example the following console command:
grep -s -r -e "Error:" -e "Warning:" *.log | www
would open a browser (or add a new tab into an already opened browser window) with the output of the grep.
Here is a screen of
dmesg | www
command with BIOS pattern marked as yellow:

Here is a bit of history that might be of interest to developers.

Few years ago, as I was studying I/O area of the C++ standard library, I wrote a small tool that transformed its STDIN input into a temporary HTML file. Though by that time I had only the educational goal, I found this tool to be surprisingly useful in my working environment – as I often have to analyze logs (typically after some grep operations on them) and due to my dual-monitor setup I find it very convenient to issue some command on one screen and see the results on the other.

The only major change before I stopped working on it was adding a ShellExecute call in the end – so that the newly generated HTML file was opened automatically in the default browser (which must be chrome which is chrome in my case). I just put it into a local SCM and kept it for few years without changes.

Recently, as I often rely on LoadRunner logs analysis while handling support cases, few more ideas came to mind as we discussed this tool with my colleague – what if we enrich the generated HTML file with a bit of JavaScript to allow some grep-like functionality right in the browser window? What if we colorize particular filters, so that it will be easy to visually find the interesting spots in the logs? That gave me enough motivation to invest more time in the project and the colleague helped me with the JavaScript part. I also added an imaginary TODO item to ensure that the tool compiles and works on Linux, as I use it on my laptop. Now, a simple one-file educational application became cross-platform with OS-transparent and OS-agnostic layers. I also made a bit of interesting research on how to embed binary resources into Linux executable modules (as I wanted to simulate Windows Portable Executable Resources behavior which allows trivial embedding of whatever you want into the executable).

Next battle was for Unicode support. Or, to put it correctly, total lack of such support in the language. I tried to avoid WinAPIs for that and to be able to handle windows-1251, UTF-16 BE/LE and UTF-8 encodings – I had to spent few nights reading quite a number of materials on iwfstream and codecvt (not the best reading, imo), which is neither simple to use nor beautiful conceptually. And still, removing BOM-checks in C++ classic iwfstream-based code was a great pain. And, OMG!, after I ensured it works on Windows – it just didn’t work on Linux… though it was a personal free-time development, instead of bringing satisfaction and joy – it was just nasty time waste. Finally, though I’ve got to a stable point with support of cp1251 and UTF8, I just didn’t feel I want to proceed any further.

And here comes Python!

Just for sport interest, I tried to reach the same functionality by using Python 3.2 and total time spent until I got it working on two platforms I’m interested in was around 2 hours of coding and debugging.

Benefits brought by Python:
- single-file executable with no dependencies. Even though Python version had platform-agnostic parts too, they were typically one-liners and there was no reason to move them into separate modules. In C++ I had to implement platform-agnostic (and even linker-agnostic for Linux) resource embedding, cross-platform code to open browser and it naturally resulted in multiple headers/source files.
- ability to modify HTML template with JavaScript payload – again, in Python it is native to have multi-line string literals – so, go on and just edit the script. In C++, it was simple too, but only because HTML template had been intentionally kept as an external resource linked into an executable. On windows, it is possible to modify an embedded resource externally with a tool like Resource Hacker. I don’t have a clue on whether anything as simple is available on Linux.
- Unicode. Python 3.x rulez here: standard distribution of the language has plenty of encodings supported out of the box, BOMs are removed automagically granted you’ve specified the correct encoding. It was pity to conclude that pure C++ with its standard library totally suck at this part.
- Performance – for a tiny tool like this I won’t even bother with runtime performance. Productivity is what matters here and Python is an obvious winner here. I admit it wasn’t clever for me to write such a program in C++ in the first place, but as I said it was an educational project and … I like to code in C++ anyway, at least when it’s not a unicode related code.

Dealing with several Python versions in Sublime Text 2

Sublime Text 2 contains the following build system definition for Python (refer to ~/.config/sublime-text-2/Packages/Python/Python.sublime-build):

{
    "cmd": ["python", "-u", "$file"],
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python"
}

Thus default Python interpreter is used (which is typically a symlink to concrete version). In case if you have several versions of Python installed – you’d want to control which version of Python to invoke on Sublime Text’s “Build” command.

It can be easily done by adding custom build system in Tools > New Build System… menu with a slightly changed definition, e.g:

{
    "cmd": ["python3.2", "-u", "$file"],
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python"
}

After saving this file, e.g. as Python3.sublime-build, and restarting Sublime Text editor it would be possible to specify the version of interpreter you need in Tools > Build System menu.

If you’re interested in Russian version of this entry – take a look here on my impulse9 blog

Python ctypes as a handy tool for C/C++ developers

I have recently learned how useful Python ctypes library might be when it comes to cross-platform testing of dynamic libraries. ctypes handles library loading and allows easy function invocation in case if library exposes C-like interfaces. In addition, library provides a way of using non-trivial data types as function call arguments (such as struct, union, C-style arrays and their combinations).

Solution is cross-platform, it significantly simplifies development and testing of new functions addition to existing dynamic libraries at early stages, so that there are no other clients for those functions ready yet. Also, it can be used as a good sanity measure for automated build regression checks.

As an example, here is a small use case: suppose I have to check the following function from a dynamic library (.dll for Windows, .so on Linux) written in C/C++:

int get_ips(const unsigned short af, struct ipaddress_info ** ppaddrinfo);

Where ipaddress_info is defined like this:

typedef struct ipaddress_info {
    unsigned short family;
    unsigned char addr[16];
    unsigned int scope_id;
} IPADDRESS_INFO;

So given the definition above – you’d have a DLL on Windows (or a shared library on Linux) that exports get_ips function. And here is the sample Python script that is capable to load that library, perform a call to get_ips and display the contents of a returned array of structures:

import os
import sys
from ctypes import *
from socket import AF_INET, AF_INET6, AF_UNSPEC

# Python representation of a C struct defined
# in a shared library being tested
class ipaddress_info(Structure):
    _fields_ = [
        ("family",      c_ushort),
        ("addr",        c_ubyte * 16),
        ("scope_id",    c_uint),
    ]

def bytes2str(v):
    s = ""
    for x in v:
        s += "%.02X " % (x)
    return s

class dynlib_bridge:
    def __init__(self, dll_path, working_dir="."):
        tmp = os.getcwd()
        os.chdir(working_dir)
        self.invoke = cdll.LoadLibrary(dll_path)
        os.chdir(tmp)

class test_case_base:
    def __init__(self, real_values):
        self.__inout__ = dict()
        self.__result__ = None
        for arg_name, arg_value in real_values.items():
            self.__inout__[arg_name] = arg_value
    def actual(self, name):
        return self.__inout__[name]
    def result(self):
        return self.__result__
    def setResult(self, v):
        self.__result__ = v
        
# performs a positive test case
def test_pos(testobj):
    result = testobj.run()
    if result:
        print ("[tc+] OK  :", testobj.describe())
    else:
        print ("[tc+] FAIL:", testobj.describe())    

# performs a negative test case
def test_neg(testobj):
    result = testobj.run()
    if not result:
        print ("[tc-] OK  :", testobj.describe())
    else:
        print ("[tc-] FAIL:", testobj.describe())    

# wraps invokation of a shared library function
class get_ips(test_case_base):
    def __init__(self, dll, af):
        super(get_ips, self).__init__({
            'af': c_ushort(af),
            'ppaddrinfo': POINTER(ipaddress_info)(),
            })
        self.dll = dll
    def run(self):
        invoke_result = self.dll.invoke.get_ips(
            self.actual('af'), 
            byref(self.actual('ppaddrinfo')))
        self.setResult(invoke_result)
        return (invoke_result > 0)
    def describe(self):
        p = self.actual('ppaddrinfo')
        ips = []
        if self.result() > 0:
            for i in range(self.result()):
                ips.append("#%d: family = %d, scope = %d, %s" % (
                    i, 
                    p[i].family, 
                    p[i].scope_id, 
                    bytes2str(p[i].addr)))
        s = "result: %.02d\n%s" % (self.result(), "\n".join(ips))
        return s
        
if __name__ == "__main__":
    # you'd probably have to use LD_LIBRARY_PATH to make this work on Linux
    path_to_lib = sys.argv[1]
    path_to_env = sys.argv[2]
    print ("lib path :", path_to_lib)
    print ("env path :", path_to_env)
    dll = dynlib_bridge(path_to_lib, working_dir=path_to_env)
    
    test_pos(get_ips(dll, AF_UNSPEC))
    test_pos(get_ips(dll, AF_INET))
    test_pos(get_ips(dll, AF_INET6))
    test_neg(get_ips(dll, 99))

The first argument is a library file name; second – is the location from which library load should be performed (it is sufficient on Windows, on Linux you might need to deal with LD_LIBRARY_PATH if your library has its own dependencies).

For simplicity, I removed most of error handling code and some not very interesting things like calling another library function to perform memory cleanup. Also, code assumes cdecl calling convention (which actually was my case for both tested platforms)

If you’re interested in Russian version of this entry – take a look here on my impulse9 blog

“IPv6 Learning Roadmap” from MSFT

Recently, while trying to improve a bit my  ipv6 understanding, I came upon this reference for those who already know ipv4 stuff and need to become acquainted with ipv6 - http://technet.microsoft.com/en-us/library/gg250710(WS.10).aspx.

And here is the corresponding blog post, which seems to be quite useful for developers dealing with windows networking, http://blogs.technet.com/b/ipv6/archive/2010/11/02/ipv6-learning-roadmap-now-available.aspx

workaround for MSTSC window minimization curse

As I was involved in development of a functional testing automation tool, we’ve been often addressed by customers using RDP to perform testing automation on remote machine. What they would like to do is to connect to the remote machine, invoke a testing tool there and start the testing, then either minimize the RDP client screen or even disconnect from the session (leaving the testing process in “background”).

The problem with such pattern was that Microsoft Terminal Services Client used output suppression for minimized MSTSC windows, i.e. notified the terminal server that client is not interested in updates from server. According to Remote Desktop Protocol specs:

Controlling Server Graphics OutputA client connected to a server and displaying graphics data may need to request that the server resend the graphics data for a collection of rectangular regions of the session screen area, or stop sending graphics data for a period of time (perhaps when the client is minimized). These two tasks are accomplished by having the client send the Refresh Rect PDU and Suppress Output PDUs, respectively.

Server’s reaction was to switch to winlogon screen for security reasons. the whole point is good – server’s station is in safe state and no extra traffic is generated, BUT the test would probably fail. Functional testing tools rely on mouse / keyboard control APIs, which don’t work on secured stations. Moreover, some applications under test do not work in this case either (as they might also rely on APIs that don’t work, e.g. GetCursorPos).

I’ve raised this topic (and definitely I wasn’t the first one), but without success. Idea was simple – all we need is the ability to configure client’s behavior on minimization, i.e. just don’t send “suppress output” hint to server. But there wasn’t an official way to do that for mstsc, though there was such a patch for alternative open-source client rdesktop.

Now I am not working in the area of functional testing automation tools development, but still it was nice to know that solution to this problem has been made public (though some companies already had this information). You can easily find it by googling about RemoteDesktop_SuppressWhenMinimized (or, for the lazy ones: stackoverflow link, Running Tests in Minimized Remote Desktop Windows).

---

This article has the  following references:

• MSDN: [MS-RDPBCGR]: Remote Desktop Protocol: Basic Connectivity and Graphics Remoting Specification
• StackOverflow topic: Scheduled execution of Testcomplete fails – Stack Overflow
• TestComplete KB: Running Tests in Minimized Remote Desktop Windows
• Old discussion thread: workaround for MSTSC window minimization curse

If you’d like to read the original Russian version of this article just click here.

blog expansion?

..well, I’m not feeling quite comfortable with wordpress so far.

I’ll most probably keep this blog for posts in English. Readers reading in Russian would probably prefer my newly created blog impulse9.blogspot.com.

Wow, I starting to behave like if I had readers :)