Wrong keyboard layout

May 26th, 2008

How many times have you typed an entire URL looking at your keyboard only to lift your eyes and see that you’ve used the wrong keyboard layout? How many times have you typed your email into some obscure form on the internet only to find that you’ve typed some unintelligible gibberish in Hebrew, Russian or Farsi?

Well, no more! Recaps, that tiny utility that lets you switch your keyboard layout using the almost forgotten Capslock key, can now fix text you’ve typed with the wrong language selected. Just hit Ctrl-Capslock as soon as you discover your mistake and the text will be replaced with what you actually intended to type. I’ve only tested it with Hebrew and Russian, the languages I personally use on a regular basis, but it should work with any language supported by Windows. Let me know if it doesn’t.

Digging into Python’s PYC files

January 25th, 2008

One of the first things we needed to do when we started working on Testuff, was to figure out how are we going to update the installed desktop clients. This is one of those problems that seems to usually fall under the NIH syndrome, and like many others before me, I invented my own scheme. The gist of it is a version.xml file that sits alongside the setup file for the newest release and looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<update-info version="0.8.0[1212]">
    <update file="TestuffSetup.exe" from-version="all"/>
    <update file="TestuffUpdate.exe" from-version="0.7.1[1110]"/>
    <md5hashes>
        <file md5="3a23dd6eff6fd6c1d0fbfcbfb0d57221" path="async.pyc"/>
        <file md5="0d1ea490a18c65cec7ba8715b5ea9e69" path="atexit.pyc"/>
        <file md5="166723a4330a98b573119326fc689322" path="base64.pyc"/>
        <file md5="01c1bda049936de570ed922424c057a8" path="BeautifulSoup.pyc"/>
    </md5hashes>
</update-info>

When the Testuff client launches, it gets the version.xml file from the server and compares its version to the version attribute of the update-info tag. If the client’s version is wrong, it checks the update tags to see which update it should download and install. We generate two separate setup files – one to update the most recent version to the new one called and another to update all the other (older) versions.

Aside from the info about which version of the client should use which update file, version.xml also contains the MD5 hashes for each file in the distribution. That might seem like a lot of wasted space and time, but it’s actually there for a very good reason. When our setup building script is creating TestuffUpdate.exe, it too downloads version.xml from our server. It then tries to determine which files have changed or have been added since the last version by comparing the MD5 hashes in version.xml to the the hashes of the actual files that have been generated by the build. Any file that is different is added to the update so we can be sure we haven’t missed any essential component in the update.

Recently I discovered that our update files are much larger than they should be. We release a new version with just a couple of fixes in a single module, and the size of the update is half the size of the full install. As it turned out, that most of the PYC files were marked as changed and added to the update. That didn’t seem right, especially for things like threading.pyc, which is a Python module that shouldn’t change unless you upgrade to a different version of Python, which we didn’t (still stuck at 2.4.4 I’m afraid). That got me curious enough to go digging in the, apparently undocumented, binary structure of the PYC files.

This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).

The first thing I did was compare the two threading.pyc files – the one from the current distribution and the one just generated by the build script. The result showed there was difference in only two bytes:

D:GooliDevTempcompare>fc /b threading-old.pyc threading-new.pyc
Comparing files threading-old.pyc and threading-new.pyc
00000004: 6E CE
00000005: F7 6A

Only two bytes differ, and they are right at the beginning of the file? That looks suspiciously like a version or a timestamp in the file header. Since the PYC file structure is undocumented, I went looking for the details in Python’s source code, but the answer was actually closer to home – in the compiler package. A file called pycodegen.py in Python\Lib\compiler contains the following code:

def getPycHeader(self):
    mtime = os.path.getmtime(self.filename)
    mtime = struct.pack(‘&lt;i’, mtime)
    return self.MAGIC + mtime

So, the PYC header file contains a magic number that identifies the Python release and the modification time of the original source file as the number of seconds since the epoch. That shouldn’t be a problem – the threading module hasn’t changed and should have the same timestamp. But as we’ve seen, the PYC files were different. How can that be?

Acting on a hunch, I wrote a short script to read the header from the PYC file and print the embedded date:

import os, os.path
import struct
import timedef print_internal_date(filename):
    f = open(filename, "rb")
    data = f.read(8)
    mtime = struct.unpack("&lt;i", data[4:])
    print time.asctime(time.gmtime(mtime[0]))print_internal_date("threading-old.pyc")
print_internal_date("threading-new.pyc")

Which printed the following results:

Mon Mar 13 22:51:26 2006
Mon Mar 13 12:51:26 2006

Notice anything odd about them? They are exactly 10 hours apart. At first I thought I might actually be looking at two different versions of threading.py, but the chances of two edits being exactly 10 hours apart right down to the second is practically non-existent. It had to be something with time zones. I live and work in Israel, which is at GMT+2:00. The default timezone for Windows is Pacific time, which is GMT-8:00. Exactly 10 hours apart. However, no matter how I tweak the Regional Settings on my computer, all the PYC files I generate here have the same timestamp. Perhaps it has to do with the timezone you have set when you install Python. If I ever find out, I’ll let you know.

But that wasn’t the point of this post. The point was to figure out what PYC files look inside and we did that, at least in part – they start with a magic number that is different for each Python version (check out the comments in import.c), and they have an embedded timestamp of the source code they got generated from after that. The rest is generated by the marshal module and can be read by it to get the code objects and the global data in the module.

Another thing to be learned from this is that we really should always build the Testuff client on the same machine, which is why I’m heading to the office right now to burn a copy of the VMWare image I created with everything needed to build Testuff. We got a new version with a couple of important fixes to our Mantis support to release today.

A very simple Win+R enhancement

January 7th, 2008

I use the Win+R combination to run almost all the applications I use. Hit Win+R, type firefox, and Firefox runs. Hit Win+R, type winword, and Word starts. Hit Win+R…, well you get the idea.

The Run dialog is great to quickly access applications you use, but it’s only usable for programs that put their folder in the PATH environment variable, which not all programs do. I’ve been adding programs I liked to the PATH for years when a friend of mine showed me a neat trick. You create a folder called Shortcuts somewhere (say C:\Shortcuts) and put it at the start of your PATH environment variable and then create shortcuts to your favorite programs in that folder. All those shortcuts are now available to you via using Win+R. A nice bonus is the fact that you can name the shortcuts any way you like. I usually have a www.lnk file pointing to Firefox, note.lnk pointing to Notepad and word.lnk pointing to Word.

There’s only one thing missing here to make this trick into an fully featured program launcher. We need a way to quickly create shortcuts to programs, preferably by right-clicking on them in Windows Explorer. You can do that without installing anything using a couple of obscure Windows features – the Send To menu in the right-click menu of each file and the fact that Windows can run *.vbs files with VBScript code in it using something called WSH.

Here’s the source of a script that creates a link to whichever was passed to it in the command line (executable, folder, etc.).

SHORTCUTS_PATH = "C:\Shortcuts"

If WScript.Arguments.Count < 1 Then
    WScript.Echo("Missing parameters")
Else
    ‘ create shorcut name
    commandPath = WScript.Arguments(0)
    dot = InStrRev(commandPath, ".")
    If dot = 0 Then dot = Len(commandPath)
    slash = InStrRev(commandPath, "\")
    If slash = 0 Then slash = 1
    commandDir = Mid(commandPath, 1, slash-1)
    shortcutName = Mid(commandPath, slash+1, dot-slash-1)

    ‘ ask for shortcut name
    shortcutName = InputBox("Enter shortcut name", "Shortcut", shortcutName)

    ‘ create shortcut
    shortcutPath = SHORTCUTS_PATH & "\" & shortcutName & ".lnk"
    Set WshShell = WScript.CreateObject("WScript.Shell")
    Set oShellLink = WshShell.CreateShortcut(shortcutPath)
    oShellLink.TargetPath = commandPath
    oShellLink.WindowStyle = 1
    oShellLink.IconLocation = commandPath & ", 0"
    oShellLink.WorkingDirectory = commandDir
    oShellLink.Save
End If

The Send To menu is actually a folder located at C:\Documents and Settings\<YourUser>\SendTo. This folder contains links to various programs and you can add shortcuts to your own programs there. When you right-click a file, and choose one of the Send To items, the corresponding program is called, and given the name of the file you right-clicked as a command line parameter. So, if you put a link to the script above in the Send To folder, you’ll be able to easily create links in the C:\Shortcuts folder.

In case you lost me somewhere along the way, and in case I ever need to set this up myself again, here are the instructions to set things up:

  1. Create C:\Shortcuts
  2. Create C:\Shortcuts\create_shortcut.vbs and paste the above code into it.
  3. Add C:\Shortcuts to the start of your PATH environment variable (Win+Pause > Advanced > Environment Variables).
  4. Hit Win+R, type sendto, hit Enter. That should get you to your Send To folder.
  5. Create a shortcut here to C:\Shortcuts\create_shortcut.vbs and call it Shortcuts.

To add a program to the launcher:

  1. Right click a program or a folder (you can do it even on programs in your Start Menu).
  2. Send To > Shortcuts.
  3. In the dialog change the shortcut name to something short and catchy.
  4. Test it : Hit Win+R, type the shortcut name and hit Enter.

Right to left (BiDi) support in Firefox

December 31st, 2007

Whenever I install Firefox on a new computer, I find myself hitting Ctrl-Shift-X to switch the text box direction to now avail. Unfortunately the way to turn this option on is quite hidden.

You have to navigate to about:config, type bidi in the filter box, and double click the line that says bidi.browser.ui. That will set its value to true. All you have to do now is restart Firefox and the View->Switch Page Direction and Edit->Switch Text Direction menu items will be added along with the Ctrl-Shift-X shortcut.

Hope this helps some poor soul someday.

Win32: GetFocus across process boundries

December 29th, 2007

I’ve been using Miranda for all my chats for a few years now, since the time it was a nightmare to install and configure. The chief reason I like it is the fact that it supports Hebrew amazingly well, via the TabSRMM plugin. You can even set the right-to-left settings for the chat window on a person-by-person basis.

However, the latest version of Miranda broke my language switching gizmo, Recaps. Every once in a while the CapsLock key would stop switching the language in the Miranda chat window, and even restarting Recaps didn’t help. The basic thing Recaps does upon detecting that CapsLock has been pressed is send an WM_INPUTLANGCHANGEREQUEST message the the current foreground window, which is obtained via the GetForegroundWindow Win32 API function. Apparently I overlooked the fact that the MSDN specifically says the aforementioned message is sent to the window that has the keyboard focus, which I clear wasn’t doing. GetForegroundWindow just gets you the current top-level window, not the actual control that has the keyboard focus.

All I had to do was just replace the GetForegroundWindow call something that gets the window that has the focus and be done with it. And GetFocus seemed to be just what I needed. Only there was a catch:

The GetFocus function retrieves the handle to the window that has the keyboard focus, if the window is attached to the calling thread’s message queue.

That isn’t good enough of course as the whole point was finding out which window had the focus when that window belonged to a different application, running in a different process, in a different thread. There are a few techniques to inject code into a remote process, but they all require creating a separate DLL and tricking the target process into loading it either via hooks or using the CreateRemoteThread/LoadLibrary trick. I was getting ready to dive into that dark abyss when I stumbled upon the AttachThreadInput function:

Windows created in different threads typically process input independently of each other. That is, they have their own input states (focus, active, capture windows, key state, queue status, and so on), and they are not synchronized with the input processing of other threads. By using the AttachThreadInput function, a thread can attach its input processing to another thread. This also allows threads to share their input states, so they can call the SetFocus function to set the keyboard focus to a window of a different thread. This also allows threads to get key-state information. These capabilities are not generally possible.

So, all I had to do was call AttachThreadInput to connect my main UI thread to the thread responsible for the current foreground window, call GetFocus to find out which window has the focus, and then call AttachThreadInput again to give the control of the input back to the original thread.

Here’s the code to do that:

HWND RemoteGetFocus()
{
    HWND hwnd = GetForegroundWindow();
    DWORD remoteThreadId = GetWindowThreadProcessId(hwnd, NULL);
    DWORD currentThreadId = GetCurrentThreadId();
    AttachThreadInput(remoteThreadId, currentThreadId, TRUE);
    HWND focused = GetFocus();
    AttachThreadInput(remoteThreadId, currentThreadId, FALSE);
    return focused;
}

This function will only work if called from a thread that has created a message queue, which is the case for a standard Win32 main thread that runs in a message loop.

Miranda and Recaps now play well together.

Yay!

Testuff is finally usable

December 20th, 2007

It’s a very exciting time for us here at Testuff as our product has finally reached a point where it can actually be used by real people. You can’t use testing software unless you can report bugs, and until now we could only report bugs to Trac. Now, with the addition of support for Bugzilla and FogBugz, we really do integrate with several bug trackers as our home page claims, and the new test editor make it a breeze to add and edit test cases.

I’m going to talk about some of the new things we did for this version in the rest of this post, but if you just want to take a look at the new Testuff, just grab it from our site.

Eating Bonzo for breakfastNew Test Editor
They say eating your own dog food is important. Although I personally prefer a good steak, you can’t argue with some of the best minds of the software industry. Following the canine oriented software practice we finally sat down to create some tests for Testuff in, well, Testuff. And boy, was it awkward to write tests with our test editor. So we wrote a new test editor, which makes typing tests easy. Just separate the steps with a blank line and the editor will do the rest. You can also copy-paste tests into the new editor from almost anywhere – Word, Excel, the web, etc.

We helped manually convert a bunch of tests for some of our clients who had a few hundred test cases written down in Excel. Took about half an hour to get it done including reorganizing the tests making sure there were no duplicates.

Integrate early, integrate often
It was kind of ridiculous that the Testuff front page said we supported integration with various bug trackers and when you scrolled down to the list, you saw we actually only supported Trac. We started out with Trac as we use it ourselves, but we now added support for two more popular bug trackers. We’re are going to add support for even more bug trackers, as that seems to be what our users want most.

Bugzilla is probably the oldest guy on the block. With almost 10 years of experience under their belts, the guys there know how to manage defects and have seen it all. A interesting recent post seems to indicate Bugzilla may be dying along side Perl, but it’s still widely used and loved by many however ugly I may think it’s UI is.

Many software developers know the name of Joel Spolsky. I was first introduced to his excellent articles when I was a young officer in the Israeli army trying to lead a team of soldiers-developers through the mud of an enormous C++/MFC HR planning system. Some of the articles were amusing, some interesting and a few were a real eye-opener to me. His company, FogCreek, makes a bug tracking and project management software called FogBugz. Personally, I prefer the more Spartan approach of Trac and like to tinker with it’s Python code base, but FogBugz has many followers and it is now offered as an on-demand service, which makes it a great match with Testuff.

Those damn proxies
Being a desktop application with a web-based backend, the Testuff client needs to access our server in order to work. It turned out that accessing the internet through a proxy is a challenging task, which Python doesn’t help with nearly enough. I ended up wrapping the WinHTTP library with ctypes code and using it to connect to the internet using the settings in Internet Explorer. That seems to be working with all our clients so far, but it’s not a portable solution. I’ll be taking a look at pycurl next, to see if it works as well as promised on their website. I’ll post a full report on the proxy fighting process sometime soon.

Oh, yeah, and you can finally change the password you use to connect to Testuff. That doesn’t have anything to do with the proxy support, but this stupid omission we made in previous versions doesn’t justify it’s own heading, right?

What’s next?
Right now we’re hard at work on our next version, 0.8. There will be support for even more bug trackers that our customers asked for (Mantis, Elementool, Jira and maybe another one or two if we have the time), and, if I can fix that damn segfault, a Linux version of the Testuff client.

You can download Testuff from www.testuff.com/download.