Well, no more! Recaps, that tiny utility that lets you switch your keyboard layout using the almost forgotten Capslock key, can now fix text you’ve typed with the wrong language selected. Just hit Ctrl-Capslock as soon as you discover your mistake and the text will be replaced with what you actually intended to type. I’ve only tested it with Hebrew and Russian, the languages I personally use on a regular basis, but it should work with any language supported by Windows. Let me know if it doesn’t.

<?xml version="1.0" encoding="UTF-8"?>
<update-info version="0.8.0[1212]“>
    <update file=”TestuffSetup.exe” from-version=”all”/>
    <update file=”TestuffUpdate.exe” from-version=”0.7.1[1110]“/>
    <md5hashes>
        <file md5=”3a23dd6eff6fd6c1d0fbfcbfb0d57221″ path=”async.pyc”/>
        <file md5=”0d1ea490a18c65cec7ba8715b5ea9e69″ path=”atexit.pyc”/>
        <file md5=”166723a4330a98b573119326fc689322″ path=”base64.pyc”/>
        <file md5=”01c1bda049936de570ed922424c057a8″ path=”BeautifulSoup.pyc”/>
    </md5hashes>
</update-info>

When the Testuff client launches, it gets the version.xml file from the server and compares its version to the version attribute of the update-info tag. If the client’s version is wrong, it checks the update tags to see which update it should download and install. We generate two separate setup files - one to update the most recent version to the new one called and another to update all the other (older) versions.

Aside from the info about which version of the client should use which update file, version.xml also contains the MD5 hashes for each file in the distribution. That might seem like a lot of wasted space and time, but it’s actually there for a very good reason. When our setup building script is creating TestuffUpdate.exe, it too downloads version.xml from our server. It then tries to determine which files have changed or have been added since the last version by comparing the MD5 hashes in version.xml to the the hashes of the actual files that have been generated by the build. Any file that is different is added to the update so we can be sure we haven’t missed any essential component in the update.

Recently I discovered that our update files are much larger than they should be. We release a new version with just a couple of fixes in a single module, and the size of the update is half the size of the full install. As it turned out, that most of the PYC files were marked as changed and added to the update. That didn’t seem right, especially for things like threading.pyc, which is a Python module that shouldn’t change unless you upgrade to a different version of Python, which we didn’t (still stuck at 2.4.4 I’m afraid). That got me curious enough to go digging in the, apparently undocumented, binary structure of the PYC files.

This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).

The first thing I did was compare the two threading.pyc files - the one from the current distribution and the one just generated by the build script. The result showed there was difference in only two bytes:

D:GooliDevTempcompare>fc /b threading-old.pyc threading-new.pyc
Comparing files threading-old.pyc and threading-new.pyc
00000004: 6E CE
00000005: F7 6A

Only two bytes differ, and they are right at the beginning of the file? That looks suspiciously like a version or a timestamp in the file header. Since the PYC file structure is undocumented, I went looking for the details in Python’s source code, but the answer was actually closer to home - in the compiler package. A file called pycodegen.py in Python\Lib\compiler contains the following code:

So, the PYC header file contains a magic number that identifies the Python release and the modification time of the original source file as the number of seconds since the epoch. That shouldn’t be a problem - the threading module hasn’t changed and should have the same timestamp. But as we’ve seen, the PYC files were different. How can that be?

Acting on a hunch, I wrote a short script to read the header from the PYC file and print the embedded date:

Which printed the following results:

Mon Mar 13 22:51:26 2006
Mon Mar 13 12:51:26 2006

Notice anything odd about them? They are exactly 10 hours apart. At first I thought I might actually be looking at two different versions of threading.py, but the chances of two edits being exactly 10 hours apart right down to the second is practically non-existent. It had to be something with time zones. I live and work in Israel, which is at GMT+2:00. The default timezone for Windows is Pacific time, which is GMT-8:00. Exactly 10 hours apart. However, no matter how I tweak the Regional Settings on my computer, all the PYC files I generate here have the same timestamp. Perhaps it has to do with the timezone you have set when you install Python. If I ever find out, I’ll let you know.

But that wasn’t the point of this post. The point was to figure out what PYC files look inside and we did that, at least in part - they start with a magic number that is different for each Python version (check out the comments in import.c), and they have an embedded timestamp of the source code they got generated from after that. The rest is generated by the marshal module and can be read by it to get the code objects and the global data in the module.

Another thing to be learned from this is that we really should always build the Testuff client on the same machine, which is why I’m heading to the office right now to burn a copy of the VMWare image I created with everything needed to build Testuff. We got a new version with a couple of important fixes to our Mantis support to release today.

The Run dialog is great to quickly access applications you use, but it’s only usable for programs that put their folder in the PATH environment variable, which not all programs do. I’ve been adding programs I liked to the PATH for years when a friend of mine showed me a neat trick. You create a folder called Shortcuts somewhere (say C:\Shortcuts) and put it at the start of your PATH environment variable and then create shortcuts to your favorite programs in that folder. All those shortcuts are now available to you via using Win+R. A nice bonus is the fact that you can name the shortcuts any way you like. I usually have a www.lnk file pointing to Firefox, note.lnk pointing to Notepad and word.lnk pointing to Word.

There’s only one thing missing here to make this trick into an fully featured program launcher. We need a way to quickly create shortcuts to programs, preferably by right-clicking on them in Windows Explorer. You can do that without installing anything using a couple of obscure Windows features - the Send To menu in the right-click menu of each file and the fact that Windows can run *.vbs files with VBScript code in it using something called WSH.

Here’s the source of a script that creates a link to whichever was passed to it in the command line (executable, folder, etc.).

The Send To menu is actually a folder located at C:\Documents and Settings\<YourUser>\SendTo. This folder contains links to various programs and you can add shortcuts to your own programs there. When you right-click a file, and choose one of the Send To items, the corresponding program is called, and given the name of the file you right-clicked as a command line parameter. So, if you put a link to the script above in the Send To folder, you’ll be able to easily create links in the C:\Shortcuts folder.

In case you lost me somewhere along the way, and in case I ever need to set this up myself again, here are the instructions to set things up:

1. Create C:\Shortcuts
2. Create C:\Shortcuts\create_shortcut.vbs and paste the above code into it.
3. Add C:\Shortcuts to the start of your PATH environment variable (Win+Pause > Advanced > Environment Variables).
4. Hit Win+R, type sendto, hit Enter. That should get you to your Send To folder.
5. Create a shortcut here to C:\Shortcuts\create_shortcut.vbs and call it Shortcuts.

To add a program to the launcher:

1. Right click a program or a folder (you can do it even on programs in your Start Menu).
2. Send To > Shortcuts.
3. In the dialog change the shortcut name to something short and catchy.
4. Test it : Hit Win+R, type the shortcut name and hit Enter.

You have to navigate to about:config, type bidi in the filter box, and double click the line that says bidi.browser.ui. That will set its value to true. All you have to do now is restart Firefox and the View->Switch Page Direction and Edit->Switch Text Direction menu items will be added along with the Ctrl-Shift-X shortcut.

Hope this helps some poor soul someday.

However, the latest version of Miranda broke my language switching gizmo, Recaps. Every once in a while the CapsLock key would stop switching the language in the Miranda chat window, and even restarting Recaps didn’t help. The basic thing Recaps does upon detecting that CapsLock has been pressed is send an WM_INPUTLANGCHANGEREQUEST message the the current foreground window, which is obtained via the GetForegroundWindow Win32 API function. Apparently I overlooked the fact that the MSDN specifically says the aforementioned message is sent to the window that has the keyboard focus, which I clear wasn’t doing. GetForegroundWindow just gets you the current top-level window, not the actual control that has the keyboard focus.

All I had to do was just replace the GetForegroundWindow call something that gets the window that has the focus and be done with it. And GetFocus seemed to be just what I needed. Only there was a catch:

The GetFocus function retrieves the handle to the window that has the keyboard focus, if the window is attached to the calling thread’s message queue.

That isn’t good enough of course as the whole point was finding out which window had the focus when that window belonged to a different application, running in a different process, in a different thread. There are a few techniques to inject code into a remote process, but they all require creating a separate DLL and tricking the target process into loading it either via hooks or using the CreateRemoteThread/LoadLibrary trick. I was getting ready to dive into that dark abyss when I stumbled upon the AttachThreadInput function:

Windows created in different threads typically process input independently of each other. That is, they have their own input states (focus, active, capture windows, key state, queue status, and so on), and they are not synchronized with the input processing of other threads. By using the AttachThreadInput function, a thread can attach its input processing to another thread. This also allows threads to share their input states, so they can call the SetFocus function to set the keyboard focus to a window of a different thread. This also allows threads to get key-state information. These capabilities are not generally possible.

So, all I had to do was call AttachThreadInput to connect my main UI thread to the thread responsible for the current foreground window, call GetFocus to find out which window has the focus, and then call AttachThreadInput again to give the control of the input back to the original thread.

Here’s the code to do that:

This function will only work if called from a thread that has created a message queue, which is the case for a standard Win32 main thread that runs in a message loop.

Miranda and Recaps now play well together.

Yay!

I’m going to talk about some of the new things we did for this version in the rest of this post, but if you just want to take a look at the new Testuff, just grab it from our site.

Eating Bonzo for breakfast
They say eating your own dog food is important. Although I personally prefer a good steak, you can’t argue with some of the best minds of the software industry. Following the canine oriented software practice we finally sat down to create some tests for Testuff in, well, Testuff. And boy, was it awkward to write tests with our test editor. So we wrote a new test editor, which makes typing tests easy. Just separate the steps with a blank line and the editor will do the rest. You can also copy-paste tests into the new editor from almost anywhere – Word, Excel, the web, etc.

We helped manually convert a bunch of tests for some of our clients who had a few hundred test cases written down in Excel. Took about half an hour to get it done including reorganizing the tests making sure there were no duplicates.

Integrate early, integrate often
It was kind of ridiculous that the Testuff front page said we supported integration with various bug trackers and when you scrolled down to the list, you saw we actually only supported Trac. We started out with Trac as we use it ourselves, but we now added support for two more popular bug trackers. We’re are going to add support for even more bug trackers, as that seems to be what our users want most.

Bugzilla is probably the oldest guy on the block. With almost 10 years of experience under their belts, the guys there know how to manage defects and have seen it all. A interesting recent post seems to indicate Bugzilla may be dying along side Perl, but it’s still widely used and loved by many however ugly I may think it’s UI is.

Many software developers know the name of Joel Spolsky. I was first introduced to his excellent articles when I was a young officer in the Israeli army trying to lead a team of soldiers-developers through the mud of an enormous C++/MFC HR planning system. Some of the articles were amusing, some interesting and a few were a real eye-opener to me. His company, FogCreek, makes a bug tracking and project management software called FogBugz. Personally, I prefer the more Spartan approach of Trac and like to tinker with it’s Python code base, but FogBugz has many followers and it is now offered as an on-demand service, which makes it a great match with Testuff.

Those damn proxies
Being a desktop application with a web-based backend, the Testuff client needs to access our server in order to work. It turned out that accessing the internet through a proxy is a challenging task, which Python doesn’t help with nearly enough. I ended up wrapping the WinHTTP library with ctypes code and using it to connect to the internet using the settings in Internet Explorer. That seems to be working with all our clients so far, but it’s not a portable solution. I’ll be taking a look at pycurl next, to see if it works as well as promised on their website. I’ll post a full report on the proxy fighting process sometime soon.

Oh, yeah, and you can finally change the password you use to connect to Testuff. That doesn’t have anything to do with the proxy support, but this stupid omission we made in previous versions doesn’t justify it’s own heading, right?

What’s next?
Right now we’re hard at work on our next version, 0.8. There will be support for even more bug trackers that our customers asked for (Mantis, Elementool, Jira and maybe another one or two if we have the time), and, if I can fix that damn segfault, a Linux version of the Testuff client.

You can download Testuff from www.testuff.com/download.

Then there’s the CapsLock key. I don’t think anybody uses it nowadays, and even the touch typists seem to just HOLD THE SHIFT WITH THEIR PINKY and type what needs to be in capital letters. I wrote a small program called Recaps a while ago that converts CapsLock into a language switching key. Now I can’t live without it. I find myself instinctively hitting CapsLock to switch languages never thinking about it, even on computers I didn’t install it on. Needless to say it’s one of the first things I install on a computer I need to work on.

I talked to an old friend of mine last night who said he was using Recaps and spreading it around but he was missing a feature. When there were more three or more languages installed on the computer, Recaps would just cycle through all of them, like Alt-Shift does. Most times however, you only use two languages at any given time, typically English and your native tongue, and only need to switch between these two.

Doing this in Win32 API was a bitch, but I finally got a tray icon and a small menu to work. The menu shows the list of languages currently installed on your computer with check boxes next to them. Hitting CapsLock now only cycles through the languages that are currently enabled and even saves the active languages between runs.

You can download source and binaries for the new 0.3 version from my Recaps page.

I’d love to know if anybody finds it as useful as I do.

Sarah Siegel writes:

“Channa, I don’t want to forget to ask you for a word today. How do you say, ‘Bull’ in Kannada?”
“Gooli, Ma’am.”

And the urban dictionary says:

Goolies

Noun:
1. Private parts
2. Family Jewels

“I kicked him in the goolies”

Balls of bull. Nice!

The problem becomes more complex when you need to separate the tokens you find into different kinds, words and numbers, for instance. We’ll use a well known lyric as our sample text:

The first thing we need to do is build a regular expression that recognizes words and another one that recognizes numbers. Although there are shorter ways to build those regular expressions, I like the less obscure form:

We could now use findall on the string and get all the numbers and words out of it.

But wait, you say, that isn’t what we wanted at all! We need to get the tokens in the order of their appearance in text and still get the type of each token. Something along the lines of

would be really nice.

In order to do that, we’ll need to combine both regular expressions into one and iterate on the result of findall examining each token to decide on its type.

As you can see, the result of the call to findall is a list of tuples, each containing a single match. If you look closely at the way I’ve combined the two regular expressions, you’ll see that each part is surrounded with parenthesis and that there’s a pipe (|) between the expressions. The compound regular expression matches either a number rf a word and each tuple in the return value of findall contains the matches for each parenthesized part of the regexp. However, since we combined the parts using a pipe (|), only one of the parts matches each time.

Using that knowledge we can now construct a simple loop that shows the token type for each of the words in the lyric:

We now have most of the knowledge we need to build ourselves a lexer that will take a list of regular expressions and some text and return (or even better, generate) an list of tokens and their types. We’ll need to combine the regular expressions for each token into one big regex using pipes, scan the string, and gather the tokens and their types.

Our usage code looks like this:

And here is the code for the lexer itself:

Some notes on the implementation are in order. I’ve used the little known (?P<name>…) syntax for naming the parenthesized groups of regular expressions. Using that syntax the expression (?P<word>[A-Za-z]) matches a word and that match is accessible with match.group(’word’) where match is a re.Match object.

In order to speed things up a bit, I’ve compiled the regular expression when the Lexer object is created, used the finditer function instead of findall, and made parse a generator instead of a list returning function.

Using this simple lexer implementation it was quite simple to create a Python-to-HTML converter with syntax highlighting that works well enough to highlight the code of the highlighter itself!

The code for the lexer and syntax highlighter example are available here and on my snippets page. You can also see the result of running the syntax highlighter on itself here.

Enjoy lexing and let me know if you found this useful.

Would you host your website with a company whose offices look like this?

I don’t really know what to think about this. Many “serious” companies I’ve delat with in the past provide crappy service although their offices are sparky clean and they don’t do silly things like consume enough alcohol to get evicted. On the other hand it does seem ensettling that the company that you rely on to keep your data safe behaves like a college fraternity.

I did write about them however as others have done and that’s got to be worth something. After all, there’s no such thing as bad press, right?