The integration allows for an automated testing script to report the results of its run to the Testuff server. In order for the results to be grouped, displayed and summarized correctly, the automation script needs to tell the server which test it ran, and whether the test has passed or failed. A long discussion emerged on what the best way to uniquely identify tests.

After quite a bit of back and forth, we’ve settled on permalinks, those more-or-less-readable URLs that are in common use in blogs. The idea of a permalink is to take the title (of a blog post or a test) and replace any characters that aren’t numbers or letters with an underscore or a hyphen. Using this simple scheme, “Unicode and permalinks” becomes “unicode-and-permalinks”, which is quite suitable for use in a URL.

The implementation is a simple regular expression:

While this code works perfectly for the English language, it doesn’t work at all if string is a Unicode string containing something in Hebrew, Russian or Polish – language that some of our customers use. And so, I set out to write code that will essentially behave like the regular expression above, but will work for letters and numbers in all the languages of the world.

Fortunately the Unicode standard includes a rarely used classification of characters into various categories. For each given character we can find out whether it is an uppercase letter, a lowercase letter, and number, a punctuation mark and so on. Surprisingly, Python includes a module called unicodedata that contains all that information. The function category accepts a character and returns a string that tells us what the character is: “Lu” denotes an uppercase letter, “Nd” denotes a decimal digit, etc.

All that remains to be done is go over the characters in the title, keep the letters and numbers, and replace all the other characters with a dash or an underscore. The regular expression at the end replaces any sequence of underscores into a single underscore to make the resulting URLs even nicer to look at.

[Update] Or, as Almad correctly pointed out, you could just use the re module support for Unicode and be done with it in two lines, which kind of takes the air out of this post.

There’s one other thing to consider when dealing with Unicode permalinks. If you’re a native speaker of a language other than English, you’ve probably seen URLs that in your own language in Wikipedia.

From the looks of it, URLs can include characters in any language. Right?

Wrong.

RFC3986 defines the syntax for URLs (actually URIs, but that’s a moot point) explicitly and states which characters are allowed in a URL. This includes little more than English letters and numbers from the lower half of the ASCII chart.

If you look at the headers your browser passes when you access such a URL, you’ll see that it encodes all the characters with percent encoding, so neither the browser nor the web server is violating the standard. This is what the server saw when I navigated to the main Hebrew page of Wikipedia:

GET /wiki/%D7%A2%D7%9E%D7%95%D7%93_%D7%A8%D7%90%D7%A9%D7%99 HTTP/1.1
Host: he.wikipedia.org

In order to understand what this percent encoding means, you need to know a bit about Unicode. Basically, the Unicode URL is encoded in UTF8 and each byte of the UTF8-encoded string is encoded using percent encoding. The browser apparently recognized this specific encoding scheme (which isn’t documented anywhere I could fine) and displays nice internationalized URLs for the user.

If you want to support such URLs in your server, you’ll probably need to write some code to translate the percent-encoded URLs into their actual Unicode representation.

This is how the Python repository looks like via http://svn.python.org/projects/python:

It says Revision 65620 at the top and I’ve always wondered if you could access another revision in the same simple way. Turns out there is a way.

All you need to do is add !svn/bc/REVISION to the URL:

http://svn.python.org/projects/!svn/bc/5000/python shows revision 5000 of the Python Subversion repository.

<?xml version="1.0" encoding="UTF-8"?>
<update-info version="0.8.0[1212]">
    <update file="TestuffSetup.exe" from-version="all"/>
    <update file="TestuffUpdate.exe" from-version="0.7.1[1110]"/>
    <md5hashes>
        <file md5="3a23dd6eff6fd6c1d0fbfcbfb0d57221" path="async.pyc"/>
        <file md5="0d1ea490a18c65cec7ba8715b5ea9e69" path="atexit.pyc"/>
        <file md5="166723a4330a98b573119326fc689322" path="base64.pyc"/>
        <file md5="01c1bda049936de570ed922424c057a8" path="BeautifulSoup.pyc"/>
    </md5hashes>
</update-info>

When the Testuff client launches, it gets the version.xml file from the server and compares its version to the version attribute of the update-info tag. If the client’s version is wrong, it checks the update tags to see which update it should download and install. We generate two separate setup files – one to update the most recent version to the new one called and another to update all the other (older) versions.

Aside from the info about which version of the client should use which update file, version.xml also contains the MD5 hashes for each file in the distribution. That might seem like a lot of wasted space and time, but it’s actually there for a very good reason. When our setup building script is creating TestuffUpdate.exe, it too downloads version.xml from our server. It then tries to determine which files have changed or have been added since the last version by comparing the MD5 hashes in version.xml to the the hashes of the actual files that have been generated by the build. Any file that is different is added to the update so we can be sure we haven’t missed any essential component in the update.

Recently I discovered that our update files are much larger than they should be. We release a new version with just a couple of fixes in a single module, and the size of the update is half the size of the full install. As it turned out, that most of the PYC files were marked as changed and added to the update. That didn’t seem right, especially for things like threading.pyc, which is a Python module that shouldn’t change unless you upgrade to a different version of Python, which we didn’t (still stuck at 2.4.4 I’m afraid). That got me curious enough to go digging in the, apparently undocumented, binary structure of the PYC files.

This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).

The first thing I did was compare the two threading.pyc files – the one from the current distribution and the one just generated by the build script. The result showed there was difference in only two bytes:

D:GooliDevTempcompare>fc /b threading-old.pyc threading-new.pyc
Comparing files threading-old.pyc and threading-new.pyc
00000004: 6E CE
00000005: F7 6A

Only two bytes differ, and they are right at the beginning of the file? That looks suspiciously like a version or a timestamp in the file header. Since the PYC file structure is undocumented, I went looking for the details in Python’s source code, but the answer was actually closer to home – in the compiler package. A file called pycodegen.py in Python\Lib\compiler contains the following code:

So, the PYC header file contains a magic number that identifies the Python release and the modification time of the original source file as the number of seconds since the epoch. That shouldn’t be a problem – the threading module hasn’t changed and should have the same timestamp. But as we’ve seen, the PYC files were different. How can that be?

Acting on a hunch, I wrote a short script to read the header from the PYC file and print the embedded date:

Which printed the following results:

Mon Mar 13 22:51:26 2006
Mon Mar 13 12:51:26 2006

Notice anything odd about them? They are exactly 10 hours apart. At first I thought I might actually be looking at two different versions of threading.py, but the chances of two edits being exactly 10 hours apart right down to the second is practically non-existent. It had to be something with time zones. I live and work in Israel, which is at GMT+2:00. The default timezone for Windows is Pacific time, which is GMT-8:00. Exactly 10 hours apart. However, no matter how I tweak the Regional Settings on my computer, all the PYC files I generate here have the same timestamp. Perhaps it has to do with the timezone you have set when you install Python. If I ever find out, I’ll let you know.

But that wasn’t the point of this post. The point was to figure out what PYC files look inside and we did that, at least in part – they start with a magic number that is different for each Python version (check out the comments in import.c), and they have an embedded timestamp of the source code they got generated from after that. The rest is generated by the marshal module and can be read by it to get the code objects and the global data in the module.

Another thing to be learned from this is that we really should always build the Testuff client on the same machine, which is why I’m heading to the office right now to burn a copy of the VMWare image I created with everything needed to build Testuff. We got a new version with a couple of important fixes to our Mantis support to release today.

However, the latest version of Miranda broke my language switching gizmo, Recaps. Every once in a while the CapsLock key would stop switching the language in the Miranda chat window, and even restarting Recaps didn’t help. The basic thing Recaps does upon detecting that CapsLock has been pressed is send an WM_INPUTLANGCHANGEREQUEST message the the current foreground window, which is obtained via the GetForegroundWindow Win32 API function. Apparently I overlooked the fact that the MSDN specifically says the aforementioned message is sent to the window that has the keyboard focus, which I clear wasn’t doing. GetForegroundWindow just gets you the current top-level window, not the actual control that has the keyboard focus.

All I had to do was just replace the GetForegroundWindow call something that gets the window that has the focus and be done with it. And GetFocus seemed to be just what I needed. Only there was a catch:

The GetFocus function retrieves the handle to the window that has the keyboard focus, if the window is attached to the calling thread’s message queue.

That isn’t good enough of course as the whole point was finding out which window had the focus when that window belonged to a different application, running in a different process, in a different thread. There are a few techniques to inject code into a remote process, but they all require creating a separate DLL and tricking the target process into loading it either via hooks or using the CreateRemoteThread/LoadLibrary trick. I was getting ready to dive into that dark abyss when I stumbled upon the AttachThreadInput function:

Windows created in different threads typically process input independently of each other. That is, they have their own input states (focus, active, capture windows, key state, queue status, and so on), and they are not synchronized with the input processing of other threads. By using the AttachThreadInput function, a thread can attach its input processing to another thread. This also allows threads to share their input states, so they can call the SetFocus function to set the keyboard focus to a window of a different thread. This also allows threads to get key-state information. These capabilities are not generally possible.

So, all I had to do was call AttachThreadInput to connect my main UI thread to the thread responsible for the current foreground window, call GetFocus to find out which window has the focus, and then call AttachThreadInput again to give the control of the input back to the original thread.

Here’s the code to do that:

This function will only work if called from a thread that has created a message queue, which is the case for a standard Win32 main thread that runs in a message loop.

Miranda and Recaps now play well together.

Yay!

Then there’s the CapsLock key. I don’t think anybody uses it nowadays, and even the touch typists seem to just HOLD THE SHIFT WITH THEIR PINKY and type what needs to be in capital letters. I wrote a small program called Recaps a while ago that converts CapsLock into a language switching key. Now I can’t live without it. I find myself instinctively hitting CapsLock to switch languages never thinking about it, even on computers I didn’t install it on. Needless to say it’s one of the first things I install on a computer I need to work on.

I talked to an old friend of mine last night who said he was using Recaps and spreading it around but he was missing a feature. When there were more three or more languages installed on the computer, Recaps would just cycle through all of them, like Alt-Shift does. Most times however, you only use two languages at any given time, typically English and your native tongue, and only need to switch between these two.

Doing this in Win32 API was a bitch, but I finally got a tray icon and a small menu to work. The menu shows the list of languages currently installed on your computer with check boxes next to them. Hitting CapsLock now only cycles through the languages that are currently enabled and even saves the active languages between runs.

You can download source and binaries for the new 0.3 version from my Recaps page.

I’d love to know if anybody finds it as useful as I do.

The problem becomes more complex when you need to separate the tokens you find into different kinds, words and numbers, for instance. We’ll use a well known lyric as our sample text:

The first thing we need to do is build a regular expression that recognizes words and another one that recognizes numbers. Although there are shorter ways to build those regular expressions, I like the less obscure form:

We could now use findall on the string and get all the numbers and words out of it.

But wait, you say, that isn’t what we wanted at all! We need to get the tokens in the order of their appearance in text and still get the type of each token. Something along the lines of

would be really nice.

In order to do that, we’ll need to combine both regular expressions into one and iterate on the result of findall examining each token to decide on its type.

As you can see, the result of the call to findall is a list of tuples, each containing a single match. If you look closely at the way I’ve combined the two regular expressions, you’ll see that each part is surrounded with parenthesis and that there’s a pipe (|) between the expressions. The compound regular expression matches either a number rf a word and each tuple in the return value of findall contains the matches for each parenthesized part of the regexp. However, since we combined the parts using a pipe (|), only one of the parts matches each time.

Using that knowledge we can now construct a simple loop that shows the token type for each of the words in the lyric:

We now have most of the knowledge we need to build ourselves a lexer that will take a list of regular expressions and some text and return (or even better, generate) an list of tokens and their types. We’ll need to combine the regular expressions for each token into one big regex using pipes, scan the string, and gather the tokens and their types.

Our usage code looks like this:

And here is the code for the lexer itself:

Some notes on the implementation are in order. I’ve used the little known (?P<name>…) syntax for naming the parenthesized groups of regular expressions. Using that syntax the expression (?P<word>[A-Za-z]) matches a word and that match is accessible with match.group(’word’) where match is a re.Match object.

In order to speed things up a bit, I’ve compiled the regular expression when the Lexer object is created, used the finditer function instead of findall, and made parse a generator instead of a list returning function.

Using this simple lexer implementation it was quite simple to create a Python-to-HTML converter with syntax highlighting that works well enough to highlight the code of the highlighter itself!

The code for the lexer and syntax highlighter example are available here and on my snippets page. You can also see the result of running the syntax highlighter on itself here.

Enjoy lexing and let me know if you found this useful.

I haven’t posted too much here lately and for a good reason. Arik and I have been hard at work to release the first public beta of our test management service called Testuff. Developing software is hard enough when you have plenty of resources but when you are a one- or two-man shop with limited funds it’s even harder. We’ve built Testuff to help small companies and mISV’s like ourselves manage and run their software tests. We’ve based it on the SaaS model so you don’t have to install any servers, but we also made a rich desktop client for it so you could enjoy a better user experience. If you’re doing any sort real development for actual, breathing clients, you should try it out.

</marketing>

It’s been a week since the public release and although we made some marketing efforts (like this post) we’re still not getting enough traffic to our site. Only a few people have actually downloaded and tried to use our application and I think there’s only one name on that list that I don’t know. I realize we should be doing more marketing and getting the word out to as many people as we can but I don’t seem to be able to get past my perfectionism. I’m looking at Testuff now and it is (aside from some bugs and quirks) a fine achievement. It is quite convenient, rather pretty and has some really cool features like recording the video of the application you’re testing so you could reproduce the bugs with ease. However, since I’ve been working on it for so long, I’ve gotten u sed to all the cool things by now and I am already cultivating a new vision in my mind. A cleaner interface, less features, a faster bug video recorder, an ability to email a test to your friends who could run it and report the recording of the bug directly and so on. I’m struggling because I’ve promised my partner I’d write emails to some key figures in the micro ISV world (people like Bob Walsh, Eric Sink, Joel Spolsky, Ian Landsman and Andy Brice). But how can I describe the wonders of Testuff to them when I’m already thinking about the next version and the one after it?

Another thing I’m worried about is the fact that although every developer and QA I’ve talked to was very excited about Testuff, very few have visited the site and tried it out, not to mention started using it on their own team. Price shouldn’t be an obstacle as we’re giving it out for free right now and I don’t think there is a lack of need for a service like this. Something is amiss however and I still haven’t figured out what it is.

I’d love to hear any thoughts you may have on the subject and any advice you might have. You’ll probably need to install Testuff to do that (Ha! Gotcha!) so you’d better head on to the Testuff download page.

Python has only the most basic support for secure SSL and HTTPS and if you know anything about how SSL works, you’ll know that support doesn’t provide enough security. I’ll leave the discussion of SSL, TLS, HTTPS and other related protocols and technologies to people who actually know something about it (any good links I should put here?), but the following quote from the Python documentation should put even the uninitiated on their toes:

class HTTPSConnection(host[, port, key_file, cert_file])

A subclass of HTTPConnection that uses SSL for communication with secure servers. Default port is 443. key_file is the name of a PEM formatted file that contains your private key. cert_file is a PEM formatted certificate chain file.

Warning: This does not do any certificate verification!

The red color is mine, but the warning is there (at least in Python 2.4.4 – I’ve been a bit slow to adopt 2.5 yet, but I don’t think it has changed).

What that means is that although you might think you’re using a secure connection when you’re using HTTPSConnection you really aren’t. At least not as secure as you thought. Although all the data transferred between you and the server will be encrypted, you won’t actually know you’re talking to the right server and wil be vulnerable to the man-in-the-middle attack.

But fear not, because M2Crypto comes to the rescue. M2Crypto is a Python library based on the well known OpenSSL library which does all the right cryptographic magic in all the right ways. M2Crypto has a compatible HTTPSConnection class that should work as a drop-in replacement of the one in httplib and actually authenticate the server correctly.

Now that we’ve got all this unimportant stuff out of the way, lets get our hands a dirty with building the library on Windows.

Tools you’ll need

Here are the programs you’ll need installed before you dig in:

1. Python 2.4 or later – might work with earlier versions, but I haven’t tested it with anything but Python 2.4.4.
2. Microsoft Visual Studio 2003 – this is the version that Python 2.4/2.5 is built with and this is the version you need to build M2Crypto. I don’t think any other (including 2005) will work.
3. ActivePerl 5.8.7 – that’s the version I used, but I guess any reasonable Perl will do.
4. Command prompt – you don’t need to install it, but you’re going to be using it a lot so you’d best be familiar with it.

Building OpenSSL for Windows

The first thing we’ll need to do is build us a fresh OpenSSL DLL.

1. Download the latest OpenSSL source package from http://www.openssl.org/source/.
2. Unzip and untar the package somewhere and open a command prompt there.
3. > perl Configure VC-WIN32 –prefix=c:/openssl
4. > ms\do_masm
5. > nmake -f ms\ntdll.mak
6. > nmake -f ms\ntdll.mak install

If something doesn’t work, refer to the INSTALL.W32 file in  the OpenSSL source package. I followed the intructions there to the letter and they worked.

Building M2Crypto for Windows

M2Crypto uses a tool called SWIG to help write the Python code that wraps the OpenSSL library that is written in C, so we’ll have to download and install it.

Let’s go.

1. Download the latest SWIG Windows binaries from http://www.swig.org/download.html .
2. Unzip and untar the SWIG package to some directory and add that directory to your PATH.
3. Download the latest M2Crypto sources from http://chandlerproject.org/bin/view/Projects/MeTooCrypto.
4. Unzip and untar the M2Crypto source somewhere and open a command prompt there.
5. > python setup.py build_ext –openssl c:/openssl
6. > python setup.py bdist_wininst

That last command will create a nice M2Crypto-0.18.win32-py2.4.exe file in the dist subdirectory which you can run to install M2Crypto in the Python site-packages directory.

To test your build, run python and do import M2Crypto. If you get an error that says ‘ImportError: DLL load failed with error code 182′, it’s because the M2Crypto library can’t find the OpenSSL DLLs. You’ll need to place the libeay32.dll and ssleay32.dll files somewhere python can find them. The directory in which your script resides is a good bet.

But Python is an interpreted language and it’s slow. You don’t feel it until you do something stupid like going through all the pixels in a bitmap and converting them into gray scale. I tried doing that once to create grayed out versions of my button images in a wxPython app I was building. I couldn’t belive how slow that was.  Then again, the slowness might have been due to the calls to the wxWidgets C++ layer and back.

Anyway..

I was completely unaware of the fact that the guys at YouTube, the mega site that serves millions of users daily, use Python. And they don’t just use it, their whole damn server side runs on it.

Here’s a lecture by Cuong Do Cuong, the engineering manager at YouTube. Apparently, he was there from the very start and he tells an interesting story of dealing with exponential growth and handling the scalability issues as they arised.

[Update: The lecture is a bit boring, but the real interesting stuff is during the Q&A at the last 10 minutes or so. You should watch them.]

The amazing thing is that the guys that did all kinds of insanely amazing things to acommodate their growth rate, including hacking the lighttpd source code to improve the way it does multithreading, still use Python for their server side code. According to Coung, it never was the bottleneck and the speed problems they did have were easily solved by throwing in several more servers. The speed of development in Python on the other hand, helped them respond quickly to the changes and implement new ideas almost on the spot.

When I chose Python as the main language for Tuzig about two years ago, I knew very little about it. I knew only that everything I tried thus far (C++, Java and .NET) wasn’t going get us quickly enough to where we were going and that Python looked very promising. I haven’t looked back since. I usually try to choose the best tool for the job, but Python seems to excel in many areas : from small one-off scripts to complex GUI applications (thanks to wxPython) to web applications. I can even write Python stored procedures for PostgreSQL.

And speedwise, if it’s good enough for YouTube, it should be good enough for me.

If you don’t know Python, you should take a look at DiveIntoPython. It’s a great book by Mark Pilgrim that is written for experienced programmers and focuses on the things unique to Python instead of explaining, like many other books, what a for loop is. It’s also freely available on the net so can dive right in.

I wrote a small class that to help me out. This class prints the current call stack every time somebody tries to print something. When the class was ready, it took me just a few seconds to locate the evil print.

The class uses the builtin traceback module’s format_stack function to get the stack info and prints it. One thing I learned while doing that is that a print statement is actually translated to two writes. Don’t know if that means anything, but that’s the reason for the comparison trick in the code.

This is how you’d use the class:

Enjoy.