The integration allows for an automated testing script to report the results of its run to the Testuff server. In order for the results to be grouped, displayed and summarized correctly, the automation script needs to tell the server which test it ran, and whether the test has passed or failed. A long discussion emerged on what the best way to uniquely identify tests.

After quite a bit of back and forth, we’ve settled on permalinks, those more-or-less-readable URLs that are in common use in blogs. The idea of a permalink is to take the title (of a blog post or a test) and replace any characters that aren’t numbers or letters with an underscore or a hyphen. Using this simple scheme, “Unicode and permalinks” becomes “unicode-and-permalinks”, which is quite suitable for use in a URL.

The implementation is a simple regular expression:

While this code works perfectly for the English language, it doesn’t work at all if string is a Unicode string containing something in Hebrew, Russian or Polish – language that some of our customers use. And so, I set out to write code that will essentially behave like the regular expression above, but will work for letters and numbers in all the languages of the world.

Fortunately the Unicode standard includes a rarely used classification of characters into various categories. For each given character we can find out whether it is an uppercase letter, a lowercase letter, and number, a punctuation mark and so on. Surprisingly, Python includes a module called unicodedata that contains all that information. The function category accepts a character and returns a string that tells us what the character is: “Lu” denotes an uppercase letter, “Nd” denotes a decimal digit, etc.

All that remains to be done is go over the characters in the title, keep the letters and numbers, and replace all the other characters with a dash or an underscore. The regular expression at the end replaces any sequence of underscores into a single underscore to make the resulting URLs even nicer to look at.

[Update] Or, as Almad correctly pointed out, you could just use the re module support for Unicode and be done with it in two lines, which kind of takes the air out of this post.

There’s one other thing to consider when dealing with Unicode permalinks. If you’re a native speaker of a language other than English, you’ve probably seen URLs that in your own language in Wikipedia.

From the looks of it, URLs can include characters in any language. Right?

Wrong.

RFC3986 defines the syntax for URLs (actually URIs, but that’s a moot point) explicitly and states which characters are allowed in a URL. This includes little more than English letters and numbers from the lower half of the ASCII chart.

If you look at the headers your browser passes when you access such a URL, you’ll see that it encodes all the characters with percent encoding, so neither the browser nor the web server is violating the standard. This is what the server saw when I navigated to the main Hebrew page of Wikipedia:

GET /wiki/%D7%A2%D7%9E%D7%95%D7%93_%D7%A8%D7%90%D7%A9%D7%99 HTTP/1.1
Host: he.wikipedia.org

In order to understand what this percent encoding means, you need to know a bit about Unicode. Basically, the Unicode URL is encoded in UTF8 and each byte of the UTF8-encoded string is encoded using percent encoding. The browser apparently recognized this specific encoding scheme (which isn’t documented anywhere I could fine) and displays nice internationalized URLs for the user.

If you want to support such URLs in your server, you’ll probably need to write some code to translate the percent-encoded URLs into their actual Unicode representation.

<?xml version="1.0" encoding="UTF-8"?>
<update-info version="0.8.0[1212]">
    <update file="TestuffSetup.exe" from-version="all"/>
    <update file="TestuffUpdate.exe" from-version="0.7.1[1110]"/>
    <md5hashes>
        <file md5="3a23dd6eff6fd6c1d0fbfcbfb0d57221" path="async.pyc"/>
        <file md5="0d1ea490a18c65cec7ba8715b5ea9e69" path="atexit.pyc"/>
        <file md5="166723a4330a98b573119326fc689322" path="base64.pyc"/>
        <file md5="01c1bda049936de570ed922424c057a8" path="BeautifulSoup.pyc"/>
    </md5hashes>
</update-info>

When the Testuff client launches, it gets the version.xml file from the server and compares its version to the version attribute of the update-info tag. If the client’s version is wrong, it checks the update tags to see which update it should download and install. We generate two separate setup files – one to update the most recent version to the new one called and another to update all the other (older) versions.

Aside from the info about which version of the client should use which update file, version.xml also contains the MD5 hashes for each file in the distribution. That might seem like a lot of wasted space and time, but it’s actually there for a very good reason. When our setup building script is creating TestuffUpdate.exe, it too downloads version.xml from our server. It then tries to determine which files have changed or have been added since the last version by comparing the MD5 hashes in version.xml to the the hashes of the actual files that have been generated by the build. Any file that is different is added to the update so we can be sure we haven’t missed any essential component in the update.

Recently I discovered that our update files are much larger than they should be. We release a new version with just a couple of fixes in a single module, and the size of the update is half the size of the full install. As it turned out, that most of the PYC files were marked as changed and added to the update. That didn’t seem right, especially for things like threading.pyc, which is a Python module that shouldn’t change unless you upgrade to a different version of Python, which we didn’t (still stuck at 2.4.4 I’m afraid). That got me curious enough to go digging in the, apparently undocumented, binary structure of the PYC files.

This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).

The first thing I did was compare the two threading.pyc files – the one from the current distribution and the one just generated by the build script. The result showed there was difference in only two bytes:

D:GooliDevTempcompare>fc /b threading-old.pyc threading-new.pyc
Comparing files threading-old.pyc and threading-new.pyc
00000004: 6E CE
00000005: F7 6A

Only two bytes differ, and they are right at the beginning of the file? That looks suspiciously like a version or a timestamp in the file header. Since the PYC file structure is undocumented, I went looking for the details in Python’s source code, but the answer was actually closer to home – in the compiler package. A file called pycodegen.py in Python\Lib\compiler contains the following code:

So, the PYC header file contains a magic number that identifies the Python release and the modification time of the original source file as the number of seconds since the epoch. That shouldn’t be a problem – the threading module hasn’t changed and should have the same timestamp. But as we’ve seen, the PYC files were different. How can that be?

Acting on a hunch, I wrote a short script to read the header from the PYC file and print the embedded date:

Which printed the following results:

Mon Mar 13 22:51:26 2006
Mon Mar 13 12:51:26 2006

Notice anything odd about them? They are exactly 10 hours apart. At first I thought I might actually be looking at two different versions of threading.py, but the chances of two edits being exactly 10 hours apart right down to the second is practically non-existent. It had to be something with time zones. I live and work in Israel, which is at GMT+2:00. The default timezone for Windows is Pacific time, which is GMT-8:00. Exactly 10 hours apart. However, no matter how I tweak the Regional Settings on my computer, all the PYC files I generate here have the same timestamp. Perhaps it has to do with the timezone you have set when you install Python. If I ever find out, I’ll let you know.

But that wasn’t the point of this post. The point was to figure out what PYC files look inside and we did that, at least in part – they start with a magic number that is different for each Python version (check out the comments in import.c), and they have an embedded timestamp of the source code they got generated from after that. The rest is generated by the marshal module and can be read by it to get the code objects and the global data in the module.

Another thing to be learned from this is that we really should always build the Testuff client on the same machine, which is why I’m heading to the office right now to burn a copy of the VMWare image I created with everything needed to build Testuff. We got a new version with a couple of important fixes to our Mantis support to release today.

I’m going to talk about some of the new things we did for this version in the rest of this post, but if you just want to take a look at the new Testuff, just grab it from our site.

Eating Bonzo for breakfast
They say eating your own dog food is important. Although I personally prefer a good steak, you can’t argue with some of the best minds of the software industry. Following the canine oriented software practice we finally sat down to create some tests for Testuff in, well, Testuff. And boy, was it awkward to write tests with our test editor. So we wrote a new test editor, which makes typing tests easy. Just separate the steps with a blank line and the editor will do the rest. You can also copy-paste tests into the new editor from almost anywhere – Word, Excel, the web, etc.

We helped manually convert a bunch of tests for some of our clients who had a few hundred test cases written down in Excel. Took about half an hour to get it done including reorganizing the tests making sure there were no duplicates.

Integrate early, integrate often
It was kind of ridiculous that the Testuff front page said we supported integration with various bug trackers and when you scrolled down to the list, you saw we actually only supported Trac. We started out with Trac as we use it ourselves, but we now added support for two more popular bug trackers. We’re are going to add support for even more bug trackers, as that seems to be what our users want most.

Bugzilla is probably the oldest guy on the block. With almost 10 years of experience under their belts, the guys there know how to manage defects and have seen it all. A interesting recent post seems to indicate Bugzilla may be dying along side Perl, but it’s still widely used and loved by many however ugly I may think it’s UI is.

Many software developers know the name of Joel Spolsky. I was first introduced to his excellent articles when I was a young officer in the Israeli army trying to lead a team of soldiers-developers through the mud of an enormous C++/MFC HR planning system. Some of the articles were amusing, some interesting and a few were a real eye-opener to me. His company, FogCreek, makes a bug tracking and project management software called FogBugz. Personally, I prefer the more Spartan approach of Trac and like to tinker with it’s Python code base, but FogBugz has many followers and it is now offered as an on-demand service, which makes it a great match with Testuff.

Those damn proxies
Being a desktop application with a web-based backend, the Testuff client needs to access our server in order to work. It turned out that accessing the internet through a proxy is a challenging task, which Python doesn’t help with nearly enough. I ended up wrapping the WinHTTP library with ctypes code and using it to connect to the internet using the settings in Internet Explorer. That seems to be working with all our clients so far, but it’s not a portable solution. I’ll be taking a look at pycurl next, to see if it works as well as promised on their website. I’ll post a full report on the proxy fighting process sometime soon.

Oh, yeah, and you can finally change the password you use to connect to Testuff. That doesn’t have anything to do with the proxy support, but this stupid omission we made in previous versions doesn’t justify it’s own heading, right?

What’s next?
Right now we’re hard at work on our next version, 0.8. There will be support for even more bug trackers that our customers asked for (Mantis, Elementool, Jira and maybe another one or two if we have the time), and, if I can fix that damn segfault, a Linux version of the Testuff client.

You can download Testuff from www.testuff.com/download.

I finally sat down today to write those Testuff emails I talked about and just as I was getting into the mood of doing that I spotted this post. Apparently Ian keeps track of whoever mentions his name and had quite a few things to say about our offer and our site. Thanks Ian!

Most of the comments about Testuff are dead on and we’re defeinitely going to address them, both on the website and in our application. Following are a couple of items I want to elaborate on.

Ian mentioned that it is unclear why he needed to download something:

It’s also a bit confusing which parts are online and why I’m downloading something. Clearing that up a bit would be useful.

We’re going to change the site to convey it better, but I do want to answer it right here for those who might have the same question. Testuff is a hybrid application with a rich GUI front end and a web-based backend. That means you have to download the client application to use it, but everything is stored online and can be shared between several people. That is similar to how services like iTunes and Chandler work.

Ian also said that we need to state clearly that Testuff integrated with existing bug trackers:

Your site makes it appear that the bugs are logged with you , though I found one random note that suggests it actually integrates with commercial bug trackers. This is a huge point, nobody wants to log bugs with you. You should prominently display the names of the bug trackers you support all over the place. That way when I come to your site I can see my bug trackers name and know you support it right away and that this improves my existing bug tracker not replaces it.

Yes, we are going to integrate with existing bug trackers, but we haven’t done that yet. That’s why there’s only a random note about it on the site and it’s not in H1 on the main page. We’re hard at work on Trac integration with Bugzilla and Fogbugz on our feature list for the coming weeks. The selling point of actually improving your existing bug tracker is a great spin. After all, everybody uses a bug tracker these days (even if it is a simple excel sheet) and having the video records of the bugs in it could be huge!

Our original concept for building Testuff was to create something akin to TestDirector, but lighter, simpler and more useful for small companies. Using what we know about testing and QA we built a tool you could manage your testing process with – create tests, run them, record the results, and see reports about the quality of your product. That feature list seems to strike a note with the larger companies that already have a team of testers in place who are looking for tools to imrpove their processes. Smaller companies and mISV on the other hand, which we’re eager to please, seem to have less interest in test managament and are more excited by a better way to reproduce bugs.

Testuff is a young service and is a work in progress. I am very eager to hear more comments and thoughts on the subject, especially the negative ones as you learn the most from those. I promise to address each and every one.

I haven’t posted too much here lately and for a good reason. Arik and I have been hard at work to release the first public beta of our test management service called Testuff. Developing software is hard enough when you have plenty of resources but when you are a one- or two-man shop with limited funds it’s even harder. We’ve built Testuff to help small companies and mISV’s like ourselves manage and run their software tests. We’ve based it on the SaaS model so you don’t have to install any servers, but we also made a rich desktop client for it so you could enjoy a better user experience. If you’re doing any sort real development for actual, breathing clients, you should try it out.

</marketing>

It’s been a week since the public release and although we made some marketing efforts (like this post) we’re still not getting enough traffic to our site. Only a few people have actually downloaded and tried to use our application and I think there’s only one name on that list that I don’t know. I realize we should be doing more marketing and getting the word out to as many people as we can but I don’t seem to be able to get past my perfectionism. I’m looking at Testuff now and it is (aside from some bugs and quirks) a fine achievement. It is quite convenient, rather pretty and has some really cool features like recording the video of the application you’re testing so you could reproduce the bugs with ease. However, since I’ve been working on it for so long, I’ve gotten u sed to all the cool things by now and I am already cultivating a new vision in my mind. A cleaner interface, less features, a faster bug video recorder, an ability to email a test to your friends who could run it and report the recording of the bug directly and so on. I’m struggling because I’ve promised my partner I’d write emails to some key figures in the micro ISV world (people like Bob Walsh, Eric Sink, Joel Spolsky, Ian Landsman and Andy Brice). But how can I describe the wonders of Testuff to them when I’m already thinking about the next version and the one after it?

Another thing I’m worried about is the fact that although every developer and QA I’ve talked to was very excited about Testuff, very few have visited the site and tried it out, not to mention started using it on their own team. Price shouldn’t be an obstacle as we’re giving it out for free right now and I don’t think there is a lack of need for a service like this. Something is amiss however and I still haven’t figured out what it is.

I’d love to hear any thoughts you may have on the subject and any advice you might have. You’ll probably need to install Testuff to do that (Ha! Gotcha!) so you’d better head on to the Testuff download page.