Python 2.4: Support is already built-in in Gvim 7.2

Python 2.5: Install Gvim 7.2 and replace vim.exe and gvim.exe with the files in gvim72python25.zip

Python 2.6: Install Gvim 7.2 and replace vim.exe and gvim.exe with the files in gvim72python26.zip

Python 3.1: I wasn’t able to build Gvim 7.2 with Python 3.1 (most probably due to the backwards incompatible changes in Python 3)

To determine what version of Python your Gvim installation supports use the following command in Gvim:

python import sys; print sys.version

I’ve been a  long time proponent of digital reading. I started out with a (slightly darker) gray on (a slightly Plighter) gray display of an old  Palm III. I read the entire Hitchhiker’s Guide to the Galaxy series (which I incidentally didn’t like very much) on that device straining my eyes to see the badly drawn fonts. When the old PDA broke, I moved on to a Palm Tungsten T3 and have been reading books on it since using the wonderful Plucker software. I read about a hundred books over the last 4 years, almost all of which were in digital format on and I enjoyed every minute of it. Most of the time when I’m reading I forget I’m holding a digital device in my hands and not a real book, except that I find it’s much more convenient to hold the PDA. The first thing I noticed about digital reading is that I could hold the device and turn pages using just one hand. I was taking the train to work at the time, which was so crammed I had to stand most of the way. I found that I could stand on the train squeezed between odd smelling passengers, hold on to a pole with one hand and hold the PDA in the other comfortably reading and turning pages with my thumb. Paper backs on the other hand, I can’t even figure out how to hold properly with both hands. If you open a paperback all the way, you ruin the book with an ugly crease on the back of the cover. If you don’t, you have to awkwardly hold it half open sticking your thumb in the middle and struggle to read the inner part of the pages. Perhaps I’m overly modern in my approach, but I fail to see the sentiment in holding, owning and reading paperbacks printed on the cheapest recycled grayish paper with the cheapest possible ink in a font that’s either too small or too large, but is never the right size. In modern terms, I’d say the user interface of the common paperback leaves much to be desired. Note, I’m not talking about a rare edition of a classic whose pages were manually arranged with love and caring. I’m talking about those pulp fiction books we all read and forget about a day later which are printed at a factory somewhere in China and shipped in containers for us to read. There’s no sentimental value in most of those books either in form or content and we don’t read most of them more than once.

I believe the psychological aspects of holding a book, smelling it (ew!) and turning its pages will last at most a single generation once a proper technology is available. One of the things I like best about reading from a PDA is the immediate access to more information about what I’m reading. Even when I read something as simple as a Vince Flynn thriller I sometimes use a dictionary or turn to Wikipedia for some more info on a term, a place or a person. This is one thing Amazon got right with their Kindle – it’s always online and you have full access to Wikipedia, a dictionary and other internet resources. The experience of reading a book is so much richer when you can instantly find out more about what you are reading. This is even more important for scientific or professional books. The ability to directly jump to a cited source and read the original material the author based her claims on is quite amazing as I imagine you know from reading stuff on the web. The feeling is only enhanced by the ability to do that while in bed or sprawled on a sofa.

A final point that came to me while writing this is that the digitization of books can help books evolve into a different non-linear media. The result might be similar to the hypertext we see on the web, but perhaps more akin to the Dungeons and Dragons books that were common in the early nineties. In those books you could choose the storyline by jumping to some page in the book according to the author’s instructions. Perhaps the Interactive Fiction style of games will return with a vengeance and allow us to be more active participants in the books we read.

I don’t believe paper books will disappear and be completely replaced by digital books. After all, the TV hasn’t completely replaced the radio and CDs and MP3 players haven’t replaced vinyl records. However, when new technology emerges, the old ones usually mutate and find the niche they serve best. After all, people watch TV at home but still prefer the radio when they are driving. They listen to old jazz records on vinyl, but take their MP3 player when they jog (which might be a bit inconvenient carrying a record player). In a similar manner, paper books and digital books will coexist each finding its niche. Meanwhile, I’ll be ordering myself a brand new Sony PRS-700 and hoping that somebody ports a dictionary program and a Wikipedia browser to it so I’ll be able to enjoy a truly enhanced digital reading experience.

The integration allows for an automated testing script to report the results of its run to the Testuff server. In order for the results to be grouped, displayed and summarized correctly, the automation script needs to tell the server which test it ran, and whether the test has passed or failed. A long discussion emerged on what the best way to uniquely identify tests.

After quite a bit of back and forth, we’ve settled on permalinks, those more-or-less-readable URLs that are in common use in blogs. The idea of a permalink is to take the title (of a blog post or a test) and replace any characters that aren’t numbers or letters with an underscore or a hyphen. Using this simple scheme, “Unicode and permalinks” becomes “unicode-and-permalinks”, which is quite suitable for use in a URL.

The implementation is a simple regular expression:

While this code works perfectly for the English language, it doesn’t work at all if string is a Unicode string containing something in Hebrew, Russian or Polish – language that some of our customers use. And so, I set out to write code that will essentially behave like the regular expression above, but will work for letters and numbers in all the languages of the world.

Fortunately the Unicode standard includes a rarely used classification of characters into various categories. For each given character we can find out whether it is an uppercase letter, a lowercase letter, and number, a punctuation mark and so on. Surprisingly, Python includes a module called unicodedata that contains all that information. The function category accepts a character and returns a string that tells us what the character is: “Lu” denotes an uppercase letter, “Nd” denotes a decimal digit, etc.

All that remains to be done is go over the characters in the title, keep the letters and numbers, and replace all the other characters with a dash or an underscore. The regular expression at the end replaces any sequence of underscores into a single underscore to make the resulting URLs even nicer to look at.

[Update] Or, as Almad correctly pointed out, you could just use the re module support for Unicode and be done with it in two lines, which kind of takes the air out of this post.

There’s one other thing to consider when dealing with Unicode permalinks. If you’re a native speaker of a language other than English, you’ve probably seen URLs that in your own language in Wikipedia.

From the looks of it, URLs can include characters in any language. Right?

Wrong.

RFC3986 defines the syntax for URLs (actually URIs, but that’s a moot point) explicitly and states which characters are allowed in a URL. This includes little more than English letters and numbers from the lower half of the ASCII chart.

If you look at the headers your browser passes when you access such a URL, you’ll see that it encodes all the characters with percent encoding, so neither the browser nor the web server is violating the standard. This is what the server saw when I navigated to the main Hebrew page of Wikipedia:

GET /wiki/%D7%A2%D7%9E%D7%95%D7%93_%D7%A8%D7%90%D7%A9%D7%99 HTTP/1.1
Host: he.wikipedia.org

In order to understand what this percent encoding means, you need to know a bit about Unicode. Basically, the Unicode URL is encoded in UTF8 and each byte of the UTF8-encoded string is encoded using percent encoding. The browser apparently recognized this specific encoding scheme (which isn’t documented anywhere I could fine) and displays nice internationalized URLs for the user.

If you want to support such URLs in your server, you’ll probably need to write some code to translate the percent-encoded URLs into their actual Unicode representation.

To check the firmware version, go to http://192.168.2.1 (that’s the IP of the router in its default configuration) and click Status Info at the top of the page that appears. The firmware version is written under Runtime Code Version.

The 1.57 version apparently has a bug, which did not exist in previous versions and has been fixed in newer ones. The latest official firmware release is 1.51, and can be found at the Edimax site. I’ve also found a 1.58 version on an Israeli’s dealer’s site here.

Direct link to 1.51 (edimax.com): BR6204Wg_1.51.zip (unzip it before upgrading)
Direct link to 1.58 (pikok.co.il): EdiEngBR6204Wg_1.58.bin

I’ve tested both 1.51 and 1.58, and they both fix the problem.

Upgrading the firmware is actually quite easy.

Go to http://192.168.2.1, select System Tools at the top, then Firmware Upgrade on the left and follow the instructions on the screen.

Hope this helps somebody someday.

That is so annoying!

It took me over 20 minutes of Googling to find how to disable that feature, so I’m documenting it here.

The option that controls it in Firefox is Tools > Options > Advanced > General > Browsing > Use autoscrolling.

Apparently “auto scrolling” is the name of this obscure feature, which is now forever disabled on all my machines.

This is how the Python repository looks like via http://svn.python.org/projects/python:

It says Revision 65620 at the top and I’ve always wondered if you could access another revision in the same simple way. Turns out there is a way.

All you need to do is add !svn/bc/REVISION to the URL:

http://svn.python.org/projects/!svn/bc/5000/python shows revision 5000 of the Python Subversion repository.

Well, no more! Recaps, that tiny utility that lets you switch your keyboard layout using the almost forgotten Capslock key, can now fix text you’ve typed with the wrong language selected. Just hit Ctrl-Capslock as soon as you discover your mistake and the text will be replaced with what you actually intended to type. I’ve only tested it with Hebrew and Russian, the languages I personally use on a regular basis, but it should work with any language supported by Windows. Let me know if it doesn’t.

<?xml version="1.0" encoding="UTF-8"?>
<update-info version="0.8.0[1212]">
    <update file="TestuffSetup.exe" from-version="all"/>
    <update file="TestuffUpdate.exe" from-version="0.7.1[1110]"/>
    <md5hashes>
        <file md5="3a23dd6eff6fd6c1d0fbfcbfb0d57221" path="async.pyc"/>
        <file md5="0d1ea490a18c65cec7ba8715b5ea9e69" path="atexit.pyc"/>
        <file md5="166723a4330a98b573119326fc689322" path="base64.pyc"/>
        <file md5="01c1bda049936de570ed922424c057a8" path="BeautifulSoup.pyc"/>
    </md5hashes>
</update-info>

When the Testuff client launches, it gets the version.xml file from the server and compares its version to the version attribute of the update-info tag. If the client’s version is wrong, it checks the update tags to see which update it should download and install. We generate two separate setup files – one to update the most recent version to the new one called and another to update all the other (older) versions.

Aside from the info about which version of the client should use which update file, version.xml also contains the MD5 hashes for each file in the distribution. That might seem like a lot of wasted space and time, but it’s actually there for a very good reason. When our setup building script is creating TestuffUpdate.exe, it too downloads version.xml from our server. It then tries to determine which files have changed or have been added since the last version by comparing the MD5 hashes in version.xml to the the hashes of the actual files that have been generated by the build. Any file that is different is added to the update so we can be sure we haven’t missed any essential component in the update.

Recently I discovered that our update files are much larger than they should be. We release a new version with just a couple of fixes in a single module, and the size of the update is half the size of the full install. As it turned out, that most of the PYC files were marked as changed and added to the update. That didn’t seem right, especially for things like threading.pyc, which is a Python module that shouldn’t change unless you upgrade to a different version of Python, which we didn’t (still stuck at 2.4.4 I’m afraid). That got me curious enough to go digging in the, apparently undocumented, binary structure of the PYC files.

This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).

The first thing I did was compare the two threading.pyc files – the one from the current distribution and the one just generated by the build script. The result showed there was difference in only two bytes:

D:GooliDevTempcompare>fc /b threading-old.pyc threading-new.pyc
Comparing files threading-old.pyc and threading-new.pyc
00000004: 6E CE
00000005: F7 6A

Only two bytes differ, and they are right at the beginning of the file? That looks suspiciously like a version or a timestamp in the file header. Since the PYC file structure is undocumented, I went looking for the details in Python’s source code, but the answer was actually closer to home – in the compiler package. A file called pycodegen.py in Python\Lib\compiler contains the following code:

So, the PYC header file contains a magic number that identifies the Python release and the modification time of the original source file as the number of seconds since the epoch. That shouldn’t be a problem – the threading module hasn’t changed and should have the same timestamp. But as we’ve seen, the PYC files were different. How can that be?

Acting on a hunch, I wrote a short script to read the header from the PYC file and print the embedded date:

Which printed the following results:

Mon Mar 13 22:51:26 2006
Mon Mar 13 12:51:26 2006

Notice anything odd about them? They are exactly 10 hours apart. At first I thought I might actually be looking at two different versions of threading.py, but the chances of two edits being exactly 10 hours apart right down to the second is practically non-existent. It had to be something with time zones. I live and work in Israel, which is at GMT+2:00. The default timezone for Windows is Pacific time, which is GMT-8:00. Exactly 10 hours apart. However, no matter how I tweak the Regional Settings on my computer, all the PYC files I generate here have the same timestamp. Perhaps it has to do with the timezone you have set when you install Python. If I ever find out, I’ll let you know.

But that wasn’t the point of this post. The point was to figure out what PYC files look inside and we did that, at least in part – they start with a magic number that is different for each Python version (check out the comments in import.c), and they have an embedded timestamp of the source code they got generated from after that. The rest is generated by the marshal module and can be read by it to get the code objects and the global data in the module.

Another thing to be learned from this is that we really should always build the Testuff client on the same machine, which is why I’m heading to the office right now to burn a copy of the VMWare image I created with everything needed to build Testuff. We got a new version with a couple of important fixes to our Mantis support to release today.

The Run dialog is great to quickly access applications you use, but it’s only usable for programs that put their folder in the PATH environment variable, which not all programs do. I’ve been adding programs I liked to the PATH for years when a friend of mine showed me a neat trick. You create a folder called Shortcuts somewhere (say C:\Shortcuts) and put it at the start of your PATH environment variable and then create shortcuts to your favorite programs in that folder. All those shortcuts are now available to you via using Win+R. A nice bonus is the fact that you can name the shortcuts any way you like. I usually have a www.lnk file pointing to Firefox, note.lnk pointing to Notepad and word.lnk pointing to Word.

There’s only one thing missing here to make this trick into an fully featured program launcher. We need a way to quickly create shortcuts to programs, preferably by right-clicking on them in Windows Explorer. You can do that without installing anything using a couple of obscure Windows features – the Send To menu in the right-click menu of each file and the fact that Windows can run *.vbs files with VBScript code in it using something called WSH.

Here’s the source of a script that creates a link to whichever was passed to it in the command line (executable, folder, etc.).

The Send To menu is actually a folder located at C:\Documents and Settings\<YourUser>\SendTo. This folder contains links to various programs and you can add shortcuts to your own programs there. When you right-click a file, and choose one of the Send To items, the corresponding program is called, and given the name of the file you right-clicked as a command line parameter. So, if you put a link to the script above in the Send To folder, you’ll be able to easily create links in the C:\Shortcuts folder.

In case you lost me somewhere along the way, and in case I ever need to set this up myself again, here are the instructions to set things up:

1. Create C:\Shortcuts
2. Create C:\Shortcuts\create_shortcut.vbs and paste the above code into it.
3. Add C:\Shortcuts to the start of your PATH environment variable (Win+Pause > Advanced > Environment Variables).
4. Hit Win+R, type sendto, hit Enter. That should get you to your Send To folder.
5. Create a shortcut here to C:\Shortcuts\create_shortcut.vbs and call it Shortcuts.

To add a program to the launcher:

1. Right click a program or a folder (you can do it even on programs in your Start Menu).
2. Send To > Shortcuts.
3. In the dialog change the shortcut name to something short and catchy.
4. Test it : Hit Win+R, type the shortcut name and hit Enter.

You have to navigate to about:config, type bidi in the filter box, and double click the line that says bidi.browser.ui. That will set its value to true. All you have to do now is restart Firefox and the View->Switch Page Direction and Edit->Switch Text Direction menu items will be added along with the Ctrl-Shift-X shortcut.

Hope this helps some poor soul someday.