Comments on: Unicode and permalinks http://www.gooli.org/blog/unicode-and-permalinks/ on software development and related issues Thu, 11 Mar 2010 09:00:39 -0600 http://wordpress.org/?v=2.8.2 hourly 1 By: video encoding http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-12778 video encoding Wed, 22 Jul 2009 23:57:47 +0000 http://www.gooli.org/blog/?p=155#comment-12778 thank you, great post. i never knew it could be done, and this could really help me in my new project. thank you, great post.

i never knew it could be done, and this could really help me in my new project.

]]> By: Lawrence Sheed http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-8054 Lawrence Sheed Tue, 17 Feb 2009 17:47:11 +0000 http://www.gooli.org/blog/?p=155#comment-8054 @Love Encounter Flow. Quote "...no rfc that would govern usage and announcement of encodings in urls" Actually, there is: RFC 3987 Non Ascii (and valid) Characters are assumed to be UTF-8, and should be encoded in percent encoding. This article at the W3C http://www.w3.org/International/articles/idn-and-iri/ talks about this. I'm in the middle of a discussion about URI implementation in an open source CMS at the moment about this (which I how I found this link) Lawrence / Computer Solutions Design China. @Love Encounter Flow.

Quote “…no rfc that would govern usage and announcement of encodings in urls”

Actually, there is: RFC 3987

Non Ascii (and valid) Characters are assumed to be UTF-8, and should be encoded in percent encoding.

This article at the W3C http://www.w3.org/International/articles/idn-and-iri/ talks about this.

I’m in the middle of a discussion about URI implementation in an open source CMS at the moment about this (which I how I found this link)

Lawrence / Computer Solutions Design China.

]]> By: andy mckay http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-6264 andy mckay Mon, 27 Oct 2008 13:12:35 +0000 http://www.gooli.org/blog/?p=155#comment-6264 Perhaps a quick peek at Plone code might help that takes utf-8 and runs it through a decode to form a nice asciid url. Tthere's going to be problems with it but Plone uses this to make urls. Here's some sample code, https://svn.plone.org/svn/plone/CMFPlone/tags/2.5.5/UnicodeNormalizer.py and a sample that does it in JSONP: http://clearwind-labs.appspot.com/ Perhaps a quick peek at Plone code might help that takes utf-8 and runs it through a decode to form a nice asciid url. Tthere’s going to be problems with it but Plone uses this to make urls. Here’s some sample code, https://svn.plone.org/svn/plone/CMFPlone/tags/2.5.5/UnicodeNormalizer.py and a sample that does it in JSONP: http://clearwind-labs.appspot.com/

]]> By: The Cave » Blog Archive » Unicode Permalinks http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-5731 The Cave » Blog Archive » Unicode Permalinks Sat, 27 Sep 2008 17:08:52 +0000 http://www.gooli.org/blog/?p=155#comment-5731 [...] Even better, solid information on uncommon (and poorly understood) Unicode handling in Python. [...] [...] Even better, solid information on uncommon (and poorly understood) Unicode handling in Python. [...]

]]> By: gooli http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-5618 gooli Tue, 23 Sep 2008 12:39:40 +0000 http://www.gooli.org/blog/?p=155#comment-5618 Damn! I didn't know the re module could do that. Damn! I didn’t know the re module could do that.

]]> By: Almad http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-5617 Almad Tue, 23 Sep 2008 12:16:20 +0000 http://www.gooli.org/blog/?p=155#comment-5617 Is it not sufficient to use re.sub("\W", "_", re.UNICODE)? Is it not sufficient to use re.sub(”\W”, “_”, re.UNICODE)?

]]> By: Love Encounter Flow http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-5590 Love Encounter Flow Mon, 22 Sep 2008 12:55:32 +0000 http://www.gooli.org/blog/?p=155#comment-5590 oops, that alternative percent-encoding would be %ua34f and so on, the u being an indicator that four digits are used and the character set referred to is unicode. oops, that alternative percent-encoding would be %ua34f and so on, the u being an indicator that four digits are used and the character set referred to is unicode.

]]> By: Love Encounter Flow http://www.gooli.org/blog/unicode-and-permalinks/comment-page-1/#comment-5585 Love Encounter Flow Mon, 22 Sep 2008 11:22:13 +0000 http://www.gooli.org/blog/?p=155#comment-5585 there's even more things to be aware of in regard to unicode in urls: (1) some browsers, including firefox 2, may chose to send *some* urls not in utf-8, but in a legacy encoding such as the system default or plain latin-1. for ffx2, this would appear to be true whenever the characters entered by the user happen to be encodable as latin-1 etc. there is no rfc that would govern usage and announcement of encodings in urls (what a joke), so you have to guess yourself when doing url decoding (i always use a routine that first tries utf-8, then latin-1 or similar as a fallback. web application frameworks surprisingly often fail in doing that for me). (2) there is yet another percent-encoding style using unicode character ids, using four hex digits à la %a34f (see http://en.wikipedia.org/wiki/Percent-encoding#Non-standard_implementations); this has been rejected by the w3c (probably on the grounds that this would definitely make things too easy—which is presumably why the standards bodies adopted http://en.wikipedia.org/wiki/Punycode, by far the weirdest character encoding standard ever released). (3) on the bright side, there is a healthy tendency among browser vendors to show, in the address bar, the intended, not the encoded likeness of the url typed in. the ffx2 locationbar² extension does it, google chrome does it, flock does it (the fine people over at ffx3 have sadly missed the trend so far—although your screenshots seem to indicate otherwise?). this is an important feature to get readable urls for the people of the world. hopefully, with browser vendors paying more attention to this issue, encoding problems will also become less of a burden in the future. there’s even more things to be aware of in regard to unicode in urls: (1) some browsers, including firefox 2, may chose to send *some* urls not in utf-8, but in a legacy encoding such as the system default or plain latin-1. for ffx2, this would appear to be true whenever the characters entered by the user happen to be encodable as latin-1 etc. there is no rfc that would govern usage and announcement of encodings in urls (what a joke), so you have to guess yourself when doing url decoding (i always use a routine that first tries utf-8, then latin-1 or similar as a fallback. web application frameworks surprisingly often fail in doing that for me). (2) there is yet another percent-encoding style using unicode character ids, using four hex digits à la %a34f (see http://en.wikipedia.org/wiki/Percent-encoding#Non-standard_implementations); this has been rejected by the w3c (probably on the grounds that this would definitely make things too easy—which is presumably why the standards bodies adopted http://en.wikipedia.org/wiki/Punycode, by far the weirdest character encoding standard ever released). (3) on the bright side, there is a healthy tendency among browser vendors to show, in the address bar, the intended, not the encoded likeness of the url typed in. the ffx2 locationbar² extension does it, google chrome does it, flock does it (the fine people over at ffx3 have sadly missed the trend so far—although your screenshots seem to indicate otherwise?). this is an important feature to get readable urls for the people of the world. hopefully, with browser vendors paying more attention to this issue, encoding problems will also become less of a burden in the future.

]]>