Comments on: A simple lexer in Python http://www.gooli.org/blog/a-simple-lexer-in-python/ on software development and related issues Tue, 07 Oct 2008 23:29:42 +0000 http://wordpress.org/?v=2.5.1 By: Fırat KÜÇÜK http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-2143 Fırat KÜÇÜK Tue, 22 Jan 2008 21:04:35 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-2143 good design. congratulations. good design. congratulations.

]]> By: gooli http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1813 gooli Mon, 03 Dec 2007 12:45:47 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1813 I am, but apparently it is only suitable for lexing languages that are similar to shell scripts. I am, but apparently it is only suitable for lexing languages that are similar to shell scripts.

]]> By: Norbert Klamann http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1812 Norbert Klamann Mon, 03 Dec 2007 12:08:01 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1812 Are you aware of the 'shlex' module ? Are you aware of the ’shlex’ module ?

]]> By: gooli http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1562 gooli Wed, 24 Oct 2007 21:14:34 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1562 I don't think lexing gets that much more complex, at least as long as you aren't concerned with performance too much. There might be some issues with precedence and you could add support for different start conditions to better support things like C multiline comments. Semantical parsing on the other hand, is a much more complicated matter. That aperiot module looks really cool. Thanks for the link. I don’t think lexing gets that much more complex, at least as long as you aren’t concerned with performance too much. There might be some issues with precedence and you could add support for different start conditions to better support things like C multiline comments. Semantical parsing on the other hand, is a much more complicated matter.

That aperiot module looks really cool. Thanks for the link.

]]> By: Kumar McMillan http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1561 Kumar McMillan Wed, 24 Oct 2007 21:07:52 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1561 unfortunately, lexical parsing doesn't remain "simple" for very long! Last year at Pycon I saw an excellent presentation on aperiot, an easy to use lexical parsing module: http://moncs.cs.mcgill.ca/people/eposse/projects/aperiot/atglance.dtml I've never had a good reason to try it out though. unfortunately, lexical parsing doesn’t remain “simple” for very long! Last year at Pycon I saw an excellent presentation on aperiot, an easy to use lexical parsing module:
http://moncs.cs.mcgill.ca/people/eposse/projects/aperiot/atglance.dtml
I’ve never had a good reason to try it out though.

]]> By: gooli http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1549 gooli Mon, 22 Oct 2007 07:19:52 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1549 @James I did look at the tokenize module, but it can only tokenize Python code. That's great it all you need to do is syntax highlight Python code. But it's not a generic syntax highlighter. The implementation seems to be similar to what I've done although much more complete of course. @James
I did look at the tokenize module, but it can only tokenize Python code. That’s great it all you need to do is syntax highlight Python code. But it’s not a generic syntax highlighter. The implementation seems to be similar to what I’ve done although much more complete of course.

]]> By: kib2 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1547 kib2 Sun, 21 Oct 2007 20:22:11 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1547 Thanks a lot for this great article, I've found it very clear and usefull. Thanks a lot for this great article, I’ve found it very clear and usefull.

]]> By: James Thiele http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1546 James Thiele Sun, 21 Oct 2007 17:55:48 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1546 Have you looked at the tokenize module in the standard Python library? Have you looked at the tokenize module in the standard Python library?

]]> By: gooli http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1543 gooli Sun, 21 Oct 2007 07:14:23 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1543 The scanner interface in the sre module is absolutely cool! It's similar to my own implementation but is more efficient using the re.Match's lastindex property. Thanks for the info. The scanner interface in the sre module is absolutely cool! It’s similar to my own implementation but is more efficient using the re.Match’s lastindex property.

Thanks for the info.

]]> By: Andrew Dalke http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1541 Andrew Dalke Sat, 20 Oct 2007 23:48:39 +0000 http://www.gooli.org/blog/a-simple-lexer-in-python/#comment-1541 You might also be interested in the undocumented-and-not-official "scanner" interface in the re module. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/457664 The PLY parser uses this sort of approach, but it's hard to find the details because of the extra work it does to do better error reporting. It also works around limitations in re, like not supporting over 100 tokens, and checking that each regular expression is valid before doing the "|" merge. Ie, consider 'a(b' and 'c)d' as the two input patterns. Separate they are not valid, but together they are, along with an extra '()'. You might also be interested in the undocumented-and-not-official “scanner” interface in the re module. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/457664

The PLY parser uses this sort of approach, but it’s hard to find the details because of the extra work it does to do better error reporting. It also works around limitations in re, like not supporting over 100 tokens, and checking that each regular expression is valid before doing the “|” merge. Ie, consider ‘a(b’ and ‘c)d’ as the two input patterns. Separate they are not valid, but together they are, along with an extra ‘()’.

]]>