This is a huge advance for Windows GUI programming with WxPython. Now WxPython applications on Windows can host an IE window, and control the contents programmatically via the document property.

The WxPython team has been pretty quiet about it. The changes don't seem to have made it into the docs (although they're mentioned in the change log and the sample code has been updated).

Here's a simple example of a dialog box with an HTML window. The dialog intercepts clicks on the links, and uses them to set the color of the text by accessing its css property.

Here's the code (htmlwin.zip).

The method BeforeNavigate2 intercepts clicks on links, and uses the link information to get the desired color. Then it finds the element in the DOM with an id of "text", and sets that element's color to the link color.

The possibilities of this technique are huge. Although it has the disadvantage of tying you to Windows, if your application is going to be Windows-only anyway, it lets you write an application with many of the benefits of a Web app, and very few of the drawbacks.

Here's a screenshot of the above code in action:

Here's how I created a custom function for fuzzy glossary searches, using SQLAlchemy for the ORM, with SQLite as the database engine. Assuming you've got your SessionClass object, create a session, and get a connection object:

Speedup

Just for fun, I compared the speed of (1) using custom functions in SQLite with (2) keeping the records as a Python array, and getting the matches in pure Python using a list comprehension. I found that the SQLAlchemy version is about 8 times faster. In the test, I created a glossary of 44,732 records using random word pairs, and got the fuzzy substrings for a query sentence.

Since the fuzzy-matching code and database code are written in C, the SQLAlchemy version is probably approaching near-C speeds, with the only slowdown being the overhead of calling them from Python (which is pretty minimal; most of the work is done elsewhere).

More importantly, the SQLAlchemy version easily meets my performance target of a 50,000-record search in 0.25 seconds, while the native Python version falls pretty far short.

Also interestingly, I found that psyco didn't speed up either version at all, and in fact made both slightly slower. Another demonstration that you should profile rather than applying psyco as a panacea.

Here's the code used for the test.

I recently created a server application to share Felix memories/glossaries over a local network. After a simple test application, I was confident that CherryPy would serve my needs, so I bought CherryPy Essentials (Sylvain Hellgegouarch) and started hacking.

The story of a website

The book uses a practical-minded, show-me-the-code style that I personally like. The book tells the story of a website, starting with an introduction to CherryPy, then walking us through the creation of a sophisticated Web 2.0 website, adding more elements as our understanding grows.

While I found the style of taking us through the development of a website useful, this approach has an obvious weakness: if you don't use the same components as the author, the book will be less relevant. For example, the author chooses Dejavu for his ORM, while I prefer SQLAlchemy; he chooses Kid for his templating engine, while I prefer Mako; and he chooses MochiKit for his JavaScript library, while I prefer jquery.

While this prevented me from using the book's code whole cloth, it did show me how to build a website the CherryPy way, using CherryPy's fantastic features and integrating them with the other technologies used in a modern Web application. In all I'm quite pleased with the value I got out of the book.

Contents

Chapter 1: Introduction to CherryPy
Chapter 2: Download and Install CherryPy
Chapter 3: Overview of CherryPy
Chapter 4: CherryPy in Depth
Chapter 5: A Photoblog Application
Chapter 6: Web Services
Chapter 7: The Presentation Layer
Chapter 8: Ajax
Chapter 9: Testing
Chapter 10: Deployment

Chapter 1: Introduction to CherryPy

This chapter provides some introduction to CherryPy's history and community. It can be safely skipped by the impatient.

Chapter 2: Download and Install CherryPy

Not too much of interest here either. If you can find the book, I'm sure you can find CherryPy.

Chapter 3: Overview of CherryPy

Here's where things start getting interesting. We start with a simple "shout" style application, then walk through the anatomy of a basic CherryPy application: configuration, static files, URL routing, and page handlers.

The author also provides a list of the built-in library modules, which I found to be fairly useless since he doesn't tell us how to use them or even really much about what they might be good for.

Chapter 4: CherryPy in Depth

True to its title, this chapter talks in depth about the various capabilities and tools provided with CherryPy, including material on extending and hooking the CherryPy engine. If you've got experience developing websites, chapters 3 and 4 are really all you need to get cracking on a cool website with all the fixings.

Chapter 5: A Photoblog Application

For the remainder of the book, we'll be building a photo blogging application, fully Web 2.0 buzzword compliant with web services, JavaScript, and AJAX goodness.

The chapter provides a high-level outline of the application we'll be building, and then discusses some of the various ORMs available for Python.

Chapter 6: Web Services

This cool chapter describes how to build Web services (including RESTful services) using CherryPy. This served as a major inspiration for creating the API to my server, although I decided not to use REST.

Chapter 7: The Presentation Layer

This chapter goes over templating libraries for Python (the only thing more numerous in Python than templating libraries are Web frameworks) and how to integrate them with CherryPy, then settles on Kid as the engine to use for the application.

Chapter 8: Ajax

Although the author uses MochiKit as his JavaScript library, he describes the AJAX communication process at a fairly low level, so it was easy to transfer the concepts over to jquery. After briefly describing JSON as the mechanism for transferring data between JavaScript and Python, the chapter goes on to describe how we'll be using AJAX in our photo blogging application.

Chapter 9: Testing

You've got to give props to a framework book that devotes a chapter to testing. As with other components, although I prefer nose and mechanize for unit/web testing to the author's unittest and webtest, the concepts were what mattered and I had no trouble following along.

I also appreciated the section on performance/load testing using FunkLoad, as well as function testing with Selenium, which I wasn't very familiar with.

Chapter 10: Deployment

This chapter is also pure gold in a framework book. It discusses the various deployment and configuration options, including deploying behind Apache and lighttpd. I found the section on supporting SSL to be particularly helpful, as my next step for my server will be hosting it myself via SSL.

Conclusion

This book is a great practical guide to building websites with CherryPy, using best practices for the framework and getting the most out of the huge array of functionality that it provides.

The Pakt Publishing Ebook

I bought this book as a PDF from Pakt Publishing. I prefer having programming books in electronic format, because I typically read them at my computer while experimenting with the code in the book. And since I live in Japan, the electronic format is also a great way to get my grubby hands on technical books quickly and cheaply.

Pakt ebooks, I found, have an annoying security feature: copy and paste is disabled, thwarting my preferred tactic of copying and pasting code snippets to run, and terms to google. Sure, the code samples are provided, but that adds a lot of hassle to what should be a simple operation.

I wish Pakt would discontinue this practice. It only hurts paying customers, since the pirates will no doubt have little trouble stripping out that protection and repackaging the book in any format they like.

Python is for people who want to program, not REAL WORLD programmers.

(Python encourages a sense of fun, and people on the comp.lang.python group tend to like to have fun taking the piss out of trolls.)

Not that "real world" programmers don't use Python — just that people who place a lot of value on being a "real world" programmer are probably using Java or C# or something. Meanwhile, people use Python because it's a great language and they love programming in it, not because it will make people think they're "real world" programmers.

This goes back to the Python Paradox described by Paul Graham: python programmers are generally good hires, not because Python is (necessarily) a better language than Java or C#, but because

…people don't learn Python because it will get them a job; they learn it because they genuinely like to program and aren't satisfied with the languages they already know.

This is becoming less of a sure thing as Python gains popularity and starts to look good on "real world" resumes, but I think it's still true that people use Python because they like it.

Since the app was working fine without py2exe, I brilliantly deduced that maybe something was getting broken in the packaging process.

It turns out that I need to include "mako.cache" in my packages, as pointed out by this blog post.

So the relevant section of my setup dictionary now looks like this:

Everything's working now — lovely stuff!

Googling on this error failed to produce any hits, so I had to (gasp!) do some actual research to figure this out. Here's hoping that I can save the next poor soul from that horror.

In this post I want to pull that together with code to get a word count from an HTML file.

What needs counting

What needs counting depends to some extent on what you need the word count for, but here I'm going to be assuming that the word count is going to be used to count billable/localizable content.

In that scenario, you've got to count the text in the title tag, as well as the visible text in the body, and certain other localizable content: img alt attributes, a title attributes, and input value attributes (am I missing any?).

The Code

The code for counting the actual text is in the above link. Here we need code to extract the text from the HTML file, and to accumulate the counts for all the chunks we've extracted.

Here's the Segment class for accumulating counts:

Next, the code for extracting (segmenting) the text from an HTML file. For this, you'll need the excellent Beautiful Soup module.

And here's some code to get the actual wordcount:

Here are the docstats and htmlseg modules, and here is an online tool using the code for the HTML word counts.

The code is written in a somewhat functional style. It's short and, dare I say, kind of elegant. But it turns out that this code is quite a bit slower than an imperative style using collections.defaultdict.

Here are the two functions:

Here are the helper functions:

The groupby version is shorter than the defaultdict version, and I'd say that it's simpler and more readable as well. Because it's shorter, the groupby version is less likely to contain bugs. In particular, the defaultdict version has a mutable local variable (used as an accumulator in the for loop), which is a classic source of bugs. The groupby version is also likely to be easier to maintain because it's shorter and simpler.

But the defaultdict version of the function winds up being considerably faster.

The times it took to run these functions 10 times on my computer, retrieving the top 50 most frequent words for "/python25/readme.txt", are as follows (seconds rounded to 4 decimal places).

The defaultdict version is 1.4x faster than the groupby version. This gap grows even further when psyco is used, making the defaultdict version nearly 1.6x as fast. I'd say that most of the reason for the slowness is that the groupby version of the function performs two sorts, compared to one sort in the defaultdict version.

(The psyco speedup for the defaultdict version comes from the for loop; changing get_words to return a generator expression eliminates the speedup. The speedup for the groupby version comes from the freq list comprehension; changing this to a generator expression eliminates its speedup.)

So which one should I use?

It's pretty common for Python code written in a functional style to be slower than equivalent code written in an imperative style. Nevertheless, I tend to prefer the more functional style of programming, switching to a more imperative style (or other forms of optimization) if performance isn't satisfactory.

It is easier to optimize correct code, than correct optimized code.

–Yves Deville

A big question here is how to tell if the functional version is fast enough. My general rule of thumb is that the user would be prepared to wait up to two seconds for a typical "grovel through these files and tell me something interesting" command that's performed infrequently (how frequently do you need to get word frequencies from files?). For a more common action, the wait time should be under a second, with < .5 seconds being optimal (this includes GUI responsiveness but not Web page loading).

Given the times above, and assuming that the user will search no more than 50 files of sizes comparable to Python's README file, then either version of the function is sufficient. If we assume that the user will search up to 100 files, or files substantially larger than the README file, then only the imperative version is acceptable (and we may need to optimize this further if our demands are higher than this).

That's why it's so important to profile and test Python programs from the very beginning. I keep a suite of test cases that I profile with every build (performed at least daily), noting trends in performance and optimizing when the code has gelled and bottlenecks remain.

Here is the test code:

The whole shebang:

Here are some examples from the interactive interpreter.

A list of numbers

And a function that does this for any iterable:

Top 20 most frequent words in a file

Here's a function that will do this.

Edit: Thanks to Kirit's comment below, I added code to check for UTF-32.

Usage:

Discussion: Why check for a BOM?

You might ask, why check for a BOM if chardet already does this? This is because although chardet will correctly detect the BOM, it won't tell you that it found it, so you won't know to chop it off before processing the text. Which means that you'd have to check for a BOM anyway in most cases.

There are several platforms for programming Windows GUI applications in Python. Below I outline a few of them, with a simple "hello world" example for each. Where I've lifted the example from another site, there's a link to the source.

Tkinter

Tkinter is the ubiquitous GUI toolkit for Python. It's cross platform and easy to use, but it looks non-native on just about every platform. There are various add-ons and improvements you can find to improve the look and feel, but the basic problem is that the toolkit implements its own widgets, rather than using the native ones provided on the platform.

Pros

• Most portable GUI toolkit for Python
• Very easy to use, with pythonic API

Cons

• Non-native look and feel out of the box

Hello world example (code source):

wxPython

wxPython is probably the most popular GUI toolkit for Python. It's a wrapper for the wxWidgets C++ toolkit, and as such it betrays a few unpythonic edges (like lumpy case, getters and setters, and funky C++ errors creeping up occasionally). There are a few pythonification efforts on top of wxPython, such as dabo and (the now apparently moribund) wax.

Pros

• Highly cross platform
• Relatively mature and robust
• Uses native Windows widgets for authentic look and feel

Cons

• Must include large wx runtime when packaging with py2exe (adds ~7 MB)
• Cross platform nature makes accessing some native platform features (like ActiveX) difficult to impossible

Hello world example (code source):

.NET with IronPython

IronPython is a .NET implementation of Python. As of 1.0 it has full support for Python 2.4 features, and the 2.0 version will duplicate the Python 2.5 feature set. Although there are many CPython libraries/modules that won't run under IronPython (namely, the ones relying on compiled extensions that have not yet been ported), this lack is partially made up by the huge .NET library.

One cool thing about IronPython is that you can easily create lightweight .exe files that you can ship off to your friends — although you pay for this with a dependency on the .NET runtime, which you can't count on random Windows users to have installed.

Of course, when you go the IronPython route, you take all that comes with it: the good things, like access to .NET libraries and possibly the easiest/cleanest optimization path of any Python implementation (C#); and the bad things, like dependence on the .NET runtime and danger of getting caught on the MS upgrade treadmill.

Another way of getting at the .NET libraries is Python.NET, which adds two files to your Python directory to enable you to call the CLR from CPython.

Pros

• Leverage .NET libraries
• Easily create .exe files

Cons

• Depends on .NET runtime

Hello world example (code source):

PyQT

PyQT is probably the third most widely used GUI toolkit, after wxPython and Tkinter. It has a dual commercial/GPL license (Edit: but it does let you use other open-source licenses; see comments below). I have to admit that this made it a non-starter for me: I don't want to pay for my toolkit when there are others just as good or better that are free; and when I do release open-source software, I want to choose my own license. For others, the GPL might be a non-issue or a plus, so I've left it off my pro/con list.

Pros

• Highly cross platform
• Very easy to use
• Highly mature
• Decent looking widgets

Cons

• Somewhat non-native look and feel (though much better than Tkinter)
• Must include large runtime when packaging with py2exe

Hello world example (from PyQT docs):

Pyglet

Pyglet is kind of the new kid on the block in terms of GUI toolkits, but it sure made a splash. It implements its own windowing system, but with no dependencies other than Python (for Python 2.5 users). You will need OpenGL to do decent 3D graphics, but that's hardly a black mark for pyglet — other libraries would love to make it this easy.

Pros

• High degree of freedom for GUI creation
• Only depends on Python
• Large number of widgets

Cons

• Purposely doesn't duplicate the native platform look and feel
• Although there are a lot of widgets, you'll have to roll your own for many things the platform gives you for free.

Hello world example (slightly modified from code source):

Win32 with ctypes

Of course, all you really need to write GUI applications on Windows with Python is your trusty ctypes module and a well worn copy of Petzold. The benefit of this style is that you're working right down at the system API level, with nothing to get in your way. The disadvantage is that you're working right down at the system API level, with nothing to relieve you from all that boilerplate (unless you write your own abstraction layer on top; see Venster, below…).

Pros

• Enables high level of control
• Straightforward if familiar with Win32 API
• No added complexity or buried functionality due to need to be cross-platform
• Lightest of all Windows GUI programming methods using Python

Cons

• All the complexity and inconsistency of Win32 API in gory detail
• Lack of high-level libraries (have to write more code)

Hello world example (long, ain't it?):