I had a quick go with the W3 Total Cache WordPress plug-in, but it seems rather biased to running Apache (which I’m not) and I experienced some strange errors that I failed to immediately fix. Rather than wrestle with it, or try other WordPress caches, I decided to get to grips with Varnish. This is something I’d been meaning to do for ages, and of course it isn’t limited to WordPress – Varnish is a fabulous caching solution for whatever site you’re building.

The challenge with deploying Varnish to any site is to ensure you cache as much as you can while ensuring dynamic content is fresh when required. With WordPress as an example, the cases are fairly obvious. While logged in to the admin screens you don’t want to cache anything, but the outside world reading your posts should get cached pages whenever possible. In their simplest forms, these cases almost work out of the box with Varnish, but there are a few gotchas. There is also scope to cache more aggressively if you choose to, and most importantly you’ll want to ensure that when content is altered, the changes are visible to everyone immediately.

Cookie gotcha no.1

By default, if you send a cookie to Varnish it assumes quite sensibly that you are in some way expecting dynamic content from the back end. e.g. you may be an admin user, logged in to the site. Of course Varnish can’t actually know what the cookie is for, so when your browser sends a completely redundant Google Analytics UTM cookie, you’ll miss the cache. On this site, that means Varnish would only serve you a maximum of one cached page per visit.

Anonymous session cookies

If a visitor comments on a WordPress site, they are pseudo-logged-in by way of a simple cookie exchange. This means the site can do niceties like pre-populate comment forms for commenting again. These niceties are exactly that, so if you overlook this functionality and block those cookies you can serve more cached pages. In my case, I am happy to make this trade.

The following snippet from my configuration shows how I’ve maximised anonymous user caching while ensuring admin users always get dynamic content

sub vcl_recv {
  # ...
  # admin users always miss the cache
  if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
    return (pass);
  }
  # ignore any other cookies
  unset req.http.Cookie;
  return (lookup);
}

Purging when content changes

HTTP defines a wealth of cache control headers that allow applications and clients to specify what is to be cached and for how long, etc.. but there is no standard way to tell a cache that certain content is to be purged. This is the main challenge of deploying Varnish to a site like WordPress.  A post may be updated, or a user may comment. You’ll want to purge any affected pages from the cache immediately, regardless of their original expiry. Putting that business logic into your VCL configuration would be hugely impractical – it’s really your application’s jurisdiction.

I looked at various purging techniques, and by far the best approach in my opinion is for your application to talk directly to the Varnish administration CLI which listens on the network on a different port to the cache itself. As long as your backend can access your front end by a secure, internal address, you can execute commands on all your Varnish front ends any time you want from your application.

To achieve this, I first had a look at the wp-varnish plug-in which uses the HTTP PURGE method. I don’t like this approach at all; it’s very limited, too closely coupled to the VCL config, and also this plug-in doesn’t purge all pages relating to changes, such as tag and archive indexes.

So, I rolled my own

I started by writing a VarnishAdminSocket class, which talks to Varnish from PHP. This was pretty easy to do, because the varnishadm program uses a very basic text-based protocol. This isn’t designed for WordPress, this is designed for use in any PHP application.

My php-varnish toolkit is on Github. It includes a WordPress plug-in which I am using on this site and you can see the VCL configuration I am using too.

I have a like/hate relationship with WordPress. That is to say that it does a lot, it has a great admin area and there’s a large community producing themes and plug-ins. However, I am a PHP developer of many years, and every time I come into direct contact with the core WordPress code-base I end up being sick in my mouth.

What’s to hate? Well it kind of reminds me of that old BMW advert (or was it Audi?) the one with the swan – On the surface it glides effortlessly, but under the water it’s an ugly son of a bitch, churning away to get the job done.  I had the misfortune of working on a nasty WPMU project last year and found myself looking under the bonnet more than was good for me. Not only is WordPress a horribly written piece of PHP, (and possibly responsible for many a bad word said about PHP itself) it has a legacy of incremental improvements alongside a legacy of community-contributed code that strives to remain as reverse compatible as possible.

If you don’t want to read my rantings, skip to the bit at the end, otherwise … WordPress, let me count the ways a hate thee.

1. Base URL hell

I wanted to get as much of my static assets onto a CDN as I could – a topic I’ll cover in full later. This means serving static files (such as images and css) from a different base URL. It should be simple to upload your theme and plug-in directories to another server and ensure that WordPress points to the correct assets in your HTML. It is not simple. Just look at the hideousness of determining base URLs in WordPress. This simple task should not require a complex plug-in to cover all possible legacy issues. Furthermore, WordPress allows all plug-ins to mess around with base URLs, which means one plug-in might completely undo the work of another.

Oh, by the way – WordPress defines the constant WP_CONTENT_URL before it loads any plug-ins, so the only way to override this is to define it in wp-config.php.

2. Canonical domain ridiculousness

If you access a WordPress site from a domain that is not hard-wired into the site’s database, it will redirect you to the ‘correct’ URL. This is the default behaviour. So, if you want to host the same exact site from a different domain, or different port, (e.g. for debugging a distributed system) you will have to write a WordPress plug-in to override this behaviour. Why does this annoy me?

1. It is not PHP’s job to decide what domain a site lives on – that is the job of the web server. A domain redirect like this should be done at server config level.
2. No system should hard-wire its domain into database-saved preferences. This is nuts, and WordPress does it in multiple places.

3. Magic quotes are the Devil\\\’s work

Magic quotes are so utterly evil that they have been removed from PHP 5.3. They are testament to the amateur reputation of PHP. However, even in earlier versions of PHP, magic quotes could be easily disabled, and anyone who knows what they’re doing with PHP will have been doing so  for years. Furthermore any system written properly would insist on them being disabled. Not so with WordPress. In fact, being able to disable magic quotes poses a problem for WordPress. How can it guarantee to work if you can forcibly disable magic quotes?

Answer: By forcing them even more forcibly. WordPress 3 forces all script input through its own wp_magic_quotes function. Funnily enough this isn’t documented, but it looks like this.

4. mysqli support – or not.

The core WordPress code-base uses the old mysql database driver. When I say old, consider that the mysqli (improved) extension was introduced for versions of MySQL > 4.1.3  in PHP 5, which came out of beta in 2004. WordPress does have a drop-in replacement system though, which allows you to place your own db.php file in wp-content. Here’s the third-party-developed mysqli adapter for WordPress that I am using on this site.

So that’s okay then, right? Well, judging by the required location for the db.php file, this adapter system looks rather like an after-thought. Using the alternative database adapter means calling require_wp_db() rather than just including the original wp-includes/wp-db.php file. Great – except is every plug-in out there doing this? Even WordPress itself isn’t fully patched to support this. At the time of writing, the “famous 5-minute installer” uses the mysql adapter only – I had to patch wp-admin/install.php to be able to install WordPress with only mysqli on my system.

5. Depreciated code handling

Watch out for depreciated function names. For example: WordPress changed wp_specialchars to esc_html. This will result in notices being raised in older themes, which you’ll only see if you enable WP_DEBUG. It’s a good idea to find-and-replace these if you’re upgrading an old theme to WP3. I don’t really hate WordPress for this, but it seems a bit of a pointless change, and isn’t it a bit late in the day for being pedantic about naming conventions? A whim like this just contributes to the amount of legacy in the system and the amount of code devoted to handling depreciation. There’s a 2,500 line file devoted to depreciating functions that are to be removed “later”. Making a clean break with WordPress 3 would have prevented many people from upgrading, so I understand why they do this; I just don’t like the effect it has on the code-base.

I could go on, but I’m hungry. If you want more, see what this guy has to say on the topic.

The bit at the end

All in all, I’m disappointed that WordPress 3 wasn’t taken as an opportunity to improve upon its own legacy. It is fairly clear that popularity with amateurs, ‘ease of use’ and backward compatibility is more important to the team than creating good, robust software that developers will appreciate. Drupal sets a much better example. I’m not saying I don’t have any gripes with Drupal, but they are much happier to take reduced backward compatibility on the chin when they release new versions. This means more incompatible modules, but such is the price of progress.

It’ s great, here’s a pic of the iPhone interface (courtesy of @pepijndevos)

WPTouch iPhone screen grab

 Part 3 – Hacking the library

Code libraries essentially provide abstraction. Abstraction is good. But anyone who regularly uses third party code in their work has at some point hit the knowledge wall; that point where the abstracted nature of the library leaves you helpless in resolving an apparent problem. You have limited choices – Wait for a patch, hit the forums, or hack it yourself. The latter is probably the worst thing to do, but deadlines are deadlines.

To hack or not to hack

I’d like to point out straight away that I don’t condone the hacking of third party code. If you hack a library it is much harder to update if a new version comes out.  The biggest risk is that you may destabilize another part if the system without realizing it. The author knows much more about the code than you, so unless it’s a project you’re deeply involved in it’s probably best to leave it alone. Ultimately if a reputable library is causing you problems, the chances are that either you’re using it wrongly, or you’re using the wrong tool for the job. However, theory and practice don’t always correlate, so bring on the anecdotes …

Smarty

Smarty is almost synonymous with PHP templating engine. Smarty and I go way back and I’ve generally had a good experience, so I decided to use it recently in a bespoke CMS for one of our clients. I used the disk caching features extensively and the CMS back office was designed to clear the cache whenever content is edited. It worked great, and when the client’s server team asked how the application would cope across multiple web servers all accessing a common NFS, I said it should be fine. I figured that the OS would take care of file locking and other such low level stuff that I generally don’t have to worry about, but mainly I was putting my trust in Smarty – A project that’s been around this long surely would have died out if it couldn’t cope with such a common server configuration. I was wrong. As it turns out, Smarty cannot cope with multiple servers reading and writing from the same cache. The reason for this took me hours to work out.. PHP has a file status cache, which will cause some serious confusion when multiple processes are modifying the same files during a script execution. The library didn’t seem to make any attempt to recover from errors that might be caused by this -

- so I hacked it. 

I’ll post my full solution at a later date, but basically it involves clearing the stat cache at certain, critical points.

WordPress

Another very popular PHP project – you are looking at right now. I’ve installed it a number of time on various different servers, but today I ran into trouble. To my astonishment even the latest version of WordPress (2.7) doesn’t support the mysqli extension. Only the old, arguably obsolete, mysql extension is supported. Depending on your server, you may not be able to install the old extension. This is the situation I ended up in today. I failed to find any trustworthy patches out there -

- so I hacked it.

It was fortunately fairly easy to replace all mysql_* function calls with mysqli_* function calls, but only time will tell if it will cause problems.

If you have any library hacking anecdotes, please post a comment