At The Economist, we've been struggling some to keep our database changes 100% in code. We're required to automate everything using hook_update and hook_install. We don't want to be pushing database imports around, and we want our build scripts to create a current database for testing purposes. Also, we want to see how the database is changing over time. How we do it? The answer might not surprise you--the Rake of Drupal is Drush.

After nearly 24 hours of struggling with our new production cluster, two weeks of testing, two months of development, two months of design, and a really long time of seriously headache-inducing brainstorming, Observer.com is finally running Drupal 6, with a totally new look and a totally new philosophy. I owe a post about how we did what we did and what tools we used, but that's tomorrow. I will also provide the source code for every custom module we wrote for the new site as soon as I figure out how to license it. This post is just a thank you note to our staff, partners, and members of the Drupal community who were particularly helpful--on IRC, in person, and some just by writing awesome code.

Lately I've been wrapped up pretty seriously helping dww migrate Project Issue Tracking to Drupal 6 as part of the Drupal.org sprint in Boston (I'll be joining them on Wednesday). Today I continued work on this issue, specifically converting an old hard coded AHAH routine to use #ahah, eliminating custom security checks and an entire custom Javascript file, not to mention fixing the sheer brokenness of the thing in Drupal 6. I won't walk you through the whole #ahah process; you can get that here. All I want to do is use #ahah to change the structure of a form--and do it it right. Here's what I learned.

Every night for the next month, while I am sleeping, all of the data in the Drupal 5 version of Observer.com will migrate to the Drupal 6 version. This means that our editors will always get to see yesterday's content on the build site of the Drupal 6 version and we will have witnessed more than 30 (hopefully) successful data migrations and have plenty of time to make adjustments. When we're ready to launch the site, we'll just let the data migrate as usual, then dump the newly created database and files directory, install them on the production MySQL and NFS servers, and change Apache's configuration to point to the Drupal 6 version--it couldn't be easier, and it won't take me longer than thirty minutes. So the why of this should be pretty obvious who's done this sort of migration before. But the how isn't so clear, or it wouldn't have taken me so long to get right.

So I just submitted my first core patch, followed quick by several revisions. It felt as good as I was told it would. Still tingling.

Anyways, this is an issue that's got to be resolved--it's been stuck for months on one particular problem, which I don't think should be a problem at all. It's very problematic for developers of large scale sites to be unable to adjust the expiration sent by Drupal to the client. My goal in this patch is to give developers this ability and intentionally not address the issue--which, again, has delayed this patch for months--of how reverse proxies are going to deal with it. That's not Drupal's job--it's the job of whoever is connecting Drupal with the reverse proxy, and any attempt to solve this on the Drupal Core level will require not using PHP Sessions for users that aren't authenticated. Turn the page for the proof.

Some comments I received for my post about coming to terms with Drupal requested a list of Modules that I trust. I considered several approaches, including starting a database of modules with my reviews. I decided against that--I would hate to sound authoritative on which Drupal modules are good and which are bad because I haven't used so many of them, so I'm just going to discuss my experience with some common modules and hope that it helps. This is absolutely just my subjective opinion having run these modules for over a year on Observer.com and Politicker.com. I will vouch for these modules to the extent they will only ruin your life if you use them badly. Of course, your mileage may vary wildly, and if you're running or starting a site of any scale, please don't just take my word for it.

I used to really hate Drupal.

That's probably a surprising thing for me to say. I'm the lead developer of Observer.com and Politicker.com, both of which use Drupal heavily. Observer.com was a very early instance of a high-profile newspaper running on Drupal, and it was built by Moshe Weitzman, a very early instance of a core Drupal Contributor. I saw Moshe most recently at Do it with Drupal in New Orleans where we shared the floor at a case study of Observer.com. After our presentation, he asked me to share some details of my transformation from a Drupal hater to a Drupal lover. I hope this might be of some use to other programmers coming into Drupal for the first time, and to the Drupal community as it continues to mature.