Drupalcon - part 2... testing


There were a lot of very interesting sessions at Drupalcon, 4 full days packed with them in fact. Some of my personal favorites were openid, form api 3, panels 2, and the Drupal association panel. One of the things that really stood out (and I also really enjoyed) was the configuration management discussions. A decent percentage of the talks covered or had some overlap with configuration management and testing. Automated testing is a huge issue for "Enterprise" Drupal right now. There's not really a "safe" way to upgrade a Drupal site at the moment. A major upgrade pretty much goes as follows:

  1. backup
  2. upgrade the code
  3. update the database
  4. cross your fingers
  5. tap your toes
  6. hope it works.
  7. repeat if necessary (*always* repeat if necessary ;-))

Once you've upgraded you can manually go through the pages and try to find any glitches in the site, and if you're lucky you'll get them. Maybe the users of the site will find and report the errors. If you're lucky you won't lose any data. There's currently no unit testing, regession testing or smoke testing (sure, there's some test code... but as far as I'm concerned the amount that exists is really only proof of concept that it can be done). There's roughly half a dozen companies that put the majority of paid development into Drupal and are also hiring up most of the good Drupal developers. These companies have a vested interest in getting automated testing working in Drupal.

At one of the more ad hoc sessions (which included people from most of the big Drupal comapnies and prominent community members) NowPublic promised a large chunk of one of their developers' time to work exclusively on writing unit tests. There was obviously a lot of support for this. This session was more or less a discussion of how to approach the problem and solve it and I think some good ideas and points came out of it.

Writing tests is a bitch. I can say this as I use to write *very* extensive tests for code I wrote (python has some very nice tools for unit testing that I miss in almost every other language I use). The test code can often be significantly larger than the code it's testing. It was kind of fun at first, but all the typing and copy-and-pasting quickly started aggrevating the carpel tunnel ;). Writing the tests is the easy part. Keeping the tests up to date is the tough part. Getting developers to keep improving the code at the same rate while forcing them to write and update the test code on a mostly volunteer project... I don't even know if that's possible.

There's a popular idea in the Drupal business community that if you develop something, give it back to the community and let the community maintain it. I think this works great for both the companies developing Drupal and the volunteers working on the project. Keeping their code a secret hurts both parties. Let the company pay for the initial development of the code, and if there's interest someone will pick it up. Unfortunately, I don't see this working for testing. Even if the initial set of tests is paid for, I really can't see the "community" picking up the maintenance of the tests. It's a bitch, and doesn't scratch enough peoples' itch.

Regardless, automated testing in Drupal will be a Good Thing. And what do I know, maybe volunteer developers will be totally into keeping the test code up to date. "All tests passed" messages make people feel warm and fuzzy inside :).

One last point... since I'm in a ranty mood. This is to the person who likes to preach "...every time you touch a contrib module ... run coder module ... write and update simpletest tests ... yada yada". Perhaps it's about time you put your money where your mouth is and start getting your own developers to do it. :-p

Peanut Gallery

Drupal is using the

Scott Hadfield

Drupal is using the simpletest and selenium for their testing framework. Everything is in place and it's easy to run tests and add new tests. Now we just need to get over the overhead of actually writing all the tests.

Actually I'm not too worried about python testing anymore, since I always write perfect python code ;-). But there's also the new simpletest automator (http://drupal.org/project/simpletest_automator) which automates building of tests, i don't even understand how it works yet.

A good example of an

A good example of an open-source library with extensive unit tests is Boost.

Check the links on the right side of the page under Regression.

For unit testing to be a win,
1) They must be easy to run all of the time
2) They must be easy to add new tests

Other than, it is hard to get up to date unit tests in a professional, closed-source environment. I would imagine it would be just as hard to do in a volunteer, open-source environment.

What did you need to

What did you need to copy-and-paste for unit tests in Python?

In the C/C++ world, macros are usually used to cut down on the typing. There must be a way to automate some of this code writing in the Python world.