On Tue, 2005-11-15 at 06:32 -0500, Tom Allison wrote:

...

What I am thinking of is to run those tests before anything new is released. On the other hand I cannot even estimate how much work would be needed for this ...

Udo Rader

You could develop a test suite using something like perl. The approach would be run beat the hell out of all the functionality that dovecot provides and validate the input/output against what is expected. It's a black-box style of testing, but it would help indicate any potential problems. It was through my own version of this that I found there are some limitations to how many UIDS I can transmit at one time. But who expects >10,000 UIDS in a single call?

Between running IMAP calls to localhost and inspecting the local filesystem directly, I suspect a really nice test suite could be developed. Interestingly, most of it could apply to more than just dovecot if you seperate testing for IMAP/Maildir standard formats from dovecot particulars (like indexing?).

I see two possiblities: write an external "testclient" that does all the nasty things MUAs are doing to dovecot or to extend the codebase to include (unit?) test cases.

Major advantage of the external test suite indeed would be that one could use it against any MDA and I even found a couple of testsuites for POP3/IMAP servers, yet I am not sure that this is enough.

If for example I wanted to test if dovecot still correctly accesses an SQL database after a code update, a simple POP3/IMAP protocol test would probably not be enough to reveal any real problems. Or more general, it would be a nice thing to be able to test if a certain config option still works as before (I am thinking of locking methods on various platforms, access to the various userdb/passdb backends, sasl, ...).

Maybe a new "make test" target during compilation would suffice for that ...

Udo Rader

-- bestsolution.at EDV Systemhaus GmbH http://www.bestsolution.at

You'll probably also want to redesign dovecot a bit for testability. In the electrical engineering world, this means bringing important circuit nodes that are functional chokepoints to the outside of the package where test equipment can be attached to them. It also means adding a way around some layers of external interface, so the core functionality can be attached to a "life support system" where all of its inputs and outputs, now reworked into convenient test plugs and sockets, can be controlled at once by the "test harness".

In the software world, it works the same way. If there's an important state machine at the core of dovecot, make its state directly readable and writable. You may want to write some state-flushing and loading functions so that you can quickly reset to a known state, or step forward to a given state without going through a bunch of intermediate states. The external interface you might remove for dovecot would be the socket interface, so that you can test ten different versions of dovecot at once, on the same machine, under automated control, without any of them colliding on pathnames or IP ports.

You'll find redesign for testing has the beneficial side effect of cleaning up the interfaces within pieces of the core, and finding and removing linkages you didn't realize were there and didn't want.

For projects larger than dovecot, design for testability ends up embeding a scripting language into the application (don't write one, use guile, perl, or tcl) for greater control of internal pieces/parts. In a small C application like dovecot, you may get somewhere by inspecting internal state with gdb and full symbols rather than a scripting language. Send this verb in the input, check that internal variable with gdb, read the contents of that Maildir file. Because the test is automated, it can be added to the compile, build, and install scripts. You get to a point where you are testing the same functionality from several different viewpoints, and then it's hard for bugs to creep in without lots of warning from tests failing.

Enjoy! Brian