[Dovecot] Test suite? Was [Dovecot-news] 1.0.rc21 released

[Dan Price]

On Mon 05 Feb 2007 at 09:27AM, Eric Rostetter wrote:
> >I won't implement a unit test suite anytime soon though, so for now if
> >someone's interested, an IMAP-based test suite would be possible to
> >implement quite easily with some perl scripts or something. Even better
> >if such test suite was made server-independent.
> 
> As I said, I wouldn't expect _you_ alone to implement a unit test suite,
> as it would be way too big of a project.  But is is something the
> community could do for the CVS HEAD code, so it would be there for
> dovecot 2.x when that comes due.  But it is up to you if you want this
> or not.
> 
> Creating an IMAP test is going to be harder than you would think also,
> if you want to test mbox, maildir, and dbox all over local disk, NFS,
> AFS, etc.  And even then, as has been pointed out, all you've tested
> is the IMAP functions (no pop3, deliver, sieve, etc).  I'd see it
> more as a stop gap method if done ad hoc.

I think it's not the best advice to tell a software developer (any
developer) that creating a test suite will be harder than they think...
That's just one more reason not to start on it. :)

Others pointed out that testing every codepath will be hard; I think the
problem is much more decomposable than that.  The goal should be first
to test the parts of the server which most influence data integrity,
and then to work up from there.

Things like testing with NFS, AFS, exotic plugins, etc. can be done in
part by the user community--- that is to say, before a major release of
dovecot the community can run the test suite in a wide variety of
configurations.

Another approach is to play to your strength: Dovecot is pretty cleanly
organized into different modules.  That means that a set of C programs
could be written to exercise those interfaces, improving the likelihood
that individual components will work as planned.

Here are two reasonable summaries of what testing can do for a project,
written by some of my colleagues:

http://blogs.sun.com/bill/entry/zfs_and_the_all_singing
http://blogs.sun.com/ahl/entry/on_testing

        -dp

-- 
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp