On Mon 05 Feb 2007 at 09:27AM, Eric Rostetter wrote:
I won't implement a unit test suite anytime soon though, so for now if someone's interested, an IMAP-based test suite would be possible to implement quite easily with some perl scripts or something. Even better if such test suite was made server-independent.
As I said, I wouldn't expect _you_ alone to implement a unit test suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Creating an IMAP test is going to be harder than you would think also, if you want to test mbox, maildir, and dbox all over local disk, NFS, AFS, etc. And even then, as has been pointed out, all you've tested is the IMAP functions (no pop3, deliver, sieve, etc). I'd see it more as a stop gap method if done ad hoc.
I think it's not the best advice to tell a software developer (any developer) that creating a test suite will be harder than they think... That's just one more reason not to start on it. :)
Others pointed out that testing every codepath will be hard; I think the problem is much more decomposable than that. The goal should be first to test the parts of the server which most influence data integrity, and then to work up from there.
Things like testing with NFS, AFS, exotic plugins, etc. can be done in part by the user community--- that is to say, before a major release of dovecot the community can run the test suite in a wide variety of configurations.
Another approach is to play to your strength: Dovecot is pretty cleanly organized into different modules. That means that a set of C programs could be written to exercise those interfaces, improving the likelihood that individual components will work as planned.
Here are two reasonable summaries of what testing can do for a project, written by some of my colleagues:
http://blogs.sun.com/bill/entry/zfs_and_the_all_singing http://blogs.sun.com/ahl/entry/on_testing
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp