Quoting Dan Price dp@eng.sun.com:
I think it's not the best advice to tell a software developer (any developer) that creating a test suite will be harder than they think... That's just one more reason not to start on it. :)
Yeah, there is some logic in that... ;)
Others pointed out that testing every codepath will be hard; I think the problem is much more decomposable than that. The goal should be first to test the parts of the server which most influence data integrity, and then to work up from there.
Well, the idea of a unit test suite is that the smallest units influence the data integrity the most, and everything works up from there.... Your believe system may differ...
Things like testing with NFS, AFS, exotic plugins, etc. can be done in part by the user community--- that is to say, before a major release of dovecot the community can run the test suite in a wide variety of configurations.
Yes, that is always the goal of a test suite. But at the same time, sometimes you need code to check for specific things for different environments...
Another approach is to play to your strength: Dovecot is pretty cleanly organized into different modules. That means that a set of C programs could be written to exercise those interfaces, improving the likelihood that individual components will work as planned.
That is exactly what a unit test suite does. If you looked up the executive summary of a unit test suite, it would match what you say above.
Here are two reasonable summaries of what testing can do for a project, written by some of my colleagues:
http://blogs.sun.com/bill/entry/zfs_and_the_all_singing http://blogs.sun.com/ahl/entry/on_testing
I think everyone agrees testing is good, and most people would agree doing testing correctly is not easy. The problem is of course people are lazy, and we take shortcuts... Part of human nature. The good thing to remember is we can, if we decide to, overcome human nature and reach a level higher than our base nature...
-dp
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!