[Dovecot] Test suite? Was [Dovecot-news] 1.0.rc21 released

[Bart Smaalders]

Eric Rostetter wrote:
> Quoting Ethan Sommer <sommere at gac.edu>:
> 
>>> Maybe a test suite next time,
>> A test suite sounds like an excellent idea, which I might be interested
>> in implementing. How would such a beat work.
> 
> Best practices is a unit test suite, to test every function/routine
> as thoroughly as possible.
> 
>> I'd imagine a program which takes a "test case" which comes with
>> 1. an initial mailbox
>> 2. a series of IMAP commands to execute on the mailbox
>> 3. a final test which has a IMAP command to run and a regular
>> expression which its result should match.
> 
> This might be one group of tests, but it wouldn't be sufficient by
> itself.
> 
>> I would think that non-programmers could come up with test cases given
>> such a system.
> 
> Non-programmers can at least suggest tests that should exist for others
> to code.  In effect, every bug report should spawn a new unit test when
> it is fixed.  So there is always opportunity for non-programmers to
> at least suggest tests if not code them.
> 
>> Off the top of my head, the downsides I see are: doesn't test deliver,
>> regular expressions might wind up being too specific and generate false
>> failures when doing regression tests, requires knowledge of IMAP
>> protocol.
> 
> That is why a complete set of unit tests is better.  It tests everything.
> Of course, it takes more time to develop and maintain.
> 
>> What do people think?
> 
> I think any project this size, with the goals it has, shoujld certainly
> create a unit test suite.  But then, Timo only has so much time, and
> volunteers would be very helpful here.  It could almost be started
> as a separate project inside the main project in CVS.
> 
>>

One of the ways of testing dovecot might be to utilize any test suites
available for common imap clients such as thunderbird, pine, ...  the 
idea of extending the test suite to catch all bugs found after release
is a good one.

- Bart