[Dovecot] a proposal to increase the stability

[Tom Allison]

Udo Rader wrote:
> On Tue, 2005-11-15 at 06:32 -0500, Tom Allison wrote:
> 
>>>What I am thinking of is to run those tests before anything new is
>>>released. On the other hand I cannot even estimate how much work would
>>>be needed for this ...
>>>
>>>Udo Rader
>>>
>>
>>You could develop a test suite using something like perl.
>>The approach would be run beat the hell out of all the functionality 
>>that dovecot provides and validate the input/output against what is 
>>expected.  It's a black-box style of testing, but it would help indicate 
>>any potential problems.   It was through my own version of this that I 
>>found there are some limitations to how many UIDS I can transmit at one 
>>time.  But who expects >10,000 UIDS in a single call?
>>
>>Between running IMAP calls to localhost and inspecting the local 
>>filesystem directly, I suspect a really nice test suite could be 
>>developed.  Interestingly, most of it could apply to more than just 
>>dovecot if you seperate testing for IMAP/Maildir standard formats from 
>>dovecot particulars (like indexing?).
> 
> 
> I see two possiblities: write an external "testclient" that does all the
> nasty things MUAs are doing to dovecot or to extend the codebase to
> include (unit?) test cases.

I was thinking of a testclient based on well established IMAP client 
libraries from language of your choice.  Perl has a lot of this already.

> Major advantage of the external test suite indeed would be that one
> could use it against any MDA and I even found a couple of testsuites for
> POP3/IMAP servers, yet I am not sure that this is enough.

It's "never" enough to some and sufficient for others.

> If for example I wanted to test if dovecot still correctly accesses an
> SQL database after a code update, a simple POP3/IMAP protocol test would
> probably not be enough to reveal any real problems. Or more general, it
> would be a nice thing to be able to test if a certain config option
> still works as before (I am thinking of locking methods on various
> platforms, access to the various userdb/passdb backends, sasl, ...).

I think you are missing my point.
You would build a test that would do something on both ends of the black 
box operation that is dovecot's IMAP server.

I would fully expect dovecot to fail accessing a SQL database after a 
code update if that really was the case.  You would trap that error and 
voila! you have a test case for it.
If you were interested something like this, then you could seed the 
database with whatever information you wanted to be accessed (or missed) 
via perl, run the IMAP client commands, validate the output and confirm 
the database information.
You have to go through a sequence on each test:
guarantee that all the necessary data/configurations are in place.  This 
may mean that the test script changes the configuration files, restarted 
dovecot, and does the testing.
Run the IMAP commands.
validate their output.
validate the database state.