[Dovecot] 1.0.rc21 released
http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz.sig
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
- Cache file handling could have crashed rc20Timo,
Thank you for the quick response. Rc21 is now in place here and there was no screeching tires and breaking glass with it (unlike rc20). Time to test the read/unread/reply flags in imap.
Jeff Earickson Colby College
On Fri, 2 Feb 2007, Timo Sirainen wrote:
Date: Fri, 02 Feb 2007 16:18:36 +0200 From: Timo Sirainen tss@iki.fi To: dovecot-news@dovecot.org Cc: dovecot@dovecot.org Subject: [Dovecot] 1.0.rc21 released
http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz.sig
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
- Cache file handling could have crashed rc20
Hi Timo,
Maybe a test suite next time, but I've been happy since RC7 when I upgraded from POP3-Daemon. Dovecot actually saved me from buying a new server, because my server load on my email server went from 1.5 to 0.2 overnight after upgrading. Dovecot rocks! Keep up the good work.
Steffen
Timo Sirainen skrev:
http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz.sig
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
- Cache file handling could have crashed rc20
Dovecot-news mailing list Dovecot-news@dovecot.org http://dovecot.org/cgi-bin/mailman/listinfo/dovecot-news
Silicom Web (Steffen) wrote:
Hi Timo,
Maybe a test suite next time, A test suite sounds like an excellent idea, which I might be interested in implementing. How would such a beat work.
I'd imagine a program which takes a "test case" which comes with
- an initial mailbox
- a series of IMAP commands to execute on the mailbox
- a final test which has a IMAP command to run and a regular expression which its result should match.
I would think that non-programmers could come up with test cases given such a system. Even if its only used as a way to say "here, run this test case which shows how things are broken now" and then the test case is kept around to make sure there aren't any regressions in newer versions, I would think that such a system would be very useful.
Off the top of my head, the downsides I see are: doesn't test deliver, regular expressions might wind up being too specific and generate false failures when doing regression tests, requires knowledge of IMAP protocol.
What do people think?
-- Ethan Sommer Systems Administrator Gustavus Adolphus College 507-933-7042 sommere@gac.edu
Quoting Ethan Sommer sommere@gac.edu:
Maybe a test suite next time, A test suite sounds like an excellent idea, which I might be interested in implementing. How would such a beat work.
Best practices is a unit test suite, to test every function/routine as thoroughly as possible.
I'd imagine a program which takes a "test case" which comes with
- an initial mailbox
- a series of IMAP commands to execute on the mailbox
- a final test which has a IMAP command to run and a regular expression which its result should match.
This might be one group of tests, but it wouldn't be sufficient by itself.
I would think that non-programmers could come up with test cases given such a system.
Non-programmers can at least suggest tests that should exist for others to code. In effect, every bug report should spawn a new unit test when it is fixed. So there is always opportunity for non-programmers to at least suggest tests if not code them.
Off the top of my head, the downsides I see are: doesn't test deliver, regular expressions might wind up being too specific and generate false failures when doing regression tests, requires knowledge of IMAP protocol.
That is why a complete set of unit tests is better. It tests everything. Of course, it takes more time to develop and maintain.
What do people think?
I think any project this size, with the goals it has, shoujld certainly create a unit test suite. But then, Timo only has so much time, and volunteers would be very helpful here. It could almost be started as a separate project inside the main project in CVS.
-- Ethan Sommer Systems Administrator Gustavus Adolphus College 507-933-7042 sommere@gac.edu
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Eric Rostetter wrote:
Quoting Ethan Sommer sommere@gac.edu:
Maybe a test suite next time, A test suite sounds like an excellent idea, which I might be interested in implementing. How would such a beat work.
Best practices is a unit test suite, to test every function/routine as thoroughly as possible.
I'd imagine a program which takes a "test case" which comes with
- an initial mailbox
- a series of IMAP commands to execute on the mailbox
- a final test which has a IMAP command to run and a regular expression which its result should match.
This might be one group of tests, but it wouldn't be sufficient by itself.
I would think that non-programmers could come up with test cases given such a system.
Non-programmers can at least suggest tests that should exist for others to code. In effect, every bug report should spawn a new unit test when it is fixed. So there is always opportunity for non-programmers to at least suggest tests if not code them.
Off the top of my head, the downsides I see are: doesn't test deliver, regular expressions might wind up being too specific and generate false failures when doing regression tests, requires knowledge of IMAP protocol.
That is why a complete set of unit tests is better. It tests everything. Of course, it takes more time to develop and maintain.
What do people think?
I think any project this size, with the goals it has, shoujld certainly create a unit test suite. But then, Timo only has so much time, and volunteers would be very helpful here. It could almost be started as a separate project inside the main project in CVS.
One of the ways of testing dovecot might be to utilize any test suites available for common imap clients such as thunderbird, pine, ... the idea of extending the test suite to catch all bugs found after release is a good one.
- Bart
On Sun, 2007-02-04 at 20:56 -0600, Eric Rostetter wrote:
Quoting Ethan Sommer sommere@gac.edu:
Maybe a test suite next time, A test suite sounds like an excellent idea, which I might be interested in implementing. How would such a beat work.
Best practices is a unit test suite, to test every function/routine as thoroughly as possible.
I think the real problem is how to test all the code paths. Especially testing against race conditions isn't really possible without inserting some checkpoints into the code to block at specific points so the test can even work. I'm more interested in that and in checking error handling paths than just calling all the public functions with different parameters and checking if they work.
I won't implement a unit test suite anytime soon though, so for now if someone's interested, an IMAP-based test suite would be possible to implement quite easily with some perl scripts or something. Even better if such test suite was made server-independent.
Quoting Timo Sirainen tss@iki.fi:
On Sun, 2007-02-04 at 20:56 -0600, Eric Rostetter wrote:
Quoting Ethan Sommer sommere@gac.edu:
Maybe a test suite next time, A test suite sounds like an excellent idea, which I might be interested in implementing. How would such a beat work.
Best practices is a unit test suite, to test every function/routine as thoroughly as possible.
I think the real problem is how to test all the code paths. Especially
A good unit test suite should test all code paths...
testing against race conditions isn't really possible without inserting some checkpoints into the code to block at specific points so the test
Yes, unit tests are not always the best at race conditions. Sometimes you can write a unit test for this, but often, you can't...
can even work. I'm more interested in that and in checking error handling paths than just calling all the public functions with different parameters and checking if they work.
Well, it is your project, so you get the ultimate decision on things.
I won't implement a unit test suite anytime soon though, so for now if someone's interested, an IMAP-based test suite would be possible to implement quite easily with some perl scripts or something. Even better if such test suite was made server-independent.
As I said, I wouldn't expect _you_ alone to implement a unit test suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Creating an IMAP test is going to be harder than you would think also, if you want to test mbox, maildir, and dbox all over local disk, NFS, AFS, etc. And even then, as has been pointed out, all you've tested is the IMAP functions (no pop3, deliver, sieve, etc). I'd see it more as a stop gap method if done ad hoc.
Instead, I'd propose setting up a system for unit tests, and then within that structure create your IMAP testing. Then over time, more tests will be added, until eventually you have a full test suite with a defined setup and structure.
But again, it's your project, so you get the last word...
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
On 5.2.2007, at 17.27, Eric Rostetter wrote:
As I said, I wouldn't expect _you_ alone to implement a unit test
suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Well, sure a test suite would be nice. It would be also nice if
someone with some experience in them would create the beginnings of
it. I've some vague ideas about how it could work, but my past test
suite experience is pretty small.
Creating an IMAP test is going to be harder than you would think also, if you want to test mbox, maildir, and dbox all over local disk, NFS, AFS, etc. And even then, as has been pointed out, all you've tested is the IMAP functions (no pop3, deliver, sieve, etc). I'd see it more as a stop gap method if done ad hoc.
I was thinking that the IMAP test suite could be useful also outside
Dovecot. There's been talk about such test suite once in a while in
imap-protocol list. So it could check for both basic protocol
compliance and for specific bugs in different servers.
Quoting Timo Sirainen tss@iki.fi:
On 5.2.2007, at 17.27, Eric Rostetter wrote:
As I said, I wouldn't expect _you_ alone to implement a unit test suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Well, sure a test suite would be nice. It would be also nice if someone with some experience in them would create the beginnings of it. I've some vague ideas about how it could work, but my past test suite experience is pretty small.
I'm willing to help. How best to start? Should we discuss possible method on the list? Or should someone (e.g. myself) just start with what they want to do, without discussion? Or should we start private thread for this? Or a new mailing list?
Creating an IMAP test is going to be harder than you would think also, if you want to test mbox, maildir, and dbox all over local disk, NFS, AFS, etc. And even then, as has been pointed out, all you've tested is the IMAP functions (no pop3, deliver, sieve, etc). I'd see it more as a stop gap method if done ad hoc.
I was thinking that the IMAP test suite could be useful also outside Dovecot. There's been talk about such test suite once in a while in imap-protocol list. So it could check for both basic protocol compliance and for specific bugs in different servers.
Sounds interesting. More of a separate project, but interesting.
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
On Tue, 2007-02-06 at 10:33 -0600, Eric Rostetter wrote:
Quoting Timo Sirainen tss@iki.fi:
On 5.2.2007, at 17.27, Eric Rostetter wrote:
As I said, I wouldn't expect _you_ alone to implement a unit test suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Well, sure a test suite would be nice. It would be also nice if someone with some experience in them would create the beginnings of it. I've some vague ideas about how it could work, but my past test suite experience is pretty small.
I'm willing to help. How best to start? Should we discuss possible method on the list? Or should someone (e.g. myself) just start with what they want to do, without discussion? Or should we start private thread for this? Or a new mailing list?
I think this mailing list is good enough for talking about it. That allows other people to comment on it who wouldn't otherwise bother to join a new list. And I don't think there are going to be that many messages about it anyway :)
As for talking first vs. just doing it, I don't know. I'm not sure if I have much comments about it, but probably best to show me early enough what you've done in case there's something I really don't like :)
All of lib/ should be pretty easy to test with quite simple tests.
Testing all the code paths in lib-index will be a huge job. I think the best way to do it would be to build some special index/log/cache files for each test. Testing for race conditions could be done some day by inserting some unittest_checkpoint("name"); calls in the code and compiling with a special -DUNITTEST flag.
Testing lib-storage then.. Well, it's somewhat related to lib-index since the index file and mail storage is synced at the same time. Also maildir has a lot of race condition possibilities that can be difficult to test.
I think lib-index and lib-storage tests are going to be the most useful ones, since that's where the bugs usually have been. Don't bother adding tests to master since I've already rewritten it (not yet in CVS). login-common is also more or less rewritten. lib-auth will hopefully go away too.
On Mon 05 Feb 2007 at 09:27AM, Eric Rostetter wrote:
I won't implement a unit test suite anytime soon though, so for now if someone's interested, an IMAP-based test suite would be possible to implement quite easily with some perl scripts or something. Even better if such test suite was made server-independent.
As I said, I wouldn't expect _you_ alone to implement a unit test suite, as it would be way too big of a project. But is is something the community could do for the CVS HEAD code, so it would be there for dovecot 2.x when that comes due. But it is up to you if you want this or not.
Creating an IMAP test is going to be harder than you would think also, if you want to test mbox, maildir, and dbox all over local disk, NFS, AFS, etc. And even then, as has been pointed out, all you've tested is the IMAP functions (no pop3, deliver, sieve, etc). I'd see it more as a stop gap method if done ad hoc.
I think it's not the best advice to tell a software developer (any developer) that creating a test suite will be harder than they think... That's just one more reason not to start on it. :)
Others pointed out that testing every codepath will be hard; I think the problem is much more decomposable than that. The goal should be first to test the parts of the server which most influence data integrity, and then to work up from there.
Things like testing with NFS, AFS, exotic plugins, etc. can be done in part by the user community--- that is to say, before a major release of dovecot the community can run the test suite in a wide variety of configurations.
Another approach is to play to your strength: Dovecot is pretty cleanly organized into different modules. That means that a set of C programs could be written to exercise those interfaces, improving the likelihood that individual components will work as planned.
Here are two reasonable summaries of what testing can do for a project, written by some of my colleagues:
http://blogs.sun.com/bill/entry/zfs_and_the_all_singing http://blogs.sun.com/ahl/entry/on_testing
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp
Quoting Dan Price dp@eng.sun.com:
I think it's not the best advice to tell a software developer (any developer) that creating a test suite will be harder than they think... That's just one more reason not to start on it. :)
Yeah, there is some logic in that... ;)
Others pointed out that testing every codepath will be hard; I think the problem is much more decomposable than that. The goal should be first to test the parts of the server which most influence data integrity, and then to work up from there.
Well, the idea of a unit test suite is that the smallest units influence the data integrity the most, and everything works up from there.... Your believe system may differ...
Things like testing with NFS, AFS, exotic plugins, etc. can be done in part by the user community--- that is to say, before a major release of dovecot the community can run the test suite in a wide variety of configurations.
Yes, that is always the goal of a test suite. But at the same time, sometimes you need code to check for specific things for different environments...
Another approach is to play to your strength: Dovecot is pretty cleanly organized into different modules. That means that a set of C programs could be written to exercise those interfaces, improving the likelihood that individual components will work as planned.
That is exactly what a unit test suite does. If you looked up the executive summary of a unit test suite, it would match what you say above.
Here are two reasonable summaries of what testing can do for a project, written by some of my colleagues:
http://blogs.sun.com/bill/entry/zfs_and_the_all_singing http://blogs.sun.com/ahl/entry/on_testing
I think everyone agrees testing is good, and most people would agree doing testing correctly is not easy. The problem is of course people are lazy, and we take shortcuts... Part of human nature. The good thing to remember is we can, if we decide to, overcome human nature and reach a level higher than our base nature...
-dp
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
I think that the infinite loop problem you fixed in rc21 fixed my load problem that I was having. I might actually not have to revert back to rc7 this time. :)
It will be really nice if the dovecot code can be scanned by: Coverity (http://www.coverity.com/) and Klocwork (http://www.klocwork.com/)
The have scanned the samba code: http://www.samba.org/samba/history/samba-3.0.23.html The service I think was free and the results are great: http://scan.coverity.com/
Maybe Timo can talk to those companies and get a free scan ;)
HTH Oliver
-- Oliver Schulze L. | Get my e-mail after a captcha in: Asuncion - Paraguay | http://tinymailto.com/oliver
On 7.2.2007, at 8.00, Oliver Schulze L. wrote:
It will be really nice if the dovecot code can be scanned by: Coverity (http://www.coverity.com/) and Klocwork (http:// www.klocwork.com/)
Dovecot's code was scanned with Coverity a few weeks ago. It found a
couple of minor bugs in error handling paths, and I did some code
cleanups because of it.
Just great, thanks for replying Timo
Oliver
Timo Sirainen wrote:
Dovecot's code was scanned with Coverity a few weeks ago. It found a couple of minor bugs in error handling paths, and I did some code cleanups because of it.
-- Oliver Schulze L. | Get my e-mail after a captcha in: Asuncion - Paraguay | http://tinymailto.com/oliver
For what it's worth in kudos -- one of my beta testers immediately replied with the below after upgrading rc19 to rc21 (Maildir, NFS/fnctl, PAM using nss_ldap):
"Troy, whatever it was that you did has sped up my use of Thunderbird tremendously. I'm back to moving a month's worth of email with no timeout errors. Huge huge HUGE workflow and efficiency improvement!"
woo! ;) -te
Timo Sirainen wrote:
http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz http://dovecot.org/releases/dovecot-1.0.rc21.tar.gz.sig
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
- Cache file handling could have crashed rc20
-- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com
On 2007/02/02 15:18, Timo Sirainen tss@iki.fi wrote:
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
Hi Timo,
I have noticed there are no CVS tags for the releases after rc16.
Max
On 6.2.2007, at 9.03, Max Kellermann wrote:
On 2007/02/02 15:18, Timo Sirainen tss@iki.fi wrote:
Just one fix. Maybe the one big thing in Dovecot v2.0.* will be a test suite, which is run before any release. :)
Hi Timo,
I have noticed there are no CVS tags for the releases after rc16.
I keep forgetting to add them..
participants (11)
-
Bart Smaalders
-
Dan Price
-
Eric Rostetter
-
Ethan Sommer
-
Jeff A. Earickson
-
Marc Perkel
-
Max Kellermann
-
Oliver Schulze L.
-
Silicom Web (Steffen)
-
Timo Sirainen
-
Troy Engel