[Dovecot] writing an MDA, help me avoid corrupted indexes
Hello.
First, I want to thank those responsible for Dovecot IMAP as it was a real treat to implement in comparison to my mostly-failed attempts with Cyrus. The passwd-file authentication was exactly what I needed as I really didn't want to have to deal with PAM or LDAP but where I am setting up a server with virtual accounts.
I'm using Postfix as an MTA, and it quite handily passes the mail off to SpamAssassin. From there I need an MDA to dump any mails detected as spam into a common spam folder, discard any mails with viruses (antivir) and send a notification to me, and then drop good mail into the appropriate maildir directory for the virtual users to get via both POP3 and IMAP.
I could conceivably use procmail to do that MDA, and I've done similar things with procmail in the past, but my growing experience with procmail indicates that procmail is too ferocious for what simple work I need done, and too difficult to make work right in a virtual-user environment. So I've written my own little MDA with a few lines of shell that will take the mail from Postfix+SpamAssassin and ultimately deposit mails in the maildirs.
So far so good, except that I keep seeing corrupt indexes... probably because I'm not touching the indexes at all with the MDA. My guess is that in order for the MDA to properly deal with maildir it needs to adjust the indexes after depositing new mail. Okay. So can someone either give me a pointer to the proper documentation on this, including the index file formats, or perhaps just explain it to me? Maybe I'm missing a larger picture of maildir handling?
I've glanced through http://www.dovecot.org/doc/index.txt but I notice that the index files are not ASCII, so I'm not sure where to start at understanding that documentation.
Any and all help would be appreciated.
Thanks,
Lee.
Lee Howard wrote:
Hello.
I could conceivably use procmail to do that MDA, and I've done similar things with procmail in the past, but my growing experience with procmail indicates that procmail is too ferocious for what simple work I need done, and too difficult to make work right in a virtual-user environment. So I've written my own little MDA with a few lines of shell that will take the mail from Postfix+SpamAssassin and ultimately deposit mails in the maildirs.
Have you checked out Dovecot's own LDA? It includes index updating as, really, its primary purpose, but also Sieve filtering. If you just want the LDA to filter on headers set by your spam/virus scanners, it's ample.
So far so good, except that I keep seeing corrupt indexes... probably because I'm not touching the indexes at all with the MDA. My guess is that in order for the MDA to properly deal with maildir it needs to adjust the indexes after depositing new mail. Okay. So can someone either give me a pointer to the proper documentation on this, including the index file formats, or perhaps just explain it to me? Maybe I'm missing a larger picture of maildir handling?
My understanding was that Dovecot fixes corrupt indexes as a matter of course, and isn't too bothered with it. What level do you have your logging set to?
Do you, by chance, have your mail partition mounted 'noatime'?
I've glanced through http://www.dovecot.org/doc/index.txt but I notice that the index files are not ASCII, so I'm not sure where to start at understanding that documentation.
Any and all help would be appreciated.
As a further note, have you checked out the patches that allow Postfix to use the Dovecot auth daemon, so you can have SMTP AUTH without resorting to the horror that is Cyrus SASL?
-- Curtis Maloney cmaloney@cardgate.net
Curtis Maloney wrote:
Have you checked out Dovecot's own LDA?
No, but I just did, and it looks like exactly what I need... except that it doesn't support passwd-file auth. Or am I missing that?
The website documentation on this matter is noticeably outdated.
Do you, by chance, have your mail partition mounted 'noatime'?
No, but I think that my indexing problem is that my self-made LDA/MDA is not following maildir spec in file naming and construction. I'd prefer to use deliver if I could.
As a further note, have you checked out the patches that allow Postfix to use the Dovecot auth daemon, so you can have SMTP AUTH without resorting to the horror that is Cyrus SASL?
I'll look into this when it's time to get ASMTP going.
Thanks,
Lee.
On Wed, 2005-12-14 at 10:03 -0800, Lee Howard wrote:
Curtis Maloney wrote:
Have you checked out Dovecot's own LDA?
No, but I just did, and it looks like exactly what I need... except that it doesn't support passwd-file auth. Or am I missing that?
It uses dovecot-auth directly, so you don't have to configure users for the LDA separately.
The website documentation on this matter is noticeably outdated.
Do you, by chance, have your mail partition mounted 'noatime'?
No, but I think that my indexing problem is that my self-made LDA/MDA is not following maildir spec in file naming and construction. I'd prefer to use deliver if I could.
As long as the filenames are always unique (eg. inode numbers get reused), there should be no problem. Dovecot doesn't require any index updates either from MDA. What errors are you getting?
Timo Sirainen wrote:
On Wed, 2005-12-14 at 10:03 -0800, Lee Howard wrote:
Curtis Maloney wrote:
Have you checked out Dovecot's own LDA?
No, but I just did, and it looks like exactly what I need... except that it doesn't support passwd-file auth. Or am I missing that?
It uses dovecot-auth directly, so you don't have to configure users for the LDA separately.
I'm was trying to follow the instructions here:
http://wiki.dovecot.org/moin.cgi/LDA
I didn't realize that I had to create a "socket listen" in auth default... I was trying to use the other "socket listen" instances that are found in the example config for other uses.
Once I got that sorted out I got the socket fine. But because permissions wouldn't allow it, I had to put the socket somewhere other than /var/run/dovecot/auth-master, and that was easy enough with dovecot.conf, but the dovecot-deliver.conf file shown on the Wiki indicates an auth_socket_path option which doesn't seem to have an effect. I had to look at the code to see that it's modified with the -a option now.
Okay, so that's now all sorted out on this end. My own LDA now uses Dovecot deliver, and it works well... I don't seem to have the problems that I had just sticking the mail file into the maildir new directory on my own.
If I could offer a suggestion here it would be to get some more accurate documentation out there for deliver... even if it is just runtime "--help" or some such messages.
As long as the filenames are always unique (eg. inode numbers get reused), there should be no problem. Dovecot doesn't require any index updates either from MDA.
This certainly doesn't appear to be the case.
What errors are you getting?
Well, I'm not getting those errors any more because I'm using Dovecot deliver ;-)
However, when I was just sticking the files into the maildir new directory "by hand" (yes, they used unique filenames)...
With the dovecot-0.99.13-4.FC2 binary installation I was getting "Corrupted index file" errors pointing at the .imap.index file in the maildir and with a "Fixed duplicate" error message the mail would get moved to the cur directory and the POP3 client would never download it.
With the 1.0alpha5 tarball installation that I performed (after uninsalling the binary RPM installation) I didn't get the "Corrupted index file" messages, but I did get the "Fixed duplicate" messages, and the mail would still get moved to the cur directory.
Having now installed both 0.99 and 1.0alpha I can see obvious improvements in the newer version. However, admittedly I found 0.99 less complicated to configure than 1.0alpha. I think that the formatting of the config file is a bit too programmer-esque and would be more user-friendly with context-style (.ini) entries like this:
[auth default] passdb = passwd-file passwd-file-path = /usr/local/etc/dovecot-users .......
[pop3] uidl_format = %08Xu%08Xv ......
In any case, thank you for all that you do. Dovecot is a wonderfully better than Cyrus from an installation and maintenance perspective.
Lee.
Lee Howard wrote:
Curtis Maloney wrote:
Have you checked out Dovecot's own LDA?
No, but I just did, and it looks like exactly what I need... except that it doesn't support passwd-file auth. Or am I missing that?
erm... it's not the place of the LDA to check the user's password. Why would you want to check their password for mail being delivered _to_ them?
The website documentation on this matter is noticeably outdated.
Although it's been around for a while now, it's still considered quite new in Dovecot, so this isn't entirely surprising. Perhaps you'd like to contribute your findings and tips once you're set up?
Do you, by chance, have your mail partition mounted 'noatime'?
No, but I think that my indexing problem is that my self-made LDA/MDA is not following maildir spec in file naming and construction. I'd prefer to use deliver if I could.
Ah, well... yes, I can see that could cause problems :)
Certainly, I'd go with Dovecot's own LDA instead of rolling your own. Then you know it makes filenames the way Dovecot likes, and get the bonus of updated indexes.
Can't wait until I get the chance to move to 1.0 (am still running 0.99 here :/ ) ... deliver, shared folders, oh the bliss! :)
-- Curtis Maloney cmaloney@cardgate.net
On Wed, 14 Dec 2005, Lee Howard wrote:
No, but I think that my indexing problem is that my self-made LDA/MDA is not following maildir spec in file naming and construction. I'd prefer to use deliver if I could.
Actually, the filenames are not to matter at all. The spec gives just a convention to work-a-round concurrent write access and speed up mail reading etc.
Bye,
-- Steffen Kaiser
Steffen Kaiser wrote:
On Wed, 14 Dec 2005, Lee Howard wrote:
No, but I think that my indexing problem is that my self-made LDA/MDA is not following maildir spec in file naming and construction. I'd prefer to use deliver if I could.
Actually, the filenames are not to matter at all. The spec gives just a convention to work-a-round concurrent write access and speed up mail reading etc.
The naming convention that I was using was this... sender_email:receiver_email:unix_time:pid:random_number. So maybe something like:
spammer@nohost.org:faxguy@howardsilvan.com:1134651499:8722:23976
I understand that the naming convention shouldn't matter. In fact, mbox2mdir produces filenames outside the spec that don't cause problems. However, for some reason the naming convention that I chose to use caused issues.
Thanks,
Lee.
On Thu, 2005-12-15 at 07:03 -0800, Lee Howard wrote:
The naming convention that I was using was this... sender_email:receiver_email:unix_time:pid:random_number. So maybe something like:
spammer@nohost.org:faxguy@howardsilvan.com:1134651499:8722:23976
I understand that the naming convention shouldn't matter. In fact, mbox2mdir produces filenames outside the spec that don't cause problems. However, for some reason the naming convention that I chose to use caused issues.
':' isn't allowed. Dovecot assumes that the base filename ends in the first ':'.
participants (5)
-
Brian Bartholomew
-
Curtis Maloney
-
Lee Howard
-
Steffen Kaiser
-
Timo Sirainen