[Dovecot] Blank lines in mailboxes
Hello,
we use dovecot on a regular basis. Sometimes I end up with a user telling me he does not get mail. When I look after the problem, I always find leading blank lines in the mbox file. We are using postfix with dovecot and amavisd as spam scanner. Amavisd never accesses the mbox files directly so this can be ruled out. I suspected procmail but now it happened also with accounts where no procmail is active. I found only a list thread where this happened always, i.e. it was a configuration error. (Meaning the mailbox was correct but dovecot would not read it). Here ist is only occasionally and therefore hard to track . Does anybody have the same problem or has hints to a solution ? We are running postfix 2.1.4 and dc 0.99.10.6 ; just now I am upgrading to pf 2.2.2. and dc 0.99.14.
Yours, Jakob Curdes
On Tue, 2005-04-26 at 12:02 +0200, Jakob Curdes wrote:
we use dovecot on a regular basis. Sometimes I end up with a user telling me he does not get mail. When I look after the problem, I always find leading blank lines in the mbox file.
At the beginning of the file?
0.99.x's mbox handling isn't too good and it's known to cause this sometimes. 1.0-stable releases have rewritten mbox support which fixes this.
Timo Sirainen schrieb:
On Tue, 2005-04-26 at 12:02 +0200, Jakob Curdes wrote:
we use dovecot on a regular basis. Sometimes I end up with a user telling me he does not get mail. When I look after the problem, I always find leading blank lines in the mbox file.
At the beginning of the file?
0.99.x's mbox handling isn't too good and it's known to cause this sometimes. 1.0-stable releases have rewritten mbox support which fixes this.
Yes, at the beginning. Sometimes it's one line, sometimes 20 or 30. What would I have to change before switching to the 1.0 branch ? We use maildir, mbox for the inboxes, ssl, no other peculiarities.
Yours, Jakob Curdes
At the beginning of the file?
0.99.x's mbox handling isn't too good and it's known to cause this sometimes. 1.0-stable releases have rewritten mbox support which fixes this.
But why does this phenomenon start now ? We had this once in a while, now I have 3-4 mailboxes per day that are affected. Could it be a load-related problem ?
Yours, JC
On Tue, 2005-04-26 at 16:19 +0200, Jakob Curdes wrote:
At the beginning of the file?
0.99.x's mbox handling isn't too good and it's known to cause this sometimes. 1.0-stable releases have rewritten mbox support which fixes this.
But why does this phenomenon start now ? We had this once in a while, now I have 3-4 mailboxes per day that are affected. Could it be a load-related problem ?
No idea. If I knew what the problem was and when it would happen, I would have fixed it.
No idea. If I knew what the problem was and when it would happen, I would have fixed it.
Ok, when I upgrade to 1.0-stable, do I have to chenge something ? I have to get rid of this problem quickly as the phone keeps ringing here. I try to monitor my maillog but obviously I do not see every LF error.
Yours, JC
Is there a way to restrict access to imap on a per-user basis ? Situation is as following : internal server w/many users, some users work also from home. I would like to make their mailboxes accessible from outside via imaps, but only for selected users with strong passwords. If dovecot would run via inetd, I could use hosts.allow patterns for this. Any idea or is this just not possible ? (I know the easiest way would be to connect via VPN but thst is not possible at the moment).
Yours, Jakob Curdes
On Tue, 2005-04-26 at 17:50 +0200, Jakob Curdes wrote:
Is there a way to restrict access to imap on a per-user basis ? Situation is as following : internal server w/many users, some users work also from home. I would like to make their mailboxes accessible from outside via imaps, but only for selected users with strong passwords. If dovecot would run via inetd, I could use hosts.allow patterns for this. Any idea or is this just not possible ? (I know the easiest way would be to connect via VPN but thst is not possible at the moment).
It had been possible to run Dovecot via inetd at some point, but looks like it's pretty broken right now. I tried to fix it but there are just too may bugs with it :)
But if it's enough to do the check after authentication, you can use with 1.0-stable/test:
protocol imap { mail_executable = /usr/sbin/tcpd /usr/libexec/dovecot/imap }
or with 0.99.x (although I'm not sure if it actually allows parameters.. if not, replace it with a binary that execs them):
imap_executable = /usr/sbin/tcpd /usr/libexec/dovecot/imap
Another way that would already work is to creatively use %r variable and SQL passdb.
Another way that would already work is to creatively use %r variable and SQL passdb.
Can you elaborate on that one ? The inetd way would not be my favorite at all. A third way would be to use iptables with user recognition, but this would affect only the imap or pop process, not the login as it runs under "dovecot".
JC
On Tue, 2005-04-26 at 23:06 +0200, Jakob Curdes wrote:
Another way that would already work is to creatively use %r variable and SQL passdb.
Can you elaborate on that one ?
something like:
pass_query = select user, password from users where user = '%u' and (allow_external_access = 'Y' or '%r' like '10.%')
On Tue, 2005-04-26 at 17:33 +0200, Jakob Curdes wrote:
No idea. If I knew what the problem was and when it would happen, I would have fixed it.
Ok, when I upgrade to 1.0-stable, do I have to chenge something ? I have to get rid of this problem quickly as the phone keeps ringing here. I try to monitor my maillog but obviously I do not see every LF error.
http://wiki.dovecot.org/moin.cgi/UpgradingDovecot
..and get the few minutes old 1.0-stable which fixes a stupid little bug causing bad performance problems.
:-)) I read that one already. Problem is that I could not figure out what
"Maildir-only: The filename that stores user subscriptions has changed from .subscriptions to subscriptions (The former .customflags file is not used at all in the 1.0-stable implementation.)"
means. Do I have to change all the .subscription filenames to prevent people losing their folder subscriptions?
Yours, Jakob Curdes
On Tue, 2005-04-26 at 21:39 +0200, Jakob Curdes wrote:
:-)) I read that one already. Problem is that I could not figure out what
"Maildir-only: The filename that stores user subscriptions has changed from .subscriptions to subscriptions (The former .customflags file is not used at all in the 1.0-stable implementation.)"
means. Do I have to change all the .subscription filenames to prevent people losing their folder subscriptions?
Only with maildir. With mbox it's still called .subscriptions.
On Tue, Apr 26, 2005 at 06:16:43PM +0300, Timo Sirainen wrote:
On Tue, 2005-04-26 at 16:19 +0200, Jakob Curdes wrote:
At the beginning of the file?
0.99.x's mbox handling isn't too good and it's known to cause this sometimes. 1.0-stable releases have rewritten mbox support which fixes this.
But why does this phenomenon start now ? We had this once in a while, now I have 3-4 mailboxes per day that are affected. Could it be a load-related problem ?
No idea. If I knew what the problem was and when it would happen, I would have fixed it.
Does it detect writing to a zero-length existing file?
You might consider being liberal when opening an mbox file, and skipping any blank lines prior to the first "From_", although this might be unsettling to some people.
Kinda funny: I once encountered a UNIX mail utility that would always append messages to mbox files with a leading blank line even if the file was empty (i.e. it would create new mbox files with a leading blank line), but would refuse to open any mbox files that began with a blank line. Dunno how it survived in the wild... I suspect it was some broken interim version, but at any rate I hacked it to keep it from writing the blank line, *and* to skip blank lines when opening up an mbox.
mm
Kinda funny: I once encountered a UNIX mail utility that would always append messages to mbox files with a leading blank line even if the file was empty (i.e. it would create new mbox files with a leading blank line), but would refuse to open any mbox files that began with a blank line. Dunno how it survived in the wild... I suspect it was some broken interim version, but at any rate I hacked it to keep it from writing the blank line, *and* to skip blank lines when opening up an mbox.
Would you be able to disclose the name of that utility in case I am using it inadvertently ??
JC
On Tue, Apr 26, 2005 at 06:43:54PM +0200, Jakob Curdes wrote:
Kinda funny: I once encountered a UNIX mail utility that would always append messages to mbox files with a leading blank line even if the file was empty (i.e. it would create new mbox files with a leading blank line), but would refuse to open any mbox files that began with a blank line. Dunno how it survived in the wild... I suspect it was some broken interim version, but at any rate I hacked it to keep it from writing the blank line, *and* to skip blank lines when opening up an mbox.
Would you be able to disclose the name of that utility in case I am using it inadvertently ??
Yes but then I'd have to correct the inaccurate memory :-)
As it turns out, it really was a case of one utility (elm) writing to any empty mbox with the blank line, which it itself could read but other programs barfed on. So there were patches to more than one program.
The one writing the blank lines was elm 2.5PL3. I don't have any idea how current that version is; I haven't used elm in a while. But while I'm here, the patch was:
---cut--- *** savecopy.c.orig Tue Aug 1 17:03:25 2000 --- savecopy.c Wed Mar 12 11:03:11 2003
*** 268,274 **** --- 268,284 ---- #ifdef MMDF fputs(MSG_SEPARATOR, fp_copy); #else +
- # ifndef MV_COMM /* mem 20030311 */ putc('\n', fp_copy);
- # else
/* MV_COMM -- If it's an empty file, don't start out with a newline,else we create a file that some mailers can't read!*/if ( ftell( fp_copy ) > 0L )putc('\n', fp_copy);- # endif /* MV_COMM */
- #endif /* MMDF */ time(&now); fprintf(fp_copy, "From %s %s", user_name, ctime(&now)); ---cut---
I no longer seem to have that specific patch to other utilitie(s) (probably because the problem went away once elm was patched), but I do have some others:
/bin/mail had issues reading some "From_" line date formats;
mutt-1.4.2.1 had at least one eggregious "From_"-line reading issue, which is that it would get fooled by From lines which didn't follow a blank line.
I think that none of those are against anywhere-near-current versions, though.
Probably way off topic, sorry..
mm
The one writing the blank lines was elm 2.5PL3. I don't have any idea how current that version is; I haven't used elm in a while. But while I'm here, the patch was:
Wow. I wonder how many people still know what "elm" is....
Probably way off topic, sorry..
mm
I fear the same. Anyway elm is _not_ the culprit in my case ...
JC
On Tue, 2005-04-26 at 12:29 -0400, Mark E. Mallett wrote:
No idea. If I knew what the problem was and when it would happen, I would have fixed it.
Does it detect writing to a zero-length existing file?
Yep.
You might consider being liberal when opening an mbox file, and skipping any blank lines prior to the first "From_", although this might be unsettling to some people.
Maybe.. But it shouldn't happen so there shouldn't be any need to do that.. :) I think a better idea is to just upgrade to 1.0-stable nowadays if someone has this problem.
Kinda funny: I once encountered a UNIX mail utility that would always append messages to mbox files with a leading blank line even if the file was empty (i.e. it would create new mbox files with a leading blank line), but would refuse to open any mbox files that began with a blank line. Dunno how it survived in the wild... I suspect it was some broken interim version, but at any rate I hacked it to keep it from writing the blank line, *and* to skip blank lines when opening up an mbox.
Sounds like first versions of Dovecot ;)
--On Tuesday, April 26, 2005 12:02 PM +0200 Jakob Curdes jc@info-systems.de wrote:
we use dovecot on a regular basis. Sometimes I end up with a user telling me he does not get mail. When I look after the problem, I always find leading blank lines in the mbox file.
The one time I've encountered corruption, the problem was a long string of nulls at the top, not blank lines.
--On Tuesday, April 26, 2005 9:38 AM -0700 Kenneth Porter shiva@sewingwitch.com wrote:
The one time I've encountered corruption, the problem was a long string of nulls at the top, not blank lines.
Second time this happened, last night. User's "Trash" folder had a string of 15174 nul's at the top. Using Mozilla 1.8b on Win2kSP4 against dovecot-0.99.13-4.FC2.
On Apr 28, 2005, at 15:14, Kenneth Porter wrote:
--On Tuesday, April 26, 2005 9:38 AM -0700 Kenneth Porter shiva@sewingwitch.com wrote:
The one time I've encountered corruption, the problem was a long string of nulls at the top, not blank lines.
Second time this happened, last night. User's "Trash" folder had a string of 15174 nul's at the top. Using Mozilla 1.8b on Win2kSP4 against dovecot-0.99.13-4.FC2.
Is the Folder accessed by dovecod via NFS?
If that is the case, this bug might be caused by the NFS client caching filesystem metadata.
For example, if dovecot thinks it is opening a 1200 byte file (by believing stat(2) to give correct info), while the file on the NFS server actually is 1800 bytes, strange things might happen, depending on how the different info from stat(2) and actually reading from the file is used / handled.
It might help adjusting the mount options (actimeo?).
Regards, Frode Nordahl
--On Saturday, April 30, 2005 6:39 PM +0200 Frode Nordahl frode@nordahl.net wrote:
Is the Folder accessed by dovecod via NFS?
Nope, this is a local ext3 partition.
participants (5)
-
Frode Nordahl
-
Jakob Curdes
-
Kenneth Porter
-
Mark E. Mallett
-
Timo Sirainen