[Dovecot] DoveCot IMAP and "inconsistent state" messages
I need some help troubleshooting this problem. It only shows up with IMAP connections. I initially thought it was related to SquirrelMail (because it gives me an 'EXPUNGE' error), but after attempting to send IMAP commands directly to the server as shown below, I'm thinking there is something else going on.
The system configuration is PostFix as the MTA delivering to procmail in the mailbox (maildir format), with Dovecot handling POP3 and IMAP
Postfix is version 2.3.3 Dovecot is version 1.0.rc15 procmail is version 3.22 2001/09/10 kernel is 2.6.18-53.1.14.el5, CentOS 5
There is additional info about the system configuration after the command sequence and log entries below. I've left my typos in, just in case they somehow bear on the problem. I omitted about 2050 messages from the mailbox FETCH listing, since they were all identical but for the message number, and I didn't figure you'd want to read all of that. :)
Command sequence: [me@myserver~]$ telnet locahost 143 locahost/143: Name or service not known [me@myserver~]$ telnet localhost 143 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'.
- OK myserver awaiting command
login me mypass login BAD Error in IMAP command received by server. a001 login me mypass a001 OK Logged in. a002 list "Mail" "*" a002 OK List completed. a003 list "Folders" "*" a003 OK List completed. a004 list "Inbox" a004 BAD Error in IMAP command LIST: Invalid arguments. a005 list "Inbox" "*"
- LIST (\HasNoChildren) "." "INBOX"
a005 OK List completed. a006 list "" "*"
- LIST (\HasNoChildren) "." "Drafts"
- LIST (\HasNoChildren) "." "Spam"
- LIST (\HasNoChildren) "." "Trash"
- LIST (\HasNoChildren) "." "Sent"
- LIST (\HasNoChildren) "." "INBOX"
a006 OK List completed.
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)]
Flags permitted.
- 2088 EXISTS
- 1 RECENT
- OK [UNSEEN 2088] First unseen.
- OK [UIDVALIDITY 1190322195] UIDs valid
- OK [UIDNEXT 89700] Predicted next UID
a007 OK [READ-WRITE] Select completed. a008 FETCH l:* FLAGS a008 BAD Error in IMAP command FETCH: Invalid messageset a008 FETCH 1:* FLAGS
- 1 FETCH (FLAGS (\Seen))
- 2 FETCH (FLAGS (\Seen))
- 3 FETCH (FLAGS (\Seen))
- 4 FETCH (FLAGS (\Seen))
- 5 FETCH (FLAGS (\Seen))
- 6 FETCH (FLAGS (\Seen))
. . .
- 2083 FETCH (FLAGS (\Seen))
- 2084 FETCH (FLAGS (\Seen))
- 2085 FETCH (FLAGS (\Seen))
- 2086 FETCH (FLAGS (\Seen))
- 2087 FETCH (FLAGS (\Seen))
- 2088 FETCH (FLAGS (\Recent))
- BYE Mailbox is in inconsistent state, please relogin.
Connection closed by foreign host.
These are the corresponding entries from my mail syslog during the same time period. I've included the postfix entries, as they might be relevant to the problem.
Mar 31 13:55:14 myserver dovecot: imap-login: Login: user=<me>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Mar 31 13:55:33 myserver dovecot: pop3-login: Login: user=<me>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy Mar 31 13:55:35 myserver dovecot: POP3(me): Disconnected: Logged out top=0/0, retr=1/18590, del=2/2089, size=10863040 Mar 31 13:56:29 myserver postfix/smtpd[26589]: connect from unknown[79.165.160.211] Mar 31 13:56:30 myserver postfix/smtpd[26589]: 1A57E2F7E6: client=unknown[79.165.160.211] Mar 31 13:56:30 myserver postfix/cleanup[26646]: 1A57E2F7E6: message-id=000901c89360$061afdf8$11e5c1ae@qawbud Mar 31 13:56:35 myserver postfix/qmgr[2254]: 1A57E2F7E6: from=amsonson@impsat.com.ar, size=61232, nrcpt=1 (queue active) Mar 31 13:56:36 myserver postfix/smtpd[26651]: connect from localhost.localdomain [127.0.0.1] Mar 31 13:56:36 myserver postfix/smtpd[26651]: C98262F893: client=localhost.localdomain [127.0.0.1] Mar 31 13:56:36 myserver postfix/cleanup[26646]: C98262F893: message-id=000901c89360$061afdf8$11e5c1ae@qawbud Mar 31 13:56:36 myserver postfix/smtpd[26651]: disconnect from localhost.localdomain [127.0.0.1] Mar 31 13:56:36 myserver amavis[26499]: (26499-03) Passed SPAMMY, [79.165.160.211] [79.165.160.211] amsonson@impsat.com.ar -> me@digitalvoice.com, Message-ID: 000901c89360$061afdf8$11e5c1ae@qawbud, mail_id: 47TfQi44++zM, Hits: 17.608, size: 61232, queued_as: C98262F893, 1372 ms Mar 31 13:56:36 myserver postfix/smtp[26648]: 1A57E2F7E6: to=me@mydomain.com, orig_to=postmaster@digitalvoice.com, relay=127.0.0.1[127.0.0.1]:10024, delay=6.8, delays=5.4/0.01/0.01/1.4, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as C98262F893)Mar 31 13:56:36 myserver postfix/qmgr[2254]: C98262F893: from=amsonson@impsat.com.ar, size=62235, nrcpt=1 (queue active) Mar 31 13:56:36 myserver postfix/qmgr[2254]: 1A57E2F7E6: removed Mar 31 13:56:36 myserver postfix/local[26653]: C98262F893: to=me@mydomain.com, relay=local, delay=0.12, delays=0.1/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) Mar 31 13:56:36 myserver postfix/qmgr[2254]: C98262F893: removed Mar 31 13:58:07 myserver dovecot: pop3-login: Login: user=<someone>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy Mar 31 13:58:07 myserver dovecot: POP3(someone): Disconnected: Logged out top=0/0, retr=0/0, del=0/291, size=11570114 Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,) Mar 31 14:00:58 myserver dovecot: IMAP(me): Disconnected: Mailbox is in inconsistent state, please relogin.
output of dovecot -n: # /etc/dovecot.conf listen: * login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: myserver awaiting command mail_location: maildir:~/.Maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postdrop master:
So, now I've got two questions:
- Is accessing a mailbox via POP3 and IMAP at the same time considered to be a Bad Thing?
- Is having procmail drop messages directly into the directory that the IMAP process is scanning a Bad Thing?
- Is there a better/preferred way to have Postfix/procmail deliver mail to the maildir that will keep Dovecot IMAP happy? (There don't seem to be any issues with pure POP3 access).
Thanks in advance for your help, and if there's anything else you need to know that I can provide, please ask. I've tried to provide as much information up front as I can.
Later, Chris
Charles Marcus wrote:
On 3/31/2008, Chris Richards (gizmo@giz-works.com) wrote:
Dovecot is version 1.0.rc15
Upgrade please - rc15 is very old...
Err, that's the newest thing in the yum repository, and if I go compiling code that isn't 'official' (i.e. doesn't come from the yum repository), I'll get myself in an awful lot of trouble. Management seems to be under the strange belief that only code from the repository it is 'safe'.
So, the question is, how hard do I need to fight in order to get this done?
Later, Chris
on 3-31-2008 1:31 PM Chris Richards spake the following:
Charles Marcus wrote:
On 3/31/2008, Chris Richards (gizmo@giz-works.com) wrote:
Dovecot is version 1.0.rc15
Upgrade please - rc15 is very old...
Err, that's the newest thing in the yum repository, and if I go compiling code that isn't 'official' (i.e. doesn't come from the yum repository), I'll get myself in an awful lot of trouble. Management seems to be under the strange belief that only code from the repository it is 'safe'.
So, the question is, how hard do I need to fight in order to get this done?
Later, Chris
If you are using a RHEL or clone, atrpms.net has a newer version in their repository.
Scott Silva wrote:
on 3-31-2008 1:31 PM Chris Richards spake the following:
Charles Marcus wrote:
On 3/31/2008, Chris Richards (gizmo@giz-works.com) wrote:
Dovecot is version 1.0.rc15
Upgrade please - rc15 is very old...
Err, that's the newest thing in the yum repository, and if I go compiling code that isn't 'official' (i.e. doesn't come from the yum repository), I'll get myself in an awful lot of trouble. Management seems to be under the strange belief that only code from the repository it is 'safe'.
So, the question is, how hard do I need to fight in order to get this done?
Later, Chris
If you are using a RHEL or clone, atrpms.net has a newer version in their repository.
Oh bloody hell. WHY am I fighting with a dovecot version that is 1 1/2 flippin' years old?!?!
Suits. I HATES 'em, I tells ya!
I'll be back when I've figured out how to convince the CTO that this stupid policy is causing a lot of unnecessary grief, considering this issue is probably already fixed. Thanks for your time, guys.
Later, Chris
At 3:31 PM -0500 3/31/08, Chris Richards wrote:
Charles Marcus wrote:
On 3/31/2008, Chris Richards (gizmo@giz-works.com) wrote:
Dovecot is version 1.0.rc15
Upgrade please - rc15 is very old...
Err, that's the newest thing in the yum repository, and if I go compiling code that isn't 'official' (i.e. doesn't come from the yum repository), I'll get myself in an awful lot of trouble. Management seems to be under the strange belief that only code from the repository it is 'safe'.
So, the question is, how hard do I need to fight in order to get this done?
That's a question about the competence of the people maintaining that repository. Presumably these would be the people who blessed a pre-release version of Dovecot almost 18 months ago, in a period when such versions were being released every few days *due to bugs*, and who have not updated their build at any time since. It seems to me that these are not people who should be tasked or trusted with being the gatekeepers of software deployment, as that seems to be demonstrably beyond their competence.
I don't say that as a slur on the people running the CentOS project, but rather to note that the packaging service they offer is unsuited to the role your management has given it in your company's environment.
More to the point:
At 3:10 PM -0500 3/31/08, Chris Richards wrote:
So, now I've got two questions:
- Is accessing a mailbox via POP3 and IMAP at the same time considered to be a Bad Thing?
No.
- Is having procmail drop messages directly into the directory that the IMAP process is scanning a Bad Thing?
No.
- Is there a better/preferred way to have Postfix/procmail deliver mail to the maildir that will keep Dovecot IMAP happy? (There don't seem to be any issues with pure POP3 access).
Buffer overflow... was only expecting 2 questions. :)
But again, the answer is: No.
Note the nature of the problem: a new message has been assigned a UID that isn't higher than all other existing UID's in that mailbox. That is a violation of the IMAP spec (RFC3501 sec. 2.3.1.1) and Dovecot did it. Nothing else sets UID's for messages. Once Dovecot has noticed that it has made that mistake, it really needs to kick out any IMAP sessions with that mailbox selected, because they are fairly likely to have some impression that the <mailbox><UIDVALIDITY><UID> triple including that new UID refers to a particular old and probably expunged message and may end up trying to work with it as if the new message is the old one. Dovecot will provide a new UIDVALIDITY value to any later sessions selecting that mailbox to prevent that sort of mistake, and any clients seeing that will have to resynch their mappings of messages to UID triple.
The only solution to Dovecot making that sort of error is for Dovecot to be fixed. The release announcements for later 1.0rc* versions seem to describe such fixes.
bill@scconsult.com
Hi,
El Martes, 1 de Abril de 2008 a las 04:14, Bill Cole escribió:
That's a question about the competence of the people maintaining that repository. Presumably these would be the people who blessed a pre-release version of Dovecot almost 18 months ago, in a period when such versions were being released every few days *due to bugs*, and who have not updated their build at any time since. It seems to me that these are not people who should be tasked or trusted with being the gatekeepers of software deployment, as that seems to be demonstrably beyond their competence.
RedHat (and CentOS) has his own policy about releases, and more or less it is: "no update will break a working instalation". So they try to port any security patches to their running versions -and this is a lot of work, they have their own forked version of almost any package!-, but almost never add any new funcionality. This policy has one great point: it's easy to understand, and it gives few surprises. And this is great most of the times.
Pre-1.0 Dovecot is the kind of software that doesn't fit well in that policy: a lot of changes, and no standard stable version. So they chose one version -1.0rc15 in this case-, because their other option was not including dovecot. That's exactly why I'm compiling dovecot from source, but I usually like the default policy.
Aaaaaaaaaagur.
Joseba Torre. CIDIR Bizkaia.
Joseba Torre wrote:
Hi,
El Martes, 1 de Abril de 2008 a las 04:14, Bill Cole escribió:
That's a question about the competence of the people maintaining that repository. Presumably these would be the people who blessed a pre-release version of Dovecot almost 18 months ago, in a period when such versions were being released every few days *due to bugs*, and who have not updated their build at any time since. It seems to me that these are not people who should be tasked or trusted with being the gatekeepers of software deployment, as that seems to be demonstrably beyond their competence.
RedHat (and CentOS) has his own policy about releases, and more or less it is: "no update will break a working instalation". So they try to port any security patches to their running versions -and this is a lot of work, they have their own forked version of almost any package!-, but almost never add any new funcionality. This policy has one great point: it's easy to understand, and it gives few surprises. And this is great most of the times.
Pre-1.0 Dovecot is the kind of software that doesn't fit well in that policy: a lot of changes, and no standard stable version. So they chose one version -1.0rc15 in this case-, because their other option was not including dovecot. That's exactly why I'm compiling dovecot from source, but I usually like the default policy.
Aaaaaaaaaagur.
I guess it will drawn groans, but this is largely the reason that I chose Gentoo for my server distros.
Roughly speaking with most main distros you get a "stuck in stone" set of packages + bug fixes for those packages each week. This causes few surprises, but it means that if you need some new feature you are immediately into compiling from source. Additionally big version upgrades seem to be approximately once a year and it appears in most cases this is a take the server down, upgrade it, bring it up and check the configs kind of process (ie fairly major)
Gentoo goes the other way - everything remains permanently unstable (if you want to think in those terms). Every time you upgrade you get the latest and greatest versions... There is a package masking process so that there isn't "too" much suprise with stuff breaking and local patches are applied to keep upgrades as simple as possible. This falls somewhere between a Redhat weekly upgrade kind of stability and a redhat "operating system version upgrade" experience - ie I wouldn't trust it to run unattended every sunday without someone watching it, yet on the other hand I can take a year old server and upgrade all the software on it to latest stable in an hour without (usually) taking it down.
Pays your money and takes your choice I guess... Suits my needs though (I often need new features and up to date software)
Ed W
On 4/1/2008, Ed W (lists@wildgooses.com) wrote:
I guess it will drawn groans, but this is largely the reason that I chose Gentoo for my server distros.
No groan here... I love the flexibility Gentoo gives... you can easily leave the primary system at 'stable', then just set certain packages to unstable for the latest/greatest.
My servers have been running nonstop for over 3 years, with just a few minor hiccups now and then that are quickly resolved by digging through logs and/or hitting google and/or the forums...
To be fair, most distros *do* have the ability to use extra 'unstable' repos, at least for most major packages, and if I wasn't using Gentoo, I would at *least* be using those...
Charles Marcus wrote:
No groan here... I love the flexibility Gentoo gives... you can easily leave the primary system at 'stable', then just set certain packages to unstable for the latest/greatest.
My servers have been running nonstop for over 3 years, with just a few minor hiccups now and then that are quickly resolved by digging through logs and/or hitting google and/or the forums...
To be fair, most distros *do* have the ability to use extra 'unstable' repos, at least for most major packages, and if I wasn't using Gentoo, I would at *least* be using those...
My other box is Gentoo, and I quite like it. The biggest problem I had with it was that about a year ago when I was give control of it, it hadn't been synced in like 3 years, and it was so woefully out of date that when I tried to emerge -upDN world, it couldn't reliably upgrade because some packages no longer existed, including core packages (and the system profile).
Other than that, the only problem I've had was when a Metalog (sysloger) update came out that caused my entire system to hang at boot because the portage package didn't properly move a couple of files.
Later, Chris
Chris Richards wrote:
My other box is Gentoo, and I quite like it. The biggest problem I had with it was that about a year ago when I was give control of it, it hadn't been synced in like 3 years, and it was so woefully out of date that when I tried to emerge -upDN world, it couldn't reliably upgrade because some packages no longer existed, including core packages (and the system profile).
I was once in this position with a redhat box and it turned out that you can't even update it from 3 years ago because they make you re-install every couple of years to put on a new OS...
I can kind of imagine what might of happened though - was it a hardened profile by any chance? There was some trickiness with upgrading Python a couple of years back (as in if you were that far out of date) where portage needed a newer version than the older one would install. It was fairly simple to work around if you were familiar with the issues, but yes I agree it wasn't ideal.
Other than that, the only problem I've had was when a Metalog (sysloger) update came out that caused my entire system to hang at boot because the portage package didn't properly move a couple of files.
Sounds like you aren't using vservers yet?
I build a minimal server on the bare iron and then immediately tar it up and copy it into /vservers/template. Then I use the vserver project to make it simple to "boot" this chrooted version and customise it a little and that then forms the basis for all my real servers. I usually keep about 3 template servers, one vanila-ish install, another setup for PHP apps, and another for some rails apps.
Additionally you can easily test out your latest upgrade by simply copying a vserver somewhere, boot it, run the upgrade and then shut it down again. Bonus points for using a central package dir so that actually when you go back to your proper vserver and run the upgrade it actually uses the binary packages and updates in a few seconds...
I bind mount all the dirs in my vservers which contain data to some other central storage. This means for example my dovecot vserver is quite small and quick to take a copy, but when you are inside it I bind mount all my maildirs into place. This makes it much simpler to copy vservers around and boot them up optionally pointing at the same live data as the original vserver (at the same time if you wish)
There is nothing stopping you from starting to convert your current servers to this setup. Just get a compatible kernel on there at your next opportunity. Then grab a roughly suitable stage 4 and unpack it somewhere. "boot it" and recompile it to the state you actually want as your base template. Then copy it a couple of times and start moving live services into each vserver one by one. So you can have DNS in one, mail in another, amavis scanning in another, etc. It's probably fairly easy to move services one by one this way without any great hassle and eventually you will be all converted except that the base OS is more messy than it needs. Still it will then be easy to migrate the vservers between real machines and you can clean down that physical server and easily rebuild the base os without anyone noticing...
Good luck
Ed W
On Mon, 2008-03-31 at 15:10 -0500, Chris Richards wrote:
Dovecot is version 1.0.rc15
I agree with others that it would be a good idea to upgrade from rc15, but..
Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,)
This is a procmail configuration problem: http://wiki.dovecot.org/MailboxFormat/Maildir#procmail
Timo Sirainen wrote:
On Mon, 2008-03-31 at 15:10 -0500, Chris Richards wrote:
Dovecot is version 1.0.rc15
I agree with others that it would be a good idea to upgrade from rc15, but..
I'm working on this....
Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,)
This is a procmail configuration problem: http://wiki.dovecot.org/MailboxFormat/Maildir#procmail
Ok, As I read this, basically you're saying that I need to append the / to my delivery dirs? Or that I need to append the / and only deliver to Maildir, instead of Maildir/new?
Here's my .procmailrc # .procmailrc # routes incoming mail to appropriate destinations PATH=/usr/bin:/usr/sbin:/bin:/sbin MAILDIR=$HOME/.Maildir # all mailboxes live here DEFAULT=$MAILDIR/new/ # This is where incoming mail goes LOGFILE=$HOME/.procmail_log DELIVER="/usr/lib/dovecot/deliver" SHELL=/bin/sh :0:
- ^X-Spam-Flag: YESS
- ^X-Spam-Level: *******************
$MAILDIR/.Spam/new/
I changed the .procmailrc to append the / to the dir. Is that all I need to do?
Thanks, Chris
Chris Richards wrote:
Ok, As I read this, basically you're saying that I need to append the / to my delivery dirs? Or that I need to append the / and only deliver to Maildir, instead of Maildir/new?
Here's my .procmailrc # .procmailrc # routes incoming mail to appropriate destinations PATH=/usr/bin:/usr/sbin:/bin:/sbin MAILDIR=$HOME/.Maildir # all mailboxes live here DEFAULT=$MAILDIR/new/ # This is where incoming mail goes LOGFILE=$HOME/.procmail_log DELIVER="/usr/lib/dovecot/deliver" SHELL=/bin/sh :0:
- ^X-Spam-Flag: YESS
- ^X-Spam-Level: *******************
$MAILDIR/.Spam/new/
I changed the .procmailrc to append the / to the dir. Is that all I need to do?
Thanks, Chris
Never mind. I've figured this out. The above configuratioon seems to do Bad Things to mail delivery. I've change my config to the following, and things seem to be working:
# .procmailrc # routes incoming mail to appropriate destinations PATH=/usr/bin:/usr/sbin:/bin:/sbin MAILDIR=$HOME/.Maildir # all mailboxes live here DEFAULT=$MAILDIR/new/ # This is where incoming mail goes LOGFILE=$HOME/.procmail_log DELIVER="/usr/lib/dovecot/deliver" SHELL=/bin/sh :0:
- ^X-Spam-Flag: YESS
- ^X-Spam-Level: *******************
$MAILDIR/.Spam/new/
Thanks, Chris
Timo Sirainen wrote:
On Tue, 2008-04-01 at 14:55 -0500, Chris Richards wrote:
DEFAULT=$MAILDIR/new/ # This is where incoming mail goes $MAILDIR/.Spam/new/
Do these really work? They're not writing mails to new/new/ directory? I think they should have been without the "new/" part.
Yeah, that was what I discovered. *g*
Later, Chris
Chris Richards wrote:
Chris Richards wrote: Never mind. I've figured this out. The above configuratioon seems to do Bad Things to mail delivery. I've change my config to the following, and things seem to be working:
# .procmailrc # routes incoming mail to appropriate destinations PATH=/usr/bin:/usr/sbin:/bin:/sbin MAILDIR=$HOME/.Maildir # all mailboxes live here DEFAULT=$MAILDIR/new/ # This is where incoming mail goes LOGFILE=$HOME/.procmail_log DELIVER="/usr/lib/dovecot/deliver" SHELL=/bin/sh :0:
- ^X-Spam-Flag: YESS
- ^X-Spam-Level: *******************
$MAILDIR/.Spam/new/
Thanks, Chris
whoops. I didn't edit this properly. The proper config is below # .procmailrc # routes incoming mail to appropriate destinations PATH=/usr/bin:/usr/sbin:/bin:/sbin MAILDIR=$HOME/.Maildir # all mailboxes live here DEFAULT=$MAILDIR/ # This is where incoming mail goes LOGFILE=$HOME/.procmail_log DELIVER="/usr/lib/dovecot/deliver" SHELL=/bin/sh :0:
- ^X-Spam-Flag: YESS
- ^X-Spam-Level: *******************
$MAILDIR/.Spam/
Later, Chris
At 11:20 AM +0300 4/1/08, Timo Sirainen wrote:
On Mon, 2008-03-31 at 15:10 -0500, Chris Richards wrote:
Dovecot is version 1.0.rc15
I agree with others that it would be a good idea to upgrade from rc15, but..
Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,)
This is a procmail configuration problem: http://wiki.dovecot.org/MailboxFormat/Maildir#procmail
That's a surprise. I saw that and considered it, but I interpreted it to mean that mh delivery from procmail into a Maildir's 'new' subdirectory created a file with the inode *number* in the filename.
I was suspecting the UID lock issue fixed in rc26: http://www.dovecot.org/list/dovecot-news/2007-March/000035.html
Which does not explain how Dovecot could reuse a UID, but that just seemed weird and wrong in any case...
Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,)
This is a procmail configuration problem: http://wiki.dovecot.org/MailboxFormat/Maildir#procmail
Just wanted to say thanks for the help. You guys have made me a hero.
After reading the PostFix group and this group, and spending about a day sorting through a bunch of spam configuration stuff, I've got our mail server purring like a kitten. So I went to see the CTO this afternoon and explained the situation with the policy about RPMs. After him commenting that he had noticed a substantial drop in spam, his response was basically "You seem to have a clue; do whatever you think is appropriate."
Thanks again, guys.
Later, Chris
on 4-2-2008 1:29 PM Chris Richards spake the following:
Mar 31 14:00:58 myserver dovecot: IMAP(me): Maildir /home/me/.Maildir sync: UID inserted in the middle of mailbox (85953 > 85053, file = msg.XL7B:2,)
This is a procmail configuration problem: http://wiki.dovecot.org/MailboxFormat/Maildir#procmail
Just wanted to say thanks for the help. You guys have made me a hero.
After reading the PostFix group and this group, and spending about a day sorting through a bunch of spam configuration stuff, I've got our mail server purring like a kitten. So I went to see the CTO this afternoon and explained the situation with the policy about RPMs. After him commenting that he had noticed a substantial drop in spam, his response was basically "You seem to have a clue; do whatever you think is appropriate."
Thanks again, guys.
Later, Chris
So you did it backwards, you dazzled him with brilliance instead of baffling him with bull$h!t! ;-P Congrats and keep up the good work!
I need some help troubleshooting this problem. It only shows up with IMAP connections. I initially thought it was related to SquirrelMail (because it gives me an 'EXPUNGE' error), but after attempting to send IMAP commands directly to the server as shown below, I'm thinking there is something else going on.
Did you ever figure out this problem? I'm having what appears to be the exact same issue with SquirrelMail. Running Dovecot 1.0.14 (Maildir) and Squirrelmail 1.4.15, both from and on Debian. Procmail is occasionally dropping in mail in the background, which the previous message indicated might be related.
Debug log shows:
Jul 21 14:27:41 betty dovecot: IMAP(foobar): Maildir /var/vpopmail/domains/foobar.baz/foobar/Maildir sync: UID inserted in the middle of mailbox (4412 > 4385, file = 1214817167.16333_0.betty:2,RST) Jul 21 14:27:41 betty dovecot: IMAP(foobar): Disconnected: Mailbox is in inconsistent state, please relogin.
Symptom is us the typically unhelpful expunge error message from SM, or even connection dropped on regular fetches, and user is logged out.
Any help would be greatly appreciated, as none of my testing thus far have made any difference, and I can't seem to find any hints elsewhere. Could upgrading to 1.1 help at all? (I'd rather not try unless I know for sure)
TIA
On Tue, 2008-07-22 at 01:11 +0200, Christer Mjellem Strand wrote:
Jul 21 14:27:41 betty dovecot: IMAP(foobar): Maildir /var/vpopmail/domains/foobar.baz/foobar/Maildir sync: UID inserted in the middle of mailbox (4412 > 4385, file = 1214817167.16333_0.betty:2,RST)
Show your dovecot -n output? I suppose the users don't have direct access to these maildirs, and nothing else besides Dovecot and procmail touches them?
This error means that Dovecot lost that file and thought it was expunged. But sometimes afterwards it saw the file again.
Any help would be greatly appreciated, as none of my testing thus far have made any difference, and I can't seem to find any hints elsewhere. Could upgrading to 1.1 help at all? (I'd rather not try unless I know for sure)
v1.1 might not remove the root problem, but it will handle this better by renaming the file and showing it to client as a new message instead of returning "inconsistent state" error.
Jul 21 14:27:41 betty dovecot: IMAP(foobar): Maildir /var/vpopmail/domains/foobar.baz/foobar/Maildir sync: UID inserted in the middle of mailbox (4412 > 4385, file = 1214817167.16333_0.betty:2,RST)
Show your dovecot -n output?
Sorry, should have included that right away.
betty - ~ # dovecot -n # 1.0.14: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S listen: *:9000 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_greeting_capability: yes login_max_processes_count: 256 first_valid_uid: 89 mail_location: maildir:~/Maildir dotlock_use_excl: yes maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_process_size: 512 mail_plugins: quota imap_quota trash lazy_expunge imap_client_workarounds: outlook-idle delay-newmail namespace: type: private inbox: yes namespace: type: private separator: / prefix: .Trash/ location: maildir:~/Maildir/.Trash hidden: yes namespace: type: private separator: / prefix: .Trash/ location: maildir:~/Maildir/.Trash hidden: yes namespace: type: private separator: / prefix: .Trash/ location: maildir:~/Maildir/.Trash hidden: yes auth default: user: vpopmail verbose: yes passdb: driver: checkpassword args: /data/vpopmail/bin/vchkpw userdb: driver: prefetch plugin: quota: maildir trash: /etc/dovecot/dovecot-trash.conf lazy_expunge: .Trash/ .Trash/ .Trash/
I suppose the users don't have direct access to these maildirs, and nothing else besides Dovecot and procmail touches them?
No, this is qmail with Vpopmail, so all mail is owned by the vpopmail user. Default MDA is qmail-local, but where procmail filters are enabled, it takes over all local delivery, and never hands it back to qmail-local. I haven't actively looked for a pattern yet, but from the top of my head, all users I can think of experiencing this problem use procmail for delivery.
This error means that Dovecot lost that file and thought it was expunged. But sometimes afterwards it saw the file again.
Hm. What is the normal scenario where something like this might happen, if there is such a thing?
Any help would be greatly appreciated, as none of my testing thus far have made any difference, and I can't seem to find any hints elsewhere. Could upgrading to 1.1 help at all? (I'd rather not try unless I know for sure)
v1.1 might not remove the root problem, but it will handle this better by renaming the file and showing it to client as a new message instead of returning "inconsistent state" error.
That does sound more graceful. Squirrelmail shows an error for every dropped connection, so the end result is that users are seeing a whole bunch of error messages, without actually experiencing any problems (from what I've heard). I'd prefer to cure the problem, but if I can't, curing the symptom might be adequate.
Jul 21 14:27:41 betty dovecot: IMAP(foobar): Maildir /var/vpopmail/domains/foobar.baz/foobar/Maildir sync: UID inserted in the middle of mailbox (4412 > 4385, file = 1214817167.16333_0.betty:2,RST)
Show your dovecot -n output?
Sorry, should have included that right away.
betty - ~ # dovecot -n
[..]
lazy_expunge: .Trash/ .Trash/ .Trash/
Just upgraded to 1.1.2, and so far it's looking promising. The slightly more elaborate error messages helped me identify the probable root cause of the problem. I've got lazy_expunge configured to mimic the behavior of my old Courier, where anything deleted anywhere will simply get thrown into Trash, regardless of context. I suspect that this leads to situations where messages deleted from Trash get copied to Trash, leading them to seemingly reappear "out of nowhere".
It'd be great if Dovecot was able to handle this more gracefully, ideally by catching this, being aware of the situation, and ignoring deletions from whatever is defined as the Trash directory by lazy_expunge.
I guess my setup is sorta stupid, and probably not entirely what lazy_expunge was intended for, so if anyone has any other suggestions as to how to achieve this behavior, those would be welcome.
Thanks for your help!
participants (8)
-
Bill Cole
-
Charles Marcus
-
Chris Richards
-
Christer Mjellem Strand
-
Ed W
-
Joseba Torre
-
Scott Silva
-
Timo Sirainen