[Dovecot] Dovecot LDA munging INBOX access times?
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time! Not only do our users rely on this access time being when they last accessed their e-mail, we also rely on it to migrate "inactive" users to cheaper disk. I consider this a major bug for us and wondered if there is some reason why the LDA is doing this. I can understand that in the absence of index files it may choose to build them from scratch and therefore need to access the INBOX, but in normal cases I don't see that being necessary.
It seems to me there should be a few options for how the LDA operates:
Deliver to INBOX, do not create or update the index. This would avoid accessing the INBOX.
Deliver to INBOX, update the index only if it is already current. This would avoid accessing the INBOX. (not sure if this is possible)
Deliver to INBOX, update/create the index as necessary to make it current. This obviously may involve accessing the INBOX.
We would rather not go with option #1 -- what's the point of using Dovecot LDA in that scenario??
Comments appreciated.
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
Steven F Siirila wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time!
Umm, yeah, that's kind of the point. mbox format is one big file with all of the messages concatenated together. So if a new message is delivered, then the INBOX file has to be opened to append the new message. This isn't a bug by any stretch, but rather a different operational method than what you were expecting.
I'm guessing that UW IMAP doesn't retrieve new messages until the user asks for them (e.g. out of /var/mail/something). If you are using deliver, it actually, you know, _delivers_ the message. I think that there is a way to have the same behavior as UW IMAP, but I don't know off the top of my head how to do that (since we use maildir instead). You may want to check the list archives first, rather than claiming dovecot is doing something wrong...
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
On Wed, Jun 13, 2007 at 08:30:52PM -0400, John Peacock wrote:
Steven F Siirila wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time!
Umm, yeah, that's kind of the point. mbox format is one big file with all of the messages concatenated together. So if a new message is delivered, then the INBOX file has to be opened to append the new message.
Opened as WRITE ONLY APPEND. That doesn't update last access time.
This isn't a bug by any stretch, but rather a different operational method than what you were expecting.
I'm not claiming this is a bug, it is a major issue for us though.
I'm guessing that UW IMAP doesn't retrieve new messages until the user asks for them (e.g. out of /var/mail/something). If you are using deliver, it actually, you know, _delivers_ the message. I think that there is a way to have the same behavior as UW IMAP, but I don't know off the top of my head how to do that (since we use maildir instead).
It isn't the UW IMAP behavior, it is the former MTA's deliver behavior that is the issue. Most MTAs (actually MDAs) simply append to the inbox, which does not update the last access time.
You may want to check the list archives first, rather than claiming dovecot is doing something wrong...
Again, I'm not making such a claim. I have a problem that needs resolution, and I seek input from others who may have run into this same issue in the past.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
On Wed, 2007-06-13 at 17:52 -0500, Steven F Siirila wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time!
Well, there are two things:
It always makes sure that the mbox file ends with "\n". I guess this isn't all that important, but I'm not really interested in just removing the code.
If index files are fully synced, Dovecot writes X-UID: header. It also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first message, which causes Dovecot to read() the file. The nextuid update isn't really required, but I think some other bug shows up if it isn't done.
It would be possible to check the atime before any reading is done and then later after message is saved it could be updated back. Hmm. Well, attached a patch that seems to work. I'm not sure if I want to add it to v1.0.1.
On Thu, Jun 14, 2007 at 06:22:45AM +0300, Timo Sirainen wrote:
On Wed, 2007-06-13 at 17:52 -0500, Steven F Siirila wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time!
Well, there are two things:
- It always makes sure that the mbox file ends with "\n". I guess this isn't all that important, but I'm not really interested in just removing the code.
If that were the only reason for opening in read mode I'd lobby for this to be changed. :)
- If index files are fully synced, Dovecot writes X-UID: header. It also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first message, which causes Dovecot to read() the file. The nextuid update isn't really required, but I think some other bug shows up if it isn't done.
Do these operations apply to Deliver or just IMAP/POP?
Not knowing enough about how the indexes work.... Question: Is it possible for Deliver to append a message w/o writing an X-UID header, leaving that operation to the IMAP/POP client code, and still maintain an updated index?
If not, are there any options I can provide to Dovecot LDA to make it function without index file updating?
It would be possible to check the atime before any reading is done and then later after message is saved it could be updated back. Hmm. Well, attached a patch that seems to work. I'm not sure if I want to add it to v1.0.1.
Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure).
What we have is a daemon called "mailattrd" on our mail servers which runs on a TCP port and simply does a stat() on a user's mailbox and returns the atime, mtime, ctime values of said mailbox. That "service" is used by a variety of our applications. We were thinking of another possibility: changing Dovecot IMAP/POP to update the last access time (open in READ) of a specific file other than the user's INBOX. However, we'd rather not maintain local modifications unless absolutely necessary. Any thoughts on alternatives which might be mailbox-format-independent?
Thanks Timo for your super and timely support -- it is greatly appreciated!!
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
OK, I have some more sane thoughts after we discussed our local situation. See inline comments.
On Wed, Jun 13, 2007 at 10:51:02PM -0500, Steven F Siirila wrote:
On Thu, Jun 14, 2007 at 06:22:45AM +0300, Timo Sirainen wrote:
On Wed, 2007-06-13 at 17:52 -0500, Steven F Siirila wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time!
Well, there are two things:
- It always makes sure that the mbox file ends with "\n". I guess this isn't all that important, but I'm not really interested in just removing the code.
If that were the only reason for opening in read mode I'd lobby for this to be changed. :)
- If index files are fully synced, Dovecot writes X-UID: header. It also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first message, which causes Dovecot to read() the file. The nextuid update isn't really required, but I think some other bug shows up if it isn't done.
Do these operations apply to Deliver or just IMAP/POP?
Not knowing enough about how the indexes work.... Question: Is it possible for Deliver to append a message w/o writing an X-UID header, leaving that operation to the IMAP/POP client code, and still maintain an updated index?
This is still an open question, but not as important to be answered now.
If not, are there any options I can provide to Dovecot LDA to make it function without index file updating?
To answer my own question, one could probably use a very large value for 'mbox_min_index_size'.
It would be possible to check the atime before any reading is done and then later after message is saved it could be updated back. Hmm. Well, attached a patch that seems to work. I'm not sure if I want to add it to v1.0.1.
Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure).
For the mbox case (our immediate concern) your patch seems perfect, provided that it is performing the read of the access time and the re-setting of the access time during the time that the INBOX is locked. Timo, is this the case?
What we have is a daemon called "mailattrd" on our mail servers which runs on a TCP port and simply does a stat() on a user's mailbox and returns the atime, mtime, ctime values of said mailbox. That "service" is used by a variety of our applications. We were thinking of another possibility: changing Dovecot IMAP/POP to update the last access time (open in READ) of a specific file other than the user's INBOX. However, we'd rather not maintain local modifications unless absolutely necessary. Any thoughts on alternatives which might be mailbox-format-independent?
What we are considering doing is updating our local "mailattrd" to lock using the same mechanism Dovecot is before reading the access time:
lock /var/mail/user (open RDONLY, fcntl SETLKW)
stat /var/mail/user (to get last access time)
unlock /var/mail/user (close)Unfortunately, it is more overhead. But it should be 100% accurate. This should give us an accurate "last time the user accessed their INBOX". I'm not sure how this would work with other mailbox formats, but we'll cross that bridge when we get there.
Thanks Timo for your super and timely support -- it is greatly appreciated!!
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
On Fri, 2007-06-15 at 14:07 -0500, Steven F Siirila wrote:
Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure).
For the mbox case (our immediate concern) your patch seems perfect, provided that it is performing the read of the access time and the re-setting of the access time during the time that the INBOX is locked. Timo, is this the case?
That's right.
What we are considering doing is updating our local "mailattrd" to lock using the same mechanism Dovecot is before reading the access time:
lock /var/mail/user (open RDONLY, fcntl SETLKW) stat /var/mail/user (to get last access time) unlock /var/mail/user (close)
Unfortunately, it is more overhead. But it should be 100% accurate.
That should work. Or I guess you saw my other mail already too? That doesn't require creating the lock, but should be 100% accurate as well.
On Sun, Jun 17, 2007 at 01:24:46PM +0300, Timo Sirainen wrote:
On Fri, 2007-06-15 at 14:07 -0500, Steven F Siirila wrote:
Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure).
For the mbox case (our immediate concern) your patch seems perfect, provided that it is performing the read of the access time and the re-setting of the access time during the time that the INBOX is locked. Timo, is this the case?
That's right.
Great! Will you be incorporating this patch into a future release? (Right now we have incorporated it as a local mod.)
What we are considering doing is updating our local "mailattrd" to lock using the same mechanism Dovecot is before reading the access time:
lock /var/mail/user (open RDONLY, fcntl SETLKW) stat /var/mail/user (to get last access time) unlock /var/mail/user (close)
Unfortunately, it is more overhead. But it should be 100% accurate.
That should work. Or I guess you saw my other mail already too? That doesn't require creating the lock, but should be 100% accurate as well.
Since there is no way to "check" the lock for fcntl locks, I think we're left with no choice but to use the above code. Thanks again!
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
On Mon, 2007-06-25 at 14:39 -0500, Steven F Siirila wrote:
On Sun, Jun 17, 2007 at 01:24:46PM +0300, Timo Sirainen wrote:
On Fri, 2007-06-15 at 14:07 -0500, Steven F Siirila wrote:
Hmm. I wonder how much of a timing window is left; this could be a viable option for us. However, I would like to consider our future options for mailbox formats other than mbox (we will eventually migrate I'm sure).
For the mbox case (our immediate concern) your patch seems perfect, provided that it is performing the read of the access time and the re-setting of the access time during the time that the INBOX is locked. Timo, is this the case?
That's right.
Great! Will you be incorporating this patch into a future release? (Right now we have incorporated it as a local mod.)
It's already in the v1.1 code tree.
On Wed, 2007-06-13 at 22:51 -0500, Steven F Siirila wrote:
- If index files are fully synced, Dovecot writes X-UID: header. It also updates nextuid field in X-IMAP: / X-IMAPbase: header of the first message, which causes Dovecot to read() the file. The nextuid update isn't really required, but I think some other bug shows up if it isn't done.
Do these operations apply to Deliver or just IMAP/POP?
All.
Not knowing enough about how the indexes work.... Question: Is it possible for Deliver to append a message w/o writing an X-UID header, leaving that operation to the IMAP/POP client code, and still maintain an updated index?
That's how messages are saved if index files aren't up-to-date. So the X-UID: adding is just an optimization.
If not, are there any options I can provide to Dovecot LDA to make it function without index file updating?
You could set:
protocol lda { mail_location = ...:INDEX=MEMORY }
What we have is a daemon called "mailattrd" on our mail servers which runs on a TCP port and simply does a stat() on a user's mailbox and returns the atime, mtime, ctime values of said mailbox.
You could do something like this:
do { st1 = stat(mbox) while (mbox.lock exists) wait; st2 = stat(mbox) } while (st1.mtime != st2.mtime || st1.atime != st2.atime);
With some small lock timeout value.
That "service" is used by a variety of our applications. We were thinking of another possibility: changing Dovecot IMAP/POP to update the last access time (open in READ) of a specific file other than the user's INBOX. However, we'd rather not maintain local modifications unless absolutely necessary.
Mount with noatime and have Dovecot update the atime itself? I'm not sure if the patch I sent is enough to update atime always.
Any thoughts on alternatives which might be mailbox-format-independent?
There aren't really mailbox-independent ways to do this. With maildir you'd probably stat() new/ and see if atime > ctime.
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time! Not only do our users
If you're willing to pass on delivery-time indexing, and you assign one unix uid per account, then you can instead consider using other delivery agents - and just use dovecot to serve.
I use procmail on my system; and on some mailboxes, I use deliver; some I deliver direct via procmail to reduce deliver times to accomodate the flow rates that hit those specific mailboxes (alerts in a very large production environment that are still smtp based).
On Wed, Jun 13, 2007 at 08:43:26PM -0700, Jason Fesler wrote:
We are running Dovecot 1.0.0 using mbox format (currently in the midst of conversion from UW IMAP). We discovered today that the Dovecot LDA is accessing the user's INBOX at delivery time! Not only do our users
If you're willing to pass on delivery-time indexing, and you assign one unix uid per account, then you can instead consider using other delivery agents - and just use dovecot to serve.
True -- however, two points: 1) we probably won't stay with the 'mbox' format forever, and 2) we would like to use the Sieve plugin, which I believe requires use of Dovecot LDA.
I use procmail on my system; and on some mailboxes, I use deliver; some I deliver direct via procmail to reduce deliver times to accomodate the flow rates that hit those specific mailboxes (alerts in a very large production environment that are still smtp based).
We also have a very large production environment. Thanks for the ideas.
--
Steven F. Siirila Office: Lind Hall, Room 130B Internet Services E-mail: sfs@umn.edu Office of Information Technology Voice: (612) 626-0244 University of Minnesota Fax: (612) 626-7593
True -- however, two points: 1) we probably won't stay with the 'mbox' format forever, and 2) we would like to use the Sieve plugin, which I believe requires use of Dovecot LDA.
Please set up a system using that plugin and make sure it does everything you need it to. I had multiple problems with specific features not being available, plus no timsieved, which then begs the problem of - how will your users manage sieve instructions. I finally mangled a squirrelmail plugin to use local disk for storage instead of talking to timseived and ripped out all the incompatible bits - it wasn't pretty.
I don't want to discourage you :-) but before you get too far committed, make sure that your needs are met.
We also have a very large production environment. Thanks for the ideas.
Good luck :-).
participants (4)
-
Jason Fesler
-
John Peacock
-
Steven F Siirila
-
Timo Sirainen