On Tue, 2007-02-06 at 19:05 +0100, Jürgen Herz wrote:

...

  + pop3: Commit the transaction even if client didn't QUIT so

cached

...

    data gets saved.

Might be I don't understand what transaction you mean. But isn't the change a violation of RFC 1939 chapter 6?

It doesn't expunge any mails, so no. But RETRed messages are marked with \Seen flag now, previously they weren't. POP3 spec doesn't say anything about seen flags, and I think this way might be better.

Jürgen Herz schrieb:

Timo Sirainen wrote:

...

On Tue, 2007-02-06 at 19:05 +0100, Jürgen Herz wrote:

...

Might be I don't understand what transaction you mean. But isn't the change a violation of RFC 1939 chapter 6? It doesn't expunge any mails, so no. But RETRed messages are marked with \Seen flag now, previously they weren't. POP3 spec doesn't say anything about seen flags, and I think this way might be better.

I think that's a hole in the RFC, though "update state" could be interpreted more widely than just for expunge. But updating the flags seems quite harmless, yes.

Not a hole in RFC from POP's perspective - POP3 simply has no notion of "Seen" flags, since it follows a fetch-and-remove model.

I wonder if this change does good - on the other hand, given that Dovecot does not allow querying the "LAST" message which was removed several POP3 revisions ago, it will probably not harm fetchmail which will fall back to UIDL if LAST does not work.

Best regards, Matthias Andree

Timo Sirainen schrieb:

Dovecot does support LAST optionally, but is it even then a problem? Looking at RFC1081 it still seems that LAST should return the highest RETRed message, regardless of whether the connection was QUIT or not.

Yes. The fundamental problem with LAST is that the server must assume that a message RETR'd is a message delivered, which is untrue and can cause message loss. I wish Dovecot wouldn't implement LAST at all.

And I'm certainly not bothering teaching fetchmail to do RETR, RSET and other tricks to keep LAST workable. We have UIDL, and we've had it for over a decade now, and it works well.

RSET command then.. Currently I'm rollbacking the transaction there.

Correct. Same for a broken connection. Only the UPDATE state (after a QUIT command) is allowed to update any state.

I'm not sure if it's wanted or not. If LAST is enabled it doesn't matter anyway, since all the \Seen flags are removed. So I guess I'll change the code so it doesn't rollback with LAST enabled since it only loses cache data.

Whatever cache data you're referring to...

In any case, I'd rather retrieve a message twice than only a corrupted copy if the connection breaks amidst the retrieval, so RSET, broken connections, whatever -> rollback, pretending the broken connection had never happened.

It *is* ugly, and I'd encourage you to remove "LAST" support altogether. It's foul debris from the past century.

Best, Matthias

Hi Mart,

Hmm, is this fsync_disable=yes real option?

dovecot: Error: Error in configuration file /etc/dovecot.conf line 284: Unknown setting: fsync_disable

I think it's only available in CVS HEAD. Timo announced backporting it to 1.0 branch in http://dovecot.org/list/dovecot/2007-January/018953.html but obviously has not done so yet.

Regards

Benjamin

-- Benjamin Dabelow dabelow@tuxoft.de Offenburger Str. 29 tuxoft.de 69126 Heidelberg Germany

On Tue, 2007-02-06 at 20:38 +0200, Timo Sirainen wrote:

m: mmap_disable=yes f: fsync_disable=yes

imaptest clients=1 secs=30 Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc m : 1502 1502 1501 1501 762 405 405 511 1501 : 1495 1495 1494 1494 780 407 407 513 1494 f+m : 1572 1572 1571 1571 802 439 439 534 1571 f : 1540 1540 1539 1539 770 416 416 517 1539

Except with 10 concurrent clients accessing the box, it does give a bit larger improvements:

imaptest clients=10 secs=30 seed=1 Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1273 1272 1262 1262 633 493 493 1406 1263 m 1316 1315 1305 1305 659 509 508 1392 1305 f 1312 1311 1306 1304 650 494 493 1467 1301 f+m 1644 1641 1634 1633 769 728 727 1810 1634

Without fsync_disable=yes the values between mmap_disable=yes/no were pretty much the same, sometimes one was faster than the other.

On Tue, 2007-02-06 at 11:41 -0800, Dan Price wrote:

Because ZFS gathers writes into transaction groups and schedules them for asynchronous write to the platter, we often see very large mailboxes complete their writes very fast (example: 0.3 seconds for 128MB mailbox fully rewritten). When dovecot calls fsync on the mailbox, ZFS is forced to commit all of those changes to the platter right then and there, and forces the application to wait. (ZFS will make sure all the changes get to the platter in the next 10 seconds or so anyway).

What about if you remove the fsync() call only in lib-storage/index/mbox/mbox-sync.c? I think I'll go and remove that anyway, since it doesn't really guarantee anything in there. The other fsyncs and fdatasyncs are more useful in guaranteeing data integrity.

On Tue, 2007-02-06 at 11:55 -0800, Dan Price wrote:

Out of curiosity, why do you feel that the other fsyncs are more beneficial? At least on ZFS, you don't have the problem of stale dirty data sitting around in memory (and not on the platter) for long periods of time.

Well it's not a real problem in many cases. Its only purpose is to prevent problems that could happen in case the computer dies before the data was written to disk. Dovecot uses it in two situations:

1. When mail is saved, it's fsync()ed before the client is told that the save succeeded. This is probably more important with deliver than in IMAP APPEND, because deliver would lose an incoming mail that was never even seen by the user.

Actually I'm missing one fsync() here with maildir. I should open the new/ directory and fsync() it too. Currently I'm just fsyncing the individual maildir files.

2. There are several places in Dovecot where it updates the file by first writing it to a temporary file, then fsyncs it, then rename()s it over the destination file. The point here is that in case of a crash it doesn't leave broken files lying around.

Looks like I'm not fsyncing enough in this case either. subscriptions file and maildir-keywords file should be fsynced.

On Tue, 2007-02-06 at 11:41 -0800, Dan Price wrote:

When dovecot calls fsync on the mailbox, ZFS is forced to commit all of those changes to the platter right then and there, and forces the application to wait.

BTW. There are two potential problems here:

1. The application needs to wait, which makes it appear slower to the user.

2. The system I/O load in general could grow, which would make the system slower for everyone.

I think 1) isn't that big of an issue normally since the delays should be fractions of a second, but 2) is.

On 6.2.2007, at 22.51, Dan Price wrote:

...

2. The system I/O load in general could grow, which would make the system slower for everyone.

I think 1) isn't that big of an issue normally since the delays
should be fractions of a second, but 2) is.

We're in agreement on (1), but I don't think I understand (2).

I'm mostly thinking that if the data isn't written to disk before
it's updated again, there's no need to write it twice then.

I think any IMAP daemon would want to get its bits out to the
platter in a reasonable time frame so that changes to the box don't sit in
dirty VM or buffer cache pages in the event of power fail or catastrophe.

Yes, but doesn't that normally happen with all filesystem in
"reasonable amount of time" anyway? I thought it was always around
few seconds to a minute or so, but I don't know this for a fact.

On 6.2.2007, at 23.35, Troy Engel wrote:

Timo Sirainen wrote:

...

I'm mostly thinking that if the data isn't written to disk before
it's updated again, there's no need to write it twice then.

Would this create a race condition (sic) if the same IMAP mailbox
were logged into from two different locations? (i.e. Thunderbird
and SquirrelMail)

No, kernel handles all of this correctly internally. File locking is
what prevents the files from being corrupted, and Dovecot does that.