[Dovecot] CONDSTORE + QRESYNC extensions
I wrote a mostly-working CONDSTORE support for Dovecot today. You can try it with:
hg clone http://hg.dovecot.org/dovecot-condstore/
There are a few problems left:
STORE FLAGS.SILENT should be sending FETCH MODSEQ updates
When SELECTing a mailbox while CONDSTORE hasn't yet been enabled in the session, Dovecot returns a higher HIGHESTMODSEQ than what's used after CONDSTORE is enabled. This is because modseqs aren't tracked until client has told us that it wants them. So Dovecot doesn't waste time trying to figure out the correct HIGHESTMODSEQ.. Something probably should be done about this.
STORE (UNCHANGEDSINCE xx) doesn't work correctly. The full fix would require adding a new transaction log record type containing the modseq, but currently it doesn't even bother refreshing the mailbox state before checking what the modseqs are.
The modseq numbers are calculated as:
modseq = ((indexid + log_file_seq) << 32) | log_file_offset
Where:
- indexid = A growing unique index file ID, usually the same as the UNIX timestamp of index file creation.
- log_file_seq = dovecot.index.log file sequence. When the .log is rotated to .log.2 the new .log file's sequence is increased (so indexid and log_file_seq uniquely identify a log file)
- log_file_offset = offset in dovecot.index.log where the flag change record was written.
The numbers are a bit large, but this way if index files are deleted the modseqs always grow (so nothing breaks), because the newly created index file will have a larger indexid.
The modseqs are also stored to dovecot.index file in "modseq" extension records. The highest modseq is stored in header extension.
QRESYNC would require keeping track of expunges. This could be implemented by just going through records in log files to find out what messages have been expunged since the given modseq (= log seq+offset). Although I'm not sure if it would be better to have a separate expunge log. To allow rarely (e.g. once a week?) syncing clients the .log files may have to be rotated less often, so they would take more space. Also reading through them to find expunges would be slower than reading a separate log having only expunges. But writing a separate expunge log would add extra writes and there's no guarantee that it's ever even read.
Other than that QRESYNC should be easy to implement.
QRESYNC would require keeping track of expunges. This could be implemented by just going through records in log files to find out what messages have been expunged since the given modseq (= log seq+offset). Although I'm not sure if it would be better to have a separate expunge log. To allow rarely (e.g. once a week?) syncing clients the .log files may have to be rotated less often, so they would take more space. Also reading through them to find expunges would be slower than reading a separate log having only expunges. But writing a separate expunge log would add extra writes and there's no guarantee that it's ever even read.
Yeah, expunges are one of the fairly crucial things to track here in order to support sync between servers. You can always infer creation of new messages by seeing that they exist on one server and not the other, but you can't differentiate between a delete on one server and a creation on the other without some kind of "tombstone" record to record the delete.
Iff it turns out to have a performance effect perhaps it would make sense to have it as a configurable option? I'm sure someone will ask for it to be configurable per folder or some incredible granulatity like that, but I'm thinking that for most purposes having a per server option will be plenty...?
Just my 2p..
Cheers
Ed W
On Thu, 2008-03-13 at 17:57 +0200, Timo Sirainen wrote:
I wrote a mostly-working CONDSTORE support for Dovecot today. You can try it with:
Recreated the repository, delete the old one and pull it again.
- STORE FLAGS.SILENT should be sending FETCH MODSEQ updates
Fixed.
- When SELECTing a mailbox while CONDSTORE hasn't yet been enabled in the session, Dovecot returns a higher HIGHESTMODSEQ than what's used after CONDSTORE is enabled. This is because modseqs aren't tracked until client has told us that it wants them. So Dovecot doesn't waste time trying to figure out the correct HIGHESTMODSEQ.. Something probably should be done about this.
Fixed.
- STORE (UNCHANGEDSINCE xx) doesn't work correctly. The full fix would require adding a new transaction log record type containing the modseq, but currently it doesn't even bother refreshing the mailbox state before checking what the modseqs are.
Not fixed.
Also:
- If CONDSTORE isn't enabled before mailbox is selected and then CONDSTORE enabling command is sent, send latest HIGHESTMODSEQ to client.
QRESYNC would require keeping track of expunges. This could be implemented by just going through records in log files to find out what messages have been expunged since the given modseq (= log seq+offset).
Implemented it this way. Some issues left with it:
- CLOSE and EXPUNGE always sends HIGHESTMODSEQ, shouldn't be needed if nothing was expunged
- CLOSE [HIGHESTMODSEQ] has race condition (just wrote about it to Lemonade list)
- SELECT with seqset+uidset parameters does only the minimal expunge restriction described by the spec. It would be possible to do it better, but I spent too much time trying to figure out the algorithm at the same time as coding it and it didn't really work. So looks like I'll have to think about it properly first before trying to implement it. :)
On Sat, 2008-03-15 at 17:06 +0200, Timo Sirainen wrote:
- STORE (UNCHANGEDSINCE xx) doesn't work correctly. The full fix would require adding a new transaction log record type containing the modseq, but currently it doesn't even bother refreshing the mailbox state before checking what the modseqs are.
Refreshing is now done, but there's a small race condition that can't be fixed without index file format changes. I'll leave this for later.
- If CONDSTORE isn't enabled before mailbox is selected and then CONDSTORE enabling command is sent, send latest HIGHESTMODSEQ to client.
Fixed.
- CLOSE and EXPUNGE always sends HIGHESTMODSEQ, shouldn't be needed if nothing was expunged
Fixed.
- CLOSE [HIGHESTMODSEQ] has race condition (just wrote about it to Lemonade list)
Fixed.
- SELECT with seqset+uidset parameters does only the minimal expunge restriction described by the spec. It would be possible to do it better, but I spent too much time trying to figure out the algorithm at the same time as coding it and it didn't really work. So looks like I'll have to think about it properly first before trying to implement it. :)
Not fixed.
participants (2)
-
Ed W
-
Timo Sirainen