[Dovecot] index synchronization during local delivery
Hi there,
Now our system will take dovecot as the whole imap back end, and keep the LDA of ours, But our system has our own UID generation mechanism (in dovecot it's incremental from 1), which will be done while local delivery, So when local deliveries, it will first generate the UID, move the mail into maildir, then synchronize the mailbox index, which is part of dovecot LDA does. The question is what API or something else about index synchronization I can integrate into our own LDA program easily? Seems mailbox_copy() would be the possibility?
Thanks.
On 14.2.2012, at 5.31, Andy YB Hu wrote:
Now our system will take dovecot as the whole imap back end, and keep the LDA of ours, But our system has our own UID generation mechanism (in dovecot it's incremental from 1), which will be done while local delivery, So when local deliveries, it will first generate the UID, move the mail into maildir, then synchronize the mailbox index, which is part of dovecot LDA does. The question is what API or something else about index synchronization I can integrate into our own LDA program easily? Seems mailbox_copy() would be the possibility?
If you want it to be free of race conditions:
- Write the mail to tmp/
- Create dovecot-uidlist.lock
- Generate the UID and move the mail to new/ or cur/
- Append a line to dovecot-uidlist with your UID and the Maildir filename
- Delete dovecot-uidlist.lock
Although it sounds like the UID is generated before you can even create uidlist lock? That means IMAP client can at the same time generate a conflicting UID with APPEND/COPY. Also remember that UIDs must always be increasing. If you try to add a new message with lower UID then Dovecot won't like it, and even if you fix that, some IMAP clients may never see the mail.
Anyway, once you have uidlist updated you can do indexing/caching with e.g. "doveadm index" command in v2.0.
I can generate the UID once I create the uidlist lock, and the UID is incremental globally.
Some questions: 1, Whether I can write the mail to tmp/ after creating dovecot-uidlist.lock? 2, I did the 5 steps manually, found that new mail is set to OLD flag, so that when SELECT command, it told there was no NEW mail. Looks like need to change code in maildir_uidlist_sync_next_uid()? 3, About the dovecot-uidlist.lock file creation and deletion, I checked the code, there are functions in dovecot like maildir_uidlist_lock_timeout and maildir_uidlist_unlock. If I did that via calling these functions, there are lots of extra codes need to call(like auth, mailbox open...). So the question is if I do that in my codes, how to handle the race condition correctly? (At least there would be other sessions want to lock the files also / already locked it while I want to create the dovecot-uidlist.lock)
Thanks.
Timo Sirainen
<tss@iki.fi>
Sent by: To
dovecot-bounces@d Dovecot Mailing List
ovecot.org <dovecot@dovecot.org>
cc
02/15/2012 10:19 Subject
AM Re: [Dovecot] index synchronization
during local delivery
Please respond to
Dovecot Mailing
List
<dovecot@dovecot.
org>
On 14.2.2012, at 5.31, Andy YB Hu wrote:
Now our system will take dovecot as the whole imap back end, and keep the LDA of ours, But our system has our own UID generation mechanism (in dovecot it's incremental from 1), which will be done while local delivery, So when local deliveries, it will first generate the UID, move the mail into maildir, then synchronize the mailbox index, which is part of dovecot LDA does. The question is what API or something else about index synchronization I can integrate into our own LDA program easily? Seems mailbox_copy() would be the possibility?
If you want it to be free of race conditions:
- Write the mail to tmp/
- Create dovecot-uidlist.lock
- Generate the UID and move the mail to new/ or cur/
- Append a line to dovecot-uidlist with your UID and the Maildir filename
- Delete dovecot-uidlist.lock
Although it sounds like the UID is generated before you can even create uidlist lock? That means IMAP client can at the same time generate a conflicting UID with APPEND/COPY. Also remember that UIDs must always be increasing. If you try to add a new message with lower UID then Dovecot won't like it, and even if you fix that, some IMAP clients may never see the mail.
Anyway, once you have uidlist updated you can do indexing/caching with e.g. "doveadm index" command in v2.0.
On 15.2.2012, at 6.13, Andy YB Hu wrote:
I can generate the UID once I create the uidlist lock, and the UID is incremental globally.
Some questions: 1, Whether I can write the mail to tmp/ after creating dovecot-uidlist.lock?
Yes, then you'll just keep the lock longer than necessary.
2, I did the 5 steps manually, found that new mail is set to OLD flag, so that when SELECT command, it told there was no NEW mail. Looks like need to change code in maildir_uidlist_sync_next_uid()?
I guess that means that the mail didn't get a \Recent flag. Wht Dovecot version is this? Are you saving the mail to new/?
3, About the dovecot-uidlist.lock file creation and deletion, I checked the code, there are functions in dovecot like maildir_uidlist_lock_timeout and maildir_uidlist_unlock. If I did that via calling these functions, there are lots of extra codes need to call(like auth, mailbox open...).
Yes, way too much work to do that way.
So the question is if I do that in my codes, how to handle the race condition correctly? (At least there would be other sessions want to lock the files also / already locked it while I want to create the dovecot-uidlist.lock)
It's a typical dotlock file. You can create it with either open(.., O_EXCL) or creating a temp file and trying to link() it to the .lock file. Or you can also directly call the file_dotlock_create() in src/lib/file-dotlock.h
Tried it, found it could trigger NFS issue.
The test is on one machine runs the script to lock the dovecot-uidlist, moving mail into cur/new, append one line to dovecot-uidlist, and unlock it. On another different machine, run one script to SELECT, another script to COPY. Found the similar issue in http://dovecot.org/pipermail/dovecot/2012-February/063539.html
Whether it means the dovecot LDA also needs to run on the same server with the backend server?
Timo Sirainen
<tss@iki.fi>
Sent by: To
dovecot-bounces@d Dovecot Mailing List
ovecot.org <dovecot@dovecot.org>
cc
02/15/2012 12:25 Subject
PM Re: [Dovecot] index synchronization
during local delivery
Please respond to
Dovecot Mailing
List
<dovecot@dovecot.
org>
On 15.2.2012, at 6.13, Andy YB Hu wrote:
I can generate the UID once I create the uidlist lock, and the UID is incremental globally.
Some questions: 1, Whether I can write the mail to tmp/ after creating dovecot-uidlist.lock?
Yes, then you'll just keep the lock longer than necessary.
2, I did the 5 steps manually, found that new mail is set to OLD flag, so that when SELECT command, it told there was no NEW mail. Looks like need to change code in maildir_uidlist_sync_next_uid()?
I guess that means that the mail didn't get a \Recent flag. Wht Dovecot version is this? Are you saving the mail to new/?
3, About the dovecot-uidlist.lock file creation and deletion, I checked the code, there are functions in dovecot like maildir_uidlist_lock_timeout and maildir_uidlist_unlock. If I did that via calling these functions, there are lots of extra codes need to call(like auth, mailbox open...).
Yes, way too much work to do that way.
So the question is if I do that in my codes, how to handle the race condition correctly? (At least there would be other sessions want to lock the files also / already locked it while I want to create the dovecot-uidlist.lock)
It's a typical dotlock file. You can create it with either open(.., O_EXCL) or creating a temp file and trying to link() it to the .lock file. Or you can also directly call the file_dotlock_create() in src/lib/file-dotlock.h
Sounds a little tricky. For backend server, director makes sense. But for LDA, it's also suggested to deploy the LDA on the same machine with backend server? Topologically, it's very possible the machine where LDA is running is separated from one of backend server. I will run the similar tests with pure dovecot LDA, instead of ours to see the result.
Timo Sirainen
<tss@iki.fi>
Sent by: To
dovecot-bounces@d Dovecot Mailing List
ovecot.org <dovecot@dovecot.org>
cc
02/16/2012 12:54 Subject
PM Re: [Dovecot] index synchronization
during local delivery
Please respond to
Dovecot Mailing
List
<dovecot@dovecot.
org>
On 16.2.2012, at 5.59, Andy YB Hu wrote:
Tried it, found it could trigger NFS issue. .. Whether it means the dovecot LDA also needs to run on the same server with the backend server?
Yes, the only 100% safe way is to do all simultaneous mailbox access via the same backend server.
On 16.2.2012, at 8.25, Andy YB Hu wrote:
Sounds a little tricky. For backend server, director makes sense. But for LDA, it's also suggested to deploy the LDA on the same machine with backend server? Topologically, it's very possible the machine where LDA is running is separated from one of backend server.
MTA/Antivirus/Antispam is often in a separate backend server, but it can do the actual mail writing by sending the mail to Dovecot via LMTP via director.
participants (2)
-
Andy YB Hu
-
Timo Sirainen