[Dovecot] Locking /var/mail/user issue with postfix and dovecot
Hello,
I have a question regarding mailbox locking and hope any one can help me to better understanding the locking of mbox
My Postfix lock option is fcntl dotlock mailbox_delivery_lock = fcntl, dotlock virtual_mailbox_lock = fcntl, dotlock
My Dovecot lock option is fcntl only mbox_write_locks = fcntl mbox_read_locks = fcntl
now, when user is download a large mail (20 MB) for example, dovecot locks /var/mail/user with fcntl until the users finish downloading the mail
ok here comes my question.
While the user is downloading the mail , a mail arrives so postfix make some thing weird
1- Postfix creates /var/mail/user.lock for 5 secs , then postfix defer the mail with reason that the /var/mail/user is locked and then delete the /var/mail/user.lock (after 5 secs)
My question is ? why postfix create user.lock although it shouldn't because already dovecot fcntl it …
1- Dovecot locks /var/mail/user using FCNTL
2- Posttix at the same time tries to FCNTL /var/mail/user .. but it fail since dovecot already fcnl it.
3- Postfix at the same time add dot lock /var/mail/user for 5 secs then remove the lock.
the question is how come postfix dot lock /var/mail although it couldn't FCNTL the file in the first place ??????? what i was expecting is
1- Dovecot locks /var/mail/user using FCNTL
2- Postfix tries to FCNTL /var/mail/user
3- POSTFIX WILL NOT CREATE DOTLOCK file unless the FCNTL is released by dovecot!!!!
Please advise if postfix will dot lock the file even if it couldn't FCNTL the file in the first place……
Regards
On 10/24/2012 3:04 PM, Robert JR wrote:
I have a question regarding mailbox locking and hope any one can help me to better understanding the locking of mbox
My Postfix lock option is fcntl dotlock mailbox_delivery_lock = fcntl, dotlock virtual_mailbox_lock = fcntl, dotlock
My Dovecot lock option is fcntl only mbox_write_locks = fcntl mbox_read_locks = fcntl <snip>
Postfix is delivering the mail to dovecot. This is done via the deliver program or lmtp which are pipes, not files. Thus, why is Postfix attempting to write files in the user's mail directory?
You write new mail to the mailbox file with either Dovecot or Postfix, not both. Fix that problem and the locking problem disappears.
-- Stan
On 2012-10-25 06:57, Stan Hoeppner wrote:
On 10/24/2012 3:04 PM, Robert JR wrote:
I have a question regarding mailbox locking and hope any one can help me to better understanding the locking of mbox My Postfix lock option is fcntl dotlock mailbox_delivery_lock = fcntl, dotlock virtual_mailbox_lock = fcntl, dotlock My Dovecot lock option is fcntl only mbox_write_locks = fcntl mbox_read_locks = fcntl
<snip>
Postfix is delivering the mail to dovecot. This is done via the deliver program or lmtp which are pipes, not files. Thus, why is Postfix attempting to write files in the user's mail directory?
You write new mail to the mailbox file with either Dovecot or Postfix, not both. Fix that problem and the locking problem disappears.
Stan, sorry but you didnot understand my question at all, dovecot in this case is reading the mailbox file while user downloading the mail and not WRITING. only postfix write when a mail arrives and DOVECOT only read the mail. And even if both write to the file, I have already set the locking option of both to FCNTL so no problem should happen. My question is postfix locking option is = FCNTL, DOTLOCK , and dovecot = FCNTL, if postfix find a file already FCNTL, why it dotlock the file 5 secs then remove the dotlock and say resource unaviable? why in the first place it dotlock the file, althought it couldnot FCNTL it in the first place since it is FCNTL by dovecot while reading the in the moment
Any help will be greatly appreciated..... Regards
Robert.
On Thu, Oct 25, 2012 at 10:23:29AM +0300, Robert JR wrote:
[..]
should happen. My question is postfix locking option is = FCNTL, DOTLOCK , and dovecot = FCNTL, if postfix find a file already FCNTL, why it dotlock the file 5 secs then remove the dotlock and say resource unaviable? why in the first place it dotlock the file, althought it couldnot FCNTL it in the first place since it is FCNTL by dovecot while reading the in the moment
You are on the wrong mailing list, this is no problem of Dovecot.
Ask this question on postfix-users (maybe you are requested to show evidence of this behaviour).
Dennis
On 2012-10-25 11:56, Dennis Guhl wrote:
On Thu, Oct 25, 2012 at 10:23:29AM +0300, Robert JR wrote:
[..]
should happen. My question is postfix locking option is = FCNTL, DOTLOCK , and dovecot = FCNTL, if postfix find a file already FCNTL, why it dotlock the file 5 secs then remove the dotlock and say resource unaviable? why in the first place it dotlock the file, althought it couldnot FCNTL it in the first place since it is FCNTL by dovecot while reading the in the moment
You are on the wrong mailing list, this is no problem of Dovecot.
Ask this question on postfix-users (maybe you are requested to show evidence of this behaviour).
Dennis\
Thanks dennis for your reply. But, This is also an dovecot issue, because how dovecot use the FCNTL lock maybe different on how Postfix use it , that's why i had to ask also here at dovecot mailist this question.
I think This behavior is understood by any one already have experience on how locking is made to /var/mail/files, I am just trying to understand how/why it happened.
My question is postfix has locking option: FCNTL, DOTLOCK, and a file is already FCNTL by dovecot, will postfix Apply the DOTLOCK and continue try to FCNTL , althugh postfix already found that the file is already FCNTL by dovecot. So it shouldnot dotlock the mbx file untill it frist FCNTL the mailbox after dovecot finished.
When i asked this question on irc postfix, they said that maybe FCNTL is used in different option with dovecot and this Might be the reason of the issue and recommended to ask here for help ..
I guess Timo will be able to respond to this on the spot ...
Thanks again. Robert JR.
On Thu, Oct 25, 2012 at 10:23:29AM +0300, Robert JR wrote:
Stan, sorry but you didnot understand my question at all, dovecot in this case is reading the mailbox file while user downloading the mail and not WRITING. only postfix write when a mail arrives and DOVECOT only read the mail. And even if both write to the file, I
I can't answer (don't know), but I can tell you that this is not true. Dovecot also writes to the file: updating message read flags and such.
Any help will be greatly appreciated.....
Maildir is not for everyone, but it does handle issues like this smoothly. The delivery agent is always able to deliver new mail.
http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
On 2012-10-25 17:28, /dev/rob0 wrote:
On Thu, Oct 25, 2012 at 10:23:29AM +0300, Robert JR wrote:
Stan, sorry but you didnot understand my question at all, dovecot in this case is reading the mailbox file while user downloading the mail and not WRITING. only postfix write when a mail arrives and DOVECOT only read the mail. And even if both write to the file, I
I can't answer (don't know), but I can tell you that this is not true. Dovecot also writes to the file: updating message read flags and such.
Any help will be greatly appreciated.....
Maildir is not for everyone, but it does handle issues like this smoothly. The delivery agent is always able to deliver new mail. We can not convert to maildir now as we have alot of users and converting each account will take a huge time .. well.. we can live with /var/mail/mailbox format , I just need to understand the locking issue and hope Timo will be able to answer our question.. Thanks again
On 10/25/2012 2:23 AM, Robert JR wrote:
On 2012-10-25 06:57, Stan Hoeppner wrote:
On 10/24/2012 3:04 PM, Robert JR wrote:
I have a question regarding mailbox locking and hope any one can help me to better understanding the locking of mbox My Postfix lock option is fcntl dotlock mailbox_delivery_lock = fcntl, dotlock virtual_mailbox_lock = fcntl, dotlock My Dovecot lock option is fcntl only mbox_write_locks = fcntl mbox_read_locks = fcntl
<snip>
Postfix is delivering the mail to dovecot. This is done via the deliver program or lmtp which are pipes, not files. Thus, why is Postfix attempting to write files in the user's mail directory?
You write new mail to the mailbox file with either Dovecot or Postfix, not both. Fix that problem and the locking problem disappears.
Stan, sorry but you didnot understand my question at all
Yes, actually I did, but I missed one part of it because I assumed you had Dovecot setup properly.
It doesn't matter if the mbox locks are write or read or both. Locks are the problem, period, because you have two daemons fighting over the same files. The fix is absolutely trivial:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated. It also enables using all the Dovecot delivery plugins such as Sieve, Quota, anti-spam, etc.
I had the same Postfix/Dovecot mbox locking problem many years ago when I first started using Dovecot. After the suggestions from the fine folks on this list I switched to LDA. It not only eliminated locking, it completely changed the character of my Dovecot install, both in performance and capabilities, as well as fixed some message flag problems, etc. If you're not using LDA with Postfix/Dovecot you're insane. ;) Some might say you're insane for using mbox but I feel it's fine for many installations. I use it myself. Our setups are very similar. To switch to LDA...
If you're using 1.2.x http://wiki.dovecot.org/LDA/Postfix
If you're using 2.x http://wiki2.dovecot.org/LDA/Postfix
-- Stan
On 10/25/2012 1:00 PM, Stan Hoeppner wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated. It also enables using all the Dovecot delivery plugins such as Sieve, Quota, anti-spam, etc.
I forgot to mention one very important feature of Dovecot LDA:
New messages delivered by Postfix are indexed by LDA as they are written to the mailbox, flags updated at this time, etc. Thus when a mailbox is opened in an IMAP MUA, new messages are displayed instantly (I don't use POP but it's probably faster as well). With your current setup it can take from a few to many seconds to show new mail, depending on message count. With LDA new message count seems to have no impact on the speed of display.
-- Stan
On 2012-10-25 21:23, Stan Hoeppner wrote:
On 10/25/2012 1:00 PM, Stan Hoeppner wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated. It also enables using all the Dovecot delivery plugins such as Sieve, Quota, anti-spam, etc.
I forgot to mention one very important feature of Dovecot LDA:
New messages delivered by Postfix are indexed by LDA as they are written to the mailbox, flags updated at this time, etc. Thus when a mailbox is opened in an IMAP MUA, new messages are displayed instantly (I don't use POP but it's probably faster as well). With your current setup it can take from a few to many seconds to show new mail, depending on message count. With LDA new message count seems to have no impact on the speed of display.
Thanks stan very much for your detailed answer, i will read about LDA to know how it works. But i still wonder why this mailbox locking issue and I hope for my referece some one at this form explain to me the issue reason. as since postfix,dovecot has the same lockign settings so why this issue happen ? I have spend 3 days searching for an answer and i couldnot find any.. I think only Timo can answer my question...
Thanks again stan
On 10/25/2012 12:38 PM, Robert JR wrote:
On 2012-10-25 21:23, Stan Hoeppner wrote:
On 10/25/2012 1:00 PM, Stan Hoeppner wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated. It also enables using all the Dovecot delivery plugins such as Sieve, Quota, anti-spam, etc.
I forgot to mention one very important feature of Dovecot LDA:
New messages delivered by Postfix are indexed by LDA as they are written to the mailbox, flags updated at this time, etc. Thus when a mailbox is opened in an IMAP MUA, new messages are displayed instantly (I don't use POP but it's probably faster as well). With your current setup it can take from a few to many seconds to show new mail, depending on message count. With LDA new message count seems to have no impact on the speed of display.
Thanks stan very much for your detailed answer, i will read about LDA to know how it works. But i still wonder why this mailbox locking issue and I hope for my referece some one at this form explain to me the issue reason. as since postfix,dovecot has the same lockign settings so why this issue happen ? I have spend 3 days searching for an answer and i couldnot find any.. I think only Timo can answer my question...
I chose to focus on the permanent and proper solution, which is eliminating your lock contention altogether and enabling maximum performance and features.
Learning about file locking problems between applications may be a noble endeavor, but at this point it's simply a waste of your time. A gazillion papers have been written about this subject over the years. If you're that interested hunt them down and read them. I'm sure Google can find 1000 of them quickly. You'll be looking for academic papers, not forum posts.
Thanks again stan
You're welcome.
-- Stan
On 2012-10-25 23:18, Stan Hoeppner wrote:
On 10/25/2012 12:38 PM, Robert JR wrote:
On 2012-10-25 21:23, Stan Hoeppner wrote:
On 10/25/2012 1:00 PM, Stan Hoeppner wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated. It also enables using all the Dovecot delivery plugins such as Sieve, Quota, anti-spam, etc. I forgot to mention one very important feature of Dovecot LDA: New messages delivered by Postfix are indexed by LDA as they are written to the mailbox, flags updated at this time, etc. Thus when a mailbox is opened in an IMAP MUA, new messages are displayed instantly (I don't use POP but it's probably faster as well). With your current setup it can take from a few to many seconds to show new mail, depending on message count. With LDA new message count seems to have no impact on the speed of display. Thanks stan very much for your detailed answer, i will read about LDA to know how it works. But i still wonder why this mailbox locking issue and I hope for my referece some one at this form explain to me the issue reason. as since postfix,dovecot has the same lockign settings so why this issue happen ? I have spend 3 days searching for an answer and i couldnot find any.. I think only Timo can answer my question...
I chose to focus on the permanent and proper solution, which is eliminating your lock contention altogether and enabling maximum performance and features.
Learning about file locking problems between applications may be a noble endeavor, but at this point it's simply a waste of your time. A gazillion papers have been written about this subject over the years. If you're that interested hunt them down and read them. I'm sure Google can find 1000 of them quickly. You'll be looking for academic papers, not forum posts.
Thanks again stan
You're welcome.
Thanks again Stan, you are very helpfull, i will start learning how to configure LDA, but hopefully i can also have an answer from Timo about why this issue happened.. i am sure he is aware of it and can explain it..
On 10/25/2012 2:42 PM, Robert JR wrote:
Thanks again Stan, you are very helpfull, i will start learning how to configure LDA, but hopefully i can also have an answer from Timo about why this issue happened.. i am sure he is aware of it and can explain it..
Probably not. You describe a dot locking "problem" with Postfix. If you want information about that you need to ask on the Postfix list, not the Dovecot list.
Wietse will answer you properly. Give it a subject "dot lock problem".
-- Stan
On 10/25/2012 1:23 PM, Stan Hoeppner wrote:
I forgot to mention one very important feature of Dovecot LDA:
New messages delivered by Postfix are indexed by LDA as they are written to the mailbox, flags updated at this time, etc. Thus when a mailbox is opened in an IMAP MUA, new messages are displayed instantly (I don't use POP but it's probably faster as well). With your current setup it can take from a few to many seconds to show new mail, depending on message count. With LDA new message count seems to have no impact on the speed of display.
Robert JR, you posted relevant information to the Postfix list that you omitted here, or at least I didn't see it. This may directly affect the advice myself and others gave you.
You stated there that you're using Dovecot POP only and not IMAP. Given the nature of POP, using LDA may not help much even if it eliminates the filesystem locking contention between processes. I don't know if Dovecot will append an mbox file while reading the entire file. This Timo will have an answer to.
Timo may also state, as he has before on this list, that:
- there are better, faster, POPers available
- Dovecot is developed primarily as an IMAP server
- the POP function has received little development for quite some time
-- Stan
At 1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
Yes, actually I did, but I missed one part of it because I assumed you had Dovecot setup properly.
It doesn't matter if the mbox locks are write or read or both. Locks are the problem, period, because you have two daemons fighting over the same files. The fix is absolutely trivial:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated.
Nonsense. deliver and imap are still separate processes accessing the same mbox, so they still need to use locks. The only difference is that since they are both dovecot programs, they will automatically be using the *same* locking strategies, and things will Just Work.
Ben
On 10/25/2012 4:24 PM, Ben Morrow wrote:
At 1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
Yes, actually I did, but I missed one part of it because I assumed you had Dovecot setup properly.
It doesn't matter if the mbox locks are write or read or both. Locks are the problem, period, because you have two daemons fighting over the same files. The fix is absolutely trivial:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated.
Nonsense. deliver and imap are still separate processes accessing the same mbox, so they still need to use locks. The only difference is that since they are both dovecot programs, they will automatically be using the *same* locking strategies, and things will Just Work.
"Nonsense" implies what I stated was factually incorrect, which is not the case. There's a difference between factual incorrectness and simply staying out of the weeds.
If you want to get into the weeds, and have me call you out for "nonsense", LDA/deliver is not a separate UNIX process. The LDA code runs within the imap process for the given user. This is what allows Dovecot to perform 'simultaneous' reads/writes to an mbox file, avoiding filesystem level locking latency. Using filesystem level locking to control read/write access between processes of own's program would be insane on many levels.
-- Stan
At 7PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
On 10/25/2012 4:24 PM, Ben Morrow wrote:
At 1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated.
Nonsense. deliver and imap are still separate processes accessing the same mbox, so they still need to use locks. The only difference is that since they are both dovecot programs, they will automatically be using the *same* locking strategies, and things will Just Work.
"Nonsense" implies what I stated was factually incorrect, which is not the case. There's a difference between factual incorrectness and simply staying out of the weeds.
What you stated was factually incorrect.
If you want to get into the weeds, and have me call you out for "nonsense", LDA/deliver is not a separate UNIX process. The LDA code runs within the imap process for the given user.
Nonsense. dovecot-lda runs in its own process, and does not involve the imap process in any way. As such it has to do locking.
If I have the following in my dovecot.conf:
mbox_read_locks = fcntl
mbox_write_locks = dotlock fcntl
namespace {
location = mbox:~/mbox
separator = /
type = private
list = yes
prefix = MBOX/
}and I run
ktrace dovecot-lda -f mauzo@localhost -m MBOX/foo < mailthen the ktrace contains
<snip> 44973 dovecot-lda CALL access(0x8021f5f68,0x6<W_OK|R_OK>) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo" 44973 dovecot-lda RET access 0 [Check the mbox exists and is accessible]
<snip> 44973 dovecot-lda CALL lstat(0x8020196c0,0x7fffffffcb60) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo.lock" 44973 dovecot-lda RET lstat -1 errno 2 No such file or directory [Look for a .lock file]
<snip> 44973 dovecot-lda CALL open(0x8020196c0,0xa02<O_RDWR|O_CREAT|O_EXCL>, 0x1b6<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH>) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo.lock" 44973 dovecot-lda RET open 9 44973 dovecot-lda CALL write(0x9,0x802019830,0x19) 44973 dovecot-lda RET write 25/0x19 44973 dovecot-lda CALL clock_gettime(0xd,0x7fffffffcbf0) 44973 dovecot-lda RET clock_gettime 0 44973 dovecot-lda CALL fstat(0x9,0x7fffffffcd90) 44973 dovecot-lda RET fstat 0 44973 dovecot-lda CALL close(0x9) 44973 dovecot-lda RET close 0 44973 dovecot-lda CALL lstat(0x8020196c0,0x7fffffffce60) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo.lock" 44973 dovecot-lda RET lstat 0 [Create a .lock file, and check it was successful]
<snip> 44973 dovecot-lda CALL open(0x8021f5f68,0x2<O_RDWR>,<unused>0xfac3c0) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo" 44973 dovecot-lda RET open 9 [Open the mbox file itself]
<snip> 44973 dovecot-lda CALL fcntl(0x9,F_SETLKW,0xffffffffffffcfd0) 44973 dovecot-lda RET fcntl 0 [Set a fcntl lock on the mbox file]
<snip> 44973 dovecot-lda CALL pread(0,0x802031000,0x1000,0) 44973 dovecot-lda RET pread 43/0x2b <snip> 44973 dovecot-lda CALL write(0x9,0x802228000,0xf5) 44973 dovecot-lda RET write 245/0xf5 <snip> 44973 dovecot-lda CALL fsync(0x9) 44973 dovecot-lda RET fsync 0 [Read from stdin and write to the mbox file]
<snip> 44973 dovecot-lda CALL lstat(0x802028440,0x7fffffffd010) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo.lock" 44973 dovecot-lda RET lstat 0 44973 dovecot-lda CALL unlink(0x802028440) 44973 dovecot-lda NAMI "/home/mauzo/mbox/foo.lock" 44973 dovecot-lda RET unlink 0 [Remove the .lock file]
<snip> 44973 dovecot-lda CALL fcntl(0x9,F_SETLK,0xffffffffffffd160) 44973 dovecot-lda RET fcntl 0 [Clear the fcntl lock]
<snip> 44973 dovecot-lda CALL close(0x9) 44973 dovecot-lda RET close 0 [Close the mbox file]
This is what allows Dovecot to perform 'simultaneous' reads/writes to an mbox file, avoiding filesystem level locking latency. Using filesystem level locking to control read/write access between processes of own's program would be insane on many levels.
I'm not sure what you mean by 'processes of [one's own] program' but it's extremely common for a process to have to take locks against another copy of itself. All traditional Unix LDAs and MUAs do this; for instance, procmail will take locks in part so that if another instance of procmail is delivering another mail to the same user at the same time the mbox won't end up corrupted.
Ben
On 10/25/2012 10:54 PM, Ben Morrow wrote:
At 7PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
On 10/25/2012 4:24 PM, Ben Morrow wrote:
At 1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated.
Nonsense. deliver and imap are still separate processes accessing the same mbox, so they still need to use locks. The only difference is that since they are both dovecot programs, they will automatically be using the *same* locking strategies, and things will Just Work.
"Nonsense" implies what I stated was factually incorrect, which is not the case. There's a difference between factual incorrectness and simply staying out of the weeds.
What you stated was factually incorrect.
If you want to get into the weeds, and have me call you out for "nonsense", LDA/deliver is not a separate UNIX process. The LDA code runs within the imap process for the given user.
Nonsense. dovecot-lda runs in its own process, and does not involve the imap process in any way. As such it has to do locking.
You apparently know your tools better than I do. Neither ps nor top show a 'dovecot-lda' or similarly named process on my systems. When I send a test message from gmail through Postfix I only see CPU or memory activity in an imap process. When I close the MUA to end the imap processes and then send a test message I don't see any CPU or memory activity in any dovecot processes, only Postfix processes, including local, and spamd. So is devecot-lda running as a sub-process or thread of Postfix' local process? Or is it part of the 'dovecot' process, and the message goes through so quick that top doesn't show any CPU usage by the 'dovecot' process?
If I have the following in my dovecot.conf: ... <snipped for readability> ...
I'm not sure what you mean by 'processes of [one's own] program' but
I.e. Dovecot has its own set of processes, Postfix has its processes, etc. With "one's one processes" I'd think it makes more sense to use IPC and other tricks to accomplish concurrent access to a file rather than filesystem locking features.
it's extremely common for a process to have to take locks against another copy of itself. All traditional Unix LDAs and MUAs do this; for instance, procmail will take locks in part so that if another instance of procmail is delivering another mail to the same user at the same time the mbox won't end up corrupted.
I guess I've given MDAs w/mbox too much credit, without actually looking at the guts. Scalable databases such Oracle, db2, etc, are far more intelligent about this, and can have many thousands of processes reading and writing the same file concurrently, usually via O_DIRECT, not buffered IO, so they have complete control over IO. This is accomplished with a record lock manager and IPC, preventing more than one process from accessing one record concurrently, but allowing massive read/write concurrency to multiple records in a file. I'd think the same concurrency optimization could be done with Dovecot.
However, as Timo has pointed out, so few people use mbox these days that he simply hasn't spent much, if any, time optimizing mbox. Implementing some kind of lock manager and client code just for mbox IO concurrency simply wouldn't be worth the time. Unless he's already done something similar with mdbox. If he has, maybe that could be 'ported' to mbox as well. But again, it's probably not worth the effort given the number of mbox users, and the fact that nobody is complaining about mbox performance. I'm certainly not. It works great here.
-- Stan
At 1AM -0500 on 26/10/12 you (Stan Hoeppner) wrote:
On 10/25/2012 10:54 PM, Ben Morrow wrote:
dovecot-lda runs in its own process, and does not involve the imap process in any way. As such it has to do locking.
You apparently know your tools better than I do. Neither ps nor top show a 'dovecot-lda' or similarly named process on my systems. When I send a test message from gmail through Postfix I only see CPU or memory activity in an imap process. When I close the MUA to end the imap processes and then send a test message I don't see any CPU or memory activity in any dovecot processes, only Postfix processes, including local, and spamd. So is devecot-lda running as a sub-process or thread of Postfix' local process? Or is it part of the 'dovecot' process, and the message goes through so quick that top doesn't show any CPU usage by the 'dovecot' process?
Assuming you have
mailbox_command = /.../dovecot-lda -a "${RECIPIENT}"or something equivalent in your Postfix configuration, dovecot-lda runs as a subprocess of local(8) under the uid of the delivered-to user.
If I have the following in my dovecot.conf: ... <snipped for readability> ...
I'm not sure what you mean by 'processes of [one's own] program' but
I.e. Dovecot has its own set of processes, Postfix has its processes, etc. With "one's one processes" I'd think it makes more sense to use IPC and other tricks to accomplish concurrent access to a file rather than filesystem locking features.
Filesystem locking, at least if NFS is not involved, is not that expensive. Successfully acquiring a flock or fcntl lock takes only a single syscall which doesn't have to touch the disk, and any form of IPC is going to need to do that. (Even something like a shared memory region will need a mutex for synchronisation, and acquiring the mutex has to go through the kernel.)
Dotlocking *is* expensive, because acquiring a dotlock is a complicated process requiring lots of syscalls, some of which have to write to disk; and any scheme involving acquiring several locks on the same file is going to be more so, especially if you can end up getting the first lock but finding you can't get the second, so then you have to undo the first and try again.
More importantly, the biggest problem with mbox as a mailbox format is that any access at all has to lock the whole mailbox. If the LDA is trying to deliver a new message at the same time as an IMAP user is fetching a completely different message, or if two instances of the LDA are trying to deliver at the same time, they will be competing for the same lock even though they don't really need to be. A file-per-message format like Maildir avoids this, to the point of being mostly lockless, but that brings its own efficiency problems; the point of dbox is to find the compromise between these positions that works best.
it's extremely common for a process to have to take locks against another copy of itself. All traditional Unix LDAs and MUAs do this; for instance, procmail will take locks in part so that if another instance of procmail is delivering another mail to the same user at the same time the mbox won't end up corrupted.
I guess I've given MDAs w/mbox too much credit, without actually looking at the guts.
I wouldn't look too hard at the details of the various ways there are of locking and parsing mbox files, or the ways in which they can go wrong. It's enough to make anyone swear off email for life :).
Scalable databases such Oracle, db2, etc, are far more intelligent about this, and can have many thousands of processes reading and writing the same file concurrently, usually via O_DIRECT, not buffered IO, so they have complete control over IO. This is accomplished with a record lock manager and IPC, preventing more than one process from accessing one record concurrently, but allowing massive read/write concurrency to multiple records in a file. I'd think the same concurrency optimization could be done with Dovecot.
However, as Timo has pointed out, so few people use mbox these days that he simply hasn't spent much, if any, time optimizing mbox. Implementing some kind of lock manager and client code just for mbox IO concurrency simply wouldn't be worth the time. Unless he's already done something similar with mdbox. If he has, maybe that could be 'ported' to mbox as well. But again, it's probably not worth the effort given the number of mbox users, and the fact that nobody is complaining about mbox performance. I'm certainly not. It works great here.
The only reason for using mbox is for compatibility with other systems which use mbox, which means you have to do the locking the same way as they do (assuming you can work out what that is). If you're going to change the locking rules you might as well change the file format at the same time, both to remove the insanity and to make it actually suitable for use as an IMAP mailstore. That's what Timo did with dbox, so if you've got your systems to the point where nothing but Dovecot touches the mail files you should seriously consider switching.
Ben
You are a well of accessible knowledge Ben. (How have I missed your posts in the past?)
On 10/26/2012 3:11 AM, Ben Morrow wrote:
Assuming you have
mailbox_command = /.../dovecot-lda -a "${RECIPIENT}"
I'm setup for system users so it's a simpler, but yes.
or something equivalent in your Postfix configuration, dovecot-lda runs as a subprocess of local(8) under the uid of the delivered-to user.
Of course that makes sense given Postfix is doing the calling. I would have assumed this but my feeble use of tools wasn't showing anything.
Filesystem locking, at least if NFS is not involved, is not that expensive. Successfully acquiring a flock or fcntl lock takes only a single syscall which doesn't have to touch the disk, and any form of IPC is going to need to do that. (Even something like a shared memory region will need a mutex for synchronisation, and acquiring the mutex has to go through the kernel.)
Thanks for this. I was under the assumption flock/fcntl were more expensive than they are. Probably because all I'd read about them was in relation to NFS (which I don't use, but I read alot like many do).
Dotlocking *is* expensive, because acquiring a dotlock is a complicated process requiring lots of syscalls, some of which have to write to disk; and any scheme involving acquiring several locks on the same file is going to be more so, especially if you can end up getting the first lock but finding you can't get the second, so then you have to undo the first and try again.
Yeah, I knew dotlocks were the worst due to disk writes, but didn't know the other details.
More importantly, the biggest problem with mbox as a mailbox format is that any access at all has to lock the whole mailbox. If the LDA is trying to deliver a new message at the same time as an IMAP user is fetching a completely different message, or if two instances of the LDA are trying to deliver at the same time, they will be competing for the same lock even though they don't really need to be. A file-per-message format like Maildir avoids this, to the point of being mostly lockless, but that brings its own efficiency problems; the point of dbox is to find the compromise between these positions that works best.
mbox locking hasn't been problem here as I split the INBOX from the user mailboxes containing IMAP folders (mbox files). We make heavy use of sieve scripts to sort on delivery, so there's not much concurrent access to any one mbox file.
The efficiency issue is why I chose mbox over maildir. Users here keep a lot of (list) mail and FTS often. The load on the spindles with maildir is simply too great and would bog down all users. The IOPS benefit of mbox in this scenario outweighs any locking issues.
I wouldn't look too hard at the details of the various ways there are of locking and parsing mbox files, or the ways in which they can go wrong. It's enough to make anyone swear off email for life :).
Heheh.
The only reason for using mbox is for compatibility with other systems which use mbox,
Not necessarily true. See above. I'm sure I'm not the only one using mbox for this reason. Dovecot is my only app hitting these mbox files.
which means you have to do the locking the same way as they do (assuming you can work out what that is). If you're going to change the locking rules you might as well change the file format at the same time, both to remove the insanity and to make it actually suitable for use as an IMAP mailstore. That's what Timo did with dbox, so if you've got your systems to the point where nothing but Dovecot touches the mail files you should seriously consider switching.
If/when I do switch mailbox formats it'll be to mdbox so FTS doesn't drop a big hammer on the spindles.
Thanks for the informative discussion Ben.
-- Stan
At 10AM +0300 on 25/10/12 you (Robert JR) wrote:
Stan, sorry but you didnot understand my question at all, dovecot in this case is reading the mailbox file while user downloading the mail and not WRITING. only postfix write when a mail arrives and DOVECOT only read the mail. And even if both write to the file, I have already set the locking option of both to FCNTL so no problem should happen. My question is postfix locking option is = FCNTL, DOTLOCK , and dovecot = FCNTL, if postfix find a file already FCNTL, why it dotlock the file 5 secs then remove the dotlock and say resource unaviable?
As Stan said earlier, this is a Postfix question. The rule for dotlocking is that you must create the .lock *before* opening the file, in case whoever has it locked will be replacing the file altogether; but with fcntl locking you must acquire the lock *after* opening the file, since that's the way the syscall works. This means that if Postfix is going to use both forms of lock, it has to acquire a dotlock before it can look for a fcntl lock.
In other words: the methods in mailbox_delivery_lock are *not* tried in order, because they can't be. Dotlock is always tried first.
You should have compatible locking settings for all your programs accessing your mboxes. If Postfix is using dotlock, Dovecot should be using dotlock as well. If you don't have any local programs (mail clients, for instance) which require dotlocks, you should probably change Postfix to just use fcntl locks.
Ben
On 2012-10-26 00:15, Ben Morrow wrote:
At 10AM +0300 on 25/10/12 you (Robert JR) wrote:
Stan, sorry but you didnot understand my question at all, dovecot in this case is reading the mailbox file while user downloading the mail and not WRITING. only postfix write when a mail arrives and DOVECOT only read the mail. And even if both write to the file, I have already set the locking option of both to FCNTL so no problem should happen. My question is postfix locking option is = FCNTL, DOTLOCK , and dovecot
FCNTL, if postfix find a file already FCNTL, why it dotlock the file 5 secs then remove the dotlock and say resource unaviable?
As Stan said earlier, this is a Postfix question. The rule for dotlocking is that you must create the .lock *before* opening the file, in case whoever has it locked will be replacing the file altogether; but with fcntl locking you must acquire the lock *after* opening the file, since that's the way the syscall works. This means that if Postfix is going to use both forms of lock, it has to acquire a dotlock before it can look for a fcntl lock.
In other words: the methods in mailbox_delivery_lock are *not* tried in order, because they can't be. Dotlock is always tried first.
You should have compatible locking settings for all your programs accessing your mboxes. If Postfix is using dotlock, Dovecot should be using dotlock as well. If you don't have any local programs (mail clients, for instance) which require dotlocks, you should probably change Postfix to just use fcntl locks.
Ben Thanks Ben for your valuable support and detailed explanation .. but according to your explanation dovecot documentation contains wrong explanation to the dotlock and that's why i was confused.
it is written as follow
** If multiple lock methods are used, which is usually the case since dotlocks aren't typically used for read locking, the order in which the locking is done is important. Consider if two programs were running at the same time, both use dotlock and fcntl locking but in different order:
Program A: fcntl locks the mbox Program B at the same time: dotlocks the mbox Program A continues: tries to dotlock the mbox, but since it's already dotlocked by B, it starts waiting Program B continues: tries to fcntl lock the mbox, but since it's already fcntl locked by A, it starts waiting Now both of them are waiting for each others locks. Finally after a couple of minutes they time out and fail the operation. **
So this means that the documentation as mentioned above is wrong and dotlock is always first even if the order is different .. Anyway i have sent this question to postfix maillist and i will wait their reply .
Thanks again Robert.
At 1AM +0300 on 26/10/12 you (Robert JR) wrote:
On 2012-10-26 00:15, Ben Morrow wrote:
As Stan said earlier, this is a Postfix question. The rule for
[Looking back at the thread it wasn't Stan, it was Dennis Guhl. Sorry about that.]
dotlocking is that you must create the .lock *before* opening the file, in case whoever has it locked will be replacing the file altogether; but with fcntl locking you must acquire the lock *after* opening the file, since that's the way the syscall works. This means that if Postfix is going to use both forms of lock, it has to acquire a dotlock before it can look for a fcntl lock.
In other words: the methods in mailbox_delivery_lock are *not* tried in order, because they can't be. Dotlock is always tried first.
You should have compatible locking settings for all your programs accessing your mboxes. If Postfix is using dotlock, Dovecot should be using dotlock as well. If you don't have any local programs (mail clients, for instance) which require dotlocks, you should probably change Postfix to just use fcntl locks.
Thanks Ben for your valuable support and detailed explanation .. but according to your explanation dovecot documentation contains wrong explanation to the dotlock and that's why i was confused.
it is written as follow
** If multiple lock methods are used, which is usually the case since dotlocks aren't typically used for read locking, the order in which the locking is done is important. Consider if two programs were running at the same time, both use dotlock and fcntl locking but in different order:
Program A: fcntl locks the mbox Program B at the same time: dotlocks the mbox Program A continues: tries to dotlock the mbox, but since it's already dotlocked by B, it starts waiting Program B continues: tries to fcntl lock the mbox, but since it's already fcntl locked by A, it starts waiting Now both of them are waiting for each others locks. Finally after a couple of minutes they time out and fail the operation. **
So this means that the documentation as mentioned above is wrong and dotlock is always first even if the order is different ..
I just checked, and you are right: Dovecot *will* use the locking strategies in the order listed. This is different from Postfix, which *will not*, so the 'Postfix' section of the MboxLocking page on the wiki is incorrect.
For compatibility with Postfix, dotlock should always be listed first.
Note that you will still frequently see Postfix acquiring a dotlock but then failing to acquire a fcntl lock. Dovecot's mbox_read_locks is usually set to just 'fcntl', which means that when Dovecot has an mbox open read-only it won't take a dotlock. This isn't really a problem, assuming you don't use any programs locally which only take dotlocks; but if that is the case you might as well configure everything to just use fcntl locks, and forget dotlocks altogether.
Stan's earlier point is fundamentally correct: if you can treat the Dovecot mailstore as a black box, with mail going in through the LDA and LMTP and mail coming out through POP and IMAP, your life will be much easier. Traditional Unix mailbox locking strategies are *completely* insane, and if all you are doing is delivering mail from Postfix and reading it from Dovecot it would be better to avoid them altogether, and switch to dbox if you can. However, if you have any other programs which touch the mail spool (local or NFS mail clients, deliveries through procmail) this may not be possible.
Ben
On 10/25/2012 11:16 PM, Ben Morrow wrote:
At 1AM +0300 on 26/10/12 you (Robert JR) wrote:
On 2012-10-26 00:15, Ben Morrow wrote:
As Stan said earlier, this is a Postfix question. The rule for
[Looking back at the thread it wasn't Stan, it was Dennis Guhl. Sorry about that.]
I prodded him a second time, might have been off-list, and he finally posted there. So call it a team effort. ;)
Wietse has already replied, and in typical fashion, asked for "concrete" evidence that Postfix was performing fcntl before dotlock, because he obviously knows better than anyone that Postfix applies a dotlock first, which you already explained here.
dotlocking is that you must create the .lock *before* opening the file, in case whoever has it locked will be replacing the file altogether; but with fcntl locking you must acquire the lock *after* opening the file, since that's the way the syscall works. This means that if Postfix is going to use both forms of lock, it has to acquire a dotlock before it can look for a fcntl lock.
In other words: the methods in mailbox_delivery_lock are *not* tried in order, because they can't be. Dotlock is always tried first.
You should have compatible locking settings for all your programs accessing your mboxes. If Postfix is using dotlock, Dovecot should be using dotlock as well. If you don't have any local programs (mail clients, for instance) which require dotlocks, you should probably change Postfix to just use fcntl locks.
<doc stuff snipped>
but if that is the case you might as well configure everything to just use fcntl locks, and forget dotlocks altogether.
Yep. Postfix can use either or both. And, surprise, recommends using maildir to avoid mailbox locking entirely.
Stan's earlier point is fundamentally correct: if you can treat the Dovecot mailstore as a black box, with mail going in through the LDA and LMTP and mail coming out through POP and IMAP, your life will be much easier. Traditional Unix mailbox locking strategies are *completely* insane, and if all you are doing is delivering mail from Postfix and reading it from Dovecot it would be better to avoid them altogether, and switch to dbox if you can. However, if you have any other programs which touch the mail spool (local or NFS mail clients, deliveries through procmail) this may not be possible.
And since this is a POP only server, users' MUAs should be deleting after download, so there shouldn't be much mail in these mbox files at any given time, making migration to maildir or dbox relatively simple.
When using Dovecot LDA you'll eliminate the filesystem level locking problems with mbox. However, you may still have read/write contention within Dovecot, such as in your 20MB download as new mail arrives example, especially if the new message has an xx MB attachment. I don't believe Dovecot is going to start appending a new message while it's still reading out the existing 20MB of emails. Depending on how long this takes Dovecot may still issue a 4xx to Postfix, which will put the new message in the deferred queue. With maildir or dbox, reading existing mail and writing new messages occurs concurrently, as each message is a different file.
-- Stan
participants (5)
-
/dev/rob0
-
Ben Morrow
-
Dennis Guhl
-
Robert JR
-
Stan Hoeppner