[Dovecot] imap locking spool?
How does IMAP lock the /var/spool/mail/user file?
Here's my problem: I'm writing a script to expire old or oversize mail through a series of archive folders. I don't want to take the whole server down to do this, nor does it seem like I would have to lock the user out entirely if I can manage to emulate the locks correctly.
I am able to lock /var/spool/mail/user (and ~user/mail/*) from procmail, dovecot popmail, and openwebmail using dotlocks.
What do I do to prevent dovecot imap from writing the spool and various mbox files while my script is running? Just posix fcntl on the various files?
I'm trying to use Mail::Box in perl which I'm having some problems with, corresponding with the developer, but if anyone has any easy way out I would appreciate the advice very much!
Mark
On Tue, 2009-03-03 at 13:54 -0800, Mark Hedges wrote:
How does IMAP lock the /var/spool/mail/user file?
mbox_read_locks and mbox_write_locks defines it.
Here's my problem: I'm writing a script to expire old or oversize mail through a series of archive folders. I don't want to take the whole server down to do this, nor does it seem like I would have to lock the user out entirely if I can manage to emulate the locks correctly.
How about doing it via IMAP? Then you wouldn't need to worry about locking or anything else.
1 select mailbox 2 search before 01-jun-2006 3 store <those messages> +flags.silent \deleted 4 expunge
On Tue, 3 Mar 2009, Timo Sirainen wrote:
On Tue, 2009-03-03 at 13:54 -0800, Mark Hedges wrote:
How does IMAP lock the /var/spool/mail/user file?
mbox_read_locks and mbox_write_locks defines it.
mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl
Unfortunately openwebmail only respects dotlocks. (or flock, but procmail doesn't use flock, or doesn't say it does.)
So what if my script is reading the messages to delete with IMAP, then a user logs into openwebmail, reads their inbox, selects messages to delete, but then my script deletes them, and then they try the delete operation on the mailbox that is no longer in sync?
Here's my problem: I'm writing a script to expire old or oversize mail through a series of archive folders. I don't want to take the whole server down to do this, nor does it seem like I would have to lock the user out entirely if I can manage to emulate the locks correctly.
How about doing it via IMAP? Then you wouldn't need to worry about locking or anything else.
1 select mailbox 2 search before 01-jun-2006 3 store <those messages> +flags.silent \deleted 4 expunge
That would be great if it meant that users could not log into openwebmail at the same time, but it doesn't, AFAICT, without establishing dotlocks. I am just assuming that users will not log in through IMAP and openwebmail at the same time.
I like my original suggestion of turning off all services while the script runs, but that's not a solution, apparently.
I'm also thinking maybe I can lock the user's password, use
verbose_proctitle to make sure they're not logged in and/or
kick the individual out with kill, then run the script,
which I've verified does block procmail delivery.
The matrix has you!
Mark
On Tue, 2009-03-03 at 14:20 -0800, Mark Hedges wrote:
On Tue, 3 Mar 2009, Timo Sirainen wrote:
On Tue, 2009-03-03 at 13:54 -0800, Mark Hedges wrote:
How does IMAP lock the /var/spool/mail/user file?
mbox_read_locks and mbox_write_locks defines it.
mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl
Unfortunately openwebmail only respects dotlocks. (or flock, but procmail doesn't use flock, or doesn't say it does.)
You can change those settings. For example:
mbox_read_locks = flock mbox_write_locks = dotlock flock
Now Dovecot is compatible with openwebmail. And with procmail both are still protected by the dotlock.
On Tue, 3 Mar 2009, Timo Sirainen wrote:
How does IMAP lock the /var/spool/mail/user file? mbox_read_locks and mbox_write_locks defines it.
mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl
Unfortunately openwebmail only respects dotlocks. (or flock, but procmail doesn't use flock, or doesn't say it does.)
You can change those settings. For example:
mbox_read_locks = flock mbox_write_locks = dotlock flock
Now Dovecot is compatible with openwebmail. And with procmail both are still protected by the dotlock.
No, openwebmail only has the choice of either dotlock or flock, but not both. So in this case, procmail could deliver using dotlock but openwebmail wouldn't know because it's only watching for flock. Welcome to my hell!
Mark
On Tue, 2009-03-03 at 14:55 -0800, Mark Hedges wrote:
On Tue, 3 Mar 2009, Timo Sirainen wrote:
How does IMAP lock the /var/spool/mail/user file? mbox_read_locks and mbox_write_locks defines it.
mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl
Unfortunately openwebmail only respects dotlocks. (or flock, but procmail doesn't use flock, or doesn't say it does.)
You can change those settings. For example:
mbox_read_locks = flock mbox_write_locks = dotlock flock
Now Dovecot is compatible with openwebmail. And with procmail both are still protected by the dotlock.
No, openwebmail only has the choice of either dotlock or flock, but not both. So in this case, procmail could deliver using dotlock but openwebmail wouldn't know because it's only watching for flock. Welcome to my hell!
Ah. Well, the last choice is then:
mbox_read_locks = dotlock mbox_write_locks = dotlock
Dotlocking for reading isn't really optimal, but shouldn't really make much of a difference usually.
On Tue, 3 Mar 2009, Timo Sirainen wrote:
You can change those settings. For example:
mbox_read_locks = flock mbox_write_locks = dotlock flock
Now Dovecot is compatible with openwebmail. And with procmail both are still protected by the dotlock.
No, openwebmail only has the choice of either dotlock or flock, but not both. So in this case, procmail could deliver using dotlock but openwebmail wouldn't know because it's only watching for flock. Welcome to my hell!
Ah. Well, the last choice is then:
mbox_read_locks = dotlock mbox_write_locks = dotlock
Dotlocking for reading isn't really optimal, but shouldn't really make much of a difference usually.
I tried that before, that would be great if it worked for imap, but I've verified that the dotlock file does not get created in /var/spool/mail/username.lock when I open the imap connection, at least for reading. It is created when I open the popmail connection (but that was already working.) That's why I'm asking.
Mar 3 15:08:53 anubis dovecot: auth(default): client in: AUTH 1 PLAIN service=imap secured lip=127.0.0.1 rip=127.0.0.1 lport=143 rport=49237 resp=<hidden>
Mar 3 15:08:53 anubis dovecot: auth(default): client out: OK 1 user=despam_test_anubis
Mar 3 15:08:53 anubis dovecot: auth-worker(default): pam(despam_test_anubis,127.0.0.1): lookup service=dovecot
Mar 3 15:08:53 anubis dovecot: auth-worker(default): pam(despam_test_anubis,127.0.0.1): #1/1 style=1 msg=Password:
Mar 3 15:08:53 anubis dovecot: auth(default): master in: REQUEST 8 22446 1
Mar 3 15:08:53 anubis dovecot: auth(default): passwd(despam_test_anubis,127.0.0.1): lookup
Mar 3 15:08:53 anubis dovecot: auth(default): master out: USER 8 despam_test_anubis system_user=despam_test_anubis uid=511 gid=100 home=/home/anubis/despam_test_anubis
Mar 3 15:08:53 anubis dovecot: IMAP(despam_test_anubis): Effective uid=511, gid=100, home=/home/anubis/despam_test_anubis
Mar 3 15:08:53 anubis dovecot: IMAP(despam_test_anubis): mbox: data=~/mail:INBOX=/var/spool/mail/despam_test_anubis
Mar 3 15:08:53 anubis dovecot: IMAP(despam_test_anubis): fs: root=/home/anubis/despam_test_anubis/mail, index=, control=, inbox=/var/spool/mail/despam_test_anubis
Mar 3 15:08:53 anubis dovecot: imap-login: Login: user=
On Mar 3, 2009, at 6:11 PM, Mark Hedges wrote:
Ah. Well, the last choice is then:
mbox_read_locks = dotlock mbox_write_locks = dotlock
Dotlocking for reading isn't really optimal, but shouldn't really make much of a difference usually.
I tried that before, that would be great if it worked for imap, but I've verified that the dotlock file does not get created in /var/spool/mail/username.lock when I open the imap connection, at least for reading. It is created when I open the popmail connection (but that was already working.)
If you set mbox_read_locks=dotlock, it should get created whenever
Dovecot is reading the mbox file. But it's created ONLY during those
times. For example opening a mailbox whose index files are up-to-date
doesn't cause dotlock to be created. And when the mbox is actually
being read, the dotlock is deleted immediately after the IMAP command
is finished.
Does openwebmail create longer existing locks?
On Tue, 3 Mar 2009, Timo Sirainen wrote:
If you set mbox_read_locks=dotlock, it should get created whenever Dovecot is reading the mbox file. But it's created ONLY during those times. For example opening a mailbox whose index files are up-to-date doesn't cause dotlock to be created. And when the mbox is actually being read, the dotlock is deleted immediately after the IMAP command is finished.
It was confusing me crazy that imap waits to establish locks until the read or write action happens. I think I got things worked out now with all the right options. Thanks for all your help!
Mark
on 3-3-2009 2:55 PM Mark Hedges spake the following:
On Tue, 3 Mar 2009, Timo Sirainen wrote:
How does IMAP lock the /var/spool/mail/user file? mbox_read_locks and mbox_write_locks defines it. mbox_read_locks = fcntl mbox_write_locks = dotlock fcntl
Unfortunately openwebmail only respects dotlocks. (or flock, but procmail doesn't use flock, or doesn't say it does.) You can change those settings. For example:
mbox_read_locks = flock mbox_write_locks = dotlock flock
Now Dovecot is compatible with openwebmail. And with procmail both are still protected by the dotlock.
No, openwebmail only has the choice of either dotlock or flock, but not both. So in this case, procmail could deliver using dotlock but openwebmail wouldn't know because it's only watching for flock. Welcome to my hell!
Mark
You could use dovecot's deliver and then you won't need to worry about procmails locking.
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
on 3-3-2009 3:15 PM Mark Hedges spake the following:
On Tue, 3 Mar 2009, Scott Silva wrote:
You could use dovecot's deliver and then you won't need to worry about procmails locking.
That's not going to happen either, a large portion of the application depends on procmail filtering, and that is totally entrenched.
Mark
Looking at their list it seems as if flock is emulated by calling the base fcntl locking routines, so you might be OK on this one.
http://openwebmail.acatysmoof.com/archive/html/owm-users/owm-users.200811/ms...
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
On Tue, 3 Mar 2009, Timo Sirainen wrote:
If you set mbox_read_locks=dotlock, it should get created whenever Dovecot is reading the mbox file. But it's created ONLY during those times. For example opening a mailbox whose index files are up-to-date doesn't cause dotlock to be created. And when the mbox is actually being read, the dotlock is deleted immediately after the IMAP command is finished.
Does openwebmail create longer existing locks?
*sigh*. No. It releases the locks as soon as it reads. (The CGI process ends.) So if my script accesses by IMAP, which also releases the locks, openwebmail can change the folder in the meantime, regardless of what lock mechanism I use.
On Tue, 3 Mar 2009, Scott Silva wrote:
Looking at their list it seems as if flock is emulated by calling the base fcntl locking routines, so you might be OK on this one.
http://openwebmail.acatysmoof.com/archive/html/owm-users/owm-users.200811/ms...
That would be great if it were true, but when I set openwebmail to use 'flock', then open the spool file directly with Mail::Box using 'POSIX' locking to keep it locked, I am still able to access openwebmail, so I could still change the file. If I use Mail::Box 'Flock' locking, it does lock me out of OWM. I think the guy means that he's imported the flock constants from Fcntl, and he thinks this is the same as using Fcntl. (A misconception that I had to battle with my predecessor's code, which assumed that a file was locked if you just imported the constants and did nothing.)
The lock code is here if anyone is interested: http://openwebmail.acatysmoof.com/dev/svnweb/index.pl/openwebmail/view/tags/...
Hrmm, rope, poison, or jumping off a bridge? At this point I would prefer a firing squad. An excuse to smoke again.
I think the only real answer is to lock the user's password, kill any dovecot processes with the username in verbose proctitle (-HUP? -KILL?), then run my expiration script on their mailboxes. OR, use horde.
Mark
On Wed, 2009-03-04 at 11:23 -0800, Mark Hedges wrote:
On Tue, 3 Mar 2009, Timo Sirainen wrote:
If you set mbox_read_locks=dotlock, it should get created whenever Dovecot is reading the mbox file. But it's created ONLY during those times. For example opening a mailbox whose index files are up-to-date doesn't cause dotlock to be created. And when the mbox is actually being read, the dotlock is deleted immediately after the IMAP command is finished.
Does openwebmail create longer existing locks?
*sigh*. No. It releases the locks as soon as it reads. (The CGI process ends.) So if my script accesses by IMAP, which also releases the locks, openwebmail can change the folder in the meantime, regardless of what lock mechanism I use.
Is it a problem if it does? As long as locking works, there shouldn't be problems regardless of who modifies the mailbox and how/when.
On 3/4/2009, Mark Hedges (hedges@scriptdolphin.org) wrote:
I think the only real answer is to lock the user's password, kill any dovecot processes with the username in verbose proctitle (-HUP? -KILL?), then run my expiration script on their mailboxes. OR, use horde.
I was going to say... apparently you must be locked into using openwebmail?
Roundcube is nice...
--
Best regards,
Charles
on 3-4-2009 11:23 AM Mark Hedges spake the following:
On Tue, 3 Mar 2009, Timo Sirainen wrote:
If you set mbox_read_locks=dotlock, it should get created whenever Dovecot is reading the mbox file. But it's created ONLY during those times. For example opening a mailbox whose index files are up-to-date doesn't cause dotlock to be created. And when the mbox is actually being read, the dotlock is deleted immediately after the IMAP command is finished.
Does openwebmail create longer existing locks?
*sigh*. No. It releases the locks as soon as it reads. (The CGI process ends.) So if my script accesses by IMAP, which also releases the locks, openwebmail can change the folder in the meantime, regardless of what lock mechanism I use.
On Tue, 3 Mar 2009, Scott Silva wrote:
Looking at their list it seems as if flock is emulated by calling the base fcntl locking routines, so you might be OK on this one.
http://openwebmail.acatysmoof.com/archive/html/owm-users/owm-users.200811/ms...
That would be great if it were true, but when I set openwebmail to use 'flock', then open the spool file directly with Mail::Box using 'POSIX' locking to keep it locked, I am still able to access openwebmail, so I could still change the file. If I use Mail::Box 'Flock' locking, it does lock me out of OWM. I think the guy means that he's imported the flock constants from Fcntl, and he thinks this is the same as using Fcntl. (A misconception that I had to battle with my predecessor's code, which assumed that a file was locked if you just imported the constants and did nothing.)
The lock code is here if anyone is interested: http://openwebmail.acatysmoof.com/dev/svnweb/index.pl/openwebmail/view/tags/...
Hrmm, rope, poison, or jumping off a bridge? At this point I would prefer a firing squad. An excuse to smoke again. You left out alcohol...
I think the only real answer is to lock the user's password, kill any dovecot processes with the username in verbose proctitle (-HUP? -KILL?), then run my expiration script on their mailboxes. OR, use horde.
Mark
Horde is probably the easiest option. I am running Horde now on my servers and it was pretty painless to install. Horde groupware webmail edition has almost all of the options that openwebmail has, and is much more stable and modern. Only one old CentOS 4 server is still running openwebmail, and I have been warning my users to switch over to Horde before I change out that server to a new one running CentOS 5.
Another plus with a standards compliant webmail system is that you can run maildir and your expiry scripts will have even less locking issues. You can just look at mtime or atime on individual mail files and move them.
-- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
participants (4)
-
Charles Marcus
-
Mark Hedges
-
Scott Silva
-
Timo Sirainen