On 03/22/2010 03:23 PM, Steffen Kaiser wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 22 Mar 2010, Proskurin Kirill wrote:

...

What Debian folk`s do in this situation?

I install from source.

Not Debian way at all. Make package from source - may be.

Yes, looks like you got some wrong information. http://packages.debian.org/lenny-backports/i386/dovecot-common/filelist lists currently the content of Dovecot v1.2.11, it contains also: /usr/lib/dovecot/managesieve /usr/lib/dovecot/managesieve-login

Ops, thank you! Usually debian pkg is very splited and I thing what they cut sieve and managesieve from base package.

-- Best regards, Proskurin Kirill

Papp Tamas put forth on 3/22/2010 9:18 AM:

On Mon, Mar 22, 2010 at 03:07:26PM +0300, Proskurin Kirill wrote:

...

Hello!

I think about migrate from FreeBSD to Linux because I need DRBD for clustering.

"Need?" There are many paths to high availability, DRDB is but one. You could stick with BSD and implement an NFS server for shared mailbox storage. NFS is very popular for this purpose, widely deployed, well understood, and supportable. I'd guess there are many users on this list using shared NFS storage.

You could also install an inexpensive iSCSI SAN storage box and use a cluster file system such as GFS for mailbox storage, although I don't know if GFS is available for BSD.

Given the ubiquity of NFS, I'd recommend you stick with BSD for now and build a reliable NFS server for mailbox storage. You will be able to get ample assistance here and around the web for this setup. Dovecot has built in locking support for NFS storage. It can be done inexpensively as well, thanks to Ebay:

http://cgi.ebay.com/HP-Proliant-DL380-G4-dual-3-4ghz-4gb-6x72gb-10k_W0QQitem...

This is but an example, as this seller doesn't ship to Hungary. I'm sure you can find something similar to build your NFS server upon.

-- Stan

On Tue, Mar 23, 2010 at 05:42:02AM -0400, Charles Marcus wrote:

On 2010-03-22 9:31 PM, Stan Hoeppner wrote:

...

Dovecot has built in locking support for NFS storage.

But it has always been problematic, according to Timo.

Have you got any references on this, apart from http://wiki.dovecot.org/NFS ?

I'm looking at migrating a courier-imap installation (FreeBSD frontends, Netapp backends, Maildir++) to dovecot. I'd be grateful of any known pitfalls I should be looking out for.

I have done some small-scale testing and it looks fine. tcpdumping the NFS traffic, I see that the FreeBSD frontend is sending "access" requests to check that its local cache is not stale. In tests with mailboxes containing 100 messages and a web IMAP frontend (atmail.org), Dovecot was generating about 1/4 of the total NFS traffic compared to courier-imap, because of how Dovecot creates a cache file containing the message headers.

I see a link to http://www.freebsd.org/cgi/query-pr.cgi?pr=123755 in the NFS wiki page. Does this problem affect only mbox over NFS, or maildir too?

I've not observed any problem with courier-imap, although courier-imap is much dumber about caching, and also at the moment the majority of the userbase are on POP3 anyway.

Regards,

Brian.

On Tue, Mar 23, 2010 at 03:19:49PM +0200, Timo Sirainen wrote:

...

I have done some small-scale testing and it looks fine.

Stress testing by running imaptest for same user's same mailbox in 2+ different servers (i.e. two NFS clients reading/writing same mailbox files) should show up quickly what kind of errors you could get. http://imapwiki.org/ImapTest

OK, I've now set this up:

 ImapTest ---> dovecot (same host) -----> NFS server
         `---> dovecot (diff host) ----'

• 172.16.23.104: dovecot 1.2.11 and ImapTest-latest. FreeBSD 7.2.
• 172.16.23.101: dovecot 1.2.11 only. FreeBSD 7.2.
• 172.16.23.103: NFS server. Ubuntu Karmic.

All three hosts are ntpd synced.

The following was needed on the FreeBSD boxes to get fcntl locking working:

nfs_client_enable="YES" rpc_lockd_enable="YES" rpc_statd_enable="YES"

(imapd worked without these, but maillog showed errors about failing to obtain locks, "operation not supported")

Test results

• Pointing a single instance of imaptest at a single host, or two instances of imaptest at the same host (with clients=5 to avoid hitting the 15 client limit) was fine. ImapTest reported no errors, and nothing out of the ordinary in maillog.

$ egrep -v "Login:|Disconnected:|Aborted login" /var/log/maillog

• Things went badly wrong with two instances of imaptest pointing at different dovecot hosts. I had seen this sort of thing when I'd previously been using dot locking, and was hoping they'd be fixed by switching to fcntl, but unfortunately not.

ImapTest reported errors including:

Error: brian@dev.example.com[8]: SELECT failed: 8.3 NO [SERVERBUG] Internal error occurred. Refer to server log for more information. [2010-03-25 10:22:23]

• 6 stalled for 16 secs in command: 11 EXPUNGE

All sorts of errors reported in maillog, including:

Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (10) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Our dotlock file /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock was overridden (locked 0 secs ago, touched 0 secs ago) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (11) Mar 25 10:22:27 freebsd-dev dovecot: IMAP(brian@dev.example.com): /mail/0/6/37/30/brian%dev.example.com/dovecot.index reset, view is now inconsistent Mar 25 10:22:46 freebsd-dev dovecot: IMAP(brian@dev.example.com): Panic: file mail-transaction-log-view.c: line 108 (mail_transaction_log_view_set): assertion failed: (min_file_seq <= max_file_seq) Mar 25 10:22:48 freebsd-dev dovecot: IMAP(brian@dev.example.com): rename(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp, /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist) failed: No such file or directory Mar 25 10:22:48 freebsd-dev dovecot: IMAP(brian@dev.example.com): unlink(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp) failed: No such file or directory

Mar 25 10:22:36 wipe-dev dovecot: IMAP(brian@dev.example.com): ftruncate(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock) failed: Stale NFS file handle

(Logs from a single test run are attached)

Interestingly, these messages imply that dovecot is still using dotlocking in some circumstances, even though I've definitely set fcntl locking.

$ grep ^lock /usr/local/etc/dovecot.conf lock_method = fcntl

$ egrep '^mail_nfs|^mmap' /usr/local/etc/dovecot.conf mmap_disable = yes mail_nfs_storage = yes mail_nfs_index = yes

All this suggests I should use some sort of 'sticky' load balancing in front so that all client conns from one IP hit the same frontend box. However, that contradicts the experience Adam McDougall has had with a similar setup:

http://dovecot.org/list/dovecot/2010-March/047815.html

It's possible that switching the Linux NFS server to a Netapp will help (which is what it will be deployed onto eventually anyway)

Adam: did you do any tuning of FreeBSD client NFS settings? And have you tried using ImapTest, or just real IMAP users?

I see there are a few tunables:

$ grep nfs /etc/defaults/rc.conf netfs_types="nfs:NFS nfs4:NFS4 smbfs:SMB portalfs:PORTAL nwfs:NWFS" # Net filesystems. nfs_client_enable="NO" # This host is an NFS client (or NO). nfs_access_cache="60" # Client cache timeout in seconds nfs_server_enable="NO" # This host is an NFS server (or NO). nfs_server_flags="-u -t -n 4" # Flags to nfsd (if enabled). nfs_reserved_port_only="NO" # Provide NFS only on secure port (or NO). nfs_bufpackets="" # bufspace (in packets) for client

I have tried rerunning with sysctl vfs.nfs.access_cache_timeout=0 but saw the same problems.

Maybe the load pattern from 'real' IMAP clients is such that these problems generally don't show in practice? (i.e. it would be unusual for a single IMAP client to make simultaneous changes to the same folder via different TCP connections)

Regards,

Brian.

...

It's possible that switching the Linux NFS server to a Netapp will help (which is what it will be deployed onto eventually anyway)

NetApp should help, but I doubt it'll remove all the problems. Also Dovecot's NFS workarounds work better for Linux NFS client than for FreeBSD..

I doubt this is related to the specific problem, but I've had issues
with FreeBSD clients when using NFS over UDP. You might try forcing
tcp from them: mount -o tcp host:/mount /mountpoint

Rick

On Thu, 2010-03-25 at 15:14 +0000, Brian Candler wrote:

On Thu, Mar 25, 2010 at 04:31:56PM +0200, Timo Sirainen wrote:

...

...

Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Our dotlock file /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock was overridden (locked 0 secs ago, touched 0 secs ago)

Wonder if dotlock_use_excl=no helps with this.

Doesn't appear to make a difference. grepping for 'dotlock':

Are you using ext4 on the Linux NFS server? Nano/microsecond resolution timestamps fix the worst caching problems.

On Thu, 2010-03-25 at 16:04 +0000, Brian Candler wrote:

Out of interest, why is dot locking used instead of fcntl here?

My laziness. It's annoying to create different code for locking files via different ways in different places. (The files are created/written to in different ways, the locking can't simply be abstracted out into generic code. At least not too easily.)

Of course the planned filesystem abstraction for v2.1/v3.0 should help with these problems in future.

On 03/25/10 06:58, Brian Candler wrote:

On Tue, Mar 23, 2010 at 03:19:49PM +0200, Timo Sirainen wrote:

...

...

I have done some small-scale testing and it looks fine.

Stress testing by running imaptest for same user's same mailbox in 2+ different servers (i.e. two NFS clients reading/writing same mailbox files) should show up quickly what kind of errors you could get. http://imapwiki.org/ImapTest

OK, I've now set this up:

  ImapTest --->  dovecot (same host) ----->  NFS server
          `--->  dovecot (diff host) ----'

• 172.16.23.104: dovecot 1.2.11 and ImapTest-latest. FreeBSD 7.2.
• 172.16.23.101: dovecot 1.2.11 only. FreeBSD 7.2.
• 172.16.23.103: NFS server. Ubuntu Karmic.

All three hosts are ntpd synced.

The following was needed on the FreeBSD boxes to get fcntl locking working:

nfs_client_enable="YES" rpc_lockd_enable="YES" rpc_statd_enable="YES"

(imapd worked without these, but maillog showed errors about failing to obtain locks, "operation not supported")

Test results

• Pointing a single instance of imaptest at a single host, or two instances of imaptest at the same host (with clients=5 to avoid hitting the 15 client limit) was fine. ImapTest reported no errors, and nothing out of the ordinary in maillog.

$ egrep -v "Login:|Disconnected:|Aborted login" /var/log/maillog

• Things went badly wrong with two instances of imaptest pointing at different dovecot hosts. I had seen this sort of thing when I'd previously been using dot locking, and was hoping they'd be fixed by switching to fcntl, but unfortunately not.

ImapTest reported errors including:

Error: brian@dev.example.com[8]: SELECT failed: 8.3 NO [SERVERBUG] Internal error occurred. Refer to server log for more information. [2010-03-25 10:22:23]

• 6 stalled for 16 secs in command: 11 EXPUNGE

All sorts of errors reported in maillog, including:

Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (10) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Our dotlock file /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock was overridden (locked 0 secs ago, touched 0 secs ago) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(brian@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (11) Mar 25 10:22:27 freebsd-dev dovecot: IMAP(brian@dev.example.com): /mail/0/6/37/30/brian%dev.example.com/dovecot.index reset, view is now inconsistent Mar 25 10:22:46 freebsd-dev dovecot: IMAP(brian@dev.example.com): Panic: file mail-transaction-log-view.c: line 108 (mail_transaction_log_view_set): assertion failed: (min_file_seq<= max_file_seq) Mar 25 10:22:48 freebsd-dev dovecot: IMAP(brian@dev.example.com): rename(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp, /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist) failed: No such file or directory Mar 25 10:22:48 freebsd-dev dovecot: IMAP(brian@dev.example.com): unlink(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp) failed: No such file or directory

Mar 25 10:22:36 wipe-dev dovecot: IMAP(brian@dev.example.com): ftruncate(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock) failed: Stale NFS file handle

(Logs from a single test run are attached)

Interestingly, these messages imply that dovecot is still using dotlocking in some circumstances, even though I've definitely set fcntl locking.

$ grep ^lock /usr/local/etc/dovecot.conf lock_method = fcntl

$ egrep '^mail_nfs|^mmap' /usr/local/etc/dovecot.conf mmap_disable = yes mail_nfs_storage = yes mail_nfs_index = yes

All this suggests I should use some sort of 'sticky' load balancing in front so that all client conns from one IP hit the same frontend box. However, that contradicts the experience Adam McDougall has had with a similar setup:

http://dovecot.org/list/dovecot/2010-March/047815.html

It's possible that switching the Linux NFS server to a Netapp will help (which is what it will be deployed onto eventually anyway)

Adam: did you do any tuning of FreeBSD client NFS settings? And have you tried using ImapTest, or just real IMAP users?

I see there are a few tunables:

$ grep nfs /etc/defaults/rc.conf netfs_types="nfs:NFS nfs4:NFS4 smbfs:SMB portalfs:PORTAL nwfs:NWFS" # Net filesystems. nfs_client_enable="NO" # This host is an NFS client (or NO). nfs_access_cache="60" # Client cache timeout in seconds nfs_server_enable="NO" # This host is an NFS server (or NO). nfs_server_flags="-u -t -n 4" # Flags to nfsd (if enabled). nfs_reserved_port_only="NO" # Provide NFS only on secure port (or NO). nfs_bufpackets="" # bufspace (in packets) for client

I have tried rerunning with sysctl vfs.nfs.access_cache_timeout=0 but saw the same problems.

Maybe the load pattern from 'real' IMAP clients is such that these problems generally don't show in practice? (i.e. it would be unusual for a single IMAP client to make simultaneous changes to the same folder via different TCP connections)

Regards,

Brian.

I use: rc.conf: nfs_client_enable="YES" # This host is an NFS client (or NO). rpc_lockd_enable="YES" rpc_statd_enable="YES"

/etc/fstab: nfsserver:/vol/mail /egr/mail nfs rw,bg,tcp,nosuid 0 0

dovecot.conf: (some other things that helped in general, not necessarily locking related, some got line wrapped) login_max_processes_count: 512 max_mail_processes: 1024 mail_max_userip_connections: 25 mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_process_size: 1024 mail_log_max_lines_per_sec: 0 auth default: worker_max_request_count: 500

internal note: lock_method is "always dotlock for maildir" according

to dovecot author #lock_method = fcntl

I have played with the access cache but ultimately nothing resulted from it so I leave it un-tuned.

I have not tried imaptest with my servers but I just let them run with real clients, as long as I am not messing around with the back end files in bad ways, I don't really get the errors you turned up in real use. I have seen plenty of them in earlier versions of dovecot before there was code to flush the FreeBSD access cache well enough. I believe I remember Timo saying something about the timestamps on the NetApp NFS server being much more fine grained than some other NFS servers which could be helping me out.

On 23.3.2010, at 11.42, Charles Marcus wrote:

On 2010-03-22 9:31 PM, Stan Hoeppner wrote:

...

Dovecot has built in locking support for NFS storage.

But it has always been problematic, according to Timo. I think 2.0 is supposed to help considerably with NFS issues though,

No, v2.0 won't help at all with NFS issues. Some future version might.