- dovecot - dovecot.org

[Dovecot] imaptest: performance testing
by Nick Urbanik 16 Mar '12

16 Mar '12

Dear Folks, Using head of imaptest with dovecot 2.1, I am attempting to put a heavy load on the server, with insufficient success. I made 15000 user accounts, put them in to a file, one per line. Then I ran imaptest with ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 pass=SECRETPASSWORD But CPU load is only reaching 20. I put an mbox containing 67 messages into ~/mail/dovecot-crlf So what is the best way to put a really heavy load on an imap server? The documentation describes testing correctness of the server without really emphasising performance. I would like to do something like dnsperf, which adds a linearly increasing load until the server is unable to cope. This machine has 24G RAM and 8 cores. I'll be grateful for any constructive suggestions. -- Nick Urbanik http://nicku.org nicku(a)nicku.org GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24

3 2

[Dovecot] Upgrading from 2.0.x to 2.1.x
by Nikolaos Milas 16 Mar '12

16 Mar '12

Hi, A quick question: Are there any incompatibilities in config settings among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require changes of any config settings (as upgrading from v1.x to 2.0.x did)? Thanks, Nick

2 2

[Dovecot] Lack of external documentation?
by Steve Campbell 16 Mar '12

16 Mar '12

Firstly, this isn't meant to be critical, and I realize the subject line probably suggest criticism, so... I was sort of forced into using dovecot as my imap/pop server due to upgrading 3 versions of OS on my mail servers. So far, that's not bad. What surprises me is that one of the first things I usually do whenever I start using different software is to purchase a book that seems to suit me. Searching all of the common places like amazon, ebay, etc for manuals turned up little to nothing on dovecot. I'm wondering why and is this so new that people just haven't written books about it yet? The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Thanks for all the help I've received so far and I think I'm really going to like dovecot. Once I get the hang of it, I'll probably reduce the amount of noise on the list by half. steve campbell

8 8

[Dovecot] v2.1.2 released
by Timo Sirainen 16 Mar '12

16 Mar '12

http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig There are a ton of proxying related improvements in this release. You should now be able to do pretty much anything you want with Dovecot proxy/director. This release also includes the initial version of dsync-based replication. I'm already successfully using it for @dovecot.fi mails, but it still has some problems. See http://dovecot.org/list/dovecot/2012-March/064243.html for some details how to configure it. + Initial implementation of dsync-based replication. For now this should be used only on non-critical systems. + Proxying: POP3 now supports sending remote IP+port from proxy to backend server via Dovecot-specific XCLIENT extension. + Proxying: proxy_maybe=yes with host=<hostname> (instead of IP) works now properly. + Proxying: Added auth_proxy_self setting + Proxying: Added proxy_always extra field (see wiki docs) + Added director_username_hash setting to specify what part of the username is hashed. This can be used to implement per-domain backends (which allows safely accessing shared mailboxes within domain). + Added a "session ID" string for imap/pop3 connections, available in %{session} variable. The session ID passes through Dovecot IMAP/POP3 proxying to backend server. The same session ID is can be reused after a long time (currently a bit under 9 years). + passdb checkpassword: Support "credentials lookups" (for non-plaintext auth and for lmtp_proxy lookups) + fts: Added fts_index_timeout setting to abort search if indexing hasn't finished by then (default is to wait forever). - doveadm sync: If mailbox was expunged empty, messages may have become back instead of also being expunged in the other side. - director: If user logged into two directors while near user expiration, the directors might have redirected the user to two different backends. - imap_id_* settings were ignored before login. - Several fixes to mailbox_list_index=yes - Previous v2.1.x didn't log all messages at shutdown. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors.

4 4

[Dovecot] problems with SSH-based clustering dovecot 2.1.1
by Michescu Andrei 16 Mar '12

16 Mar '12

Hello, I'm using dovecot 2.1.1 with vpopmail 5.4.30 with multiples domains and I have problems setting up synchronization in between multiple computers. All act like master (my clients can connect to any of the them and read their emails either via POP3 either via IMAP, inbound email gets on any of the machines). Each machine is on a different continent, there is no shared drive in between and the synchronization is supposed to be asynchronous via cron scripts. To simplify the case, let's consider 2 machines (mx1.a and mx2.a) with 3 virtual domains (a and b and c). On both machine domain a is the default domain (needs only username to connect to imap). Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. I tried any of the following commands to synchronize the 2 machines: mx1.a$ doveadm -Dv sync -u user1@a -f ssh mx2.a doveadm dsync-server -u user1@a mx1.a$ doveadm -Dv sync -u user1@a ssh mx2.a doveadm -u user1@a mx1.a$ doveadm -Dv sync -u user1@a user1@a The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22). Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u <user> and attempts to sync the first email in the default domain. Please advise. # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail }

5 24

[Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0
by Tom Talpey 16 Mar '12

16 Mar '12

I'm seeing a few warnings emitted when building for x86. They're pretty obvious, but if you want the configure options etc, I can provide those. In Dovecot 2.1.2 (I also see some of these in 2.1.1): 1) src/lib-index/mail-cache-fields.c (comparison between two last_used fields) mail-cache-fields.c: In function 'mail_cache_header_fields_read': mail-cache-fields.c:406: warning: comparison between signed and unsigned 2) src/director/user-directory.c (comparison with ioloop_time) user-directory.c: In function 'user_directory_user_is_recently_updated': user-directory.c:147: warning: comparison between signed and unsigned 3) src/replication/replicator/replicator-brain.c (comparison with ioloop_time) replicator-brain.c: In function 'doveadm_replicate': replicator-brain.c:113: warning: comparison between signed and unsigned 4) src/replication/replicator/replicator-queue.c (comparison with ioloop_time) replicator-queue.c: In function 'replicator_queue_pop': replicator-queue.c:201: warning: comparison between signed and unsigned In Pigeonhole 0.3.0: 5) src/managesieve-login/client-authenticate.c (passing size_t * not uoff_t *) client-authenticate.c: In function 'managesieve_client_auth_read_response': client-authenticate.c:214: warning: passing argument 3 of 'i_stream_get_size' from incompatible pointer type

3 3

[Dovecot] 2.1.1: doveadm backup errors
by e-frog 15 Mar '12

15 Mar '12

Hello Timo, I'm experimenting with 'doveadm backup' on 2.1.1 (latest hg, full dovecot -n output attached) and haven't managed to get it working. This is what I have done: 1. Create the directory /tmp/backup which is empty 2. Run doveadm -v backup -u testuser(a)ubuntu-test.localdomain mdbox:/tmp/backup/ Then I see the following errors: doveadm -v backup -u testuser(a)ubuntu-test.localdomain mdbox:/tmp/backup/ dsync(testuser(a)ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. dsync(testuser(a)ubuntu-test.localdomain): Info: INBOX: only in dest (guid=9e4b88178b905f4f456e0000381555a6) dsync(testuser(a)ubuntu-test.localdomain): Info: INBOX: only in source (guid=bd05451f2fbb574d40600000ec8d17cd) dsync(testuser(a)ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser(a)ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox 9e4b88178b905f4f456e0000381555a6 dsync(testuser(a)ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser(a)ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) dsync(testuser(a)ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox bd05451f2fbb574d40600000ec8d17cd dsync(testuser(a)ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) It somehow finds INBOX in destination however the backup directory is newly created and empty. Thanks, e-frog

2 5

[Dovecot] Multiple locations, 2 servers - planning questions...
by Charles Marcus 15 Mar '12

15 Mar '12

Hello all/Timo, Up until now, my main Clients office has consisted of a single location, and I have never had to deal with the situation of multiple locations for a single company. They have just told me that they are acquiring an additional floor at a building that is about 4 minutes away - but obviously far enough away that I now have to deal with supporting users in the same domain but at two disparate physical locations. These two locations will be connected via a private Gb ethernet connection, and each location will have its own internet connection (I think - still waiting on some numbers to present to the owner to see what he wants to do in that regard, but that will be my recommendation), so bandwidth for replication won't be an issue. I have a couple of months to get this done, and I am already planning on hiring Timo's new commercial support company to help with the final and actual design and implementation, but obviously first I need to know what my actual options are. Just a rough idea of what I'd like to do is: 1. Set up one dovecot server at each location (these will be VMs), so users at each location are accessing the local server for that office 2. Full replication between the two for the mail/indexes, and configure them such that each can act as a failover for the other in case one goes down for whatever reason This is my first/main question... I recall that 'dsync based replication' is actually on the map for 2.1, but, since apparently dsync can't do this now, Timo, do you have even a rough idea how much work this would be to get it working for only 2 locations (assuming it *may* be easier to get the initial support for only 2 locations, my client may be willing to pay for it if it isn't a huge amount - feel free to reply privately to this question), then you could revisit it later to make it more scalable? Or, if it is going to take more work than my client is willing to pay for (I'm hoping not, since you said it was on the map for 2.1, not 2.2+), maybe the notify plugin could be leveraged in some way to provide something 'close enough' until it is fully implemented in dsync? On that note (something 'close enough' until dsync fully supports this natively), would setting up a dsync cron job, say, every 5 or 10 minutes, be asking for trouble? Our mail server is not all that busy, really, so in 5 or 10 minutes, there wouldn't be many changes at all. If that is not recommended, although I want to avoid the hassles of NFS if at all possible, maybe there is another shared filesystem that will work ok - or... since I will be forcing users to a single server always anyway, maybe NFS or some other shared filesystem is really the best option here, and just let it take care of the syncing? and 3. Configure things such that each offices users are directed to the local server for that office, but connections will fail-over to the remote server in the case of one of them going down for whatever reason? I'm fairly sure that some combination of Dovecot Proxy/Director will accomplish this, but one concern is - for internal users, my understanding is it will redirect them via the private IP, but that would result in lots of traffic across the Gb connection between the two locations, and I'd like to eliminate that if possible - so how will this work when they are accessing it from outside the office, where each office has its own public IP? I'd prefer to not rely on users using the correct hostname (currently, we just use 'mail.example.com', and I know I could set up two new ones - office1.example.com and office2.example.com - but then I'd be relying on the users to get it right, and I'd prefer to avoid that can of worms). I guess a worst case scenario (if there is no better way) would be to do it that way, then watch the logs for users who get it wrong and are using the inter-office connection, and deal with them on a case by case basis. Thanks to any/all for reading this far and for any thoughts, suggestions and/or ideas... -- Best regards, Charles

9 19

[Dovecot] director lmtp -> smtp problem
by Jim Lawson 15 Mar '12

15 Mar '12

Hi Timo & Dovecot users, We have a 2-node director setup which front-ends for 4 nodes which share a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. Approximately 40k users, but typically only a few thousand active at any time. The director nodes run sendmail, which deliver mail "locally" using LMTP to the director, which then feeds to SMTP on the real servers (also sendmail.) Why sendmail? Because procmail is used for mail filtering and as the delivery agent. Here's the problem, on the director: Mar 14 20:40:08 imapdir2 dovecot: lmtp(10692): Connect from local Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input ->eof) Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3d99a) [0x7f79156c499a] -> /usr/lib/doveco t/libdovecot.so.0(+0x3d9e6) [0x7f79156c49e6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f791569df8f] -> dovecot/lmtp() [0x406e77] -> /usr/l ib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f79156d0044] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f79156d 0c3b] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f79156cfca8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7915 6bdfc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7914ef8cdd] -> dovecot/lmtp() [0x403d69] Mar 14 20:40:38 imapdir2 sendmail[6905]: q2D8KodI018432: SYSERR(root): timeout writing message to localhost: Broken pipe Most mail goes through OK, but some messages do not and end up queued until they run into the queue time limit. So far as I have been able to tell, all of the messages have this failure when the following conversation takes place between sendmail (on director), the Dovecot LMTP proxy, and sendmail on the backend node (SMTP): (names mangled to protect the guilty) (first, sendmail -> director LMTP) > [root@imapdir2 ~]# sendmail -qIq2EFZt1p004708 -v > > Running /var/spool/mqueue/qd2/q2EFZt1p004708 (sequence 1 of 1) > <ntssdfwe(a)pobox2.uvm.edu>... Connecting to > /var/lib/dovecot/lmtp-socket via cyrusv2... > 220 imapdir2.uvm.edu Dovecot LMTP ready > >>> LHLO imapdir2.uvm.edu > 250-imapdir2.uvm.edu > 250-8BITMIME > 250-ENHANCEDSTATUSCODES > 250 PIPELINING > >>> MAIL From:<owner-dasfqeiuasd*Abrasf*-Gorpwe**UVM*-EDU(a)LIST.UVM.EDU> > 250 2.1.0 OK > >>> RCPT To:<ntssdfwe> > >>> DATA > 250 2.1.5 OK > 354 OK > timeout writing message to localhost: Broken pipe > <ntssdfwe(a)pobox2.uvm.edu>... Deferred > Closing connection to localhost The conversation between the director (LMTP) and the backend (sendmail SMTP) goes like this: > 250-penguinc.uvm.edu Hello imapdir2.uvm.edu [132.198.100.150], pleased > to meet you > 250-ENHANCEDSTATUSCODES > 250-PIPELINING > 250-8BITMIME > 250-SIZE 10485760 > 250-ETRN > 250-AUTH DIGEST-MD5 CRAM-MD5 > 250-DELIVERBY > 250 HELP > MAIL FROM:<owner-dasfqeiuasd*Abrasf*-Gorpwe**UVM*-EDU(a)LIST.UVM.EDU> > 250 2.1.0 <owner-dasfqeiuasd*Abrasf*-Gorpwe**UVM*-EDU(a)LIST.UVM.EDU>... > Sender ok > RCPT TO:<ntssdfwe> > 552 5.2.2 User ntssdfwe mailbox is full At this point Dovecot should return the failed RCPT TO: status back to sendmail over LMTP, but instead it sits there (waiting for a timeout to expire?) and eventually dies. doveconf -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.2.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.2 (Santiago) base_dir = /var/run/dovecot/ default_client_limit = 6000 default_process_limit = 10240 director_mail_servers = penguina.uvm.edu penguinb.uvm.edu penguinc.uvm.edu penguind.uvm.edu director_servers = imapdir1.uvm.edu imapdir2.uvm.edu lmtp_proxy = yes login_trusted_networks = [REDACTED] passdb { args = proxy=y nopassword=y protocol=smtp driver = static } service anvil { client_limit = 40000 } service auth { client_limit = 45960 unix_listener auth-userdb { group = mail mode = 0660 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director service_count = 0 } service imap { process_limit = 10240 vsz_limit = 1 G } service lmtp { client_limit = 1 inet_listener lmtp { port = 24 } unix_listener /var/lib/dovecot/lmtp-socket { group = root mode = 0600 user = root } } service pop3-login { executable = pop3-login director service_count = 0 } service pop3 { process_limit = 5000 } shutdown_clients = no ssl_cert = <[REDACTED].pem ssl_key = <[REDACTED].key userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol lmtp { auth_socket_path = director-userdb } protocol imap { mail_max_userip_connections = 100 } Hope you can help, Jim Lawson

2 6

[Dovecot] v2.0.19 released
by Timo Sirainen 15 Mar '12

15 Mar '12

http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz.sig Hopefully one of the last v2.0.x releases. - IMAP: ENABLE CONDSTORE/QRESYNC + STATUS for a mailbox might not have seen latest external changes to it, like new mails. - imap_id_* settings were ignored before login. - doveadm altmove did too much work sometimes, retrying moves it had already done. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors.

1 0