May 2010 - dovecot - dovecot.org

[Dovecot] Overly long email of miscellaneous Dovecot migration questions
by Mark Moseley 26 May '10

26 May '10

Hello to the list! I've been asked to spec out the feasibility and, if feasible, plan a migration from Courier to Dovecot for both POP3 and IMAP for about 4 million mailboxes. I've been trying to absorb all the dovecot-related info I could over the past couple of weeks from the docs and from the list and I'm sure that I've muddled at least some of it, so apologies in advance if I make grossly incorrect assumptions. BTW, sorry for the giant email but I've got a bunch of questions. This is the sort of overly wordy email that makes my eyes glaze over in mailing lists. But I've got things configured to where (I think) everything will 'just work' if we migrate (barring running POP3 mailbox conversions), so I'm not looking for configuration help, just gotchas and assumption-checking mostly. Me breaking 4 million mailboxes isn't really a good move for my job security ;) * Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at. * This is a pretty fantastic piece of software. I love the neat little details that make admins' lives easier, e.g. how easy it is to use multiple SSL certs in the same server. That made me smile. I hadn't looked at dovecot in a number of years, so I was really impressed at the full feature set. It's been a while since I've played with a new server application that I've enjoyed this much. * Background: All of our mail is stored on NFS and will be for the foreseeable future, all in maildir format. Courier POP3/IMAP runs on load balanced Linux servers, so clients will be hitting multiple servers, though stickiness based on client IP address is currently done. Webmail as well is done on top of a local Courier IMAP server accessing the same NFS servers but on an entirely different pool of servers. All authentication is done against SQL. Courier has been very, very good to us over the past 9 years, but our NFS servers (12 Netapps) are unfortunately being beaten to death. We've got lots of users with thousands of emails in their inbox (i.e. in ./Maildir/cur/ alone) and I've seen plenty of mailboxes with 10's of thousands in their inbox and I've even seen some really scary ones approaching 100k emails just in cur/. Thanks to an endless cycle of industry raising mailbox limits, we actually have to support this (though not so much the 100k message ones, at least). * Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir. I know the cool thing to do would be to move the index files to a separate NFS server running entirely on SSDs for the fastest accesses possible. Building a separate index NFS cluster obviously adds complexity and performance with the index files in the maildir seems pretty great already. But I also know that real life often makes assumptions made during benchmarks look very foolish. My inclination is to try things out with the indexes in the maildir directory (and then start looking at moving to a dedicated NFS server for the indexes in a subsequent phase) but I don't want to regret that choice later. Anybody with large-scale NFS maildir stores have any advice either way? * Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things. * Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?: Since migrating POP mailboxes is going to be a mighty task, our gameplan would probably be to start migrating *just* IMAP services at first and likely on just a few servers at a time. Being load balanced, there are several scenarios where a user could theoretically access a mailbox on different servers and therefore end up hitting dovecot at one point in time and then later hitting courier or vice versa. Or they could use IMAP from one location and POP3 (during the span of time before we migrated our POP3 services) from another. I've not yet seen any side effects from switching mail servers like that (though I'm aware of what would happen if both dovecot and courier *POP3* were in service at the same time -- redownloaded mail, bleh). Has anyone seen otherwise? I'm not worried as much by the 0.1% of IMAP users who might be using advanced features but rather the other 99.9% who are using it pretty vanilla-ishly. * Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible. I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work. That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces). * Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file. Trying to update quota counts in SQL for 4 million mailboxes is a non-starter for us at this point. >From what I've done, dovecot seems pretty happy to maintain the quota using the 'maildir' quota plugin but let it be overridden by the userdb lookup. So I think I'm fine on that count, but I'd love to hear from anyone knowing of extreme gotchas doing it that way. * Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not. * One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls'). * The courier-dovecot-migrate.pl does a fine job at converting our POP3 mailboxes. We'll definitely be doing a mass run of it across all of our mailboxes. But with this many mailboxes, there's no way to get around the time lag between the script getting run and new mail showing up. Which means that there'll be some number of new messages that will re-download in POP3 when the switchover to Dovecot POP3 is done. To keep from getting too many complaints from customers, I'm thinking that I might have to write a wrapper script to do something like compare the mtime of ./Maildir/new/ against dovecot-uidlist and overwrite if ./Maildir/new/ is newer or perhaps see if the mtime of courierpop3dsizelist is newer than dovecot-uidlist and only if that's the case execute the conversion script with --overwrite. Considering we get between 500-2000 POP3 logins/sec, running any script after a POP3 login terrifies me--even a 6 line bash script. Anybody have any opinions to share on that? With a pooled architecture, it's near impossible to only have some mailboxes hit Courier and some hit Dovecot and keep slowly incrementing. At best my available units of increments to migrate are a few thousand at a time and more realistically 10-20k at a time. If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life. A big thanks to anyone who even actually reads this entire tedious email and a tremendous thanks to anyone who actually replies to any of it!

6 15

[Dovecot] with dovecot deliver amavisd not work
by fakessh 26 May '10

26 May '10

hello all reader hello list hello dovecot network since I've installed dovecot deliver. e-mails no longer pass through amavisd. amavisd no longer work. c is to say I have no anti-spam and anti virus my postconf and dovecot -n [root@r13151 ~]# postconf -n alias_database = hash:/etc/aliases , hash:/etc/postfix/aliases alias_maps = hash:/etc/aliases , hash:/etc/postfix/aliases body_checks = regexp:/etc/postfix/body_checks.cf broken_sasl_auth_clients = yes command_directory = /usr/sbin config_directory = /etc/postfix content_filter = dkimproxy:[127.0.0.1]:10029 daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 2 default_privs = nobody double_bounce_sender = no header_checks = regexp:/etc/postfix/header_checks.cf home_mailbox = Maildir/ in_flow_delay = 10 inet_interfaces = all local_recipient_maps = unix:passwd.byname $alias_maps mail_owner = postfix mail_spool_directory = /var/spool/mail mailbox_command = /usr/libexec/dovecot/deliver mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man message_size_limit = 20480000 mime_header_checks = regexp:/etc/postfix/mime_header_checks.cf mydestination = $myhostname , localhost.$mydomain, r13151.ovh.net mydomain = r13151.ovh.net mynetworks = 127.0.0.0/8 ,87.98.186.232 myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix queue_run_delay = 200s readme_directory = /usr/share/doc/postfix-2.5.4/README_FILES recipient_delimiter = + relay_domains = sample_directory = /usr/share/doc/postfix-2.5.4/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtp_sasl_security_options = noanonymous smtp_sasl_tls_security_options = noanonymous smtp_sender_dependent_authentication = yes smtp_tls_loglevel = 3 smtp_tls_note_starttls_offer = yes smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_tls_session_cache smtpd_banner = $myhostname ESMTP $mail_name ($mail_version) smtpd_client_restrictions = permit_mynetworks,reject_unknown_reverse_client_hostname,reject_unauth_pipelining, reject_non_fqdn_recipient , permit smtpd_milters = inet:[127.0.0.1]:10040 smtpd_recipient_restrictions = permit_mynetworks permit_inet_interfaces permit_sasl_authenticated reject_unverified_recipient reject_non_fqdn_sender reject_non_fqdn_recipient reject_unknown_sender_domain reject_unknown_recipient_domain reject_unknown_reverse_client_hostname reject_unauth_destination reject_unauth_pipelining reject_rbl_client zen.spamhaus.org reject_sender_login_mismatch check_policy_service unix:postgrey/socket check_sender_access hash:/etc/postfix/check_backscatterer check_sender_access hash:/etc/postfix/check_spamcannibal check_policy_service unix:private/spfpolicy reject_rhsbl_sender dbl.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client cbl.abuseat.org reject_rbl_client b.barracudacentral.org check_client_access hash:/etc/postfix/whitelist reject_rhsbl_helo dbl.spamhaus.org reject_rhsbl_client dbl.spamhaus.org reject_unknown_helo_hostname reject_invalid_helo_hostname reject_non_fqdn_helo_hostname check_client_access pcre:/etc/postfix/ptr-tld.pcre check_client_access cidr:/etc/postfix/sinokorea.cidr check_client_access cidr:/etc/postfix/taiwancidr.cidr check_client_access regexp:/etc/postfix/blacklist_clients check_client_access cidr:/etc/postfix/asian-ip.cidr reject_rbl_client relays.orbs.org reject_rbl_client in.dnsbl.org smtpd_reject_unlisted_sender = no smtpd_sasl_auth_enable = yes smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = $myhostname smtpd_sasl_path = private/auth smtpd_sasl_type = dovecot smtpd_tls_CAfile = /etc/pki/tls/certs/class3.crt smtpd_tls_ask_ccert = yes smtpd_tls_auth_only = yes smtpd_tls_cert_file = /etc/pki/tls/certs/r13151.ovh.net.crt smtpd_tls_key_file = /etc/pki/tls/private/r13151.ovh.net.key smtpd_tls_received_header = yes smtpd_tls_req_ccert = no smtpd_tls_session_cache_database = btree:/var/lib/postfix/smtpd_tls_session_cache smtpd_use_tls = yes soft_bounce = no tls_random_source = dev:/dev/urandom unknown_local_recipient_reject_code = 550 virtual_alias_domains = renelacroute.fr , nicolaspichot.fr , fakessh.eu virtual_alias_maps = hash:/etc/postfix/virtual virtual_transport = dovecot [root@r13151 ~]# dovecot -n # 1.2.11: /etc/dovecot.conf # OS: Linux 2.6.24.5-grsec-xxxx-grs-ipv4-32 i686 CentOS release 5.5 (Final) base_dir: /var/run/dovecot/ log_path: /var/log/maillog log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s managesieve listen(default): [::] listen(imap): [::] listen(pop3): [::] listen(managesieve): *:2000 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_listen(managesieve): ssl_ca_file: /etc/pki/tls/certs/root.crt ssl_cert_file: /etc/pki/tls/certs/r13151.ovh.net.crt ssl_key_file: /etc/pki/tls/private/r13151.ovh.net.key ssl_verify_client_cert: yes version_ignore: yes login_dir: /var/run/dovecot//login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login mail_location: maildir:~/Maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): autocreate mail_plugins(imap): autocreate mail_plugins(pop3): autocreate mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/managesieve imap_client_workarounds(default): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(pop3): imap_client_workarounds(managesieve): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_client_workarounds(managesieve): managesieve_logout_format(default): bytes=%i/%o managesieve_logout_format(imap): bytes=%i/%o managesieve_logout_format(pop3): bytes=%i/%o managesieve_logout_format(managesieve): bytes ( in=%i : out=%o ) lda: postmaster_address: postmaster(a)fakessh.eu hostname: r13151.ovh.net mail_plugins: autocreate cmusieve sieve mail_plugin_dir: /usr/lib/dovecot/lda sieve_global_path: /var/sieve/globalc sendmail_path: /usr/lib/sendmail auth_socket_path: /var/run/dovecot/auth-master auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 438 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 438 plugin: plugin: autocreate managesieve cmusieve sieve sieve_before: /var/sieve-scripts/roundcube.sieve sieve: ~/.dovecot.sieve sieve_dir: ~/sieve

2 1

[Dovecot] listescape und sieve
by Heiko Schlittermann 26 May '10

26 May '10

Hello, bug or feature? I'm using Dovecot 1.2.11. And because of a migration projekt I'm tempted to use the listescape plugin. If I understood well, it plugin { escape_char = "%" } should change the escape character to %. But it doesnt. Bug or feature? But more important: # sieve fileinto "a.b"; does not work as expected - the folder is visible as "a.b" in a normal imap client - because of the listescape plugin. # sieve fileinto "a\\2eb"; works. But the user won't understand this. Any ideas? Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann HS12-RIPE ----------------------------------------- gnupg encrypted messages are welcome - key ID: 48D0359B --------------- gnupg fingerprint: 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B -

2 2

[Dovecot] Dovecot 2.0beta4: Managesieve example config
by Thomas Leuxner 26 May '10

26 May '10

Hi, the 'example-config' holds an incorrect value for the listener socket: $ head -10 20-managesieve.conf ## ## ManageSieve specific settings ## # Service definitions service managesieve-login { #inet_listener sieve { # port = 4190 #} inet_listener sieve <> inet_listener managesieve Regards Thomas

5 6

[Dovecot] dovecot-sieve and managesieve for 2.0bX vanished ?
by Leon Meßner 26 May '10

26 May '10

Hi, im trying to build 2.0b5 with dovecot-sieve and possibly maangesieve. Does anyone know where i can get those for 2.0b5 ? There is no repo listed for these at http://hg.rename-it.nl/ cheers and TIA, Leon

3 3

[Dovecot] mdbox: Is there any more info on the "cleanup" step? (Dovecot 2.0beta5)
by William Blunn 26 May '10

26 May '10

The wiki page which mentions mdbox talks about a "cleanup" step for mdbox: : The space is later freed in "cleanup" step. This may be done : automatically within the session or later in a nightly cronjob when : there's less disk I/O. The cleanup first finds all files that have : refcount=0 mails. Then it goes through each file and copies the : refcount>0 mails to other dbox files (to the same files as where newly : saved messages would also go), updates the map index and finally : deletes the original file. So there is never any overwriting or file : truncation. I was wondering if there was any code to actually do this, or any command-line tools (and if either of those where they might be), or whether it was just design musings at the minute. Regards, Bill

2 1

[Dovecot] mdbox: What format is the file "storage/dovecot.map.index" in? (Dovecot 2.0beta5)
by William Blunn 26 May '10

26 May '10

I am looking at "mdbox" under Dovecot 2.0beta5. Looking at the documentation for "mdbox" I saw that we had a directory "storage" containing files "m.#" where "#" is an integer and a map file ("dovecot.map.index"). I think I expected the map file to be a plain text file with one record per line, but in fact it appears to contain binary data. According to the wiki ( http://wiki.dovecot.org/MailboxFormat/dbox ) the records go as: - map_uid: Unique growing 32 bit number for the message. - refcount: 16 bit reference counter for this message. Each time the message is copied the refcount is increased. - file_id: File number containing the message. For example if file_id=5, the message is in file m.5. - offset: Offset to message within the file. - size: Space used by the message in the file, including all metadata. The map file I have here is 1688 bytes in size (and I think there are 64 messages stored). Prime factors of 1688 are 2^3 x 211, so I think it is something other than just a straight array of fixed-size structures. Still, I can always look at the source if I really want to know :-) Regards, Bill

2 1

Re: [Dovecot] quota dict mysql not working (2.0.beta4)
by Timo Sirainen 26 May '10

26 May '10

On Mon, 2010-05-24 at 00:47 +0400, Andrey Melnikov wrote: > after mysql_query() - mysql return right data (affected rows = 1), but > there is no result to store (this is UPDATE/DELETE query), > next unconditional call to mysql_store_result() return nothing (right, > there no result) and reset affected_rows to -1ULL. > > You should modify driver_mysql_query_s() to call mysql_field_count() > first, to ensure - there is any data to read or no. But the whole point of the code is to get number of affected rows for previous UPDATE, so that if there is no quota row for user, the quota gets recalculated. This fixed it for my tests: http://hg.dovecot.org/dovecot-2.0/rev/0c15a760dab8

2 1

[Dovecot] newbie question about shared folders
by Tom Lieuallen 26 May '10

26 May '10

I'm preparing to move our site to dovecot 1.2.11 and have a question about shared folders. We're currently using UW imap's mbx mailboxes for shared folders. We accomplish the sharing by use of file system permissions and ACLs. I notice that dovecot's method of sharing is more based on IMAP ACLs and apparently file system permissions are useful, but not necessary. On the main dovecot page, it says: "Dovecot allows mailboxes and their indexes to be modified by multiple computers at the same time..." So, my question is whether I need to use the dovecot imap sharing namespace in order to safely share imap folders, or whether I can just use file system permissions as we have been using? My biggest concern is safe concurrent access to the shared folders, which I'm planning to implent with mbox files for now. thank you Tom Lieuallen Oregon State University College of Engineering

2 1

[Dovecot] Busy looping of 2.0Beta4 on NetBSD-current
by Scott Ellis 26 May '10

26 May '10

I built and install Dovecot 2.0Beta4 on my NetBSD/amd64-current system, and am seeing the user-owned imap processes busy-looping. In Dovecot 1.2.x (and 1.x and 0.x..) I force ionotify and ioloop to kqueue, so I suspected that was the problem. Turns out it's not: Using poll or none also causes the same problem. This did not happen in 1.2.8 (didn't try anything later than that). After a quick visit to gdb, I added some i_debug() statements in ioloop-kqueue.c (when I had that as the notify and loop method) which showed that the loop was in io_loop_handler_run() . I haven't done any similar debugging for the other ioloop methods, but a quick look at the 2.0 changelog for those files doesn't raise any immediate alarms. Any ideas what to look for here? Timo, any ideas? 1.2.x worked perfectly so it's something "new" in 2.0. ScottE

3 5