- dovecot - dovecot.org

[Dovecot] dsync is SLOW compared to rsync
by Jeff Gustafson 29 Mar '12

29 Mar '12

Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root@n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M /home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root@n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real 0m2.677s user 0m3.184s sys 0m1.513s [root@n24 bu]# time dsync backup -u testuser(a)domain.com \ mdbox:/home/bu/testuser real 1m9.519s user 1m7.592s sys 0m1.126s [root@n24 bu]# time dsync backup -u testuser(a)domain.com \ sdbox:/home/bu/testuser2 real 1m2.164s user 1m0.882s sys 0m0.993s [root@n24 bu]#

7 15

[Dovecot] Dovecot 1.2.9 Crash, NFS
by Müller Lukas 29 Mar '12

29 Mar '12

Hi, I'm stuck with a problem we have with dovecot. My suspicion is, that it has to do with accessing the same mailbox/mail stored on a NFS-share from two machines at the same time. setup We have to mail servers running, both run a Ubuntu 10.04, Postfix 2.70 and Dovecot 1.2.9. The mailboxes are stored in maildir format on a NFS-Share. In front of those to mail servers we have a load balancer. Unfortunately it can't be set up to use the same server for each domain, but it uses the same server for the same source-ip for at least 1 hour. Here is the output of dovecot -n: # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-33-server x86_64 Ubuntu 10.04.3 LTS nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/ca-bundle/SSL123_CA_Bundle.pem ssl_cert_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.crt ssl_key_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.key ssl_verify_client_cert: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_max_userip_connections: 25 mail_privileged_group: mail mail_location: maildir:/data/vmail/%d/%n:INDEX=/data/vmail/%d/%n/indexes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): auth default: passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf plugin: quota: maildir:storage=409600 sieve_global_path: /data/vmail/globalsieverc dict: quotadict: mysql:/etc/dovecot-dict-quota.conf problem the problem happens with a client's mailbox that is used by multiple users. >From time to time he cannot see any Emails in the mailbox, neither with his mail clients (Apple Mail) nor with in the webmail (Roundcube). Around this time I get the following entries in the log files: Mar 6 08:26:31 mail02 dovecot: IMAP(user(a)example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1524, file=1327500903.V15I5722c8M210039.mail01:2,Se) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7f37602b8b5a] -> imap(+0xaebc7) [0x7f37602b8bc7] -> imap(+0xae238) [0x7f37602b8238] -> imap(+0x497d7) [0x7f37602537d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f37602545c2] -> imap(+0x4bb06) [0x7f3760255b06] -> imap(maildir_uidlist_sync_init+0x4d) [0x7f376025652d] -> imap(+0x46ed4) [0x7f3760250ed4] -> imap(maildir_storage_sync_init+0x147) [0x7f3760251557] -> imap(imap_sync_init+0x70) [0x7f376023b190] -> imap(+0x2411e) [0x7f376022e11e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f37602c069c] -> imap(io_loop_handler_run+0x60) [0x7f37602c1000] -> imap(io_loop_run+0x18) [0x7f37602c0448] -> imap(main+0x58e) [0x7f376023bc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f375f877c4d] -> imap(+0x21979) [0x7f376022b979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 16934 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7ff81b415b5a] -> imap(+0xaebc7) [0x7ff81b415bc7] -> imap(+0xae238) [0x7ff81b415238] -> imap(+0x497d7) [0x7ff81b3b07d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7ff81b3b15c2] -> imap(maildir_uidlist_sync_init+0x105) [0x7ff81b3b35e5] -> imap(+0x46ed4) [0x7ff81b3aded4] -> imap(maildir_storage_sync_force+0x52) [0x7ff81b3ae392] -> imap(maildir_file_do+0x99) [0x7ff81b3b3cb9] -> imap(+0x4d944) [0x7ff81b3b4944] -> imap(index_mail_set_seq+0x148) [0x7ff81b3c8ed8] -> imap(index_storage_search_next_nonblock+0x162) [0x7ff81b3cd622] -> imap(mailbox_search_next_nonblock+0x20) [0x7ff81b3db2c0] -> imap(mailbox_search_next+0x26) [0x7ff81b3db316] -> imap(imap_fetch_more+0x2bf) [0x7ff81b39295f] -> imap(cmd_fetch+0x36c) [0x7ff81b38a9ec] -> imap(+0x28fad) [0x7ff81b38ffad] -> imap(+0x2908d) [0x7ff81b39008d] -> imap(client_handle_input+0x135) [0x7ff81b3902c5] -> imap(client_input+0x5f) [0x7ff81b390baf] -> imap(io_loop_handler_run+0xbd) [0x7ff81b41e05d] -> imap(io_loop_run+0x18) [0x7ff81b41d448] -> Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): imap(main+0x58e) [0x7ff81b398c5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ff81a9d4c4d] -> imap(+0x21979) [0x7ff81b388979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 13712 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7f4a311fcb5a] -> imap(+0xaebc7) [0x7f4a311fcbc7] -> imap(+0xae238) [0x7f4a311fc238] -> imap(+0x497d7) [0x7f4a311977d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f4a311985c2] -> imap(+0x47023) [0x7f4a31195023] -> imap(maildir_storage_sync_init+0x147) [0x7f4a31195557] -> imap(imap_sync_init+0x70) [0x7f4a3117f190] -> imap(+0x2411e) [0x7f4a3117211e] -> imap(+0x64c0e) [0x7f4a311b2c0e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f4a3120469c] -> imap(io_loop_handler_run+0x60) [0x7f4a31205000] -> imap(io_loop_run+0x18) [0x7f4a31204448] -> imap(main+0x58e) [0x7f4a3117fc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f4a307bbc4d] -> imap(+0x21979) [0x7f4a3116f979] Mar 6 08:42:31 mail01 dovecot: dovecot: child 24257 (imap) killed with signal 6 (core dumps disabled) Note: the first part is on a mail server, while the smaller second part is on the other one. Unfortunately I'm not able to reproduce this error. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. I have no idea how to solve this problem and any help is greatly appreciated. If you need further information, please say so. Mit freundlichen Grüssen Lukas Müller Systems Engineer _______________________________________________ NEWMEDIA Südostschweiz Newmedia AG Kasernenstrasse 1 Postfach 508, CH-7007 Chur http://www.newmedia.ch<http://www.newmedia.ch/> _______________________________________________ TYPO3 & Drupal - Wir wissen wie. Ihre professionelle Web Agentur in Chur, Ilanz, Glarus und Zürich.

2 3

Re: [Dovecot] Many messages clustered around the same date.saved value
by Joseph Tam 29 Mar '12

29 Mar '12

Timo Sirainen wrote: >> However, I noticed a strange thing: querying what would have been >> deleted >> >> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >> >> showed many date.saved values are clustered around the same timestamp, >> even among different user's Trash mailbox. >> ... >> I can't explain why many different users would have messages with the >> same (or closeby) date.saved value. > > Which mailbox format? With Maildir the date.saved is taken from > dovecot.index.cache file, and in some cases that might get dropped. If > it does, then it fallbacks to using the file's ctime. mbox. A further look into this reveals that the clustered date.saved values are the earliest values for every mailbox in the system. This timestamp is close to the time I was testing "doveadm ... -A", so the likely explanation is that I accidentally deleted/updated these values using some variation of doveadm, even though I remember confining my testing to query/search/fetch. This appears to be a case of PEBKAC. These "wrong" values shouldn't cause problems with expunge queries since they err on the side of safety. Thanks for the insight though. Joseph Tam <jtam.home(a)gmail.com>

2 1

[Dovecot] doveadm purge on clusterfs
by Jan-Frode Myklebust 29 Mar '12

29 Mar '12

Since doveadm service proxying apparently doesn't work with dovecot v2.0, we need to find a way to safely run doveadm purge on the host the user is logged into. Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? -jf

2 1

[Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook
by stonegate 29 Mar '12

29 Mar '12

Hi, i use dovecot 2.1.3 on a gentoo system. Before i installed my new imap server box, everything worked. Now with Dovecot it does not. Problem: When i receive a new email, it does not appear in my Outlook unless i have the IMAP Inbox Folder open (highlighted selection). Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize it unless i click on the inbox folder. Before that problem occured on my old system ( i think it was dovecot as well ) the inbox folder refreshed automatically and i instantly knew when i had new emails. It was kind of like with my exchange account. Does anyone have a clue what might be wrong ? Since i did not re-install outlook or something it cannot be a client thing since no settings changed on the client side. Maybe i have to do something with dovecots config file? I think it should keep the connection to the server open or something. I´d appreciate any kind of help. Thanks alot Stoney Dovecot is capable of the following things: telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. Thats my dovecot.conf: # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-gentoo-r2 x86_64 Gentoo Base System release 2.0.3 auth_mechanisms = plain login listen = * disable_plaintext_auth = no mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } #passdb { # args = * # driver = pam #} #passdb { # args = /etc/dovecot/dovecot-sql.conf.ext # driver = sql #} passdb { driver = shadow } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = </etc/ssl/dovecot/servercert.pem ssl_key = </etc/ssl/dovecot/serverkey.pem userdb { driver = passwd } #userdb { # args = /etc/dovecot/dovecot-sql.conf.ext # driver = sql #} userdb { driver = passwd -- View this message in context: http://old.nabble.com/Dovecot---IMAP---New-Mails-are-not-shown-unless-you-o… Sent from the Dovecot mailing list archive at Nabble.com.

2 1

[Dovecot] zlib_save per namespace/mailbox?
by Lutz Preßler 29 Mar '12

29 Mar '12

Hi, the zlib_save question reminds me of a wish: I think it's not possible to set zlib_save parameter per namespace (or even mailbox). Per namespace would be something for the wish list to get rid of the cron job method to compress archival mailboxes. And maybe an option to add a "Z" flag to compressed maildir message files as recommended in the wiki regarding compress crob job. Lutz

3 2

[Dovecot] Shared mailboxes with dovecot problem service=lib-storage
by Tomislav Mihalicek 29 Mar '12

29 Mar '12

Hi Could someone explain what this strings mean in dovecot 2.1.3 debug log? Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 test1(a)example.net service=lib-storage Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 test2(a)example.net service=lib-storage -- View this message in context: http://old.nabble.com/Shared-mailboxes-with-dovecot-problem-service%3Dlib-s… Sent from the Dovecot mailing list archive at Nabble.com.

2 1

[Dovecot] Namespace, prefix questions
by Steve Campbell 29 Mar '12

29 Mar '12

We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) disable_plaintext_auth = no listen = * mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mbox_write_locks = fcntl namespace { hidden = yes inbox = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = separator = / type = private } namespace { hidden = yes list = no location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = "#mbox/" separator = / type = private } namespace { hidden = yes list = no location = prefix = mail/ separator = / type = private } namespace { hidden = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = ~/mail/ separator = / type = private } namespace { hidden = yes list = no location = prefix = ~%u/mail/ separator = / type = private } passdb { driver = pam } protocols = pop3 imap ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } Thanks steve campbell Thanks steve campbell

2 2

[Dovecot] fts-solr not indexing body content
by Bård Johannessen 29 Mar '12

29 Mar '12

This could easily be me missing something, but I can't seem to get the fts-solr plugin to index message bodies. Tcpdump shows me the following being sent from Dovecot to Solr as a messages is indexed: <add><doc><field name="uid">6549</field><field name="box">fde08816e80d6b4f26650000b5f0b4b2</field><field name="user">user</field><field name="id">6549/fde08816e80d6b4f26650000b5f0b4b2/user</field><field name="body"> </field><field name="hdr">Return-path: ... As you can see, the <field name="body"/>-node contains just an empty line. The above is just a snippet to illustrate the problem. A full dump can be found at the following URL: http://db.org/temp/solr.xml.txt Full text search is configures such: plugin { fts = solr fts_solr = break-imap-search url=http://127.0.0.1:8080/solr/ } So; am I missing something, or is this a Dovecot problem? dovecot.conf: http://db.org/temp/dovecot.conf Regards, Bård Johannessen

2 4

[Dovecot] Merge mails from two mail_locations
by Patrick Westenberg 29 Mar '12

29 Mar '12

Hi guys, recently I had some trouble with my ocfs2 cluster and it unmounted itself from /var/mail. Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. Regards Patrick

2 1