March 2012 - dovecot - dovecot.org

[Dovecot] IMAP to Maildir Migration preserving UIDs?
by Gedalya 30 Mar '12

30 Mar '12

Hello all, I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. The mailbox format I would want to use is Maildir++. The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it? In either case I need this done, and I'll have to create whatever I can't find available. If there isn't anything out there that I'm yet to become aware of, then I'm looking at creating something like an offlineimap post-processing routine? Any help would be much appreciated. Gedalya

7 33

[Dovecot] Dovecot migration from any IMAP/POP3 server
by Timo Sirainen 30 Mar '12

30 Mar '12

With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: http://wiki2.dovecot.org/Migration/Dsync The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server.

2 1

[Dovecot] newbie: keep getting same emails in mail client
by neuronetv 30 Mar '12

30 Mar '12

dovecot-2.0.9-2.el6_1.1.i686 I've just set up dovecot in centos 6.2 (server install) and finally got it working (kind of). I set up a unix user (not a virtual user) sent a test email to this user but in my mail client I keep getting this test email over and over again. I don't think the fault is with the email client because other emails work fine and never duplicate and I've tweaked the account settings too, so it must be something I've done wrong in the dovecot setup. Here is my dovecot.conf file: # Dovecot configuration file protocols = pop3 imap disable_plaintext_auth = no mail_location = mbox:~/mail:INBOX=/var/spool/mail/unix-username ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem passdb { driver = pam } userdb { driver = passwd } thanks for any pointers. -- View this message in context: http://old.nabble.com/newbie%3A-keep-getting-same-emails-in-mail-client-tp3… Sent from the Dovecot mailing list archive at Nabble.com.

1 0

[Dovecot] SIS and restoring from backups
by Charles Marcus 29 Mar '12

29 Mar '12

On 2012-03-24 7:49 AM, Timo Sirainen <tss(a)iki.fi> wrote: > This is already optionally done in v2.0+dbox. MIME attachments can be > stored in plain binary form if they can be reconstructed back into > their original form. It doesn't break any signed stuff. Hey Timo, Splitting this off into a separate thread... On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? Or maybe I misunderstood the problem? -- Best regards, Charles

2 6

[Dovecot] Sieve fileinto and year/month folders.
by Xavier Beaudouin 29 Mar '12

29 Mar '12

Hi there, I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. One thing used by some of powerusers are archiving mail automatically with autocreated folders based on year + month. Is there any good way to make that with sieve... One example require "fileinto"; if address :is ["From", "To"] "dovecot(a)dovecot.org" { fileinto "INBOX.mls.%Y.%m.dovecot"; } This will fill any mails into INBOX.mls.2012.03.dovecot uppon receiving... I don't know if some sieve guru can tell me how to do that... ? Kind regards, Xavier

4 5

[Dovecot] Problem about dovecot Panic
by FABIO FERRARI 29 Mar '12

29 Mar '12

Good morning, we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some mail services in it (postfix and dovecot 2). The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). >From last week we have this dovecot problem: suddenly dovecot doesn't accept any new connections, the dovecot.log file reports lines like these Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ libdovecot.so.0 [0x36ea4362e8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) [0x36ea440b7f] -> /usr/li b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ imap [0x408179] Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 killed with signal 6 (core dumps disabled) Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command startup failed, throttling Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument and the kern.log file reports Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 can you help us? thanks in advance Fabio Ferrari

2 1

[Dovecot] dsync is SLOW compared to rsync
by Jeff Gustafson 29 Mar '12

29 Mar '12

Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root@n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M /home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root@n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real 0m2.677s user 0m3.184s sys 0m1.513s [root@n24 bu]# time dsync backup -u testuser(a)domain.com \ mdbox:/home/bu/testuser real 1m9.519s user 1m7.592s sys 0m1.126s [root@n24 bu]# time dsync backup -u testuser(a)domain.com \ sdbox:/home/bu/testuser2 real 1m2.164s user 1m0.882s sys 0m0.993s [root@n24 bu]#

7 15

[Dovecot] Dovecot 1.2.9 Crash, NFS
by Müller Lukas 29 Mar '12

29 Mar '12

Hi, I'm stuck with a problem we have with dovecot. My suspicion is, that it has to do with accessing the same mailbox/mail stored on a NFS-share from two machines at the same time. setup We have to mail servers running, both run a Ubuntu 10.04, Postfix 2.70 and Dovecot 1.2.9. The mailboxes are stored in maildir format on a NFS-Share. In front of those to mail servers we have a load balancer. Unfortunately it can't be set up to use the same server for each domain, but it uses the same server for the same source-ip for at least 1 hour. Here is the output of dovecot -n: # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-33-server x86_64 Ubuntu 10.04.3 LTS nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/ca-bundle/SSL123_CA_Bundle.pem ssl_cert_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.crt ssl_key_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.key ssl_verify_client_cert: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_max_userip_connections: 25 mail_privileged_group: mail mail_location: maildir:/data/vmail/%d/%n:INDEX=/data/vmail/%d/%n/indexes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): auth default: passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf plugin: quota: maildir:storage=409600 sieve_global_path: /data/vmail/globalsieverc dict: quotadict: mysql:/etc/dovecot-dict-quota.conf problem the problem happens with a client's mailbox that is used by multiple users. >From time to time he cannot see any Emails in the mailbox, neither with his mail clients (Apple Mail) nor with in the webmail (Roundcube). Around this time I get the following entries in the log files: Mar 6 08:26:31 mail02 dovecot: IMAP(user(a)example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:29 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1524, file=1327500903.V15I5722c8M210039.mail01:2,Se) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7f37602b8b5a] -> imap(+0xaebc7) [0x7f37602b8bc7] -> imap(+0xae238) [0x7f37602b8238] -> imap(+0x497d7) [0x7f37602537d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f37602545c2] -> imap(+0x4bb06) [0x7f3760255b06] -> imap(maildir_uidlist_sync_init+0x4d) [0x7f376025652d] -> imap(+0x46ed4) [0x7f3760250ed4] -> imap(maildir_storage_sync_init+0x147) [0x7f3760251557] -> imap(imap_sync_init+0x70) [0x7f376023b190] -> imap(+0x2411e) [0x7f376022e11e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f37602c069c] -> imap(io_loop_handler_run+0x60) [0x7f37602c1000] -> imap(io_loop_run+0x18) [0x7f37602c0448] -> imap(main+0x58e) [0x7f376023bc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f375f877c4d] -> imap(+0x21979) [0x7f376022b979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 16934 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7ff81b415b5a] -> imap(+0xaebc7) [0x7ff81b415bc7] -> imap(+0xae238) [0x7ff81b415238] -> imap(+0x497d7) [0x7ff81b3b07d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7ff81b3b15c2] -> imap(maildir_uidlist_sync_init+0x105) [0x7ff81b3b35e5] -> imap(+0x46ed4) [0x7ff81b3aded4] -> imap(maildir_storage_sync_force+0x52) [0x7ff81b3ae392] -> imap(maildir_file_do+0x99) [0x7ff81b3b3cb9] -> imap(+0x4d944) [0x7ff81b3b4944] -> imap(index_mail_set_seq+0x148) [0x7ff81b3c8ed8] -> imap(index_storage_search_next_nonblock+0x162) [0x7ff81b3cd622] -> imap(mailbox_search_next_nonblock+0x20) [0x7ff81b3db2c0] -> imap(mailbox_search_next+0x26) [0x7ff81b3db316] -> imap(imap_fetch_more+0x2bf) [0x7ff81b39295f] -> imap(cmd_fetch+0x36c) [0x7ff81b38a9ec] -> imap(+0x28fad) [0x7ff81b38ffad] -> imap(+0x2908d) [0x7ff81b39008d] -> imap(client_handle_input+0x135) [0x7ff81b3902c5] -> imap(client_input+0x5f) [0x7ff81b390baf] -> imap(io_loop_handler_run+0xbd) [0x7ff81b41e05d] -> imap(io_loop_run+0x18) [0x7ff81b41d448] -> Mar 6 08:42:31 mail02 dovecot: IMAP(user(a)example.com): imap(main+0x58e) [0x7ff81b398c5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ff81a9d4c4d] -> imap(+0x21979) [0x7ff81b388979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 13712 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail01 dovecot: IMAP(user(a)example.com): Raw backtrace: imap(+0xaeb5a) [0x7f4a311fcb5a] -> imap(+0xaebc7) [0x7f4a311fcbc7] -> imap(+0xae238) [0x7f4a311fc238] -> imap(+0x497d7) [0x7f4a311977d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f4a311985c2] -> imap(+0x47023) [0x7f4a31195023] -> imap(maildir_storage_sync_init+0x147) [0x7f4a31195557] -> imap(imap_sync_init+0x70) [0x7f4a3117f190] -> imap(+0x2411e) [0x7f4a3117211e] -> imap(+0x64c0e) [0x7f4a311b2c0e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f4a3120469c] -> imap(io_loop_handler_run+0x60) [0x7f4a31205000] -> imap(io_loop_run+0x18) [0x7f4a31204448] -> imap(main+0x58e) [0x7f4a3117fc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f4a307bbc4d] -> imap(+0x21979) [0x7f4a3116f979] Mar 6 08:42:31 mail01 dovecot: dovecot: child 24257 (imap) killed with signal 6 (core dumps disabled) Note: the first part is on a mail server, while the smaller second part is on the other one. Unfortunately I'm not able to reproduce this error. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. I have no idea how to solve this problem and any help is greatly appreciated. If you need further information, please say so. Mit freundlichen Grüssen Lukas Müller Systems Engineer _______________________________________________ NEWMEDIA Südostschweiz Newmedia AG Kasernenstrasse 1 Postfach 508, CH-7007 Chur http://www.newmedia.ch<http://www.newmedia.ch/> _______________________________________________ TYPO3 & Drupal - Wir wissen wie. Ihre professionelle Web Agentur in Chur, Ilanz, Glarus und Zürich.

2 3

Re: [Dovecot] Many messages clustered around the same date.saved value
by Joseph Tam 29 Mar '12

29 Mar '12

Timo Sirainen wrote: >> However, I noticed a strange thing: querying what would have been >> deleted >> >> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >> >> showed many date.saved values are clustered around the same timestamp, >> even among different user's Trash mailbox. >> ... >> I can't explain why many different users would have messages with the >> same (or closeby) date.saved value. > > Which mailbox format? With Maildir the date.saved is taken from > dovecot.index.cache file, and in some cases that might get dropped. If > it does, then it fallbacks to using the file's ctime. mbox. A further look into this reveals that the clustered date.saved values are the earliest values for every mailbox in the system. This timestamp is close to the time I was testing "doveadm ... -A", so the likely explanation is that I accidentally deleted/updated these values using some variation of doveadm, even though I remember confining my testing to query/search/fetch. This appears to be a case of PEBKAC. These "wrong" values shouldn't cause problems with expunge queries since they err on the side of safety. Thanks for the insight though. Joseph Tam <jtam.home(a)gmail.com>

2 1

[Dovecot] doveadm purge on clusterfs
by Jan-Frode Myklebust 29 Mar '12

29 Mar '12

Since doveadm service proxying apparently doesn't work with dovecot v2.0, we need to find a way to safely run doveadm purge on the host the user is logged into. Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? -jf

2 1