November 2011 - dovecot

[Dovecot] Corrupted transaction log file
by Peter Mogensen 09 Nov '11

09 Nov '11

Hi, I'm experiencing a problem I need some pointers to debug. I'm running Dovecot 2.0.15 and have a client which keeps causing log-entries like: Nov 4 15:10:42 mail dovecot: imap (test(a)aaaone.net) Error: Corrupted transaction log file /mail/3340444/.TestMails/dovecot.index.log seq 2: indexid changed 1320419300 -> 1320419441 (sync_offset=0) Nov 4 15:10:42 mail dovecot: imap(test(a)aaaone.net): Error: Corrupted transaction log file /mail/3340444/.TestMails/dovecot.index.log seq 2: Invalid transaction log size (4476 vs 4360): /mail/3340444/. TestMails/dovecot.index.log (sync_offset=4476) Nov 4 15:10:42 mail dovecot: imap(test(a)aaaone.net): Disconnected: IMAP session state is inconsistent, please relogin. bytes=549/100474 The client issues a lot of IMAP connections like this: A1_0 LOGIN "test(a)aaaone.net" "password" A1_1 ENABLE QRESYNC CONDSTORE * OK Waiting for authentication process to respond.. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. A1_0 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in * ENABLED QRESYNC CONDSTORE A1_1 OK Enabled. A2_0 CREATE "INBOX.folder" A2_0 OK Create completed. A3_0 SUBSCRIBE "INBOX.folder" A3_0 OK Subscribe completed. A4_0 APPEND "INBOX.folder" ($CreatedByImapClient-test) {20487+} ... more similar appends. But it only seems to be reproducible, when there's another client involved. Starting up Thunderbird with a login to the account seems to be enough to cause this. (To my knowledge this causes a series of STATUS command and then IDLE). The installation is using Maildir and lock_method = fcntl on local disks. /Peter

2 4

[Dovecot] Please advise on very fast search
by Alexander Chekalin 09 Nov '11

09 Nov '11

Hello, I try to create some kind of mail backup system. What I need is system that will store mail for the whole domain, and allow me to restore messages from/to specified email at that domain. The scheme is pretty simple: on our main mail server the SMTP server itself has a rule to send a copy of every message to 'backup(a)backupserver.host', and the backupserver.host domain is placed nearby on second server. The SMTP on second server do simple 'catchall' redirect of all messages to the single box. There is also a Dovecot that takes care for remote IMAP access to that box. And, finally, I've create some scripts to sort all messages in INBOX to folders named after message's date. So I have a lot of mailboxes inside the catchall box: INBOX 2011.11.03 2011.11.04 2011.11.05 2011.11.06 ...etc... and each folder holds messages for that day. Simply, and works perfectly. The problem is that when my archive become big (several years), it appears to be painful to find specified message(s). When someone suddenly needs to find his/her old message, it is mostly guesses like 'I think the message was between june and july of 2009, or maybe month or two before that', so I need to search all mailboxes (with 1000's messages in each). And it takes really long time. I tried to play with Dovecot indexes, but it won't help too much. The bad part is that I need to search for all emails in each message headers, not only for "From" or "To", since some messages are sent to maillists soe "To" = list address, not person's personal email. Then I tried to index messages on my own, storing info on emails into MySQL database ('email' -> 'mailbox', 'message filename'), but soon I find out that message files can be renamed by Dovecot. Could you please advice me how to speed up message search? Sorry for such a long question, hope you can help! Yours, Alexander Chekalin

3 2

Re: [Dovecot] Doveadm scrip-fu to determine alternate storage needs
by Maria Arrea 09 Nov '11

09 Nov '11

Timo, maybe you have mispelled the command? > We are using dovecot 2.0.15+mdbox+zlib and we are evaluating to buy > cheap sata-based iscsi storage / sata-NFS. We want to evaluate how > much space would be needed in alternate storage BEFORE buying it. I > have been reading about doveadm altmove. We want to move ALL messages > older than a year to alternate storage, what doveadm-script-fu could I > do to identify how much space would go from a user to alternate > storage? doveadm -A -f tab fetch size.physical before 1y would print all the message sizes. You could then sum them up with some scripting to produce the total. Unfortunately they are the uncompressed sizes, but maybe simply dividing by 2 would give approximate size? :) (You could get the compressed sizes from dovecot.map.index files with doveadm dump and mapping their contents to the older than 1 year message UIDs, but this gets difficult.)

2 1

[Dovecot] Doveadm scrip-fu to determine alternate storage needs
by Maria Arrea 09 Nov '11

09 Nov '11

Hello We are using dovecot 2.0.15+mdbox+zlib and we are evaluating to buy cheap sata-based iscsi storage / sata-NFS. We want to evaluate how much space would be needed in alternate storage BEFORE buying it. I have been reading about doveadm altmove. We want to move ALL messages older than a year to alternate storage, what doveadm-script-fu could I do to identify how much space would go from a user to alternate storage? Regards Maria

2 1

Re: [Dovecot] Bus factor of dovecot / GIT
by Peer Heinlein 09 Nov '11

09 Nov '11

Am Mittwoch, 9. November 2011, 10:29:37 schrieb Thomas Koch: Hi, > somebody from the kolab groupware project recently explained me that the > Bus- Factor[1] of around 1 would be one of their primary reasons not to > use Dovecot and stick with Cyrus. The problem is: Even it Timo will be hitten by a bus, everybody could stay on the actual Dovecot-version and everything's fine. Cyrus will IMO never reach the level, that Dovecot already has. :-) Peer P.S.: Timo! Don't leave your house. It's too dangerous for you. -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

[Dovecot] Multiple Patitions with with mdbox
by Peer Heinlein 09 Nov '11

09 Nov '11

Having > 10 TByte mailstore filesystem-checks takes too much time. At the moment we have four different partitions, but I don't like to set symlinks or LDAP-flags to sort customers and their domains to there individual mount-point. I'd like to work with mdbox:/mail/%d/%n to calculate the path automatically. How do you handle >> 10 TB mailstore? I'm very interested in the feature "alternative mailstore" with mdbox, because that makes it very easy to use at least TWO filesystems without any tricky configuration. I think I'd love to have <n> alternative mailstores. Why does dbox doesn't look for its m.*-files in more then two directorys? Sure, looking in 4 directorys would lead to four disc operations, but maybe it could be very helpful. Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 2

[Dovecot] another sizeof tweak
by Mike Abbott 08 Nov '11

08 Nov '11

Line 776 of dovecot-2.0.15/src/lib-storage/index/maildir/maildir-sync-index.c reads: memcmp(old_rec, &new_rec, sizeof(old_rec)) != 0) { Should that be sizeof(*old_rec)?

2 2

[Dovecot] Dovecot alias and mailbox problem
by Jeroen Grusewski 08 Nov '11

08 Nov '11

Hi All, My Postfix / Dovecot / MySQL is running fine but I have the following issue: When I create the following mailboxes test(a)example.com and test1(a)example.com it is working fine. I can create an alias test2(a)examples.com => test1(a)example.com that is also working without a problem, only when I create test(a)example.com => test1(a)example.com email is only delivered to test1@.... instead to both test@.... and test1@.... I believe it is properly just a setting but please help me out here ! Let me know if you need any more information before I post configs etc. Best regards, Jeroen

2 1

[Dovecot] dsync and zlib
by Peer Heinlein 08 Nov '11

08 Nov '11

Will dsync also use zlib-compression when copying mails from one side to another? Wouldn't this be a good way to compress existing mails? http://wiki2.dovecot.org/Plugins/Zlib says, there's no way to do that. But dsync should respect the zlib-Plugin... Peer -- Heinlein Professional Linux Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030/405051-42 Fax: 030/405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

[Dovecot] Failing to share folders when listescape is enabled (2.0.14)
by mailing lists 08 Nov '11

08 Nov '11

Hello, How I can share a folder with dots when the listescape plugin is enabled? In this example user001 is sharing two foders named "docs-abc" and "docs-a.b.c" to user002. The first folder (without dots) is seen by user002 but the second is not found by dovecot because it search a system folder named ".docs-a.b.c" but the system folder is ".docs-a\2eb\2ec" Nov 8 11:03:52 imap2 dovecot: imap(user002): Debug: acl vfile: file /var/virtual-maildir/user001/.docs-a.b.c/dovecot-acl not found drwx------ 2 vmail vmail 4096 2011-11-08 10:54 cur drwx------ 5 vmail vmail 4096 2011-11-08 10:57 .docs-a\2eb\2ec drwx------ 5 vmail vmail 4096 2011-11-08 10:57 .docs-abc -rw------- 1 vmail vmail 20 2011-11-08 10:57 dovecot-acl-list -rw------- 1 vmail vmail 248 2011-11-08 10:55 dovecot.index.log -rw------- 1 vmail vmail 96 2011-11-08 10:54 dovecot.mailbox.log -rw------- 1 vmail vmail 51 2011-11-08 10:55 dovecot-uidlist -rw------- 1 vmail vmail 8 2011-11-08 10:57 dovecot-uidvalidity -r--r--r-- 1 vmail vmail 0 2011-11-08 10:54 dovecot-uidvalidity.4eb8fc5b drwx------ 5 vmail vmail 4096 2011-11-08 10:57 .Drafts -rw------- 1 vmail vmail 15 2011-11-08 10:54 maildirsize drwx------ 2 vmail vmail 4096 2011-11-08 10:54 new drwx------ 5 vmail vmail 4096 2011-11-08 10:54 .Sent drwx------ 5 vmail vmail 4096 2011-11-08 10:54 .Spam -rw------- 1 vmail vmail 23 2011-11-08 10:54 subscriptions drwx------ 2 vmail vmail 4096 2011-11-08 10:54 tmp drwx------ 5 vmail vmail 4096 2011-11-08 10:54 .Trash # cat /var/virtual-maildir/user001/subscriptions Trash Sent Drafts Spam docs-abc docs-a\2eb\2ec # telnet localhost 143 Trying ::1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. . login user001 XXXXX . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA ACL RIGHTS=texk] Logged in . create docs-abc . OK Create completed. . create docs-a.b.c . OK Create completed. . setacl docs-abc user002 lrsw . OK Setacl complete. . setacl docs-a.b.c user002 lrsw . OK Setacl complete. . subscribe docs-abc . OK Subscribe completed. . subscribe docs-a.b.c . OK Subscribe completed. . logout * BYE Logging out . OK Logout completed. Connection closed by foreign host. # telnet localhost 143 Trying ::1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. . login user002 XXXXXX . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA ACL RIGHTS=texk] Logged in . subscribe shared/user001/docs-abc . OK Subscribe completed. . subscribe shared/user001/docs-a.b.c . NO Mailbox doesn't exist: shared/user001/docs-a.b.c . logout * BYE Logging out . OK Logout completed. Connection closed by foreign host. /-----------------/ # dovecot -n # 2.0.14: /etc/dovecot/dovecot.conf # OS: Linux 2.6.34.7-0.7-xen x86_64 openSUSE 11.3 (x86_64) auth_debug = yes auth_verbose = yes base_dir = /var/run/dovecot/ dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no doveadm_proxy_port = 24244 doveadm_socket_path = localhost:24244 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_debug = yes mail_fsync = always mail_gid = 5000 mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = acl quota notify expire listescape mail_uid = 5000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mmap_disable = yes namespace { inbox = yes list = yes location = prefix = separator = / subscriptions = yes type = private } namespace { list = children location = maildir:/var/virtual-maildir/%%n:INDEX=~/shared.%%n prefix = shared/%%n/ separator = / subscriptions = no type = shared } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/var/maildir/vol00/shared-mailboxes autocreate = Trash autocreate2 = Sent autocreate3 = Drafts autocreate4 = Spam autosubscribe = Trash autosubscribe2 = Sent autosubscribe3 = Drafts autosubscribe4 = Spam expire = Trash expire2 = Trash/* expire5 = Spam expire6 = Spam/* expire_dict = proxy::expire mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid from subject size vsize flags mail_plugins = " mail_log notify" quota = maildir:Quota quota_exceeded_message = Quota limit reached quota_rule2 = Trash:storage=+50M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=90%% quota-warning 90 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } service auth { unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service doveadm { inet_listener { port = 24244 } } service lmtp { inet_listener lmtp { port = 24 } unix_listener lmtp { group = vmail mode = 0666 user = vmail } } service pop3-login { inet_listener pop3 { port = 110 } } service quota-warning { executable = script /etc/dovecot/quota-warning.sh unix_listener quota-warning { user = vmail } } ssl = no userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } verbose_proctitle = yes protocol lmtp { mail_plugins = acl quota notify expire listescape sieve } protocol lda { mail_plugins = acl quota notify expire listescape sieve } protocol imap { mail_max_userip_connections = 100 mail_plugins = acl quota notify expire listescape imap_acl autocreate imap_quota } protocol pop3 { pop3_uidl_format = %v-%u } local 10.1.245.101/28/28 { doveadm_password = secret }

2 1