- dovecot - dovecot.org

[Dovecot] REpeated POP3 mail
by The Doctor 16 Nov '10

16 Nov '10

I know I have brought this up before. Question: How can one debug the pop3 process from repeated e-mail ? -- Member - Liberal International This is doctor(a)nl2k.ab.ca Ici doctor(a)nl2k.ab.ca God, Queen and country! Never Satan President Republic! Beware AntiChrist rising! http://twitter.com/rootnl2k http://www.facebook.com/dyadallee Lest we Forget!!

2 5

[Dovecot] dsync mbox->mdbox: Unexpectedly lost From-line and other issues from a big conversion.
by Axel Thimm 16 Nov '10

16 Nov '10

Hi, I'm trying to convert a 33GB mail store from mbox to compressed mdbox (largest mbox is 2.7GB). The source store is live, e.g. there are mails delivered into it and mails are being read. Actually it is my own mail. :) Although my test runs were very successful I have run into trouble with the first run on the whole store. After fighting a bit around with errors like Error: Trying to open a non-listed mailbox with guid=f59c5b31b8f3df4c70180000e7dd553b I did several reruns in case these errors would be fixed on the next iteration, but my mdbox storage wasn't growing anymore. I decided to restart after removing all old index files from the source store in case I had some corruption somewhere. Since on mbox/maildir index files are completely recreatable this would just slow down thing at the worst. Indeed many of the errors went away, and I managed to convert about 10-20%, but then I hit another array of errors that would persist even after restarting: > $ grep -v Info: dsync2.log.old* > dsync2.log.old1:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/mplayerhq.hu/ffmpeg-devel at 58706201 (cached) > dsync2.log.old1:dsync(user): Error: read(msg input) failed: Invalid argument > dsync2.log.old1:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 (cached) > dsync2.log.old1:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6486645 > dsync2.log.old2:dsync(user): Error: mdbox /home/user/mdbox/mailboxes/lists/mplayerhq.hu/ffmpeg-devel/dbox-Mails: map uid lost for uid 26483 > dsync2.log.old2:dsync(user): Error: msg guid lookup failed: Internal error occurred. Refer to server log for more information. [2010-11-15 02:36:21] > dsync2.log.old2:dsync(user): Warning: mdbox /home/user/mdbox/storage: rebuilding indexes > dsync2.log.old2:dsync(user): Error: Corrupted dbox file /home/user/mdbox/storage/m.1725 (around offset=697710): msg header has bad magic value > dsync2.log.old2:dsync(user): Warning: dbox: Copy of the broken file saved to /home/user/mdbox/storage/m.1725.broken > dsync2.log.old2:dsync(user): Warning: Transaction log file /home/user/mdbox/storage/dovecot.map.index.log was locked for 295 seconds > dsync2.log.old2:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/mplayerhq.hu/ffmpeg-devel at 58706201 (cached) > dsync2.log.old2:dsync(user): Error: read(msg input) failed: Invalid argument > dsync2.log.old2:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 (cached) > dsync2.log.old2:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6486645 > dsync2.log.old2:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 > dsync2.log.old2:dsync(user): Warning: Mailbox changes caused a desync. You may want to run dsync again. > dsync2.log.old3:dsync(user): Error: mdbox /home/user/mdbox/mailboxes/lists/mplayerhq.hu/ffmpeg-devel/dbox-Mails: map uid lost for uid 26484 > dsync2.log.old3:dsync(user): Error: msg guid lookup failed: Internal error occurred. Refer to server log for more information. [2010-11-15 09:49:26] > dsync2.log.old3:dsync(user): Warning: mdbox /home/user/mdbox/storage: rebuilding indexes > dsync2.log.old3:dsync(user): Error: Corrupted dbox file /home/user/mdbox/storage/m.1725 (around offset=758233): msg header has bad magic value > dsync2.log.old3:dsync(user): Error: link(/home/user/mdbox/storage/m.1725, /home/user/mdbox/storage/m.1725.broken) failed: File exists > dsync2.log.old3:dsync(user): Warning: Transaction log file /home/user/mdbox/storage/dovecot.map.index.log was locked for 271 seconds > dsync2.log.old3:dsync(user): Warning: Transaction log file /var/cache/dovecot/indexes/user/lists/lists.fedoraproject.org/.imap/users/dovecot.index.log was locked for 180 seconds > dsync2.log.old3:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/mplayerhq.hu/ffmpeg-devel at 58706201 (cached) > dsync2.log.old3:dsync(user): Error: read(msg input) failed: Invalid argument > dsync2.log.old3:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 (cached) > dsync2.log.old3:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6486645 > dsync2.log.old3:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 > dsync2.log.old3:dsync(user): Warning: Mailbox changes caused a desync. You may want to run dsync again. > dsync2.log.old4:dsync(user): Warning: Transaction log file /var/cache/dovecot/indexes/user/lists/lists.fedoraproject.org/.imap/scm-commits/dovecot.index.log was locked for 244 seconds > dsync2.log.old4:dsync(user): Warning: Transaction log file /var/cache/dovecot/indexes/user/lists/lists.fedoraproject.org/.imap/scm-commits/dovecot.index.log was locked for 186 seconds > dsync2.log.old4:dsync(user): Error: mdbox /home/user/mdbox/mailboxes/lists/mplayerhq.hu/ffmpeg-devel/dbox-Mails: map uid lost for uid 26485 > dsync2.log.old4:dsync(user): Error: msg guid lookup failed: Internal error occurred. Refer to server log for more information. [2010-11-15 12:40:24] > dsync2.log.old4:dsync(user): Warning: mdbox /home/user/mdbox/storage: rebuilding indexes > dsync2.log.old4:dsync(user): Error: Corrupted dbox file /home/user/mdbox/storage/m.1725 (around offset=1324287): msg header has bad magic value > dsync2.log.old4:dsync(user): Error: link(/home/user/mdbox/storage/m.1725, /home/user/mdbox/storage/m.1725.broken) failed: File exists > dsync2.log.old4:dsync(user): Warning: Transaction log file /home/user/mdbox/storage/dovecot.map.index.log was locked for 394 seconds > dsync2.log.old4:dsync(user): Warning: Transaction log file /var/cache/dovecot/indexes/user/lists/lists.fedoraproject.org/.imap/scm-commits/dovecot.index.log was locked for 249 seconds > dsync2.log.old4:dsync(user): Warning: Transaction log file /var/cache/dovecot/indexes/user/lists/lists.fedoraproject.org/.imap/scm-commits/dovecot.index.log was locked for 283 seconds > dsync2.log.old4:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/mplayerhq.hu/ffmpeg-devel at 58706201 (cached) > dsync2.log.old4:dsync(user): Error: read(msg input) failed: Invalid argument > dsync2.log.old4:dsync(user): Error: Next message unexpectedly lost from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 (cached) > dsync2.log.old4:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6486645 > dsync2.log.old4:dsync(user): Error: Unexpectedly lost From-line from mbox file /home/user/mail/lists/gnupg.org/gnupg-users at 6507197 > dsync2.log.old4:dsync(user): Warning: Mailbox changes caused a desync. You may want to run dsync again. The quoted broken mdbox file and its successor differ quite late in the file itself: > $ ls -ltrA mdbox/storage/| tail > -rw------- 1 user user 2096503 2010-11-15 01:53 m.1719 > -rw------- 1 user user 2095681 2010-11-15 01:53 m.1720 > -rw------- 1 user user 2095121 2010-11-15 01:53 m.1721 > -rw------- 1 user user 2091386 2010-11-15 01:53 m.1722 > -rw------- 1 user user 2095194 2010-11-15 01:53 m.1723 > -rw------- 1 user user 2095317 2010-11-15 01:54 m.1724 > -rw------- 1 user user 697754 2010-11-15 01:54 m.1725.broken > -rw------- 1 user user 28818176 2010-11-15 13:04 dovecot.map.index > -rw------- 1 user user 14884 2010-11-15 14:49 dovecot.map.index.log > -rw------- 1 user user 1661548 2010-11-15 14:49 m.1725 > $ cmp mdbox/storage/m.1725.broken mdbox/storage/m.1725 > mdbox/storage/m.1725.broken mdbox/storage/m.1725 differ: byte 697638, line 7642 I'm running the process as the user himself with > $ dsync -vD backup mdbox:~/mdbox while my config looks like $ doveconf -n # 2.0.7: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32.21-168.fc12.x86_64 x86_64 Fedora release 12 (Constantine) mail_location = mbox:~/mail:INBOX=/var/mail/% u:INDEX=/var/cache/dovecot/indexes/%u mail_plugins = " zlib" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl namespace { inbox = yes location = prefix = separator = / } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = bz2 } service imap { executable = imap postlogin } service postlogin { executable = script-login rawlog } ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } protocol lda { info_log_path = /var/log/dovecot-deliver.log log_path = /var/log/dovecot-deliver-errors.log mail_plugins = sieve } protocol imap { mail_plugins = " zlib imap_zlib" } I think I probably have some corrupt mbox files that give dovecot some headache. I'd like to spot them and correct them, so I tried let's convert to maildir first, which I can debug more easily. Unfortunatly the conversion speed to maildir is about 25 times slower than converting to mdbox (wow!), and it would take almost 5 days to convert to maildir. This is no sane timeframe for debugging, so I'd rather try different tactics with mdbox. One thing I'm going to try is to do an offline conversion of a snapshot of the store. Anyway I wanted to document the issues I saw in case it is something that is a bug in dovecot. -- http://thimm.gr/ - http://ATrpms.net/

2 4

[Dovecot] Email backend monitor script for Director
by Aliet Santiesteban Sifontes 16 Nov '10

16 Nov '10

Hi people, I know I saw this at some point in the list but can't find it, I need a script wich monitor the health of the email backend and if a node fails remove it from the director server, once is up again add it, I plan tu run the script at the load balancer, if you have some let me know.. thank's in advance

1 1

[Dovecot] POP users complaining about multiple copies of mail
by H. Wade Minter 16 Nov '10

16 Nov '10

Hello, We just switched from UW to Dovecot for our mail server. The transition went smoothly, except we have some users complaining about their email clients re-downloading mail or getting multiple copies of emails. When we go into their accounts, we only see one copy of the email, so I'm assuming there's something odd in the communication between their client and us that's causing the mail to get downloaded repeatedly. The clients that we've heard about are Outlook and one called "Goldmine". Has anyone run across this before? Scanning the archives didn't show me anything. This is Dovecot 1.2.12-1ubuntu8 on Ubuntu 10.10. Here's my config: # 1.2.12: /etc/dovecot/dovecot.conf # OS: Linux 2.6.35.4-rscloud x86_64 Ubuntu 10.10 log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s managesieve ssl_ca_file: /etc/ssl/certs/ca.pem ssl_cert_file: /etc/ssl/certs/wildcard.skiltech.com.pem ssl_key_file: /etc/ssl/private/wildcard.skiltech.com.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login mail_privileged_group: mail mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): imap_client_workarounds(managesieve): pop3_enable_last(default): no pop3_enable_last(imap): no pop3_enable_last(pop3): yes pop3_enable_last(managesieve): no pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_client_workarounds(managesieve): lda: postmaster_address: postmaster mail_plugins: sieve quota_full_tempfail: yes deliver_log_format: msgid=%m: %$ rejection_reason: Your message to <%t> was automatically rejected:%n%r auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/dovecot-auth mode: 432 user: postfix group: postfix plugin: sieve: ~/.dovecot.sieve sieve_dir: ~/sieve

2 5

Re: [Dovecot] proctitle woes in 2.0.7 vs. 2.0.6
by Patrick Fay 16 Nov '10

16 Nov '10

> Message: 7 > Date: Mon, 15 Nov 2010 16:45:34 +0000 > From: Timo Sirainen <tss(a)iki.fi> > Subject: Re: [Dovecot] proctitle woes with 2.0.7 vs. 2.0.6 > To: Clemens Schrimpe <csch(a)kiez.net> > Cc: Dovecot Mailing List <dovecot(a)dovecot.org> > Message-ID: <1289839534.1764.138.camel(a)kurkku.sapo.corppt.com> > Content-Type: text/plain; charset="UTF-8" > > On Sat, 2010-11-13 at 02:25 +0100, Clemens Schrimpe wrote: >>>> It seems, that while anvil has gained some verbosity, >>> >>> Yes. >>> >>>> auth and imap-login (and maybe pop-login) seem to have lost their >>>> "voice". >>>> >>>> Is there any intention behind this or did I ran into a "bug"? > > The bug was in a bit unexpected place. Fixed: > http://hg.dovecot.org/dovecot-2.0/rev/656da7e0d6b9 > Hi, Is there any chance this bug is related to MacOS proc titles being mis-identified (e.g. containing garbage) in the 1.2.x tree? Patrick

2 1

[Dovecot] Trying to building a customized auth plugin
by Antonio Perez-Aranda 16 Nov '10

16 Nov '10

I'm on dovecot 2.0.7 and I trying to building a customized auth plugin. I take passdb-passwd-file.c and userdb-passwd-file.c to try to build with a simple gcc comand out of dovecot environment as in example: export DOVECOT=/path/to/untar/dovecot-2.0.7 gcc -fPIC -shared -g -Wall -I$DOVECOT \ -I$DOVECOT/src/lib \ -I$DOVECOT/src/lib-auth \ -I$DOVECOT/src/lib-sql \ -I$DOVECOT/src/lib-settings \ -I$DOVECOT/src/lib-ntlm \ -I$DOVECOT/src/lib-master \ -I$DOVECOT/src/auth \ passdb-passwd-file.c -o passdb-passwd-file.o With this, I get errors relate with uoff_t Is possible to build this plugin by this way? Another way is patching against one plugin on dovecot RPM, but I prefer this on a separated RPM.

3 12

[Dovecot] Single-instance storage is bad for you!
by Daniel L. Miller 16 Nov '10

16 Nov '10

This single-instance storage is going to encourage bad habits! I just found myself preparing an e-mail that needs to be sent to multiple recipients - including several within my own organization - yet it's not necessary for everyone (in-house) to get the large attachment. As I was getting ready to split the message into two parts for distribution I remembered - I can just send it to EVERYBODY in-house and it only occupies a single instance - which it would anyway just being in my "sent" folder! Blasting away... You're quite the enabler, Timo! Now any mail admin who tries to lecture users on only sending attachments to those in-house personnel who really need it, because otherwise it's wasteful of precious server storage space, is going to hear the response, "Dude - don't stress out so much - just setup Dovecot with SIS!". Thanks again! -- Daniel

7 14

[Dovecot] service_count=0 for imap and pop3
by Mark Moseley 16 Nov '10

16 Nov '10

This is copy-pasted from Timo's comment in the "Todays Performance Data for 2.0.x" thread, but I didn't want to hijack that thread. In it, Timo says: <snip> service imap { service_count = 0 } (i.e. reuse imap processes) reduced the system CPU usage to almost nothing. But if you use different UIDs for different users you can't do that. </snip> Is that safe to do in imap and/or pop3? Or at least no more insecure than using service_count=0 for imap-login and pop3-login? Since I saw that comment I've been googling around and grepping the wiki but I can only find instances of using it with "service imap-login", not "service imap" or "service pop3". The mention in Timo's comment that it reduces the CPU usage by a lot got my attention, esp for pop3. We're already using service_count=0 for pop3-login and imap-login.

3 9

Re: [Dovecot] Local node indexes in a cluster backend with GFS2
by Aliet Santiesteban Sifontes 16 Nov '10

16 Nov '10

Should I set mmap_disable = yes when storing indexes in a GFS2 shared filesystem?? 2010/11/15 Aliet Santiesteban Sifontes <alietsantiesteban(a)gmail.com> > Ok, I will create a LUN also as a shared clustered storage for indexes, any > consideration to have into account when the indexes are shared by many > nodes... > thank you all... > > 2010/11/15 Timo Sirainen <tss(a)iki.fi> > > On 15.11.2010, at 6.44, Aliet Santiesteban Sifontes wrote: >> >> > mail_location = >> > sdbox:/var/vmail/%d/%3n/%n/sdbox:INDEX=/var/indexes/%d/%3n/%n >> > >> > /var/vmail is shared clustered filesystem with GFS2 shared by node1 and >> > node2 >> > >> > /var/indexes is a local filesystem at the node, so each node has his own >> > /var/indexes stuff on ext3 and raid1 for improving performance, I mean >> each >> > node a different /var/indexes of its own. >> >> This is a bad idea. With dbox the message flags are only stored in index >> files, so if you lose indexes you lose all message flags. Users won't be >> happy. >> > >

2 2

[Dovecot] (no subject)
by Radio Tron 15 Nov '10

15 Nov '10

http://aigipe.it/here.php

2 1