- dovecot - dovecot.org

[Dovecot] dsync-local crashes due to assertion failures
by Fabian Groffen 30 Dec '13

30 Dec '13

Hi, I have two servers that share nothing, running dovecot version 2.2.9. The first server hosts the original mail base, which I want to sync using dovecot to the second server. With some trial and error I managed to get synchronisation running using tcp sockets. Now for some accounts I see dsync-local crashing on the server that has the original mail base (which is still the only of the two that receives new mail). The log shows: Dec 8 03:13:14 zeus dovecot: dsync-local(someuser): Panic: file mail-index-tr ansaction-export.c: line 203 (log_append_ext_hdr_update): assertion failed: (u32 .offset + u32.size <= ext_hdr_size) Dec 8 03:13:14 zeus dovecot: dsync-local(someuser): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x70ea0) [0x7f3ee4137ea0] -> /usr/lib64/dovecot/libdovecot.so.0(+0x70efe) [0x7f3ee4137efe] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f3ee40ebd2c] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_transaction_export+0xbed) [0x7f3ee442e89d] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x89104) [0x7f3ee442d104] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit_full+0xb2) [0x7f3ee442d592] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_transaction_commit+0xc) [0x7f3ee442d63c] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0xb6) [0x7f3ee44372b6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x335d6) [0x7f3ee43d75d6] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x328bc) [0x7f3ee43d68bc] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x32d83) [0x7f3ee43d6d83] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_storage_sync_init+0xd9) [0x7f3ee43d7199] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x2e) [0x7f3ee43e7f2e] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x27) [0x7f3ee43e8047] -> dovecot/doveadm-server(dsync_brain_master_send_mailbox+0xcf) [0x41f17f] -> dovecot/doveadm-server(dsync_brain_run+0x2c8) [0x41d738] -> dovecot/doveadm-server() [0x41dd70] -> dovecot/doveadm-server() [0x42eda0] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f3ee41495a6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0xbd) [0x7f3ee414a62d] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f3ee4149008] -> dovecot/doveadm-server() [0x41bb6c] -> dovecot/doveadm-server() [0x40ee3b] -> dovecot/doveadm-server() [0x419581] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f3ee41495a6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0xbd) [0x7f3ee414a62d] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f3ee4149008] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f3ee40f1643] Dec 8 03:13:14 zeus dovecot: dsync-local(someuser): Fatal: master: service(doveadm): child 14116 killed with signal 6 (core dumps disabled) It seems that despite these crashes, some data is being transferred. Newer accounts seem not to have problems, this happens with older accounts only, for as far as I can see. I tried removing the index files, but that didn't seem to help. Any ideas on what I can do here? Both servers are configured to use maildir storage. Thanks, Fabian -- Fabian Groffen Gentoo on a different level

1 1

[Dovecot] pop3 rate limit
by Christian Rößner 30 Dec '13

30 Dec '13

Hi, we have customers with Exchange servers that are polling for new mail every minute with dozens of pop3 accounts. I am looking for a mechanism to rate limit this per user. So what I am looking for is a way to block users from polling, if a user asks for new mail more than every 5 minutes (for example). Is this possible? Can this be achieved within Dovecot or does it need external scripting? (I thought about fail2ban, but also want IPv6 support) Thanks in advance -Christian Rößner -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

2 1

[Dovecot] Sieve's spamtest always returns 0
by martin f krafft 28 Dec '13

28 Dec '13

I am a bit at a loss here with Sieve (pigeonhole) and the spamtest extension. I am using Dovecot 2.1.7 (backported to Debian squeeze), which comes with Pigeonhole 0.3.0. Messages are scanned with SpamAssassin, which adds a header like X-Spam-Status: Yes, score=84.6 required=5.0 tests=… and so I configured spamtest in conf.d/90-plugin.conf like so: sieve_spamtest_status_type = score sieve_spamtest_status_header = X-Spam-Status: [^,]*, score=(-?[[:digit:]]+\.[[:digit:]]).* sieve_spamtest_max_header = X-Spam-Status: [^,]*, score=[^[:space:]]+ required=(-?[[:digit:]]+\.[[:digit:]]).* I tested those regular expressions with sed -r, e.g. % sed -rne "s@^X-Spam-Status: [^,]*, score=-?[[:digit:]]+\.[[:digit:]] required=(-?[[:digit:]]+\.[[:digit:]]).*@\1@p" mailfile 5.0 and they work. Unfortunately, in sieve scripts, the spamtest value is always 0, which is indicative of the spamtest "not having run", which in this case I assume means that the regular expression didn't match. Am I right in assuming that the matching happens at the time of evaluation, and so adding the headers using vnd.dovecot.filter just before works? Or does the spamtest matching happen before the sieve scripts are executed? Can you see any other reason why spamtest always yields a value of 0? Thanks, -- martin | http://madduck.net/ | http://two.sentenc.es/ "those who are faithful know only the trivial side of love: it is the faithless who know love's tragedies." -- oscar wilde spamtraps: madduck.bogus(a)madduck.net

3 4

[Dovecot] Dovecot 'Submission' service?
by Charles Marcus 27 Dec '13

27 Dec '13

I do recall this being discussed, but I can't remember if it is actually in a working state now or not. According to the wiki, it is a work in progress: http://wiki2.dovecot.org/Roadmap?highlight=%28submission%29 So - is it in a functional state yet? Thanks, -- Best regards, */Charless/***

2 2

[Dovecot] Using DSync to convert from maildir to m/sdbox during migration of mail to new server
by Charles Marcus 27 Dec '13

27 Dec '13

Ok, now to work out the precise method (config for the target server and dsync command) to use to perform the migration from the old server to the new one, converting to dbox (s or m, depending on how the other thread discussing this turns out). I'll update the old server to 2.2.10 tomorrow morning so they're both at the same version. So, assuming both servers are running 2.2.10, with OldServer: mail_location = maildir:~/:INDEX=MEMORY and NewServer: mail_location = sdbox:~/sdbox or mail_location = mdbox:~/mdbox After reading and re-reading: http://wiki2.dovecot.org/Tools/Dsync I'm not so sure... First problem - apparently it can only be used one user at a time? And - do you have to actually know the users password too? If so, then this probably isn't going to be easier, and maybe even harder than just rsyncing everything, then slowly converting each user one at a time. -- Best regards, */Charles/*

1 0

[Dovecot] Migrating from Maildir to dbox+SIS... sdbox or mdbox?
by Charles Marcus 27 Dec '13

27 Dec '13

Ok... Now that I've got the niggling issues resolved, on to bigger and better things. I want to convert everyone from Maildir to dbox and enable SIS (our users use a lot of large attachments), and after some reading/googling, I think it may be better to use dsync instead of rsync for the actual migration after all since it will do this at the same time as the syncing. If I'm not mistaken, once I've figured out the correct syntax for the command, I should be able to do the conversion and migration at the same time. So, that is what I want to do, but I still have to choose between sdbox and mdbox. I tend to feel a little safer with sdbox, just because it seems more stable - and I like the idea that if a file gets corrupted somehow, then it is only that one mail that may potentially be lost, but if mdbox indexes get corrupted, you are guaranteed to lose messages (maybe a lot)... But I also really like the idea of an order of magnitude fewer actual files (would make rsync based backups much faster, etc). Timo - the last comments I was able to find from you were back in 2011... would you mind commenting on the current stated of dbox - especially mdbox, as far as reliability goes? The 'scary warnings' about how important the index files are ... well ... scares me... ;) I know there were definitely some issues with mdbox a few years ago - I found the old thread from 2011, but nothing substantial since, so... So... are there any people here, especially larger installs, using mdbox? sdbox? I'd especially be interested in hearing from anyone who has used both, and picked one over the other, and their reasons for doing so. Thanks, -- Best regards, */Charles/*

1 0

[Dovecot] Enabling SIS (single instance storage)
by Charles Marcus 27 Dec '13

27 Dec '13

Ok, didn't find much on this... The bottom of /etc/dovecot/conf.d/10-mail.conf has the following settings: #mail_attachment_dir = #mail_attachment_min_size = 128k #mail_attachment_fs = sis posix #mail_attachment_hash = %{sha1} all of which are apparently the defaults according to doveconf -d output - so no need to set them explicitly, unless I want to change them, right? So, enable the first one - but - the comments don't say anything about supporting variables. I'm assuming you can, otherwise all attachments for all domains would be in a single directory, which some admins might not like (I wouldn't)... so hopefully I can just set it to this: mail_attachment_dir = /var/vmail/%d/.attachments So how does this work during conversion? Users still on maildir simply don't engage it? But as users are converted to dbox, all attachments are de-duped, and then new mails coming in with attachments (for converted users) are de-duped at delivery? The wiki could do with a bit more on enabling and implementing it, especially what to expect when enabling it for the first time on a production system and what to expect along the way when converting existing users... -- Best regards, */Charles/*

1 0

[Dovecot] mail_location, user 'home', questions
by Charles Marcus 27 Dec '13

27 Dec '13

Hi all, Ok, another question... I always like changes to the default settings whenever possible (KISS principle)... So, according to the dovecot wiki, the default location for a maildir setup is: mail_location = maildir:~/Maildir However, when I first tried to access mail on the new box, something wasn't right, and I ended up having to change this to: mail_location = maildir:~ to get mail to display correctly. The filesystem for a users maildir currently has everything (cur/new/tmp, and the individual folders) inside that users directory, ie: /var/vmail/example.com/user/ Do I need to change this so that everything is under a 'Maildir' subdirectroy inside the users directory, like: /var/vmail/example.com/user/Maildir/ ? Also, I'm aware that it is important that even virtual users have valid 'homes', but I'm confused as to where these are (or are supposed to be) by default. My sql query currently contains: user_query = \ SELECT CONCAT('/var/vmail/', maildir) AS home \ FROM mailbox WHERE username = '%u' iterate_query = SELECT username AS user FROM mailbox So... what exactly is this saying? That the users 'home' is '/var/vmail'? Or does it automatically append %d/%u? I'm specifically wondering if the intention for the defaults is for the users home to be: /var/vmail/example.com/user and the users maildirs to be in: /var/vmail/example.com/user/Maildir Would appreciate some clarification... It wouldn't be hard to mv all of the users maildirs to a Maildir subfolder after the sync, but I want to know for sure that that is what I should/need to do... Thanks, -- Best regards, */Charles/*

1 1

[Dovecot] dovecot-postfix stack imap_client_workarounds
by thufir 27 Dec '13

27 Dec '13

To use dovecot-postfix stack with thunderbird, do I put the configuration into /usr/share/dovecot/protocols.d/impad.protocol? That would seem to be how the stack is configured. "Thunderbird To use with Thunderbird, edit the file /etc/dovecot/dovecot.conf: protocol imap { ... login_greeting_capability = yes imap_client_workarounds = tb-extra-mailbox-sep }" https://help.ubuntu.com/community/Dovecot thufir@dur:~$ thufir@dur:~$ cat /usr/share/dovecot/protocols.d/imapd.protocol protocols = $protocols imap thufir@dur:~$ thufir@dur:~$ dovecot -n # 2.0.19: /etc/dovecot/dovecot.conf # OS: Linux 3.5.0-44-generic i686 Ubuntu 12.04.3 LTS mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_cipher_list = ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 10 } protocol pop3 { mail_max_userip_connections = 10 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { deliver_log_format = msgid=%m: %$ mail_plugins = sieve postmaster_address = postmaster quota_full_tempfail = yes rejection_reason = Your message to <%t> was automatically rejected:%n%r } thufir@dur:~$ thufir@dur:~$ thanks, Thufir

2 1

[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
by Charles Marcus 27 Dec '13

27 Dec '13

Ok, hopefully there is a solution to this. I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so. Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync. Here is the series of commands I'm running: 1. stop postfix, stop dovecot on new server 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/ 3. chown vmail:vmail /var/vmail 4. start dovecot, start postfix ls -al /var/vmail/example.com/user/cur shows all of the messages that should be there, and all perms are correct. Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't. I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps. The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem. The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders. This will be a major support problem. Anyone have any ideas? -- Best regards, */Charles/*

5 16