- dovecot - dovecot.org

[Dovecot] Doubts about dsync, mdbox, SIS
by Jan-Frode Myklebust 03 Feb '12

03 Feb '12

I've been running continous dsync backups of our Maildirs for a few weeks now, with the destination dsync server using mdbox and SIS. The idea was that the destination server would act as a warm copy of all our active users data. The active servers are using Maildir, and has: $ df -h /usr/local/atmail/users/ Filesystem Size Used Avail Use% Mounted on /dev/atmailusers 14T 12T 2.2T 85% /usr/local/atmail/users $ df -hi /usr/local/atmail/users/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/atmailusers 145M 113M 33M 78% /usr/local/atmail/users very little of this is compressed (zlib plugin enabled during christmas). I'm surprised that the destination server is so large, was expecting zlib and mdbox and SIS would compress it down to much less than what we're seeing (12TB -> 5TB): $ df -h /srv/mailbackup Filesystem Size Used Avail Use% Mounted on /dev/mapper/mailbackupvg-mailbackuplv 5.7T 4.8T 882G 85% /srv/mailbackup Lots and lots of the attachement storage is duplicated into identical files, instead of hard linked. When running "doveadm purge -u $user", we're seeing lots of Error: unlink(/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab) failed: No such file or directory "/srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-057274283bb51f4f917e0000bf34f6ab" is missing, but there are 205 other copies of this file named /srv/mailbackup/attachments/c3/17/c317b32b97688c16859956f11b803e3bba434349-* with identical sha1sum. Also we see corrupted indexes during the purge. This makes me quite uncertain if dsync is a workable backup solution.. or if we can trust mdboxes. Also on the source side, during dsync, we see too many problems. Some samples: Error: Mailboxes don't have unique GUIDs: 08b46439069d3d4db0490000e671bf84 is shared by INBOX and INBOX Error: command BOX-LIST failed Error: Worker server's mailbox iteration failed Error: read() from worker server failed: EOF Error: Failed to sync mailbox INBOX.ferie 2006.: Invalid mailbox name Error: read() from proxy client failed: EOF Error: Unexpected finish reply: 1 596fec275888dbd89f6d1f5356c22db6 3720 0 \dsync-expunged 0 Error: Unexpected reply from server: 1 12200572a70726fca946da6f9378dc03 3721 0 \dsync-expunged 0 Error: Failed to sync mailbox INBOX.INBOX.Gerda: Mailbox doesn't exist: INBOX/Gerda Error: command BOX-LIST failed Error: read() failed: Broken pipe Panic: file dsync-worker-local.c: line 1678 (local_worker_save_msg_continue): assertion failed: (ret == -1) Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x367703c680] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x35) [0x367703c765] -> /usr/lib64/dovecot/libdovecot.so.0 [0x367703bb93] -> /usr/bin/dsync [0x40f48d] -> /usr/bin/dsync [0x40f589] -> /usr/bin/dsync(dsync_worker_msg_save+0x8e) [0x40eb3e] -> /usr/bin/dsync [0x40d71a] -> /usr/bin/dsync [0x40cdbf] -> /usr/bin/dsync [0x40d105] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x48) [0x3677047278] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0xd5) [0x36770485c5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x2d) [0x367704720d] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x3677035a83] -> /usr/bin/dsync(main+0x71e) [0x406c4e] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x3e3941d994] -> /usr/bin/dsync [0x406369] Do you have any idea for what our problems might be? Should we: avoid SIS ? avoid doing Maildir on one side and mdbox on the other? try other dovecot version for dsync? anything else? -jf ------------- destination server, running dovecot v2.0.14 -------- mail_attachment_dir = /srv/mailbackup/attachments mail_location = mdbox:~/mdbox mail_plugins = zlib mdbox_rotate_size = 5 M namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { driver = static } plugin { zlib_save = gz zlib_save_level = 9 } protocols = service auth-worker { user = $default_internal_user } service auth { unix_listener auth-userdb { mode = 0600 user = mailbackup } } ssl = no userdb { args = home=/srv/mailbackup/%256Hu/%d/%n driver = static } -------------/destination server -------- -jf

3 7

[Dovecot] Feature request - search and tag protocol - notmuch-alike approach
by Kirill A. Shutemov 03 Feb '12

03 Feb '12

Hi all, notmuch[1] becomes more and more popular. At least in geek community. But notmuch is local-only (except ssh)... It looks like it's hard (if even possible) to use IMAP as transport for notmuch-alike work flow. I think it would be great to create a new protocol based global search and message tagging. All core notmuch operations (search, show, count, tag) can be mapped to the new protocol (more or less) directly. Search syntax can be reused as is, I think. The tricky part is initial tagging. It requires some kind of transaction to be able to tag new messages without racing with upcoming messages. Request (command) syntax can be similar to unix command. This way we can reuse notmuch syntax even deeper. Like: search --limit 10 --sort=oldest-first tag:to-me tag:unread Replies are often structured. I think it's reasonable to use JSON for all replies. Any ideas? I've started to write a proof-of-concept, but it moves too slow, since my knowledge of dovecot internals is poor and it's hard to find time for the project. I hope somebody else has similar idea in mind and just wait opportunity to discuss the idea and start to write code. ;) [1] http://notmuchmail.org/ -- Kirill A. Shutemov

1 0

[Dovecot] converting from mbox to maildir ?
by Frank Bonnet 02 Feb '12

02 Feb '12

Hello We are planning to convert our mailhub ( freebsd 7.4 ) from mbox format to maildir format. I've read the documentation and performed some tests on another machine it is a bit long ... I would like some feedback from guys who did this operation and need some advice on what to convert first ? - first convert INBOX then convert IMAP folders ? - first convert IMAP folders then convert INBOX ? the machine use real users thru openldap ( pam_ldap + nss_ldap ) another problem is disk space. The users's email data takes about 2 Terabytes of data and I cannot duplicate as I only have 3 Tb on the raid array of the server. My idea is to use one of our NFS netapp filer during the convertion to throw the result of the convertion on an NFS mounted directory. Anyone did this before ? If yes I would be greatly interrested by their experience Thank you

2 6

[Dovecot] locking problem
by myron 02 Feb '12

02 Feb '12

I have one user who can't read his e-mail. The /var/log/mail.log file has this statement. dovecot: IMAP(<user>): Timeout while waiting for lock for transaction log file /users/<user>/mail/.imap/INBOX/dovecot.index.log I tried killing the imap daemons and removing the lock file, but it keeps coming back. Any idea what else to do? This is the dovecot file: dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-28-server x86_64 Ubuntu 10.04.3 LTS log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_cert_file: /etc/ssl/certs/ssl-cert-snakeoil.pem ssl_key_file: /etc/ssl/private/ssl-cert-snakeoil.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_privileged_group: mail mail_location: mbox:~/mail:INBOX=/var/mail/%u mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth-client mode: 432 user: postfix group: postfix --myron ================================= Myron Kowalski MoCoSIN Network/Systems Administrator Moravian College myron(a)moravian.edu (610) 861-1441

2 4

[Dovecot] Renaming user account / mailbox
by Nikolaos Milas 02 Feb '12

02 Feb '12

Hello, I am running dovecot-2.0.13-1_128.el5 x86_64 RPM on CentOS 5.7. All accounts are virtual, hosted on LDAP Server. We are using Maildir mailboxes. The question: What is the process to rename an existing account/mailbox? I would like to rename userx with email: userx(a)example.com to ux(a)example.com with a mailbox of ux (currently: userx) Of course the idea is that new mail will continue to be delivered to the same mailbox, although it has been renamed. How can I achieve it? Would it be enough (after changing the associated data in the associated LDAP entry) to simply rename the virtual user directory name, e.g. from /home/vmail/userx to /home/vmail/ux ? Thanks in advance, Nick

2 2

[Dovecot] Site-wide configurable Sieve
by Gedalya 01 Feb '12

01 Feb '12

Hi, I need some sieve actions to be specified per-user, but exposing actual sieve scripts to users is of the question. For example, I need to be able to either put spam messages into a spam folder, or discard them, based on user preference. Presumably this would be done using Sieve. As I understand, it's not possible to query a database from within a sieve script. If I can't use a single global script, I can imagine automatically generating per-user scripts when settings are saved in a GUI (e.g. webmail) and store them on the dovecot machine. This would work but I would love to hear if someone has a smarter, more robust idea. Thanks, Gedalya

1 0

[Dovecot] question of different type of Mailbox
by ml 01 Feb '12

01 Feb '12

hi folks hi timo hi master of "Fu" I just migrate my emails from one type of Maildir to Mailbox I did as I was having problems reading speed with my webmail. I did it in order to optimize when do you my current config work for me sincerely -- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC2626742 gpg --keyserver pgp.mit.edu --recv-key C2626742 http://urlshort.eu fakessh @ http://gplus.to/sshfake http://gplus.to/sshswilting http://gplus.to/john.swilting https://lists.fakessh.eu/mailman/ This list is moderated by me, but all applications will be accepted provided they receive a note of presentation

2 1

[Dovecot] LMTP : Can't handle mixed proxy/non-proxy destinations
by Alexis Lelion 01 Feb '12

01 Feb '12

Hello, In my current setup, I uses two mailservers to handle the users connections, and my emails are stored on a distant server using NFS (maildir architecture) Dovecot is both my IMAP server and the delivery agent (LMTP via postfix) To avoid indexing issues related to NFS, proxying is enabled both on IMAP and LMTP. But when a mail is sent to users that are shared between the servers, I got the subject mentionned error in the logs : Jan 25 09:05:12 mail01 postfix/lmtp[23934]: A92709300DB: to=< user_on_mail02(a)domain.com>, relay=mail01.domain.com[private/dovecot-lmtp], delay=0.07, delays=0.01/0/0/0.06, dsn=4.3.0, status=deferred (host mail01.domain.com[private/dovecot-lmtp] said: 451 4.3.0 < user_on_mail02(a)domain.com> Can't handle mixed proxy/non-proxy destinations (in reply to RCPT TO command)) >From what I saw, the mail is then put in the queue, and wait until the next time Postifx will browse the queue. The mail will then be correctly delivered on "mail02". However, the "queue_run_delay" postfix parameter is set to 900, which means that the mail will be delivered with a lag of 15 minutes. I was wondering if there was another way of handling this, for example by triggering an immediate queue lookup from postfix or forwarding a copy of the mail to the other server. Note that the postfix "queue_run_delay" was increased to 15min on purpose, so I cannot change that. I'm using dovecot 2.0.15 on Debian Squeeze, kernel 2.6.32-5-amd64. Thanks, Alexis

2 3

[Dovecot] Issues w/ Apple devices after migration to dovecot
by Armistead, Kurt 01 Feb '12

01 Feb '12

Hi guys. Recently we've moved our old (and I mean _really_ old) mailserver to a new hardware. Since I had quite some time on my hands, I decided to switch from courier to dovecot. Transition went pretty smoothly and the users experienced just a few hours' blackout. Only trouble now is that, after re-configuring some IOS handsets and tablets, not all messages are displayed correctly. For example, I had a long existing mailbox (roughly 1600 emails in INBOX and about a dozen other folders) that showed up perfectly on my iphone. Now it's been synced to the new dovecot server via imap and a whole month of messages seems to be missing. I checked on the server, via webmail and with outlook and everything is where it should be. The problem shows up on Apple devices (same thing happened on a couple of ipads). I haven't tried rebuilding the index or wiping out a phone to see if that resolves the problem, but before I do that I'd like to know if anyone experienced the same issue and how they fixed it Thanks! Kurt

2 3

[Dovecot] Uncompressing dboxes
by Lauri Alanko 01 Feb '12

01 Feb '12

Hello. I have an mdbox directory that has been created with the zlib plugin active, using bzip2 compression. I would now like to convert these into uncompressed mdbox format. How do I do this? I have tried unsetting all zlib-related configuration options, but dsync still produces compressed mdbox at the other end, too. I guess I could convert to mbox, and then back to mdbox, but for certain reasons I'm not very enthusiastic about doing anything mbox-related with dovecot any more. (Incidentally, it's a bit funny that when the zlib plugin is enabled, dsync automatically creates compressed dboxes, and even compressed maildirs (which is pretty useless), but not compressed mboxes, although that's where compression would be most beneficial.) Thanks, Lauri

2 1