August 2010 - dovecot

[Dovecot] Question about mbox_snarf and dovecot2.0
by Jonathan Siegle 18 Nov '10

18 Nov '10

Hello, I'm running revision 10622:de9d6dae7fe5 on AIX 5.3 with some local mods for our inbox hash function. I'm having a problem doing a "select inbox" when I use the mbox_snarf plugin. When I run truss on the process I see the following stat calls: 614528: kread(9, " 1 s e l e c t i n b".., 4096) = 15 614528: statx("/gpfs/inbox/14/tstem38", 0x0FFFFFFFFFFFF2F8, 176, 0) = 0 614528: statx("/gpfs/users/t/s/tstem38/IMAP//gpfs/inbox/14/.imap/tstem38", 0x0FFFFFFFFFFFED98, 176, 0) Err#2 ENOENT 614528: statx("/gpfs/users/t/s/tstem38/IMAP//gpfs/inbox/14/tstem38", 0x0FFFFFFFFFFFEDD8, 176, 0) Err#2 ENOENT 614528: statx("/gpfs/users/t/s/tstem38/IMAP", 0x0FFFFFFFFFFFEC88, 176, 0) = 0 614528: statx("/gpfs/users/t/s/tstem38/IMAP//gpfs/inbox/14/tstem38", 0x0FFFFFFFFFFFEDD8, 176, 0) Err#2 ENOENT 614528: statx("/gpfs/users/t/s/tstem38/IMAP", 0x0FFFFFFFFFFFEC88, 176, 0) = 0 and the client gets: 1 NO [CANNOT] Invalid mailbox name For pathing I have: plugin.conf: mbox_snarf = /gpfs/inbox/%Ju/%u and mail.conf:mail_location = mbox:%h:INBOX=/var/spool/mail/%Ju/%u I verified that not using the mbox_snarf works just fine. I have no namespaces enabled. Thanks, Jonathan

2 9

[Dovecot] create Sub-Subfolders
by Richard Gliebe 05 Nov '10

05 Nov '10

Hi all, this is my first post to this list, and hope someone can help me ;-) I'm running dovecot.x86_64 1.0.7-7.el5 on a CentOS release 5.5 (Final) box. when a user tries to create a sub-subfolder in Archiv, he gets the message: "Mailbox doesn't allow inferior mailboxes" Mailclient is MS Outlook 2003 SP3 is there a spezial configuration in /etc/dovecot.conf needed? MailLocation is by default in /etc/dovecot: # See doc/wiki/Variables.txt for full list. Some examples: # # mail_location = maildir:~/Maildir # mail_location = mbox:~/mail:INBOX=/var/mail/%u # mail_location = mbox:/var/mail/%d/%1n/%n:INDEX=/var/indexes/%d/%1n/%n # # <doc/wiki/MailLocation.txt> # #mail_location = Users have there IMAP folders in "~/mail" # pwd /home/bema/mail # ls -l total 552 -rw------- 1 bema bema 0 Aug 9 16:21 Archiv -rw------- 1 bema bema 0 Aug 7 12:04 Aufgaben -rw------- 1 bema bema 0 Aug 4 16:05 Drafts -rw------- 1 bema bema 0 Aug 7 12:03 Gel&APY-schte Objekte -rw------- 1 bema bema 553349 Aug 7 12:05 Gesendete Objekte -rw------- 1 bema bema 0 Aug 7 11:50 Junk-E-Mail -rw------- 1 bema bema 0 Aug 7 12:04 Postausgang -rw------- 1 bema bema 887 Aug 7 12:05 Sent -rw------- 1 bema bema 0 Aug 4 16:05 Trash many thanks Richard

5 7

[Dovecot] Auto delete older messages in SPAM folder
by Thiago Henrique 25 Oct '10

25 Oct '10

Hello, I use Cyrus in my mailservers and I intend to migrate it to Dovecot. Cyrus has a setting that purge the emails from a mailbox under a pre-defined criteria. For example: EVENTS { ... # At 12:01 delete messages older than 60 days in SPAM folder deletespam cmd="ipurge -d 60 -f user/*/SPAM" at=1201 } Is there in Dovecot any setting like this ? Thank you. -- []'s Thiago Henrique Network Administration Digirati Networks K8 Networks

2 3

[Dovecot] IMAP connection timeout value?
by Tamas Kadar 01 Oct '10

01 Oct '10

So we're having some ISP issues lately, meaning we have to reboot our cable modem like 2-3 times a day. An unfortunate side-effect I've found is, once the users max out their connection limit (which I've set to 10, and many people do), if we reboot the cable modem, they can't log in cause their connections get 'stuck' until I `doveadm kick` them, or wait for the connections to timeout which takes ??? minutes. Can I adjust that value to something like 30 seconds? Thanks and best regards, KT

5 8

[Dovecot] auth: Fatal: Unknown passdb driver 'mysql'
by Vaidas Jablonskis 29 Sep '10

29 Sep '10

Hi People, I am setting up a new dovecot server for testing first prior production upgrade from 1.2.x to 2.0.x. I used the RPMs from ATrpms: dovecot-2.0.0-1_117.el5.x86_64.rpm The errors I am getting: ----------------------------- Aug 27 18:15:29 testbox dovecot: auth: Debug: Loading modules from directory: /usr/lib64/dovecot/auth Aug 27 18:15:29 testbox dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libauthdb_ldap.so Aug 27 18:15:29 testbox dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_mysql.so Aug 27 18:15:29 testbox dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_pgsql.so Aug 27 18:15:29 testbox dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_sqlite.so Aug 27 18:15:29 testbox dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libmech_gssapi.so Aug 27 18:15:29 testbox dovecot: auth: Fatal: Unknown passdb driver 'mysql' Aug 27 18:15:29 testbox dovecot: master: Error: service(auth): command startup failed, throttling Aug 27 18:15:59 testbox dovecot: imap-login: Error: Timeout waiting for handshake from auth server. my pid=4323, input bytes=0 ------------------------------ # dovecot --build-options Build options: ioloop=epoll notify=inotify ipv6 openssl io_block_size=8192 Mail storages: cydir maildir mbox mdbox raw sdbox shared SQL driver plugins: mysql postgresql sqlite Passdb: checkpassword ldap pam passwd passwd-file shadow sql Userdb: checkpassword ldap(plugin) nss passwd prefetch passwd-file sql # ls -l /usr/lib64/dovecot/*sql* -rwxr-xr-x 1 root root 15704 Aug 20 16:51 /usr/lib64/dovecot/libdriver_mysql.so -rwxr-xr-x 1 root root 21464 Aug 20 16:51 /usr/lib64/dovecot/libdriver_pgsql.so -rwxr-xr-x 1 root root 11848 Aug 20 16:51 /usr/lib64/dovecot/libdriver_sqlite.so It is clearly seen that mysql drivers are installed and being loaded during dovecot startup. ----------------------- /etc/dovecot/conf.d/auth-sql.conf.ext passdb { driver = mysql args = /etc/dovecot/dovecot-sql.conf.ext } userdb { driver = prefetch } userdb { driver = mysql args = /etc/dovecot/dovecot-sql.conf.ext } ------------------------- Any ideas what else could be wrong? Thanks, Vaidas

2 2

[Dovecot] (Single instance) attachment storage
by Timo Sirainen 17 Sep '10

17 Sep '10

Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :) The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome. Reading attachments ------------------- dbox metadata would contain entries like (this is a wrapped single line entry): X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b So the format is: "X" 1*(<offset> <byte count> <link path>) So when reading a dbox message body, it's read as: offset=0: <first 1442 bytes from dbox body> offset=1442: <next 2742784 bytes from external file> offset=2744226: <next 152 bytes from dbox body> offset=2744378: <next 27423 bytes from external file> offset=2744378 27423: <the rest from dbox body> This is all done internally by creating a single istream that lazily opens the external files only when data is actually tried to be read from that part of the message. The link paths don't have to be in any specific format. In future perhaps it can recognize different formats (even http:// urls and such). Saving attachments separately ----------------------------- Message MIME structure is being parsed while message is saved. After each MIME part's headers are parsed, it's determined if this part should be stored into attachment storage. By default it only checks that the MIME part isn't multipart/* (because then its child parts would contain attachments). Plugins can also override this. For example they could try to determine if the commonly used clients/webmail always downloads and shows the MIME part when opening the mail (text/*, inline images, etc). dbox_attachment_min_size specifies the minimum MIME part size that can be saved as an attachment. Anything smaller than that will be stored normally. While reading a potential attachment MIME part body, it's first buffered into memory until the min. size is reached. After that the attachment file is actually created and buffer flushed to it. Each attachment filename contains a global UID part, so that no two (even identical) attachments will ever contain the same filename. But there can be multiple attachment storages in different mount points, and each one could be configured to do deduplication internally. So identical attachments should somehow be stored to same storage. This is done by taking a hash of the body and using a part of it as the path to the file. For example: mail_location = dbox:~/dbox:ATTACHMENTS=/attachments/$/$ Each $ would be expanded to 8 bits of the hash in hex (00..ff). So the full path to an attachment could look like: /attachments/04/f1/5ddf4d05177b3b4c7a7600008c4a11c1 Sysadmin can then create /attachment/00..ff as symlinks to different storages. Hashing problems ---------------- Some problematic design decisions: 1) Hash is taken from hardcoded first n kB vs. first dbox_attachment_min_size bytes? + With first n kB, dbox_attachment_min_size can be changed without causing duplication of attachments, otherwise after the change the same attachment could get a hash to a different storage than before the change. - If n kB is larger than dbox_attachment_min_size, it uses more memory. - If n kB is determined to be too small to get uniform attachment distribution to different storages, it can't be changed without recompiling. 2) Hash is taken from first n bytes vs. everything? + First n bytes are already read to memory anyway and can be hashed efficiently. The attachment file can be created without wasting extra memory or disk I/O. If everything is hashed, the whole attachment has to be first stored to memory or to a temporary file and from there written to final storage. - With first n bytes it's possible for an attacker to generate lots of different large attachments that begin with the same bytes and then overflow a single storage. If everything is hashed with a secure hash function and a system-specific secret random value is added to the hash, this attack isn't possible. I'm thinking that even though taking a hash of everything is the least efficient option, it's the safest option. It's pretty much guaranteed to give a uniform distribution across all storages, even against intentional attacks. Also the worse performance isn't probably that noticeable, especially assuming a system where local disk isn't used for storing mails, and the temporary files would be created there. Single instance storage ----------------------- All of the above assumes that if you want a single instance storage, you'll need to enable it in your storage. Now, what if you can't do that? I've been planning on making all index/dbox code to use an abstracted out simple filesystem API rather than using POSIX directly. This work can be started by making the attachment reading/writing code use the FS API and then create a single instance storage FS plugin. The plugin would work like: open(ha/sh/hash-guid): The destination storage is in ha/sh/ directory, so a new temp file can be created under it. The hash is part of the filename to make unlink() easier to handle. Since the hash is already known at open() time, look up if hashes/<hash> file exists. If it does, open it. write(): Write to the temp file. If hashes/ file is open, do a byte-by-byte comparison of the inputs. If there's a mismatch, close the hashes/ file and mark it as unusable. finish(): a) If hashes/ file is still open and it's at EOF, link() it to our final destination filename and delete the temp file. If link() fails with ENOENT (it was just expunged), goto b. If link() fails with EMLINK (too many links), goto c. b) If hashes/ file didn't exist, link() the temp file to the hash and rename() it to the destination file. c) If the hashed file existed but wasn't the same, or if link() failed with EMLINK, link() our temp file to a second temp file and rename() it over the hashes/ file and goto a. unlink(): If hashes/<hash> has the same inode as our file and the link count is 2, unlink() the hash file. After that unlink() our file. One alternative to avoid using <hash> as part of the filename would be for unlink() to read the file and recalculate its hash, but that would waste disk I/O. Another possibility would to be to not unlink() the hashes/ files immediately, but rather let some nightly cronjob to stat() through all of the files and unlink() the ones that have link count=1. This could be wastefully inefficient though. Yet another possibility would be for the plugin to internally calculate the hash and write it somewhere. If it's at the beginning of the file, it could be read from there with some extra disk I/O. But is it worth it?.. Extra features -------------- The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space: 1) The attachment could be compressed (header contains compressed-flag) 2) If base64 attachment is in a standardized form that can be 100% reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly. It would be nice if it was also possible to compress (and decompress) attachments after they were already stored. This would be possible, but it would require finding all the links to the message and recreating them to point to the new message. (Simply overwriting the file in place would require there are no readers at the same time, and that's not easy to guarantee, except if Dovecot was entirely stopped. I also considered some symlinking schemes but they seemed too complex and they'd also waste inodes and performance.) Code status ----------- Initial version of the attachment reading/writing code is already done and works (lacks some error handling and probably performance optimizations). The SIS plugin code is also started and should be working soon. This code is very isolated and can't cause any destabilization unless it's enabled, so I'm thinking about just adding it to v2.0 as soon as it works, although the config file comments should indicate that it's still considered unstable.

5 31

[Dovecot] Dovecot upgrades break Blackberry instant email (BIS)
by interfaSys sàrl 17 Sep '10

17 Sep '10

Hello, Just like when upgrading to 1.2, upgrading to 2.0.1 confuses RIM's BIS servers and users don't have instant email anymore on their Blackberrys. Under normal conditions, the BIS servers keep one open connection per email account with Dovecot. They're using IMAP IDLE to monitor changes and push them to devices very quickly. After an upgrade, all changes. The BIS servers can't maintain an idling connection anymore, they just poll the Dovecot server when they feel like it (every 2-15 minutes). There is one way to fix this on the devices: The users have to re-create the email account. I don't find this very convenient and was wondering if there was something that could be done on the Dovecot side? Last time it happened, I tried clearing the indexes and resetting file permissions, but nothing helped. Cheers, Olivier

5 14

[Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
by interfaSys sàrl 16 Sep '10

16 Sep '10

Hello, On FreeBSD, Dovecot 2.0.1 compiles fine with gcc45 and the other services have no problem, but even without any optimizations (CFLAGS) at all, pop3-login seg faults when a user tries to connect to Dovecot using POP3. As this is not the only application that has a problem with the latest version of gcc (mod_php), there may be nothing that can be done on the Dovecot side, but I thought I would mention it, in case it was due to some bug in the pop3-login code. Cheers, Olivier

2 12

[Dovecot] dsync mirror duplicates problem
by Waldemar W. 16 Sep '10

16 Sep '10

Im trying to sync 2 mailserver with dsync mirror functionality. Problem occurs when mails arrive in the same folder on both sides then dsync sometimes runs into problems and duplicates that entrys on both side(2mails then with same content). After running again the same happens again(now 4 mails with same content on every side). And then duplicates this mails on both sides on every run :( Anyone with same problem or is something that could cause this? I tried dsync with and without -f option but no change. Im running dovecot 2.0.1 together with dovecot sieve (latest hg version). thanks, Waldemar

2 3

[Dovecot] v2.0.0 released
by Timo Sirainen 15 Sep '10

15 Sep '10

http://dovecot.org/releases/2.0/dovecot-2.0.0.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.0.tar.gz.sig As promised last Friday, here's the v2.0.0 release finally. I'm cautiously optimistic that v2.0.1 won't (have to) be released for a few weeks, since there was quite a lot of testing and fixing going on in the RC stage. Remember to read http://wiki2.dovecot.org/Upgrading/2.0 before upgrading from v1.x. Although there is automatic config conversion, it can't handle everything, so reserve some extra time for upgrading. If you find an upgrade problem that isn't listed on the wiki page, please add it there or email Dovecot mailing list about it. Pigeonhole Sieve and ManageSieve support can now be installed without patching Dovecot. Unfortunately there isn't a Pigeonhole release yet for v2.0, but its Mercurial code tree should be quite stable. See http://wiki2.dovecot.org/Pigeonhole There are no changes since v2.0.rc6. The largest changes since v1.2 are: * Dovecot uses two system users for internal purposes now by default: dovenull and dovecot. You need to create the dovenull user or change default_login_user setting. * Global ACLs are now looked up using namespace prefixes. For example if you previously had INBOX. namespace prefix and a global ACL for "INBOX.Sent", it's now looked up from "INBOX.Sent" file instead of "Sent" as before. * Maildir: File permissions are no longer based on dovecot-shared file, but the mailbox directory. + Redesigned master process. It's now more modular and there is less code running as root. + Configuration supports now per-local/remote ip/network settings. + dsync utility does a two-way mailbox synchronization. + LMTP server and proxying. + Added mdbox (multi-dbox) mail storage backend. + doveadm utility can be used to do all kinds of administration functions. Old dovecotpw and *view utilities now exist in its subcommands. + imap and pop3 processes can now handle multiple connections. + IMAP: COMPRESS=DEFLATE is supported by imap_zlib plugin + director service helps NFS installations to redirect users always to same server to avoid corruption

8 12