- dovecot - dovecot.org

[Dovecot] (Single instance) attachment storage
by Timo Sirainen 17 Sep '10

17 Sep '10

Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :) The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome. Reading attachments ------------------- dbox metadata would contain entries like (this is a wrapped single line entry): X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b So the format is: "X" 1*(<offset> <byte count> <link path>) So when reading a dbox message body, it's read as: offset=0: <first 1442 bytes from dbox body> offset=1442: <next 2742784 bytes from external file> offset=2744226: <next 152 bytes from dbox body> offset=2744378: <next 27423 bytes from external file> offset=2744378 27423: <the rest from dbox body> This is all done internally by creating a single istream that lazily opens the external files only when data is actually tried to be read from that part of the message. The link paths don't have to be in any specific format. In future perhaps it can recognize different formats (even http:// urls and such). Saving attachments separately ----------------------------- Message MIME structure is being parsed while message is saved. After each MIME part's headers are parsed, it's determined if this part should be stored into attachment storage. By default it only checks that the MIME part isn't multipart/* (because then its child parts would contain attachments). Plugins can also override this. For example they could try to determine if the commonly used clients/webmail always downloads and shows the MIME part when opening the mail (text/*, inline images, etc). dbox_attachment_min_size specifies the minimum MIME part size that can be saved as an attachment. Anything smaller than that will be stored normally. While reading a potential attachment MIME part body, it's first buffered into memory until the min. size is reached. After that the attachment file is actually created and buffer flushed to it. Each attachment filename contains a global UID part, so that no two (even identical) attachments will ever contain the same filename. But there can be multiple attachment storages in different mount points, and each one could be configured to do deduplication internally. So identical attachments should somehow be stored to same storage. This is done by taking a hash of the body and using a part of it as the path to the file. For example: mail_location = dbox:~/dbox:ATTACHMENTS=/attachments/$/$ Each $ would be expanded to 8 bits of the hash in hex (00..ff). So the full path to an attachment could look like: /attachments/04/f1/5ddf4d05177b3b4c7a7600008c4a11c1 Sysadmin can then create /attachment/00..ff as symlinks to different storages. Hashing problems ---------------- Some problematic design decisions: 1) Hash is taken from hardcoded first n kB vs. first dbox_attachment_min_size bytes? + With first n kB, dbox_attachment_min_size can be changed without causing duplication of attachments, otherwise after the change the same attachment could get a hash to a different storage than before the change. - If n kB is larger than dbox_attachment_min_size, it uses more memory. - If n kB is determined to be too small to get uniform attachment distribution to different storages, it can't be changed without recompiling. 2) Hash is taken from first n bytes vs. everything? + First n bytes are already read to memory anyway and can be hashed efficiently. The attachment file can be created without wasting extra memory or disk I/O. If everything is hashed, the whole attachment has to be first stored to memory or to a temporary file and from there written to final storage. - With first n bytes it's possible for an attacker to generate lots of different large attachments that begin with the same bytes and then overflow a single storage. If everything is hashed with a secure hash function and a system-specific secret random value is added to the hash, this attack isn't possible. I'm thinking that even though taking a hash of everything is the least efficient option, it's the safest option. It's pretty much guaranteed to give a uniform distribution across all storages, even against intentional attacks. Also the worse performance isn't probably that noticeable, especially assuming a system where local disk isn't used for storing mails, and the temporary files would be created there. Single instance storage ----------------------- All of the above assumes that if you want a single instance storage, you'll need to enable it in your storage. Now, what if you can't do that? I've been planning on making all index/dbox code to use an abstracted out simple filesystem API rather than using POSIX directly. This work can be started by making the attachment reading/writing code use the FS API and then create a single instance storage FS plugin. The plugin would work like: open(ha/sh/hash-guid): The destination storage is in ha/sh/ directory, so a new temp file can be created under it. The hash is part of the filename to make unlink() easier to handle. Since the hash is already known at open() time, look up if hashes/<hash> file exists. If it does, open it. write(): Write to the temp file. If hashes/ file is open, do a byte-by-byte comparison of the inputs. If there's a mismatch, close the hashes/ file and mark it as unusable. finish(): a) If hashes/ file is still open and it's at EOF, link() it to our final destination filename and delete the temp file. If link() fails with ENOENT (it was just expunged), goto b. If link() fails with EMLINK (too many links), goto c. b) If hashes/ file didn't exist, link() the temp file to the hash and rename() it to the destination file. c) If the hashed file existed but wasn't the same, or if link() failed with EMLINK, link() our temp file to a second temp file and rename() it over the hashes/ file and goto a. unlink(): If hashes/<hash> has the same inode as our file and the link count is 2, unlink() the hash file. After that unlink() our file. One alternative to avoid using <hash> as part of the filename would be for unlink() to read the file and recalculate its hash, but that would waste disk I/O. Another possibility would to be to not unlink() the hashes/ files immediately, but rather let some nightly cronjob to stat() through all of the files and unlink() the ones that have link count=1. This could be wastefully inefficient though. Yet another possibility would be for the plugin to internally calculate the hash and write it somewhere. If it's at the beginning of the file, it could be read from there with some extra disk I/O. But is it worth it?.. Extra features -------------- The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space: 1) The attachment could be compressed (header contains compressed-flag) 2) If base64 attachment is in a standardized form that can be 100% reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly. It would be nice if it was also possible to compress (and decompress) attachments after they were already stored. This would be possible, but it would require finding all the links to the message and recreating them to point to the new message. (Simply overwriting the file in place would require there are no readers at the same time, and that's not easy to guarantee, except if Dovecot was entirely stopped. I also considered some symlinking schemes but they seemed too complex and they'd also waste inodes and performance.) Code status ----------- Initial version of the attachment reading/writing code is already done and works (lacks some error handling and probably performance optimizations). The SIS plugin code is also started and should be working soon. This code is very isolated and can't cause any destabilization unless it's enabled, so I'm thinking about just adding it to v2.0 as soon as it works, although the config file comments should indicate that it's still considered unstable.

5 31

[Dovecot] Dovecot upgrades break Blackberry instant email (BIS)
by interfaSys sàrl 17 Sep '10

17 Sep '10

Hello, Just like when upgrading to 1.2, upgrading to 2.0.1 confuses RIM's BIS servers and users don't have instant email anymore on their Blackberrys. Under normal conditions, the BIS servers keep one open connection per email account with Dovecot. They're using IMAP IDLE to monitor changes and push them to devices very quickly. After an upgrade, all changes. The BIS servers can't maintain an idling connection anymore, they just poll the Dovecot server when they feel like it (every 2-15 minutes). There is one way to fix this on the devices: The users have to re-create the email account. I don't find this very convenient and was wondering if there was something that could be done on the Dovecot side? Last time it happened, I tried clearing the indexes and resetting file permissions, but nothing helped. Cheers, Olivier

5 14

[Dovecot] SSL stops working after server upgrade
by "Ing. Daniel Rozsnyó" 17 Sep '10

17 Sep '10

Hello, after an upgrade of the dovecot server, I can not use SSL connections. Can you help me solving the issue? When googling for the error, there is an opinion that the client is broken (in my case Thunderbird, 3.1.3), but I rather see the issue in the server, because I was upgrading my gentoo box by hand, and the secure imap stopped to work after that. Is there a more verbose SSL logging than the default verbose ssl = yes? It's output is attached below. Also, maybe only a configuration detail is missing, since with this version of dovecot, the configuration files were broken into several smaller files, but for my first look they contained the options I had used before. Thanks. Sep 16 23:12:25 [dovecot] master: Dovecot v2.0.2 starting up (core dumps disabled) Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x10, ret=1: before/accept initialization [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: before/accept initialization [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv2/v3 read client hello A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client hello A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server hello A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write certificate A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write key exchange A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server done A [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [192.168.77.202] Sep 16 23:12:29 [dovecot] imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [192.168.77.202] - Last output repeated twice - Sep 16 23:12:30 [dovecot] imap-login: Warning: SSL alert: where=0x4004, ret=554: fatal bad certificate [192.168.77.202] Sep 16 23:12:30 [dovecot] imap-login: Warning: SSL failed: where=0x2002: SSLv3 read client certificate A [192.168.77.202] Sep 16 23:12:30 [dovecot] imap-login: Disconnected (no auth attempts): rip=192.168.77.202, lip=192.168.77.201, mpid=0, TLS handshaking: SSL_accept() failed: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate: SSL alert number 42

2 2

[Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
by interfaSys sàrl 16 Sep '10

16 Sep '10

Hello, On FreeBSD, Dovecot 2.0.1 compiles fine with gcc45 and the other services have no problem, but even without any optimizations (CFLAGS) at all, pop3-login seg faults when a user tries to connect to Dovecot using POP3. As this is not the only application that has a problem with the latest version of gcc (mod_php), there may be nothing that can be done on the Dovecot side, but I thought I would mention it, in case it was due to some bug in the pop3-login code. Cheers, Olivier

2 12

[Dovecot] imap processing spinning on the CPU
by Daniel J. Luke 16 Sep '10

16 Sep '10

Version: 2.0.2 OS: Mac OS X 10.5.8 IMAP Client: Apple Mail.app on 10.5.8 Since upgrading to 2.0.2 I've noticed one user ends up with 4-5 imap processes all using as much CPU as they can get. I know of one other person who is seeing this on a FreeBSD server (that person moved back to dovecot 1.2 which 'solved' the problem there). I didn't notice anything in the list archive about this, but maybe I missed it. (gdb) bt full #0 0x9534c2a8 in memchr () No symbol table info available. #1 0x001ef220 in parse_next_body_to_boundary () No symbol table info available. #2 0x001f02a8 in message_parser_parse_next_block () No symbol table info available. #3 0x0013c1cc in index_mail_cache_parse_continue () No symbol table info available. #4 0x0015c634 in mbox_save_body () No symbol table info available. #5 0x0015ca14 in mbox_save_continue () No symbol table info available. #6 0x000ecc58 in mail_storage_copy () No symbol table info available. #7 0x000f5b80 in mailbox_copy () No symbol table info available. #8 0x00003a10 in cmd_copy (cmd=0x80e820) at cmd-copy.c:66 search_ctx = (struct mail_search_context *) 0x403900 src_trans = (struct mailbox_transaction_context *) 0x403230 srcset_ctx = { str = 0x282d0, first_uid = 0, last_uid = 4294967295 } save_ctx = (struct mail_save_context *) 0x0 mail = (struct mail *) 0x81bc20 src_uidset = (string_t *) 0x282d0 copy_count_r = (unsigned int *) Cannot access memory at address 0x0 -- Daniel J. Luke +========================================================+ | *---------------- dluke(a)geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+

3 3

[Dovecot] Moving from MBOX to Maildir or DBox ?
by Frank Bonnet 16 Sep '10

16 Sep '10

Hello Actually we use MBOX format for all our IMAP access but users complain they cannot create subfolders in imap folders, of course they cannot because of MBOX format ... I need gurus advices on changing mailbox format Maildir of Dbox ? What would be the safest way to go ? Note : we extensively use Thunderbird here. Thanks

4 3

[Dovecot] dovecot as exchange server?
by Jacek Osiecki 16 Sep '10

16 Sep '10

Hi, This might be a stupid question, since my knowledge about Exchange is really poor... Since I've got iPhone, I have noticed that IMAP mailbox support is quite crippled in this device - in comparison to Exchange mailboxes... As I assume that it would be easier to convince Bill Gates to install linux than to convince Steve Jobs to change anything in his "perfect phone", I started searching for something that could serve emails similarly to Exchange. Since gmail provides Exchange access, and they definitely don't run a single huge MS Exchange server, I assume that it is somehow possible to implement Exchange protocols - or am I wrong? So, finally - if it is possible to use Exchange protocol, would there be any plans for implementing it in dovecot? Best regards, -- Jacek Osiecki joshua(a)ceti.pl GG:3828944

6 8

[Dovecot] dsync mirror duplicates problem
by Waldemar W. 16 Sep '10

16 Sep '10

Im trying to sync 2 mailserver with dsync mirror functionality. Problem occurs when mails arrive in the same folder on both sides then dsync sometimes runs into problems and duplicates that entrys on both side(2mails then with same content). After running again the same happens again(now 4 mails with same content on every side). And then duplicates this mails on both sides on every run :( Anyone with same problem or is something that could cause this? I tried dsync with and without -f option but no change. Im running dovecot 2.0.1 together with dovecot sieve (latest hg version). thanks, Waldemar

2 3

[Dovecot] Dovecot2 and mdbox problem
by Michael Grosseck 16 Sep '10

16 Sep '10

Hello, I've just installed Dovecot from the http://xi.rename-it.nl/debian/ repository. But I have a problem with the mdbox Format. When the home folder is empty Dovecot should create the folder structure if the first mail arrived or the user logs in the first time. But I get following error: Sep 16 12:30:41 smtp dovecot: lmtp(31244): Connect from local Sep 16 12:30:41 smtp dovecot: lmtp(31244, info(a)example.de) Error: file_dotlock_open() failed with file /var/mail/es030107p1/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: No such file or directory Sep 16 12:30:41 smtp dovecot: lmtp(31244, info(a)example.de) Error: Opening INBOX failed: Mailbox doesn't exist: INBOX Sep 16 12:30:41 smtp dovecot: lmtp(31244, info(a)example.de) 0ZQgDbkNkkwMegAA00WXAw: msgid=<20100916123041.4781A4F4138(a)smtp.example.de>: save failed to INBOX: Mailbox doesn't exist: INBOX Sep 16 12:30:41 smtp dovecot: lmtp(31244, info(a)example.de) Error: BUG: Saving failed to unknown storage Sep 16 12:30:41 smtp dovecot: lmtp(31244): Disconnect from local: Client quit Also if I try to create a mailbox by hand I get a similar error. doveadm(info(a)example.de): Error: file_dotlock_open() failed with file /var/mail/es030107p1/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: No such file or directory doveadm(info(a)example.de): Error: Can't create mailbox INBOX: Internal error occurred. Refer to server log for more information. [2010-09-16 14:07:42] If I now create the dbox-Mails directory and the dovecot.index.log by hand and make a doveadm -v force-resync -u info(a)example.de INBOX everthing is alright. Also I can not create a new IMAP-folder with Thunderbird. It seems Dovecot forgets to create the dbox-Mails directory and the dovecot.index.log file. Now my config: # 2.0.2 (eb5adb193064): /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-xen-amd64 x86_64 Debian 5.0.6 auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = plain base_dir = /var/run/dovecot/ disable_plaintext_auth = no first_valid_gid = 10000 first_valid_uid = 10000 last_valid_gid = 10000 last_valid_uid = 10000 login_greeting = Webfixx Dovecot ready. mail_debug = yes mail_gid = vmail mail_location = mdbox:%h/mdbox mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = liox(a)elso.mine.nu service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql }

2 2

[Dovecot] Execution of external commands from Pigeonhole sieve
by Andrés Yacopino 16 Sep '10

16 Sep '10

Does anybody knows if there are some method to call an externall command from sieve? Is xexec plugin is what i am looking for? Is it avaiable for Dovecot 1.2? __ <http://wiki.dovecot.org/Plugins/xexec>I am using dovecot 1.2 and sieve 0.1.17. I want to call ripmime from sieve to save attchments from mails in a user account. Thanks for your feedback, Greetings, -- Andrés Fernando Yacopino

2 1