- dovecot - dovecot.org

Re: [Dovecot] dsync crash
by Timo Sirainen 17 Sep '10

17 Sep '10

On Wed, 2010-09-15 at 18:34 +0400, Василий Шоков wrote: > > namespace { > hidden = no > inbox = yes > location = maildir:~/mail/inbox/:INDEX=/var/dovecot_index/%u/inbox > prefix = .INBOX. Uh. Does this really work with IMAP clients? A namespace prefix must not start with hierarchy separator. My guess is that the crash happens because of it. I think I'll add code to give an error if it's tried.

2 8

[Dovecot] dsync to move from 1.2 to 2.0
by Daniel L. Miller 17 Sep '10

17 Sep '10

I'm not certain of the correct syntax for this - and I'm reluctant to risk my live data by experimenting. I've setup a 2.0.2 server on a virtual machine, and I want to copy a live account over to it for testing. Not that it should matter, but the 1.2 machine is using maildir and I want to give sdbox a try on 2.0.2. I'm assuming (dangerous, I know) I should run dsync on the new 2.0.2 machine, and connect remotely to the 1.2. If it matters, my usernames are complete mail addresses, i.e. myname(a)mydomain.com, and are stored as /var/mail/domain/name/Maildir. Does dsync perform an imap connection to the remote server to pull the mails? Or am I confusing this with imapsync? -- Daniel

2 1

[Dovecot] Processes remaining
by Andreas Kossack 17 Sep '10

17 Sep '10

Hi all, on my system ("AIX 6.1" 64bit) after "doveadm stop" there are some processes still alive: vmail 3997878 1 0 16:54:26 - 0:00 dovecot/imap dovenull 4915236 1 0 16:54:26 - 0:00 dovecot/imap-login root 6029330 1 0 16:54:20 - 0:00 dovecot/ssl-params root 6553702 1 0 16:52:01 - 0:00 dovecot/log vmail 6881392 1 0 16:54:21 - 0:00 dovecot/imap dovecot 7078140 1 0 16:52:23 - 0:00 dovecot/auth dovenull 7733396 1 0 16:54:20 - 0:00 dovecot/imap-login root 7864502 1 0 16:54:21 - 0:00 dovecot/auth -w root 9240650 1 0 16:54:20 - 0:00 dovecot/config dovecot 12189718 1 0 16:52:01 - 0:00 dovecot/anvil After some minutes I can see less than before: root 6029330 1 0 16:54:20 - 0:00 dovecot/ssl-params root 6553702 1 0 16:52:01 - 0:00 dovecot/log dovecot 12189718 1 0 16:52:01 - 0:00 dovecot/anvil But these seem to be permanent. Restarting dovecot there is an error sometimes logged: Sep 9 17:08:17 Buckow local3:err|error dovecot: master: Error: service(auth): Socket already exists: /var/run/dovecot/auth-login And - some restarts later: root 3997884 1 0 17:07:21 - 0:00 dovecot/ssl-params root 6029330 1 0 16:54:20 - 0:00 dovecot/ssl-params root 6226004 13172840 0 17:14:04 - 0:00 dovecot/config root 6553702 1 0 16:52:01 - 0:00 dovecot/log dovecot 6881398 1 0 17:06:13 - 0:00 dovecot/anvil root 7733432 13172840 0 17:14:04 - 0:00 dovecot/log root 7864522 1 0 17:06:13 - 0:00 dovecot/log dovecot 7995430 13172840 0 17:14:04 - 0:00 dovecot/anvil dovecot 9240682 1 0 17:11:20 - 0:00 dovecot/anvil root 9306298 1 0 17:13:28 - 0:00 dovecot/ssl-params root 9371814 1 0 17:11:20 - 0:00 dovecot/log dovecot 12189718 1 0 16:52:01 - 0:00 dovecot/anvil root 13172840 1 0 17:14:04 - 0:00 /usr/sbin/dovecot What's wrong? -- Kind regards / Mit freundlichen Grüßen A. Kossack _______________________________________________________________________________ Andreas Kossack Andreas.Kossack(a)TU-Cottbus.De BTU Cottbus, Brandenburg, Germany, Walther-Pauer-Str.2, 03046 Cottbus http://wwwca.telesec.de/cgi-bin/caservice/Common/InstallRoot/DT-Root-CA-2.d… _______________________________________________________________________________ Diese Nachricht wurde aus 100% wiederverwerteten Elektronen erstellt. _______________________________________________________________________________

2 5

[Dovecot] sent-mail
by dhottinger＠harrisonburg.k12.va.us 17 Sep '10

17 Sep '10

Im trying to nail down an issue with some users sent-mail. I have had a few users complaining that their emails are not being copied to their sent-mail folder. Im not sure how widespread it is, mainly because most people dont check their sent-mail. Im running Dovecot version 1.0.7, and yes I know its outdated, but I have had zero problems thus far. Im grepping through my logfiles and dont see any error messages to indicate that there are any issues anywhere on the mailserver. Has anyone else seen any issues with sent-mail not being copied? If checked permissions on the folders and they look correct. thanks, ddh -- Dwayne Hottinger Network Administrator Harrisonburg City Public Schools "Everything should be made as simple as possible, but not simpler." -- Albert Einstein "The hottest places in Hell are reserved for those who, in times of moral crisis, preserved their neutrality." -- Dante

3 4

[Dovecot] dsync - how to speed up
by Burckhard Schmidt 17 Sep '10

17 Sep '10

Hi, I have converted some mbox mailboxes to maildir. Inbox and folders had been cpopied before into a separate directories "old-home" and old-inbox onto the new server (dovecot 2.0.2. with fix hg.dovecot.org/dovecot-2.0/rev/14a6e526900d) dsync -v -u user mirror mbox:/old-homehome/~user:INBOX=/old-inbox/user It works but it lasts "very long" time. For one mailbox with 2,4 Gb and nearly 200.000 emails it lasts 100 minutes. I would like to migrate 1.500 Gb - not possible in that way I think. Are there any possibilities making it faster? May be not to generate indices, ..., which could be done later during a first access by the user? Is it possible to run several dsync commands at the same time, of course each one for a different user? (Ressources cpu and I/O system are available.) -- Mit freundlichen Gruessen --- Burckhard Schmidt

2 1

[Dovecot] fts plugin - purpose of get_last_uid() ?
by Per Jessen 17 Sep '10

17 Sep '10

It looks like the uid returned is only used as an indication of whether the maildir is fully indexed or not, is that correct? In my setup, the index is built externally and will _always_ be up-to-date. Getting the last uid would mean running a query, which, whilst inexpensive, seems unnecessary if my understanding of the purpose of get_last_uid() is correct. thanks Per Jessen, Zürich

2 4

[Dovecot] fts_squat hanging on some messages
by Henning Rohlfs 17 Sep '10

17 Sep '10

Hello, I recently upgraded to dovecot 2.0.2 (on gentoo ~amd64) and now fts_squat's index building seems to hang for some of my mailboxes. The imap process gets stuck at 100% cpu usage and dovecot.index.search is never touched. I traced the problem to a message (attached) for one of the problematic mailboxes. Also attached are a full backtrace and the output of dovecot -n. The problem is reproducible: step 1) /usr/libexec/dovecot/imap step 2) . select Junk step 3) . search body sometext

2 2

[Dovecot] Can solaris sendmail use dovecot lda
by Jürgen Obermann 17 Sep '10

17 Sep '10

Hello, on a Solaris 9 system with dovecot 1.2.14 installed I want to replace the sendmail local delivery agent 'mail.local' with the dovcot LDA. The wiki page http://wiki.dovecot.org/LDA/Sendmail did not really help me, because there are only linux specific tipps I am not able to apply to solaris sendmail.cf. So how can I tell solaris sendmail to use dovecots LDA instead of mail.local? Greetings, Jürgen Obermann -- Hochschulrechenzentrum der | Mail: Juergen.Obermann(a)hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/ Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009 ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

2 1

[Dovecot] dsync: mbox to maildir
by Burckhard Schmidt 17 Sep '10

17 Sep '10

Hello, I started converting INBOX and folder from mbox to maildir. The old folders comming from Dovecot 1.0.13 with a mail_location mbox:%h:INBOX=/inbox/%u. The new instalation on a different server is Dovecot 2.0.1, mail-location maildir:~/maildir:INDEX=/addons/index/%u: CONTROL=~/control:LAYOUT=fs No user has logged in before. I have copied tho old Inbox and folders into /mailstore1/import-inbox and /mailstore1/import-home to the new server. my dsync command dsync -v -u account mirror mbox:/mailstore1/import-home/~account:INBOX=/mailstore1/import-inbox/account This works fine. But there are some problems too: 1. autocreate The plugin autocreate = AutoCleanSpam is activated for lda and imap. There is no user with such folder. dsync ceates two: dsync(account): Info: AutoCleanSpam: only in source dsync(account): Info: AutoCleanSpam: only in dest I saw AutoCleanSpam and AutoCleanSpam_73fc7f01efa3804cb4290000b68194d6, both empty. I deletet it with doveadm mailbox delete -u account -s "AutoCleanSpam_73fc7f01efa3804cb4290000b68194d6" Furthermore I have had before autocreate = Sent ( and some more) If the user has such a folder then again two folders are created, "Sent" which is empty and "Sent_any characters" like above. This folder contains the emails of the old folder Sent. 2. missing skip_broken_mailboxes, skip_dotdirs dsync reports errors like Error: Failed to sync mailbox xyz: Mailbox isn't a valid mbox file After a few minutes dsync stops. Or a file like .mlbxlsttmp causes the same error. I remembered my tests with convert-tool (dovecot-V1.2.12) and could use skip_broken_mailboxes skip_dotdirs. That was fine and I'm missing it. I will have to convert several thousands of users. May be I have to clean their folders before? 3. empty folder I have seen some empty folders. These have not been generated by dsync. Some user could missing them? On ballance dsync converts 10 Gb of mbox to maildir successfully. -- Regards - Burckhard Schmidt

2 2

[Dovecot] (Single instance) attachment storage
by Timo Sirainen 17 Sep '10

17 Sep '10

Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :) The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome. Reading attachments ------------------- dbox metadata would contain entries like (this is a wrapped single line entry): X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b So the format is: "X" 1*(<offset> <byte count> <link path>) So when reading a dbox message body, it's read as: offset=0: <first 1442 bytes from dbox body> offset=1442: <next 2742784 bytes from external file> offset=2744226: <next 152 bytes from dbox body> offset=2744378: <next 27423 bytes from external file> offset=2744378 27423: <the rest from dbox body> This is all done internally by creating a single istream that lazily opens the external files only when data is actually tried to be read from that part of the message. The link paths don't have to be in any specific format. In future perhaps it can recognize different formats (even http:// urls and such). Saving attachments separately ----------------------------- Message MIME structure is being parsed while message is saved. After each MIME part's headers are parsed, it's determined if this part should be stored into attachment storage. By default it only checks that the MIME part isn't multipart/* (because then its child parts would contain attachments). Plugins can also override this. For example they could try to determine if the commonly used clients/webmail always downloads and shows the MIME part when opening the mail (text/*, inline images, etc). dbox_attachment_min_size specifies the minimum MIME part size that can be saved as an attachment. Anything smaller than that will be stored normally. While reading a potential attachment MIME part body, it's first buffered into memory until the min. size is reached. After that the attachment file is actually created and buffer flushed to it. Each attachment filename contains a global UID part, so that no two (even identical) attachments will ever contain the same filename. But there can be multiple attachment storages in different mount points, and each one could be configured to do deduplication internally. So identical attachments should somehow be stored to same storage. This is done by taking a hash of the body and using a part of it as the path to the file. For example: mail_location = dbox:~/dbox:ATTACHMENTS=/attachments/$/$ Each $ would be expanded to 8 bits of the hash in hex (00..ff). So the full path to an attachment could look like: /attachments/04/f1/5ddf4d05177b3b4c7a7600008c4a11c1 Sysadmin can then create /attachment/00..ff as symlinks to different storages. Hashing problems ---------------- Some problematic design decisions: 1) Hash is taken from hardcoded first n kB vs. first dbox_attachment_min_size bytes? + With first n kB, dbox_attachment_min_size can be changed without causing duplication of attachments, otherwise after the change the same attachment could get a hash to a different storage than before the change. - If n kB is larger than dbox_attachment_min_size, it uses more memory. - If n kB is determined to be too small to get uniform attachment distribution to different storages, it can't be changed without recompiling. 2) Hash is taken from first n bytes vs. everything? + First n bytes are already read to memory anyway and can be hashed efficiently. The attachment file can be created without wasting extra memory or disk I/O. If everything is hashed, the whole attachment has to be first stored to memory or to a temporary file and from there written to final storage. - With first n bytes it's possible for an attacker to generate lots of different large attachments that begin with the same bytes and then overflow a single storage. If everything is hashed with a secure hash function and a system-specific secret random value is added to the hash, this attack isn't possible. I'm thinking that even though taking a hash of everything is the least efficient option, it's the safest option. It's pretty much guaranteed to give a uniform distribution across all storages, even against intentional attacks. Also the worse performance isn't probably that noticeable, especially assuming a system where local disk isn't used for storing mails, and the temporary files would be created there. Single instance storage ----------------------- All of the above assumes that if you want a single instance storage, you'll need to enable it in your storage. Now, what if you can't do that? I've been planning on making all index/dbox code to use an abstracted out simple filesystem API rather than using POSIX directly. This work can be started by making the attachment reading/writing code use the FS API and then create a single instance storage FS plugin. The plugin would work like: open(ha/sh/hash-guid): The destination storage is in ha/sh/ directory, so a new temp file can be created under it. The hash is part of the filename to make unlink() easier to handle. Since the hash is already known at open() time, look up if hashes/<hash> file exists. If it does, open it. write(): Write to the temp file. If hashes/ file is open, do a byte-by-byte comparison of the inputs. If there's a mismatch, close the hashes/ file and mark it as unusable. finish(): a) If hashes/ file is still open and it's at EOF, link() it to our final destination filename and delete the temp file. If link() fails with ENOENT (it was just expunged), goto b. If link() fails with EMLINK (too many links), goto c. b) If hashes/ file didn't exist, link() the temp file to the hash and rename() it to the destination file. c) If the hashed file existed but wasn't the same, or if link() failed with EMLINK, link() our temp file to a second temp file and rename() it over the hashes/ file and goto a. unlink(): If hashes/<hash> has the same inode as our file and the link count is 2, unlink() the hash file. After that unlink() our file. One alternative to avoid using <hash> as part of the filename would be for unlink() to read the file and recalculate its hash, but that would waste disk I/O. Another possibility would to be to not unlink() the hashes/ files immediately, but rather let some nightly cronjob to stat() through all of the files and unlink() the ones that have link count=1. This could be wastefully inefficient though. Yet another possibility would be for the plugin to internally calculate the hash and write it somewhere. If it's at the beginning of the file, it could be read from there with some extra disk I/O. But is it worth it?.. Extra features -------------- The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space: 1) The attachment could be compressed (header contains compressed-flag) 2) If base64 attachment is in a standardized form that can be 100% reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly. It would be nice if it was also possible to compress (and decompress) attachments after they were already stored. This would be possible, but it would require finding all the links to the message and recreating them to point to the new message. (Simply overwriting the file in place would require there are no readers at the same time, and that's not easy to guarantee, except if Dovecot was entirely stopped. I also considered some symlinking schemes but they seemed too complex and they'd also waste inodes and performance.) Code status ----------- Initial version of the attachment reading/writing code is already done and works (lacks some error handling and probably performance optimizations). The SIS plugin code is also started and should be working soon. This code is very isolated and can't cause any destabilization unless it's enabled, so I'm thinking about just adding it to v2.0 as soon as it works, although the config file comments should indicate that it's still considered unstable.

5 31