October 2006 - dovecot

[Dovecot] Clustering (replication and proxying) plans for the future
by Timo Sirainen 27 Oct '06

27 Oct '06

I wrote this mostly so that I won't forget my thoughts. Most likely won't be implemented anytime soon. I haven't thought through all the locking issues. It would require a distributed lock manager in any case. I think there are already several existing ones, so one of them could probably be used for Dovecot. Each mail server would consult a master server for getting namespace configuration and location for shared/public mailboxes. If the data is stored in some central SQL server there could just as well be multiple master servers (or even each server be their own master). Whenever trying to open a shared mailbox that doesn't exist in the current server, the imap process would automatically create a new connection for the destination server and start proxying the data. The proxy would be a somewhat dummy proxy, which just forwards the client's commands to it and doesn't even try to understand the replies from the remote server. So admin will have to be careful to use only servers which have all the capabilities that were announced by the original server. So, then the replication. Each namespace could be configured to be available either by proxying or by replication: namespace public { prefix = example/ location = proxy:imap.example.com } namespace public { prefix = example/ replicated_to = host1.example.com host2.example.com # If multi-master, we'll need to use global locking. If not, the # replicated_to servers contain only slaves. multi_master = yes location = maildir:~/Maildir } The way this would work is that each imap process connects via UNIX socket to a replication process, which gets input from all the imap processes. The replication process keeps track of how much has been sent to each other server and also the acks it has received from them. So if the network dies temporarily it'll start writing the changes to the disk until it reaches some limit, after which a complete resync will be necessary. Dovecot already writes all the changes to a mailbox first to a transaction log file and and only afterwards it updates the actual mailbox. The contents of the new mails aren't stored there though. The replication could work simply by sending the transaction logs' contents to the replicatio process which passes it onto other servers, which finally sync their local mailboxes based on that data. Since Dovecot already is able to sync mailboxes based on the transaction log's contents this should be pretty easy to implement. Of course the new mails' contents also have to be sent. This could be prioritized lower than the transaction traffic, so that each server always has very up-to-date view of the mailbox metadata, but not necessarily the contents of all the mails. If the server finds itself in a situation that it doesn't have some specific mail, it'll send a request to the replication process to fetch it ASAP from another server. The reply will then take the highest priority in the queue. Normally when opening or reading the mailbox there's really no need to do any global locking. The server will show the mailbox's state to be what it has last received from the replication process. Whenever modifying the mailbox a global lock is required. Here's also the problem that if the network is down between two servers, they both think that they got the global lock. If they did any conflicting changes, they'll have to be resolved when both of them again see each others. Dovecot's transactions in general aren't order-dependent so most of the conflicts can be resolved by just having the replication master decide which way the transactions were done and then sending the transactions back to all of the servers in that order. Adding new messages is a problem however. UIDs must be always growing and they must be unique. I think Cyrus solved this by having 64bit global UIDs which really uniquely identify the messages (ie. first 32bits contain server-specific UID), and then the 32bit IMAP UIDs. If there are any conflicts with the IMAP UIDs, they're given new UIDs (which invalidates clients' cache for those messages, but that can't be helped). Resyncing works basically by requesting a list of all the mailboxes, opening each one of them and requesting new transactions from the log files with <UID validity>, <log file sequence>, <log file offset> triplet. If UID validity has changed or log file no longer exists, a full resync needs to be done. Also if the log file contains a lot of changes, it's probably faster to do a full resync. Full resync would then be just sending the whole dovecot.index file. Conflict resolving could also require a full resync if transaction logs were lost. That'd work again by the master receiving everyone's dovecot.index files, deciding what to do with them and sending the modified version back to everyone. Each server has different transaction logs, so the log file sequences can't be directly used between servers. So the replication process probably will have to keep track of every server's log seq/offset. Also when receiving full dovecot.index files they'll have to be updated in some way. Then there's the problem of how to implement the receiving side. If the user already has a process connected to the replication process it could be instructed to update the mailboxes. But since this doesn't happen all that often, there also needs to be some replication-writer process which writes all the received transactions to the mailboxes. Now, when exactly should these replication-writer processes be created then? If all users have the same UNIX user ID, then there could be a spool of processes always running and doing the writing. Although this worries me a bit since the same process is handling different users' data. More secure way, and the only way if users have different UNIX UIDs, is to launch a separate writer process for each user. It probably isn't a great idea to do that immediately when receiving data for a user, but to queue data for the user for a while and then once enough has been received or some timeout has occurred, the writer process would be executed. There probably should be some shared in-memory cache with a memory limit of some megabytes, after which data is written to disk up to a configurable amount. Only after that is full or timeout for the user is reached, the writer process is executed and the data is sent to it. All this replication stuff could also be used to implement a remote mail storage (ie. smarter proxying). The received indexes/transactions could simply be kept in memory, and all the mails that aren't already cached in memory could just be requested from the master whenever needed. I'm not sure if there's much point in doing this over dummy proxying though.

2 2

[Dovecot] dovecot.org not resolving!
by /dev/rob0 27 Oct '06

27 Oct '06

I had to bypass my unknown sender/recipient domain restrictions to receive list mail and send this. It's hard to say without a whois server for .fi, but it appears that the problem might be with the procontrol.fi. domain. Action to fix the problem is needed ASAP! I've emailed Timo offlist; also tried to get him in IRC. -- Offlist mail to this address is discarded unless "/dev/rob0" or "not-spam" is in Subject: header

2 1

[Dovecot] POP 23 problem
by Manoj Fernandez 27 Oct '06

27 Oct '06

Hi there I am using Dovecot on Ubuntu 6 LTS forPOP3 services. The user account created at installation works fine and Iam able to log to POP3 services by telnetr. When I created a another user and tried tologin via telnet this is what happens telnet XXX.XXX.XXX.XXX 10 + OK Dovecot ready user XXXX + OK pass XXX +OK logged in Connection tohost lost and session terminates. Pls advise what could be the problem Tks Manoj Sri Lanka

1 0

[Dovecot] SpamAssassin training plugin
by Chad Parry 26 Oct '06

26 Oct '06

I found a plugin for dSPAM on this mailing list (http://thread.gmane.org/gmane.mail.imap.dovecot/4632/focus=4632) that was very useful to me. I just modified the code to work for SpamAssassin instead. The purpose of this plugin is to watch the mail that is in the user's spam folder. If the user manually moves a piece of mail into the spam folder then the mail must have been spam that was incorrectly classified. This plugin causes SpamAssassin to relearn that mail as spam. Anyone else who finds this functionality useful is welcome to follow the link below. . http://chad.parry.org/software/dovecot/ - Chad Parry

3 4

[Dovecot] i386->amd64: tls-certificate cannot be read anymore
by Andreas Barth 26 Oct '06

26 Oct '06

Hi, after rsyncing a tls-certificate from an i386-host to an amd64-one, I got this error message; Oct 25 21:13:23 alius dovecot: imap-login: Can't load private key file /etc/ssl/private/dovecot.pem: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch This happened with dovecot 1.0.rc7-1 (Debian etch) - the version on the i386 was 1.0.rc2-1bpo1. Any hints for me what to do? Cheers, Andi -- http://home.arcor.de/andreas-barth/

2 1

[Dovecot] mail_extra_groups and home directory permissions
by Joe Cooper 26 Oct '06

26 Oct '06

Hi all, I've got a virtual hosting environment, where each user lives in /home/domain/homes/user, and the mailbox is Maildir within that directory. I've been running into a permissions problem, because it seems Dovecot doesn't get the users group membership (which seems odd, but I'm not confident enough of my comprehension of Dovecot to call say it looks like a bug). In short, when Dovecot hits the /home/domain directory which is owned by domain:domain and set to 750 permissions, it gets a permission denied, despite the user having the needed group membership to traverse these directories. So, in an effort to work around this problem, I came upon the mail_extra_groups option, which seems to allow me to drop dovecot into additional groups. If I add the domain group to this list, dovecot is able to traverse to the correct directory and all works exactly as I want it to with the domain directories being set to 750 permissions. Hooray! My only question, since this is not the documented purpose of this option, is: Is it safe to do this? Am I opening up some other kind of security problem by setting this option to include all domain groups? Thanks for any thoughts anyone might have on this...

2 2

[Dovecot] "Real" users AND Virtual users
by Muhamad A. Martoprawiro 25 Oct '06

25 Oct '06

Dear all, How can I use dovecot for both "real" users and virtual users? I could configure dovecot for real users, by using these lines in /etc/dovecot.conf: default_mail_env = maildir:%h/Maildir auth default { passdb pam { } userdb passwd { } } To accept virtual users, I used these lines: default_mail_env = maildir:/usr/local/virtual/%u/ auth default { passdb sql { args = /etc/dovecot-mysql.conf } userdb sql { args = /etc/dovecot-mysql.conf } } But, again, how to accept both kind of users? I use Postfix as mail server, and use LDAP/PAM for real users that have home-directory on the system. My OS is Mandriva 2007, and I use dovecot that is bundled as an RPM package (dovecot 1.9 rc7) in the OS. The content of file /etc/pam.d/dovecot: #%PAM-1.0 auth required pam_stack.so service=system-auth account required pam_stack.so service=system-auth Thanks. Regards, Muhamad

4 6

[Dovecot] P0P3 problems
by Nick 25 Oct '06

25 Oct '06

Hi, I'm setting up Dovecot on a Linux (Ubuntu) server and IMap works but P0P3 doesn't. When I try to get the eMail, it appears to do everything it should (no errors) but nothing gets downloaded. I'm also a little confused with this: If I select either "Password authentication source" Unix passwd or shadow file (using WEBmin or text edit the config file) Dovecot errors: Unknown setting: line 648: and yet, if I take it out it works fine. How does Dovecot get the passwords if no source is included ??? Please advise, Thanks, Nick. . . 1(973)614-8385 _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ . . . . . . . . Please visit my Web Hosting/Design Business at: http://MyHost.org -=[ The prices are great - as is the service ]=- . . . . . . . . _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/ _/

9 10

[Dovecot] performance problems with rc10 and solaris 10
by Dominik Saar 25 Oct '06

25 Oct '06

Hi, i have 2 dovecot imap server running on solaris 10 The primary server with dovecot 1.0.alpha4 is running with best performance on Sun Sparc E4k5 with 8x 450MHz The secondary with rc10 has ugly performanceproblems on Sun E450 wth 4x 450MHz. Every new imap process takes permanent 25% cpu usage List the folder and fetch message are both slowy! Both servers are configred with ldap backend preventive i deleted for rc10 the complet index and controlfiles, because the spooldir is a sync from the masterserver with dovecot alpha4. I thought its a little bit slowy because of my 30.000 mails. the indexfiles are created now, but the still ugly slow .... For debugging i compiled rc10 again --with-debug=yes and start dovecot with solaris truss truss tells nothing to explain this behavor logentrys from dovecot has this warnings: --> dovecot: Oct 18 14:51:47 Warning: IMAP(saar): Growing pool 'storage' with: 1024 dovecot: Oct 18 14:51:47 Info: IMAP(saar): dirsize quota limit = 750000kB dovecot: Oct 18 14:51:47 Warning: IMAP(saar): Growing data stack with: 32768 dovecot: Oct 18 14:51:47 Warning: IMAP(saar): Growing data stack with: 65536 dovecot: Oct 18 14:51:48 Warning: IMAP(saar): Growing data stack with: 131072 dovecot: Oct 18 14:51:48 Warning: IMAP(saar): Growing pool 'uidlist record_pool' with: 32768 <-- also i compiled rc10 --with-notify=none, no result to ensure it isn´t the solaris system, i compiled alpha4. Result: good performance Any ideas? Cheers Dominik

3 10

[Dovecot] Unsolved : mbox-sync-rewrite.c : assertion failed
by Jakob Curdes 25 Oct '06

25 Oct '06

Some days ago I wrote about an inaccessible mailbox after an upgrade from 0.99.14 to 1.0rc10. Today I get this problem with another mailbox but it seems to be it is not only related to the upgrade : - user comes back from holiday, does not see his inbox : dovecot: Oct 24 18:51:09 Error: IMAP(abcd): file mbox-sync-rewrite.c: line 405 (mbox_sync_read_and_move): assertion failed: (need_space == (uoff_t)-mails[idx].space) (he accessed it last with the old dc version) Solution : moving account to other server, copy everything over to the old account and it worked. This is what I did and afterwards the mailbox was readable. Now the user goes back to his PC (I made my work from another machine) and opens his mailclient (with the still existing account) and BINGO dovecot crashes again on the mailbox as berfore. Once ist has crashed on it, it than crashes always, even if you make a new account or remove the index. So I suppose it has rather to do with cached information in the client than with the upgrade itself. But nothing the client sends should be able to crash the server ! Hoping for an idea, Jakob Curdes

3 3