- dovecot - dovecot.org

[Dovecot] [Fwd: mulitple instance run problem]
by Stewart Dean 10 Jun '07

10 Jun '07

As I explained, I want to be able to run a second instance of DC through another port to do testing while not interrupting the users using the primary instance in the the regular port. I was instructed to create a second config file with different listen ports and a separate base_dir. I did so and invoked dc with -c pointing to dovecot2.conf. Jun 7 10:04:23 mercury mail:info dovecot: Dovecot v1.0.0 starting up I pointed my client at the non-SSL port, and everything worked fine....until 3 hours later. The client wasn't getting things through, so I checked the syslog on the dc server where I saw rapidly repeating errmsgs:... Jun 7 12:51:07 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 12:58:55 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 12:58:59 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 12:59:01 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 12:59:05 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist and many more, then...... Jun 7 13:34:20 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 13:36:30 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 13:36:54 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist I kill the second instance.....but things don't clear up....rather I now get a different errmsg....which must be from the primary dc instance, since the 2nd instance and all of its children are gone (I checked the ps) Jun 7 13:39:20 mercury mail:err|error dovecot: login: Authentication process 10 89784 doesn't exist Jun 7 13:40:00 mercury mail:err|error dovecot: imap-login: Can't connect to aut h server at default: Connection refused Jun 7 13:40:06 mercury mail:err|error dovecot: imap-login: Can't connect to aut h server at default: Connection refused Jun 7 13:40:06 mercury mail:err|error dovecot: imap-login: Can't connect to aut h server at default: Connection refused Finally I do a kill -1 of the primary instance and that cleans things up Jun 7 13:45:46 mercury mail:warn|warning dovecot: SIGHUP received - reloading configuration Is there something I'm doing wrong? Have other people run two instances without problem? I tried starting the second instance and killing it immediately thereafter....that wnet OK, but it hasn't been 3 hours yet. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

4 5

[Dovecot] auth_username_format
by Sergio Souza 10 Jun '07

10 Jun '07

I configured in the archive dovecot.conf the parameter auth_username_format = “%Ld_%Ln” because the users of my server have access yours e-mail accounts based in /etc/passswd where they are registered in format domain.com_user. This configuration work without problems via pop3 and imap (squirrelmail). My server is CentOS 5.0 and dovecot is 1.0.rc15. The problem that occurs currently is: when the users go to have access the interface of change preferences (ISPCONFIG) where the supplied data of authentication are not the data of the email account (eg. user(a)domain.com) and yes of /etc/passwd the dovecot inserts underscore before the domain (eg. _domain.com_user) when it would have to make this only between the domain and the user. Exists a way to solve this problem? Thanks in advance. -- Sergio Souza

1 0

[Dovecot] Listen on several (but not all) Interfaces
by Dominik Schulz 09 Jun '07

09 Jun '07

Hi, I'm using Dovecot 1.0rc15 from Debian stable/etch. I'd like to have Dovecot listen on more than one interface. It shall listen on one IPv4 Interface and on one IPv6 Interface. The comments in the sample configuration file say, that it's not possible to listen on more than one interface at a time. Sadly it's no option to me to listen on all interfaces. So I'd like to ask if there is a chance that Dovecot will support listening on several selected interfaces anytime soon? Thanks in advance for every helpful reply. Best Regards Dominik

5 4

[Dovecot] Dot Lock probelm resolution
by Stewart Dean 08 Jun '07

08 Jun '07

For the record/archive, so some other unfortunate dovecot implementer won't spend weeks figuring out this particular way dotlocks can have problems... here is what was trashing our dotlocks Basically, I was seeing this: May 14 15:59:58 mercury mail:warn|warning dovecot: IMAP(sdean): Our dotlock file /var/spool/mail/sdean.lock was deleted (kept it 1 secs) > May 14 15:59:58 mercury mail:err|error dovecot: IMAP(sdean): file_dotlock_delete() failed with mbox file /var/spool/mail/sdean: No such file or directory Timo justifiably blamed the usual suspect, NFS, but when I dug into the maillog to see what was going on when the locks were getting trashed, in almost all cases, NFS wasn't involved. And I had recompiled procmail so that it used the same locking strategy (and checked it), deactivated NFS client attribute caching, etc., etc. It turned out that my predecessor had done something 10 years ago that was arcane, inscrutable and kludgy in the .procmailrc file. This *purposefully* trashed the dotlock. Not knowing what it did when I took over (predecessor long gone, things in disarray), I left it in, because I worried that something would break if we removed it. We have perpetuated it in creation of all new accounts. This procmailrc stanza looks like this: ####### BEGIN SYS ADMIN ADDED LINES ####### ## !!!!!! Do not remove these lines !!!!!! ## They have been placed here by System Admin ## If removed, your mail will run slower and ## generate error messages!!!!!!!! LOCKFILE #removes preexisting lockfile LOG='lockfile $DEFAULT$LOCKEXT ' TRAP="rm -f $DEFAULT$LOCKEXT" :0 $DEFAULT ####### END OF SYS ADMIN ADDED LINES ####### # IF YOU ADD STUFF, DO IT ABOVE THIS BLOCK # I sent a note to the procmail mailing list hoping to get a clue of what is doing. Here are my Qs and their As: >> 2) Is it syntactically correct? > I think so, though highly obfuscated and semantically worthless. >> 3) Is what it's doing real/worthwhile? > No. My guess is it was a kludge to work around some error with an > older version of procmail, if it ever served any function at all. Since we removed it, there have been no dotlocking errmsgs in the syslog. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

1 0

[Dovecot] User unknown in local recipient table? Dovecot LDA/Postfix
by Yu Chen 08 Jun '07

08 Jun '07

This probably is a postfix problem, but I think there are lots of postfix experts/users on this list, and have heard a lot good things about this list, so I am just giving it a try. Thanks in advance! I am using dovecot-1.0.0-8_56.src.rpm downloaded from atrpms.net, and rebuilt from it(rpmrebuild ...). Postfix is 2.4.3. I followed documents at http://wiki.dovecot.org/LDA and LDA/Postfix, and used passwd-file (created /etc/passwd.dovecot file) both as password Database and User database. I can send emails, but not receiving, keep getting the "User unknow error". Do I need to change the "local_recipient_map" in main.cf, and if so, how? Here are some related configuration: ########### dovecot -n ########### # /etc/dovecot.conf log_path: /var/log/dovecot.log ssl_ca_file: /etc/pki/dovecot/certs/ca-bundle.crt verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: Dovecot ready. verbose_proctitle: yes first_valid_uid: 102 mail_extra_groups: mail mail_location: mbox:~/mail:INBOX=/var/mail/%u:INDEX=/var/mail_indexes/%d/%n mail_debug: yes maildir_copy_with_hardlinks: yes mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): outlook-idle auth default: mechanisms: plain login executable: /usr/local/libexec/dovecot/dovecot-auth verbose: yes debug: yes debug_passwords: yes passdb: driver: passwd-file args: /etc/passwd.dovecot userdb: driver: passwd-file args: /etc/passwd.dovecot socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 384 ############### postconf -n ################ alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases broken_sasl_auth_clients = yes command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix debug_peer_level = 2 html_directory = /usr/share/doc/postfix-2.4.3-documentation/html inet_interfaces = $myhostname, localhost invalid_hostname_reject_code = 450 mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man maps_rbl_reject_code = 450 mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain myhostname = my.host.com mynetworks = myip/24, 127.0.0.0/8 newaliases_path = /usr/bin/newaliases.postfix non_fqdn_reject_code = 450 queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix-2.4.3-documentation/readme sample_directory = /etc/postfix sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtp_tls_CAfile = /usr/share/ssl/certs/ca-bundle.crt smtp_tls_cert_file = /usr/share/ssl/certs/postfix.pem smtp_tls_key_file = /usr/share/ssl/certs/postfix.pem smtp_tls_session_cache_database = btree:/var/spool/postfix/smtp_tls_session_cache smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, permit_tls_clientcerts, reject_unauth_destination, check_relay_domains smtpd_sasl_auth_enable = yes smtpd_sasl_exceptions_networks = $mynetworks smtpd_sasl_path = private/auth smtpd_sasl_security_options = noanonymous smtpd_sasl_type = dovecot smtpd_tls_CAfile = /usr/share/ssl/certs/ca-bundle.crt smtpd_tls_cert_file = /usr/share/ssl/certs/postfix.pem smtpd_tls_key_file = /usr/share/ssl/certs/postfix.pem smtpd_tls_loglevel = 2 smtpd_tls_session_cache_database = btree:/var/spool/postfix/smtpd_tls_session_cache tls_random_source = dev:/dev/urandom unknown_local_recipient_reject_code = 550 virtual_transport = dovecot Regards, C.Y =========================================== Yu Chen Howard Hughes Medical Institute Chemistry Building, Rm 182 University of Maryland at Baltimore County 1000 Hilltop Circle Baltimore, MD 21250 phone: (410)455-6347 (primary) (410)455-2718 (secondary) fax: (410)455-1174 email: chen(a)hhmi.umbc.edu ===========================================

2 1

[Dovecot] PAM info please
by Frank Bonnet 07 Jun '07

07 Jun '07

Hello I'm searching infos on using Dovecot with pam_ldap and FreeBSD 6.2 any pointers welcome :-) Thanks -- Frank

2 2

[Dovecot] Latest MANAGESIEVE patch -- how to start daemon?
by Andreas Ntaflos 07 Jun '07

07 Jun '07

Hello list, I'm using Dovecot 1.0.0 on FreeBSD 6.2-RELEASE and applied the latest MANAGESIEVE patch v3 [1] which seems to have worked smoothly, i.e. all hunks succeeded and no compilation problems. Now this may be stupid question with an obvious answer, but I am at a loss at how to start the daemon? What's the binary called that starts the sieve server? Is there a special way to invoke it? I haven't been able to find anything useful on that matter and there doesn't seem to be a TFM to R yet, the Wiki just links to [1]. What do I have to do? Did I forgot to patch something? I'd appreciate any insight on this subject! Thanks in advance! Andreas [1] http://dovecot.org/list/dovecot/2007-March/021130.html -- Andreas "daff" Ntaflos Vienna, Austria GPG Fingerprint: 6234 2E8E 5C81 C6CB E5EC 7E65 397C E2A8 090C A9B4

5 8

[Dovecot] Quota Warning Message
by Gabriele Fergola 06 Jun '07

06 Jun '07

Hi to everyone, i try to implement in dovecot the quota warning message for the user but withouot success. I do the Patch found on the site but .. nothing happened. Anyone can help me please? Workino

1 0

[Dovecot] Replication plans
by Timo Sirainen 06 Jun '07

06 Jun '07

Several companies have been interested in getting replication support for Dovecot. It looks like I could begin implementing it in a few months after some other changes. So here's how I'm planning on doing it: What needs to be replicated: - Saving new mails (IMAP, deliver) - Copying existing mails - Expunges - Flag and keyword changes - Mailbox creations, deletions and renames - Subscription list - IMAP extension changes, such as ACLs I'll first talk about only master/slave configuration. Later then master/multi-slave and multi-master. Comments? Basic design ------------ Since the whole point of master-slave replication would be to get a reliable service, I'll want to make the replication as reliable as possible. It would be easier to implement much simpler master-slave replication, but in error conditions that would almost guarantee that some messages get lost. I want to avoid that. Both slave and master would be running a dovecot-replication process. They would talk to each others via TCP connection. Master would be feeding changes (transactions) to slave, and slave would keep replying how far it's committed the changes. So there would be an increasing sequence number sent with each transaction. Master keeps all the changes in memory until slave has replied that it has committed the changes. If the memory buffer gets too large (1MB?) because the slave is handling the input too slowly or because it's completely dead, the master starts writing the buffer to a file. Once the slave is again responding the changes are read from the file and finally the file gets deleted. If the file gets too large (10MB?) it's deleted and slave will require a resync. Master always keeps track of "user/mailbox -> last transaction sequence" in memory. When the slave comes back up and tells the master its last committed sequence, this allows the master to resync only those mailboxes that had changed. If this mapping doesn't exist (master was restarted), all the users' mailboxes need to be resynced. The resyncing will also have to deal with new mailboxes, ACL changes and such. I'm not completely sure yet how the resyncing will work. Probably the only important thing here is that it would be good if it would resolve any message UID conflicts (even though they shouldn't happen, see below) by comparing either the messages' MD5 contents, or with maildir it would be enough to just compare the maildir filename. It's possible that the slave doesn't die completely but only some transactions fail. This shouldn't normally happen, so the easiest way to handle it would be to try a full resync, and if that still fails stop the whole slave. Another way would be to just mark that one user or mailbox as dirty and try to resync it once in a while. imap, pop3 and deliver would connect to dovecot-replication either using UNIX sockets or POSIX message queue. Queue could be faster, but sockets would allow better security checks. Or I suppose it would be possible to send some kind of a small authentication token in each message with queues. The communication protocol would be binary and the existing dovecot.index.log format would be used as much as possible (making it simple for slave to append changes to mailboxes' dovecot.index.log). dovecot-replication process would need read/write access to all users' mailboxes. So either it would run as root or it would need to have at least group-permission rights to all mailboxes. A bit more complex solution would be to use multiple processes each running with their own UIDs, but I think I won't implement this yet. The replication would require support for shared mailboxes. That way the mailboxes could be accessed using the configured shared namespaces, for example "shared/<username>/INBOX". (LMTP server could be implemented in a similar way BTW) Configuration ------------- I'm not yet sure how this should work. The simplest possibility would be just to configure a single slave and everything would be replicated there. But it should be possible to split users into multiple slaves (still one slave/user). The most configurable way to do this would be to have userdb return the slave host. Hmm. Actually that could work pretty well if the userdb returned slave's name and each slave would have a separate socket/queue where dovecot-replication listens in. I'm not sure if they could all still be handled by a single dovecot-replication process or if there should be a separate one for each of them. The configuration would then be something like: # slave for usernames beginning with letters A..K replication users_a_k { host = slave1.example.com # other options? } # L..Z and others replication users_l_z_rest { host = slave2.example.com } Saving ------ This is the most important thing to get right, and also the most complex one. Besides replicating mails that are being saved via Dovecot, I think also externally saved mails should be replicated when they're first seen. This is somewhat related to doing an initial sync to a slave. The biggest problem with saving is how to robustly handle master crashes. If you're just pushing changes from master to slave and the master dies, it's entirely possible that some of the new messages that were already saved in master didn't get through to slave. So when you switch to using slave, those messages are gone. The biggest problem with this is that when saving new mails, they'll be assigned UIDs that were already used by the original master. If a locally caching client had already seen the UID from the original master, it'll think the message is the original one instead of the new one. So the newly added message pretty much gets lost, and if the user could expunge the message (thinking it's the original one) without ever even seeing it. This UID conflict could be avoided by making the master tell slave whenever it's saving new mails. The latency between master-slave communication could be avoided with a bit of a trick: - When beginning to save/copy a mail, tell slave to increase UID counter by one. Slave replies with the new highest-seen-UID counter. - UID allocation is done after all the mails are saved/copied within a transaction. At this stage wait until slave has replied that it has reserved UIDs at least as high as we're currently allocating (other processes could have caused it to get even higher). - If save/copy is aborted, tell the slave to decrease the UID counter by the number of aborted messages. - UID counter desync shouldn't happen, but if it does handle that also somehow. Then the UID allocation latency depends on how long it takes to write the message, and where the message comes from. With IMAP APPEND there's probably no wait required, unless the APPEND command and the whole message arrives in one IP packet. With deliver and copying it might have to wait a while. So the UID allocation should work with minimal possible latency. It could use a separate TCP connection to slave and maybe a separate high priority process in the slave that would handle only UID allocations. The above will make sure that conflicting UIDs aren't used, but it doesn't make sure that messages themselves aren't lost. If the master's hard disk failed completely and it can't be resynced anymore, the messages saved there are lost. This could be avoided by making the saving wait until the message is saved to slave: - save mail to disk, and at the same time also send it to slave - allocate UID(s) and tell to slave what they were, wait for "mail saved" reply from slave before committing the messages to mailbox permanently This would however add more latency than the simpler UID allocation counter. I don't know if it should be optional which way is used. Perhaps I'll implement the full save-sync first and if it seems too slow then I'll implement optionally the UID allocation counter. Save messages from imap/deliver would contain at least: 1. user name (unless using UNIX sockets, then it's already known) 2. mailbox name 3. UIDVALIDITY 4a) UID 4b) global UID, filename, offset and size of the message 4a) would be used with the UID allocation counter behavior. The message would be sent after the message was saved and UIDs were allocated. 4b) would be used with the wait-slave-to-save behavior. The message would be sent immediately whenever new data was received for the message, so possibly multiple times for each message. The last block would then contain size=0. After saving all the mails in a transaction, there would be a separate "allocate UIDs x..y for global UIDs a, b, c, d, etc." message which the slave then replies and master forwards the reply to imap/deliver. If the slave is down, or if a timeout (a few seconds?) occurs while waiting for the reply from slave, imap/deliver could just go on saving the message. If this is done it's possible that if the master also dies soon, there will be UID conflicts. But since this should be a rare error condition, it's unlikely that the master will die soon after a misbehaving slave. Copying ------- Copying has similar UID conflict problems than saving. The main difference is that there's no need to transfer message contents, which makes it faster to handle copies reliably. Copying could be handled with the UID allocation counter way described above. Since the message already exists in the server, there's no way that the message gets lost. So it would be possible to allow master and slave to go into desync, because the user would still find the message from the original mailbox. Move operation is done with "copy + mark \deleted + expunge" combination. This could be problematic if the master had managed to do all that before slave saw the COPY at all. Since the message isn't yet copied to the new mailbox in slave, clients wouldn't see it there, but locally caching clients might not see the message in the original mailbox either because they had already received EXPUNGE notification for it. So this should be avoided in some way. Perhaps it would be better to just treat copies the same way as saves: Require a notification from slave before replying to client with OK message. Alternatively the wait could be done at EXPUNGE command, as described below. Expunges -------- Losing expunge transactions wouldn't be too bad, because the worst that can happen is that the user needs to expunge the message again. Except that having an expunged message suddenly appear again in the middle of the mailbox violates IMAP protocol. I don't know if/how that will break clients. Solution here would again be that before EXPUNGE notifications are sent to client we'll wait for reply from slave that it had also processed the expunge. Flag and keyword changes ------------------------ Most likely doesn't matter if some of them are lost. If someone really cares about them, it would be possible to again do the slave-waiting for either all the changes or just when some specific flags/keywords change. Other changes ------------- Mailbox creations, deletions, renames and subscription changes could all be handled in a similiar way to flag/keyword changes. Handling changes implemented by IMAP extensions such as ACL is another thing. It's probably not worth the trouble to implement a special replication handler for each plugin. These changes most likely will happen somewhat rarely though, so they could be implemented by sending the whole IMAP command to slave and have it execute the command using a regular imap process. The commands could be marked with some "replication" flag so Dovecot knows that they need to be sent to slave. Although instead of executing a new imap process for each user (probably each command), it could be possible to use one imap process and just change the mailbox name in the command. So instead of eg. "setacl mailbox (whatever)" the IMAP process would execute "setacl shared/username/mailbox (whatever)". That would require telling Dovecot what parameter contains the mailbox name. Should be pretty easy to implement, unless some IMAP extensions want multiple mailbox names or the position isn't static. I can't think of any such extensions right now though. Master/multi-slave ------------------ Once the master/slave is working, support for multiple slaves could be added. The main thing to be done here is to have the slaves select which one of them is the new master, and then they'll have to sync the mailboxes with each others, because each of them might have seen some messages that others hadn't. This is assuming that save/etc. operations that are waiting for a reply from slave would wait for only one reply from the first slave that happens to respond. The number of slave replies to wait for could be also configurable, in case you're worried of a simultaneous master+slave crash. Multi-master ------------ After master/multi-slave is working, we're nearly ready for a full multi-master operation. The master farm will need global locking. Perhaps there is a usable existing implementation which could be used. In any case there needs to be a way to keeps track of what master owns what locks. The easiest way to implement multi-master is to assume that a single user isn't normally accessed from different servers at the same time. This wouldn't be a requirement, but it would make the performance better. For most users you could handle this by making sure that load balancer transfers all connections from the same IP to the same server. Or you could use a Dovecot proxy that transfers the connections. So, once the user is accessed only from one server, the operation becomes pretty much like master/multi-slave. You'll just need to be able to dynamically change the current master server for the user. This can be done by grabbing a global per-user lock before modifying the mailbox. The lock wouldn't have to be released immediately though. If no other server needs the lock, there's no point in locking/unlocking constantly. Instead the lock server could request the current lock owner to release the lock whenever another server wants it.

8 20

[Dovecot] multiple dovecot invocations
by Stewart Dean 05 Jun '07

05 Jun '07

I need to have multiple invocations of dovecot running, one for productions and then one or more for test/dev, doing things like trying different authentication methods, getting quota working, etc. I knew how to do this with inetd, but I have been convinced that direct invocation is preferable, so I need to know how to do multiple invocations of dovecot. I had thought that perhaps I could do this: dovecot -c /usr/local/etc/dovecot2.conf where dovecot2.conf has different different listen port parms. This I did, and while it does seem to work, I get this error in the syslog: > Jun 5 14:42:43 mercury mail:err|error dovecot: Socket already exists: /usr/local/var/run/dovecot/dict-server > Jun 5 14:42:43 mercury mail:crit dovecot: execv(/usr/local/libexec/dovecot/dict) failed: No such file or directo > ry > Jun 5 14:42:43 mercury mail:err|error dovecot: child 1437914 (dict) returned error 84 (exec() failed) How do I correctly run multiple dc instances? -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

2 1