February 2012 - dovecot

[Dovecot] migrating/converting from system users -> virtual users
by Steve Platt 10 Mar '12

10 Mar '12

Hello all, We currently have a traditional mail server where all users have system accounts (ie entries in the NIS passwd map) and mbox-format mail folders in their (system) home directories. I'm trying to setup a dovecot server in which we want all users to have "virtual" accounts (in dovecot) and no entry in the passwd file at all (ie no access to the mail server). This is for security reasons, we want to imagine that users connecting to the mail server in future will not be able to access any resources on the server other than access their email messages. To this end we've set up a "vmail" user that owns all the messages in the Maildir directories. Most of this is working but I'm stuck on how to convert users' mail folders from the existing setup to the new one. I'm using the convert plugin but of course the problem is that the plugin executes as the "vmail" user and cannot access the existing mail folders that belong to the users: and I'd be worried if it could, of course! So I've converted a few users already by making copies of their old-world mailboxes, changing the ownership of the copies and pointing the convert plugin to these copies, instead of the originals. This works (modulo the dreaded "timeouted" error messages) but is very ugly. So I tried looking at the convert-tool but can't find a man page for it so am not sure what I'm doing with it! I have the idea that I should be able to run some command (as a privileged user) on the mail server and have it do the conversion for me, changing the ownership/permissions on the way. Can convert-tool do this? I'd prefer to go with the automatic (plugin) conversion if I can bodge the ownership issues somehow. Failing that, some tool or script may be the next best answer. This is Dovecot 1.2 by the way on Solaris 10. Any suggestions would be welcome! Steve Platt

2 3

[Dovecot] [PATCH] Pop3 order in courier migration script
by Christoph Bußenius 08 Mar '12

08 Mar '12

Hi, I found a problem in the courier conversion script (courier-dovecot-migrate.pl). In some cases, it does not correctly preserve the order of POP3 UIDLs. Consider the following case, when courierpop3dsizelist contains: /2 9000 1327481629 AAA1:2,S 10000 21:1327481629 AAA2:2,S 10000 22:1327481629 AAA3:2,S 10000 23:1327481629 AAA4:2,S 10000 24:1327481629 AAA5:2,S 10000 25:1327481629 BBB11:2,S 10000 11:1327481629 BBB12:2,S 10000 12:1327481629 BBB13:2,S 10000 13:1327481629 BBB14:2,S 10000 14:1327481629 BBB15:2,S 10000 15:1327481629 and courierimapuiddb contains: 1 1327480919 9000 11 BBB11 12 BBB12 13 BBB13 14 BBB14 15 BBB15 16 CCC16 17 CCC17 18 CCC18 19 CCC19 20 CCC20 21 AAA1 22 AAA2 23 AAA3 24 AAA4 25 AAA5 then call courier-dovecot-migrate.pl --to-dovecot --convert and it will create dovecot-uidlist: 3 V1327480919 N9010 21 PUID21-1327481629 W10000 :AAA1:2,S 22 PUID22-1327481629 W10000 :AAA2:2,S 23 PUID23-1327481629 W10000 :AAA3:2,S 24 PUID24-1327481629 W10000 :AAA4:2,S 25 PUID25-1327481629 W10000 :AAA5:2,S 9000 :CCC17 9001 PUID12-1327481629 W10000 :BBB12:2,S 9002 :CCC16 9003 PUID15-1327481629 W10000 :BBB15:2,S 9004 :CCC20 9005 PUID14-1327481629 W10000 :BBB14:2,S 9006 PUID13-1327481629 W10000 :BBB13:2,S 9007 :CCC18 9008 PUID11-1327481629 W10000 :BBB11:2,S 9009 :CCC19 The problem here is that the BBB files are not in the same order as in courierpop3dsizelist, and will therefore have a different order in the UIDL response in POP3. Also, the CCC files are interleaved while they should appear at the end. The reason is how the script assigns new UIDs: It does not do this in the order of courierpop3dsizelist. In fact the order is random because it depends on how perl traverses hashes. The attached patch fixes this by sorting the messages: First the POP3 messages get their new UIDs in order of their appearance in courierpop3dsizelist (so that the UIDL order will really be the same as in Courier), after that, the IMAP-only messages will be checked for UID clashes. The patch also changes how the $prev_uid variable is handled because that was also needed to always get the order right. The algorithm could probably be optimized to reduce the number of UID re-assignments to a minimum, however this would complicate the code and would only be noticed in a few cases. The patched courier-dovecot-migrate.pl will create dovecot-uidlist like this: 3 V1327480919 N9010 21 PUID21-1327481629 W10000 :AAA1:2,S 22 PUID22-1327481629 W10000 :AAA2:2,S 23 PUID23-1327481629 W10000 :AAA3:2,S 24 PUID24-1327481629 W10000 :AAA4:2,S 25 PUID25-1327481629 W10000 :AAA5:2,S 9000 PUID11-1327481629 W10000 :BBB11:2,S 9001 PUID12-1327481629 W10000 :BBB12:2,S 9002 PUID13-1327481629 W10000 :BBB13:2,S 9003 PUID14-1327481629 W10000 :BBB14:2,S 9004 PUID15-1327481629 W10000 :BBB15:2,S 9005 :CCC16 9006 :CCC17 9007 :CCC18 9008 :CCC19 9009 :CCC20 Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching

2 3

[Dovecot] duplicates with multiple To/CC and sieve redirect copy
by Adam Szpakowski 08 Mar '12

08 Mar '12

Hi, In one installation we are using sieve "redirect :copy" to create copy of the incoming email (local boss gets copy of its employees mails). There is a problem when the incoming email has multiple To/CC entries with local users. Multiple copy/redirects are created (one for each employee) and there are multiple copies in "boss" account. Each of this emails has the same Message-ID. Is there a way to detect those duplicates and discard them when saving copies into boss account? There was a discussion on the list a while ago (Feb 10/11, 2011) mentioning this problem, but with no clear solution. We are using Dovecot 2.0.15. Mail is delivered via LMTP (MDA/postfix is on separate machine then MAA/dovecot). -- Adam Szpakowski

5 10

[Dovecot] Failing: doveadm sync <--remote host--> dsync mirror
by Michael Grimm 08 Mar '12

08 Mar '12

Hi -- I'm used to dsync my two mailservers running 2.0.16. Today, I did upgrade one server to 2.1, and doveadm sync is throwing to following error: mail> doveadm sync -f -u test ssh vmail(a)remote-host.tld dsync -v -f -u test doveadm(test): Fatal: All your namespaces have a location setting. It should be empty (default mail_location) in the namespace to be converted. Same error with old running command: vmail> dsync -v -f -u test mirror ssh vmail(a)remote-host.tld dsync -v -f -u test doveadm(test): Fatal: All your namespaces have a location setting. It should be empty (default mail_location) in the namespace to be converted. Here's an excerpt of my doveconf -n: mail> doveconf -n mail_home = /var/mail/homedirs/%n mail_location = mdbox:/var/mail/maildirs/%n passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } Reverted back for the time being (production servers) until I do understand the issue. Searching Google with "Fatal: All your namespaces have a location setting." only shows closed bugs back in 2010. Do I have to upgrade both servers to get syncing running? Any hint, help is highly appreciated. Thanks, Michael

2 18

[Dovecot] testing fts-solr?
by Jean-Daniel Beaubien 05 Mar '12

05 Mar '12

hi everyone, However, how can I verify that solr is working properly? Background: - I'm trying to setup 2.1 with fts_solr and eventually test maildir/mdbox. - I used the following webpage as reference to setup solr: http://www.roessner-network-solutions.com/2012/02/19/full-text-search-with-… I'm looking at /var/log/tomcat6/catalina.out file, but see nothing telling me that it's being used (same for /var/log/mail.log, etc.) Any tips? Thanks

10 34

[Dovecot] GSSAPI auth failing for kmail
by Mark Davies 05 Mar '12

05 Mar '12

Hi, have dovecot 2.0.13 running fine on a NetBSD box, with lots of clients happily authenticating to it with GSSAPI. I've just updated an ArchLinux box to latest new version which has meant its switched from using heimdal to using mit krb5 and its kmail is now also slightly newer. Thunderbird successfully authenticates to dovecot from this machine but kmail now fails. Kmail reports: Unable to authenticate via GSSAPI. The server ... replied: SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Token header is malformed or corrupt) The dovecot log reports: Feb 27 13:42:18 bats dovecot: auth: GSSAPI(mark(a)ECS.VUW.AC.NZ,130.195.5.88): Invalid base64 data in continued response Any suggestions on how to track down what specifically has changed and how to fix it? cheers mark

2 11

[Dovecot] doveadm fetch prints duplicate results in 2.1
by Christoph Bußenius 04 Mar '12

04 Mar '12

Hi, when the private namespace has "prefix = INBOX." and you use doveadm fetch to search for "mailbox INBOX", then it prints every message twice: # doveadm fetch -u testuser 'mailbox uid guid' mailbox INBOX mailbox: INBOX uid: 1 guid: 1757c403a4ec4d4fbf040000463889b5 mailbox: INBOX uid: 1 guid: 1757c403a4ec4d4fbf040000463889b5 Without "prefix = INBOX." in the configuration, the messages are only printed once. In 2.0, they were never printed twice. doveconf -n # 2.1.1: /usr/local/dovecot/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-35-server x86_64 Ubuntu 10.04.4 LTS disable_plaintext_auth = no mail_gid = vmail mail_location = mdbox:~/mdbox mail_plugins = acl mail_uid = vmail namespace { inbox = yes location = prefix = INBOX. separator = . } passdb { args = scheme=CRYPT username_format=%u /usr/local/dovecot/etc/dovecot/users driver = passwd-file } protocols = imap service auth { unix_listener auth-userdb { group = vmail mode = 0660 } } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = username_format=%u /usr/local/dovecot/etc/dovecot/users driver = passwd-file } protocol imap { mail_plugins = acl imap_acl } Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching

2 2

[Dovecot] Dovecot clustering with dsync-based replication
by Timo Sirainen 03 Mar '12

03 Mar '12

This document describes a design for a dsync-replicated Dovecot cluster. This design can be used to build at least two different types of dsync clusters, which are both described here. Ville has also drawn overview pictures of these two setups, see http://www.dovecot.org/img/dsync-director-replication.png and http://www.dovecot.org/img/dsync-director-replication-ssh.png First of all, why dsync replication instead of block level filesystem replication? - dsync won't replicate filesystem corruption. - A cold restart of replication won't go through all of the data in the disks, but instead quickly finds out what has changed. - Split brain won't result in downtime or losing any data. If both sides did changes, the changes are merged without data loss. - If using more than 2 storages, the users' replicas can be divided among the other storages. So if one storage goes down, the extra load is shared by all the other storages, not just one. Replication mail plugin ----------------------- This is a simple plugin based on notify plugin. It listens for all changes that happen to mailboxes (new mails, flag changes, etc.) Once it sees a change, it sends an asynchronous (username, priority) notification to replication-notify-fifo. The priority can be either high (new mails) or low (everything else). Optionally the replication plugin can also support synchronous replication of new mail deliveries. In this way it connects to replication-notify UNIX socket, tells it to replicate the user with sync (=highest) priority and waits until it is done or replication_sync_timeout occurs. The IMAP/LMTP client won't see an "OK" reply until the mail is replicated (or the replication has failed). The synchronous replication probably adds a noticeable delay, so it might not be acceptable for IMAP, but might be for LMTP. So, what is listening in those replication-notify* sockets? It depends on if Dovecot is running on director-based setup or not. Aggregator ---------- When running in Dovecot director-based setup, all of the Dovecot backends (where replication plugin runs) also run "aggregator" process. Its job is very simple: It proxies the notifications from mail plugin and sends them via a single TCP connection to the replicator process running in Dovecot proxies. This is simply an optimization to avoid tons of short lived TCP connections directly from replication plugin to director server. When not running in Dovecot director setup (i.e. there is only a single Dovecot instance that handles all of the users), there is no point in having an aggregator proxy, because the replicator process is running on the same server. In this kind of setup the replicator process directly listens on the replication-notify* sockets. Replicator ---------- The initial design for replicator isn't very complex either: It keeps a priority queue of all users, and replicates those users at the top of the queue. Notifications about changes to user's mailboxes (may) move the user up in the priority queue. If the user at the top of the queue already has been replicated "recently enough", the replicator stops its work until new changes arrive or the "recently enough" is no longer that. dsync can do two types of syncs: quick syncs and full syncs. A quick sync trusts indexes and does the replication with the least amount of work and network traffic. A quick sync is normally enough to replicate all changes, but just in case something has gone wrong there's also the full sync option, which guarantees that the mailboxes end up being fully synced. A full sync is slower though, and uses more network traffic. The priority queue is sorted by: 1. Priority (updated by a notification from replication plugin) 2. If priority!=none: Last fast sync (those users are replicated first whose last replication time is oldest) 2. If priority=none: Last full sync (these users should already be fully synced, but do a full sync for them once in a while anyway) All users get added to the replication queue at replicator startup with "none" priority. The list of users is looked up via userdb iteration. If the previous replication state is found from a disk dump, it's used to update the priorities, last_*_sync timestamps and other replication state. Replicator process creates such dumps periodically [todo: every few mins? maybe a setting?]. Replicator starts replicating users at the top of the queue, setting their priorities to "none" before starting. This means that if another change notification arrives during replication, the priority is bumped up and no changes get lost. replication_max_conns setting specifies how many users are replicated simultaneously. If the user's last_full_sync is older than replication_full_sync_interval setting, a full sync is done instead of a fast sync. If the user at the top of the queue has "none" priority and the last_full_sync is newer than replication_full_sync_interval, the replication stops. [todo: it would be nice to prefer doing all the full syncs at night when there's hopefully less disk I/O] (A global replication_max_conns setting isn't optimal in proxy-based setup, where different backend servers are doing the replication. There it should maybe be a per-backend setting. Then again, it doesn't account for the replica servers that also need to do replication work. Also to properly handle this each backend should have its own replication queue, but this requires doing a userdb lookup for each user to find out their replication server, and this would need to be done periodically in case the backend changes, which can easily happen often with director-based setup. So all in all, none of this is being done in the initial implementation. Ideally the users are distributed in a way that a global replication queue would work well enough.) In director-based setup each director runs a replicator server, but only one of them (master) actually asks the backends to do the replication. The rest of them just keep track of what's happening, and if the master dies or hangs, one of the others becomes the new master. The server with lowest IP address is always the master. The replicators are connected to a ring like the directors, using the same director_servers setting. The communication between them is simply about notifications of what's happening to users' priorities. Preferably the aggregators would always connect to the master server, but this isn't required. In general there's not much that can go wrong, since it's not a problem if two replicators request a backend to start replication for the same user or if the replication queue states aren't identical. If the replication is running too slowly [todo: means what exactly?], log a warning and send an email to admin. So, how does the actual replication happen? Replicator connects to doveadm server and sends a "sync -u user@domain" command. In director-based setup the doveadm server redirects this command to the proper backend. doveadm sync ------------ This is an independent feature from all of the above. Even with none of it implemented, you could run this to replicate a user. Most of this is already implemented. The only problem is that currently you need to explicitly tell it where to sync. So, when the destination isn't specified, it could do a userdb lookup and use the returned "mail_replica" field as the destination. Multiple (sequentially replicated) destinations could be supported by returning "mail_replica2", "mail_replica3" etc. field. In NFS-based (or shared filesystem-based in general) setup the mail_replica setting is identical to mail_location setting. So your primary mail_location would be in /storage1/user/Maildir, while the secondary mail_replica would be in /storage2/user/Maildir. Simple. In non-NFS-based setup two Dovecot servers talk dsync protocol to each others. Currently dsync already supports SSH-based connections. It would also be easy to implement direct TCP-based connections between two doveadm servers. In future these connections could be SSL-encrypted. Initially I'm only supporting SSH-based connections, as they're already implemented. So what does the mail_replica setting look like in this kind of a setup? I'm not entirely sure. I'm thinking that it could be either "ssh:host" or "ssh:user@host", where user is the SSH login user (this is opposite of the current doveadm sync command line usage). In future then it could support also tcp:host[:port]. Both of these ssh: and tcp: prefixes would also be supported by doveadm sync command line usage (and perhaps the prefixless user@domain be deprecated). dsync can run without any long lived locking and it typically works fine. In case mailbox was modified during dsync, the replicas may not end up being identical, but nothing breaks. dsync currently usually notices this and logs a warning. When these conflicting changes was caused by imap/pop3/lda/etc. this isn't a problem, they've already notified replicator already to perform another sync that will fix it. Running two dsyncs at the same time is more problematic though, mainly related to new emails. Both dsyncs notice that mail X needs to be replicated, so both save it and it results in having a duplicate. To avoid this, there should be a dsync-lock. If this lock exists, dsync should wait until the previous dsync is done and then do it again, just in case there were more changes since the previous sync started. This should conclude everything needed for replication itself. High-availability NFS setup --------------------------- Once you have replication, it's of course nice if the system automatically recovers from a broken storage. In NFS-based setups the idea is to do soft mounts, so if the NFS server goes away things start failing with EIO errors, which Dovecot notices and switches to using the secondary storage(s). In v2.1.0 Dovecot already keeps track of mounted filesystems. Initially they're all marked as "online". When multiple I/O errors occur in a filesystem [todo: how many exactly? where are these errors checked, all around in the code or checking the log?] the mountpoint is marked as "offline" and the connections accessing that storage are killed [todo: again how exactly?]. Another job for replication plugin is to hook into namespace creation. If mail_location points to a mountpoint marked as "offline", it's replaced with mail_replica. This way the user can access mails from the secondary storage without downtime. If the replica isn't fully up to date, this means that some of the mails (or other changes) may temporarily be lost. These will come back again after the original storage has come back up and replication has finished its job. So as long as mails aren't lost in the original storage, there won't be any permanent mail loss. When an offline storage comes back online, its mountpoint's status is initially changed to "failover" (as opposed to "online"). During this state the replication plugin works a bit differently when the user's primary mail_location is in this storage: It first checks if the user is fully replicated, and if so uses the primary storage, otherwise it uses the replica storage. Long running IMAP protocesses check the replication state periodically and kill themselves once the user is replicated, to move back to primary storage. Once replicator notices that all users have been replicated, it tells the backends' to change the "failover" state to "online" (via doveadm server). High-availability non-NFS setup ------------------------------- One possibility is to use Dovecot proxies, which know which servers are down. Instead of directing users to those servers, it would direct them to replica servers. The server states could be handled similar to NFS setup's online vs. failover vs. offline states. Another possibility would be to do the same as above, except without separate proxy servers. Just make "mail.example.com" DNS point to two IP addresses, and if one Dovecot notices that it's not the user's primary server, it proxies to the secondary server, unless it's down. If one IP is down, clients hopefully connect to the other.

3 5

[Dovecot] fts size
by Jan-Frode Myklebust 02 Mar '12

02 Mar '12

Does anybody have any numbers for how large storage one will need for the fts indexing server? I see the wiki says 30% of mailbox size for Squat (partial=4 full=4). Is it similar for lucene/solr? Do I understand correctly if I think http://wiki2.dovecot.org/Plugins/FTS/Lucene will create an index for each user in his home directory? Will this be accounted for in the users' quota? -jf

3 5

[Dovecot] 2.1.1: subscriptions: Removing invalid entry
by Ralf Hildebrandt 01 Mar '12

01 Mar '12

Admittedly, it's not a valid folder name and dovecot is right in removing it: Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw�rfe ...but when looking at the subscription file AFTER the removal, that file contained an empty line. I don't think that's correct. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

3 2