- dovecot - dovecot.org

[Dovecot] Dovecot 1.1: Ignoring Trash/Junk for maildirsize?
by richs＠whidbey.net 03 May '08

03 May '08

Hi all, We're testing Dovecot 1.1rc3, but have run into a snag with IMAP Quota's and the way Trash and Junk folders can be handled. Our MTA (Exim) quota configuration uses this option: maildir_quota_directory_regex = ^(?:cur|new|\.(?!Junk$)(?!Trash$).*)$ This completely excludes "Junk" and "Trash" from the "maildirsize" file and quota calculations. Dovecot 1.0 could do this using "ignore=Trash" (although only on one folder). Is there any way to duplicate this behavior in Dovecot 1.1? We see that "quota_rule" can be used to set an unlimited quota (storage=0), but for some reason Dovecot still records their tallies in "maildirsize", which breaks the MTA. Thanks! -Rich

2 1

[Dovecot] Cross-Realm Krb5 Patch
by Zachary Kotlarek 03 May '08

03 May '08

I've updated my krb5 cross-realm auth patch for dovecot 1.1-rc4; there are not any (intentional) changes in functionality from the last patch, the diff just didn't apply cleanly anymore. As before the most recent version of the patch is available at: http://zinux.cynicbytrade.com/svn/servers/dovecot/cross-realm.diff.bz2 I see the cross-realm patch for Solaris made it into the mainline. What if anything can I do to encourage the mainline adoption of my non- Solaris patch? Zach

2 1

[Dovecot] Patch for zlib and maildir for 1.0.13
by Richard Platel 03 May '08

03 May '08

When using the zlib plugin with maildir and copying with hardlinks, if a compressed message is copied, the 'Z' suffix on the file isn't copied, so the new message isn't uncompressed when it's fetched. I wasn't smart enough to figure out a clean way to carry a file suffix through a copy, so I changed how the zlib-plugin detects if a message is compressed. This patch peeks at the first two bytes of the message looking for the zlib header. -- Richard Platel Tucows

3 3

[Dovecot] 1.1rc4 problem with maildir++ quota and POP3
by Laurent Papier 03 May '08

03 May '08

Hi, it looks like there is a bug (in dovecot or in my config) with quota on maildir mailboxes when accessed via POP3. The maildirsize is not created when not already present (even after a DELE + QUIT command). The quota limit (the first line of maildirsize) is not updated if values are not in sync with those in the database. If maildirsize is present, DELE commands are correctly reported in maildirsize. maildirsize is correctly updated when the same mailbox is accessed in IMAP or mail is delivered with dovecot LDA. Here is my config. Timo 'dovecot -n' does not show the lda config part... # 1.1.rc4: /etc/dovecot.conf protocols: imap pop3 ssl_disable: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: please to meet you login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_processes_count: 100 login_max_processes_count: 512 max_mail_processes: 3072 mail_uid: 1000 mail_gid: 1000 mail_location: maildir:~/ mmap_disable: yes maildir_stat_dirs: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): imap_quota quota mail_plugins(imap): imap_quota quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle imap_client_workarounds(imap): outlook-idle imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls auth default: user: nobody passdb: driver: sql args: /etc/dovecot-mysql.conf userdb: driver: sql args: /etc/dovecot-mysql.conf userdb: driver: prefetch socket: type: listen master: path: /var/run/dovecot/auth-master mode: 432 user: mailonly group: mailonly plugin: quota: maildir quota_rule2: Trash:storage=12288:messages=100 --- SQL queries ---- # prefetch auth password_query = SELECT cpasswd as password, home as userdb_home, concat('*:bytes=',quotas,':messages=',quotai) AS userdb_quota_rule FROM `mail` WHERE email='%u' # for delivery user_query = SELECT home, concat('*:bytes=',quotas,':messages=',quotai) AS quota_rule FROM `mail` WHERE email='%u' -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - <http://www.sdv.fr/>

2 1

[Dovecot] Global and personal sieve scripts
by Juan Asensio Sánchez 03 May '08

03 May '08

Hi According to Dovecot documentation (): # If there is no user-specific Sieve-script, global Sieve script is # executed if set. (v1.0.1 and older used "global_script_path") #sieve_global_path = Then, if one user has its own script, the global script won't be executed? I would like to execute first the global script (for moving SPAM to the Junk folder), and then the user scripts is they exists. Any idea? Thanks in advance.

2 1

[Dovecot] GETQUOTAROOT?
by Ralf Hildebrandt 03 May '08

03 May '08

All over a sudden the getquota plugin in SquirrelMail doesn't work anymore. I activated debugging an got: IMAP command sent: a001 GETQUOTAROOT "INBOX" IMAP response received: Array ( [0] => * QUOTAROOT "INBOX" "" [1] => * QUOTA "" () ) A manual test results in: . GETQUOTAROOT "INBOX" * QUOTAROOT "INBOX" "" * QUOTA "" () . OK Getquotaroot completed. But that mailbox DOES have a quota defined. This is dovecot-1.0.x from the Mercurial repo. I'm using: quota = maildir quota = maildir:ignore=Trash quota_warning = storage=10%% /usr/local/scripts/quota-warning 90 quota_warning2 = storage=5%% /usr/local/scripts/quota-warning 95 quota_warning3 = storage=1%% /usr/local/scripts/quota-warning 99 which works fine otherwise. -- Ralf Hildebrandt (Ralf.Hildebrandt(a)charite.de) snickebo(a)charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de In brightest day, in blackest night no evil shall escape my sight!

2 4

[Dovecot] [patch] let valgrind run on login process with GDB=1
by Diego Liziero 03 May '08

03 May '08

Not sure if this should be included in dovecot, just posting if someone feels like using valgrind. Diego. ----- diff -r ba634d2c0ab9 src/master/login-process.c --- a/src/master/login-process.c Wed Apr 30 20:18:37 2008 +0300 +++ b/src/master/login-process.c Thu May 01 00:59:10 2008 +0200 @@ -689,7 +689,8 @@ static pid_t create_login_process(struct fd_limit = 16 + listen_count + ssl_listen_count + 2 * (group->set->login_process_per_connection ? 1 : group->set->login_max_connections); - restrict_fd_limit(fd_limit); + if (getenv("GDB") == NULL) + restrict_fd_limit(fd_limit); /* make sure we don't leak syslog fd, but do it last so that any errors above will be logged */

2 1

[Dovecot] Need Help! (Bug Report)
by 이경준 03 May '08

03 May '08

Need Help!! Server : AIX 5.3 Compile Error Message configure --------------------------------- checking jfs/quota.h usability... no checking jfs/quota.h presence... yes configure: WARNING: jfs/quota.h: present but cannot be compiled configure: WARNING: jfs/quota.h: check for missing prerequisite headers? configure: WARNING: jfs/quota.h: see the Autoconf documentation configure: WARNING: jfs/quota.h: section "Present But Cannot Be Compiled" configure: WARNING: jfs/quota.h: proceeding with the preprocessor's result configure: WARNING: jfs/quota.h: in the future, the compiler will take precedence configure: WARNING: ## ---------------------------------- ## configure: WARNING: ## Report this to dovecot(a)dovecot.org ## configure: WARNING: ## ---------------------------------- ## checking for jfs/quota.h... yes checking mntent.h usability... yes make -------------------------- et/libcharset.a ../../../src/lib/liblib.a -liconv + exec /usr/vacpp/bin/xlc -g -bexpall -brtl -o convert-tool convert-tool.o ./../../src/lib-storage/register/libstorage-register.a ../../../src/lib- storage/index/maildir/libstorage_maildir.a ../../../src/lib- storage/index/mbox/libstorage_mbox.a ../../../src/lib- storage/index/libstorage_index.a ../../../src/lib-index/libindex.a ./../../src/lib-storage/libstorage.a ../../../src/lib-storage/subscription- file/libstorage_subscription_file.a ../../../src/lib-imap/libimap.a ../../.. /src/lib-mail/libmail.a ../../../src/lib-charset/libcharset.a ./../../src/lib/liblib.a -liconv -bE:.libs/libstorage-register.exp - bE:.libs/libstorage_maildir.exp -bE:.libs/libstorage_mbox.exp - bE:.libs/libstorage_index.exp -bE:.libs/libindex.exp - bE:.libs/libstorage.exp -bE:.libs/libstorage_subscription_file.exp - bE:.libs/libimap.exp -bE:.libs/libmail.exp -bE:.libs/libcharset.exp - bE:.libs/liblib.exp Target "all" is up to date. Making all in quota source='quota.c' object='quota.lo' libtool=yes DEPDIR=.deps depmode=aix /bin/sh ../../../depcomp /bin/sh ../../../libtool --tag=CC -- mode=compile /apps1/bea8/domains/eipDomain/handy/down/dovecot/imap/dovecot- 1.0.13/dovecot-cc -DHAVE_CONFIG_H -I. -I. -I../../.. -I../../../src/lib - I../../../src/lib-dict -I../../../src/lib-index -I../../../src/lib-mail - I../../../src/lib-storage -I../../../src/lib-storage/index - I../../../src/lib-storage/index/maildir -g -c -o quota.lo quota.c mkdir .libs /apps1/bea8/domains/eipDomain/handy/down/dovecot/imap/dovecot- 1.0.13/dovecot-cc -DHAVE_CONFIG_H -I. -I. -I../../.. -I../../../src/lib -I.. /../../src/lib-dict -I../../../src/lib-index -I../../../src/lib-mail -I../.. /../src/lib-storage -I../../../src/lib-storage/index -I../../../src/lib- storage/index/maildir -g -c -M quota.c -DPIC -o .libs/quota.o + exec /usr/vacpp/bin/xlc -DHAVE_CONFIG_H -I. -I. -I../../.. - I../../../src/lib -I../../../src/lib-dict -I../../../src/lib-index - I../../../src/lib-mail -I../../../src/lib-storage -I../../../src/lib- storage/index -I../../../src/lib-storage/index/maildir -g -c -M quota.c - DPIC -o .libs/quota.o "/usr/include/sys/ras.h", line 273.49: 1506-207 (W) Integer constant 0x5200000000000110ul out of range. "/usr/include/sys/ras.h", line 273.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. "/usr/include/sys/ras.h", line 276.49: 1506-207 (W) Integer constant 0x5200000000000210ul out of range. "/usr/include/sys/ras.h", line 276.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. "/usr/include/sys/ras.h", line 279.49: 1506-207 (W) Integer constant 0x5200000000000310ul out of range. "/usr/include/sys/ras.h", line 279.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. "/usr/include/sys/ras.h", line 283.49: 1506-207 (W) Integer constant 0x5200000000000120ul out of range. "/usr/include/sys/ras.h", line 283.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. "/usr/include/sys/ras.h", line 286.49: 1506-207 (W) Integer constant 0x5200000000000220ul out of range. "/usr/include/sys/ras.h", line 286.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. "/usr/include/sys/ras.h", line 289.49: 1506-207 (W) Integer constant 0x5200000000000320ul out of range. "/usr/include/sys/ras.h", line 289.1: 1506-195 (S) Integral constant expression with a value greater than zero is required. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 2.

2 1

[Dovecot] deliver: handling of quota_full_tempfail and -e
by Ulrich Zehl 03 May '08

03 May '08

Hi, while testing out dovecot 1.1, I found that when you set quota_full_tempfail = yes in dovecot.conf and use the deliver -e command line flag, no deferral reason is printed, thus breaking my fancy quota_exceeded_message. This appears to be because of the if() block in lines 1003-1009 in src/deliver/deliver.c, and should be easily fixable (if it is indeed a bug, and not intended behavior). However, I shied away from creating a patch because either there is a mismatch between the documentation and deliver's actual behavior, or I misunderstood the code (likely, since I'm not really a C programmer). The lines in question (in src/deliver/deliver.c) read: | 1003 if (error != MAIL_ERROR_NOSPACE || | 1004 getenv("QUOTA_FULL_TEMPFAIL") != NULL) { Does this mean that the corresponding block gets executed whenever quota_full_tempfail is set? If so, setting it transforms all errors into TEMPFAILs, and not only those relating to quota/space issues. I think dovecot-example.conf should at least mention this it's inteded; otherwise, the code should be changed (I'm not completely sure what Timo's intensions are, so I won't comment on possible solutions). Regards, Ulrich PS: All line numbers are taken from rev 080dd4d2fd94, which was the most recent at the time of writing.

4 3

[Dovecot] Replication protocol design
by Timo Sirainen 02 May '08

02 May '08

I'll probably be implementing multi-master replication this summer. My previous thoughts about it are here: http://dovecot.org/list/dovecot/2007-December/027284.html Below is a description of how the replication protocol will probably work. It should work just as well for master-slave and multi-master setups. Comments welcome. I'll write later a separate mail about how it'll be implemented to Dovecot. Goals ----- 1. If a single (or configurable number of) server dies, no mails must be lost that have been reported to IMAP/SMTP clients as being successfully stored. 2. Must be able to automatically recover from a server desynchronization, such as: - server has been offline for a long time - some mail files have been manually added/deleted - corrupted data/mail files if they're noticed 3. In multi-master setup if the link between servers die, the servers must be able to proceed autonomously (kind of conflicts with goal 1 though). When the link comes back, the changes are replicated as soon as possible. 4. Normal IMAP commands must not be able to cause desynchronization between servers. For example making conflicting flag changes simultaneously in two servers must not result in the servers having different flags. 5. Must perform well with at least 3 servers in a multi-master setup, preferrably still with tens of servers. 6. Latency shouldn't be increased noticeably when using servers distributed into 3 or more data centers. Must be usable over high-latency links (in optimistic async mode). 7. In normal operation send minimal incremental changes. Protocol -------- There are two major parts in the protocol: Handling the normal incremental changes and fixing desynchronization. Originally I thought that maybe the changes could be sent using the same format as Dovecot's transaction logs, but now I'm beginning to think that it's probably not that good idea. The code reuse potential is quite minimal and the format would still have to be extended in several ways so that it won't be directly compatible anyway. So I'm thinking the protocol could be something text-based. The main benefit is that text-based protocols are easier to debug. Stream compression should drop most of the extra overhead if bandwidth is a problem. The exact on-wire protocol anyway doesn't matter in the design that's discussed below. The commands have tags similar to IMAP commands, because some commands may have to be forwarded to other servers and it may take a while to get a reply. During the wait the server may process other commands. Mailbox master -------------- Each mailbox has a single master server selected. In multi-master setups the master server may be moved by having the destination server simply request it from the current master. The current master must then give it up. If link is lost to the current master, one of the remaining servers will become the new master within the remaining servers. Each server must be connected to the current master server. Since each mailbox can have a different master server, this typically means all servers are connected to each others. However it's possible to create setups where server connects to only one other server, which in turn connect to more servers. This is useful if there are bandwidth bottlenecks between some servers. This kind of a situation can also happen if in a network A-B-C the link A-C dies, but A-B and B-C continues to work. Because of this all commands must be able to function in a way that the server proxies them to the current master, instead of failing the command and trying to make it the caller's problem to resend the command to the actual master. When the cluster starts up, a single server is selected as the root master for all mailboxes. If a server doesn't know who the current mailbox master is, it asks from the root. All servers cache the currently known mailbox masters to avoid constant requests to the root. If the root server dies, another server is selected as the root. Because the new root doesn't know what masters have been requested (and asking all of them from all servers would just waste bandwidth), all the servers are expected to flush their master caches and drop their own master status. The new root doesn't respond to any requests before all servers have notified that they've dropped being a master. The master status doesn't have to be at mailbox level granularity. It could just as well be configured to move at user, domain or even global level. Perhaps this could be done dynamically, so that higher granularity is used when the master is beginning to change too often between servers. Mailbox ID ---------- Mailbox IDs are session-specific numbers dynamically assigned for user+mailbox+UIDVALIDITY combinations. All connections have different mailbox IDs. Also send and receive directions have different IDs. This allows the sender to easily replace existing IDs to point to new mailboxes without causing any confusion. MBOX: - Mailbox ID - User name - Mailbox name - Mailbox UIDVALIDITY - Mailbox UIDNEXT - Mailbox message count If the receiver finds out it has a different UIDVALIDITY, the mailbox requires a full resync. Message count and UIDNEXT may also be used to determine if replication servers are out of sync. Requesting master status ------------------------ MASTER-MOVE: - Mailbox ID - [Destination SID] (if forwarding) The command is sent to the last known master for the mailbox. The server will keep forwarding the command until it reaches the current master. During the forwarding other servers may want to request something from the master. These requests must be delayed by the forwarding servers until the move is finished. Saving messages --------------- SAVE: - Mailbox IDs - Received date - [IMAP UID] (only if we're the master) - Global UID (stays the same when copying the message) - Message text Reply: - [IMAP UID] (only if not specified in parameters) - [Current mailbox master SID] (if it was moved) If current server is not the master, the SAVE is sent to the master which gives the message its UID. The master server then replicates the message to other servers with the UID parameter set. The mailbox master may have already changed by the time server receives a save request. If server receives a SAVE without IMAP UID parameter, it's responsible for finding out the new mailbox master and sending a new SAVE request to it. Once the new master replies with the IMAP UID, the server can reply to the original SAVE request, also providing the new master SID so the future requests can be sent there directly. To be sure the message doesn't get lost, the server should not reply OK to the IMAP/SMTP client until it has received a reply from SAVE. Copying messages ---------------- COPY: - Source mailbox ID - Destination mailbox ID - Source IMAP UID - Global UID - [Destination IMAP UID] (only if we're the master) - Destination received date Reply: - [Destination IMAP UID] (only if not specified in parameters) - [Current mailbox master SID] (if it was moved) Source mailbox ID + source IMAP UID identifies the message to be copied. It's expected to contain the given global UID (which is just an extra sanity check). Otherwise it works the same way as SAVE. Since the message already exists, it's probably not necessary to wait for a reply before replying OK to originating IMAP client. Expunging messages ------------------ EXPUNGE: - Mailbox ID - UID range (No reply) Expunges also have to be sent via master server (the same way as SAVE) to avoid COPY command failing in some servers because it was just expunged. Changing message flags ---------------------- STORE: - Mailbox ID - UID range - Added flags/keywords - Removed flags/keywords - [Current modseq] (master sends) - [Highest modseq of the messages before this change] (non-master sends) - [flag: this is a CONDSTORE STORE UNCHANGEDSINCE] (non-master may send) [Reply: - UIDs where STORE was rejected to (if CONDSTORE flag was used) ] Stores also have to be sent via master server to avoid flag desynchronization. Master first checks if it has higher modseqs in the messages. Then it applies all the changes and forwards the changes to other servers. For messages that had higher initial modseqs their flags are sent to the server sending the STORE to fix a potential desync. If CONDSTORE flag is set, the change is rejected for messages that had a higher modseq. Non-masters shouldn't reply to a STORE UNCHANGEDSINCE command before the change has been replicated to master and the rejections have been processed. Mailbox synchronization ----------------------- If a mailbox is determined to have changed externally (e.g. network connection down for too long, causing replication logs to get full) the mailbox state needs to be synchronized between servers. SYNC: - UIDVALIDITY - UIDNEXT - Message count - For each message: - UID - Global UID - Modseq - Flags and keywords - Received date Reply: - (Sync finished) Receiving server compares the parameters with its own mailbox state. If it finds previously unseen global UIDs, their message texts are requested: FETCH: - Mailbox ID - UID Reply: - Message text SAVE, EXPUNGE and STORE commands are used to synchronize the mailbox. A special case is when two servers have been saving messages independently from each others. In this case it's possible that the servers have used the same UIDs for different messages (different global UIDs). These need to be resolved by giving both conflicting UIDs new unused UIDs, otherwise IMAP clients may show them as wrong messages from their caches. FIXME: If the other server had expunged a conflicting UID it still should be given a new UID. How do we find out this has happened?

4 13