- dovecot - dovecot.org

[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem
by Timo Sirainen 12 Mar '10

12 Mar '10

This is something I figured out a few months ago, mainly because this one guy at work (hi, Stu) kept telling me my multi-master replication plan sucked and we should use some existing scalable database. (I guess it didn't go exactly like that, but that's the result anyway.) So, my current plan is based on a couple of observations: * Index files are really more like memory dumps. They're already in an optimal format for keeping them in memory, so they can be just mmap()ed and used. Doing some kind of translation to another format would just make it more complex and slower. * I can change all indexing and dbox code to not require any locks or overwriting files. I just need very few filesystem operations, primarily the ability to atomically append to a file. * Index and mail data is very different. Index data is accessed constantly and it must be very low latency or performance will be horrible. It practically should be in memory in local machine and there shouldn't normally be any network lookups when accessing it. * Mail data on the other hand is just written once and usually read maybe once or a couple of times. Caching mail data in memory probably doesn't help all that much. Latency isn't such a horrible issue as long as multiple mails can be fetched at once / in parallel, so there's only a single latency wait. So the high level plan is: 1. Change the index/cache/log file formats in a way that allows lockless writes. 2. Abstract out filesystem accessing in index and dbox code and implement a regular POSIX filesystem support. 3. Make lib-storage able to access mails in parallel and send multiple "get mail" requests in advance. (3.5. Implement async I/O filesystem backend.) 4. Implement a multi-master filesystem backend for index files. The idea would be that all servers accessing the same mailbox must be talking to each others via network and every time something is changed, push the change to other servers. This is actually very similar to my previous multi-master plan. One of the servers accessing the mailbox would still act as a master and handle conflict resolution and writing indexes to disk more or less often. 5. Implement filesystem backend for dbox and permanent index storage using some scalable distributed database, such as maybe Cassandra. This is the part I've thought the least about, but it's also the part I hope to (mostly) outsource to someone else. I'm not going to write a distributed database from scratch.. This actually should solve several issues: * Scalability, of course. It'll be as scalable as the distributed database being used to store mails. * NFS reliability! Even if you don't care about any of these alternative databases, this still solves NFS caching problems. You'd keep using the regular POSIX FS API (or async FS api) but with the in-memory index "cache", so only a single server is writing to mailbox indexes at the same time. * Shared mailboxes. The filesystem API is abstracted, so it should be possible to easily add another layer to handle accessing other users' mails from both local and remote servers. This should finally make it possible to easily support shared mailboxes with system users.

17 82

[Dovecot] troubles with expire plugin
by Denis Fateyev 11 Mar '10

11 Mar '10

Hello there, Now I'm using `expire` plugin and getting troubles with cronjob (from user `exim`): /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh Fatal: setgid(12(mail)) failed with euid=93(exim), gid=93(exim), egid=93(exim): Operation not permitted (This binary should probably be called with process group set to 12(mail) instead of 93(exim)) The same error occurs when I run this command from `exim` user with sudo. My dovecot v1.2.11 configuration: dovecot.conf ----------------- ... socket listen { master { path = /var/run/dovecot/auth-master mode = 0660 user = exim group = mail } client { path = /var/spool/exim/private-auth mode = 0660 user = exim group = mail } } ... userdb-sql.conf ------------------------ # exim uid: 93, mail gid: 12 user_query = SELECT CONCAT('/var/mail/', maildir) AS home, \ CONCAT('maildir:/var/mail/', maildir) AS mail, 93 AS uid, 12 AS gid, \ CONCAT('*:storage=', quota, 'B') AS quota_rule, \ 'storage=90%% /usr/libexec/dovecot/quota_warning.sh 90' AS quota_warning \ FROM mailbox WHERE username = '%u' AND active = '1' password_query = SELECT username AS user, password, \ CONCAT('/var/mail/', maildir) AS userdb_home, \ CONCAT('maildir:/var/mail/', maildir) AS userdb_mail, \ 93 AS userdb_uid, 12 AS userdb_gid, \ CONCAT('*:storage=', quota, 'B') AS userdb_quota_rule, \ 'storage=90%% /usr/libexec/dovecot/quota_warning.sh 90' AS userdb_quota_warning \ FROM mailbox WHERE username = '%u' AND active = '1' ( maybe, is any additional information required? ) All working fine with dovecot except this issue with `expire` plugin. How to avoid it? As I see, it's working if I run this command from `root` instead of `exim`, but I don't think it will be right to leave it in the root's crontab. Thanks, --- wbr, Denis.

1 0

[Dovecot] uid fetch/store always use the last uid when range is invalid
by Jean-Baptiste Vignaud 11 Mar '10

11 Mar '10

hello all; one colleague found a strange behavior, that can be annoying : -bash-3.1$ telnet 0 143 Trying 0.0.0.0... Connected to 0 (0.0.0.0). Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN AUTH=CRAM-MD5] Dovecot ready. l login login pass l OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA] Logged in l select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 281 EXISTS * 0 RECENT * OK [UNSEEN 7] First unseen. * OK [UIDVALIDITY 1223038537] UIDs valid * OK [UIDNEXT 363] Predicted next UID * OK [HIGHESTMODSEQ 1] Highest l OK [READ-WRITE] Select completed. l uid fetch 363:* rfc822.size * 281 FETCH (UID 362 RFC822.SIZE 1559) l OK Fetch completed. and more annoying: l uid store 1000000:* +flags (\Seen \Deleted) * 281 FETCH (UID 362 FLAGS (\Deleted \Seen)) l OK Store completed. # 1.2.10: /xam/dovecot/etc/dovecot.conf # OS: Linux 2.6.22.14-72.fc6 i686 Fedora Core release 6 (Zod) ext3 base_dir: /opt/dovecot/var/run log_path: /opt/logs/dovecot/dovecot.log info_log_path: /opt/logs/dovecot-info.log protocols: imap pop3 managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): *:2000 ssl: no disable_plaintext_auth: no login_dir: /opt/dovecot/var/run/login login_executable(default): /xam/dovecot/libexec/dovecot/imap-login login_executable(imap): /xam/dovecot/libexec/dovecot/imap-login login_executable(pop3): /xam/dovecot/libexec/dovecot/pop3-login login_executable(managesieve): /xam/dovecot/libexec/dovecot/managesieve-login login_user: dovelog mail_location: maildir:/opt/vmail/%d/%n mail_cache_fields: imap.bodystructure, imap.body, mime.parts, flags, date.sent, date.received, size.virtual, size.physical mail_executable(default): /xam/dovecot/libexec/dovecot/imap mail_executable(imap): /xam/dovecot/libexec/dovecot/imap mail_executable(pop3): /xam/dovecot/libexec/dovecot/pop3 mail_executable(managesieve): /xam/dovecot/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /xam/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /xam/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /xam/dovecot/lib/dovecot/pop3 mail_plugin_dir(managesieve): /xam/dovecot/lib/dovecot/managesieve lda: postmaster_address: postmaster(a)linux15.xandmail.com sendmail_path: /opt/postfix/usr/sbin/sendmail auth_socket_path: /opt/dovecot/var/run/dovecot/auth-master sendmail_path: /xam/postfix/usr/sbin/sendmail mail_plugins: quota sieve log_path: /xam/logs/deliver/deliver.log info_log_path: /xam/logs/deliver/deliver.log auth default: mechanisms: plain cram-md5 apop user: dovecot passdb: driver: sql args: /opt/dovecot/etc/dovecot-mysql.conf userdb: driver: sql args: /opt/dovecot/etc/dovecot-mysql.conf socket: type: listen client: path: /opt/dovecot/var/run/dovecot/auth-client mode: 432 user: postfix group: postfix master: path: /opt/dovecot/var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: dict:user::proxy::quotadict quota_rule: *:storage=100M quota_rule2: *:messages=1000000 sieve: %h/sieve/dovecot.sieve sieve_global_path: /xam/dovecot/etc/sieve/default sieve_global_dir: /xam/dovecot/etc/sieve sieve_dir: %h/sieve/ mail_log_max_lines_per_sec: 0 mail_log_group_events: no mail_log_events: delete undelete expunge copy mail_log_fields: uid box msgid size dict: quotadict: mysql:/opt/dovecot/etc/dovecot-dict-quota.conf path: /opt/dovecot/var/run/dict-server

2 3

[Dovecot] Howto disable imap listener in dovecot 2.0
by Erik Logtenberg 11 Mar '10

11 Mar '10

Hi, How do I disable the imap listener in dovecot 2.0-beta-3? I only want to enable the imaps listener. In previous dovecot versions I could specify: prototols = imaps But in dovecot 2.0 this is no longer possible (imap and imaps are now both enabled with "protocols = imap"). I already made ssl a requirement with: ssl = required This is in the global section. I commented out the imap listener in master.conf, but "dovecot -a" still shows the config for the imap listener, so I suppose it's default configuration. Can I explicitly tell dovecot to not open an imap listener, but only an imaps listener? Kind regards, Erik.

3 6

[Dovecot] mogilefs
by fernando＠dfcom.com.br 10 Mar '10

10 Mar '10

Hi list, Have you ever heard or tried mogilefs? (http://danga.com/mogilefs/) Do you think suitable for dovecot/mail environments ? Regards, Fernando

2 1

[Dovecot] Limit login attempts per connection?
by Tony Nelson 10 Mar '10

10 Mar '10

Dovecot allows a large number of login attempts per connection. I'd like to reduce that number to, say, 1, and let my firewall keep the ducks at bay, but I can't find anything in /etc/dovecot.conf or by googling. How do I do it? Do I need to patch the source? dovecot-1.1.10-1.x86_64 on CentOS 5.4 -- ____________________________________________________________________ TonyN.:' <mailto:tonynelson@georgeanelson.com> ' <http://www.georgeanelson.com/>

10 17

[Dovecot] Mailing list's prefix
by Timo Sirainen 10 Mar '10

10 Mar '10

Do you think I'd break a lot of people's filters if I removed the prefix? :) Anyone strongly for/against removing it? It seems kind of annoying to me whenever I happen to think about it.

31 52

[Dovecot] vacation script
by Papp Tamás 10 Mar '10

10 Mar '10

hi All, One of my users created this sieve script via roundcube's avelsieve implementation: # This script has been automatically generated by avelsieve # (Sieve Mail Filters Plugin for Squirrelmail) # Warning: If you edit this manually, then the changes will not # be reflected in the users' front-end! #AVELSIEVE_VERSIONYTo0OntzOjU6Im1ham9yIjtpOjE7czo1OiJtaW5vciI7aTo5O3M6NzoicmVsZWFzZSI7aTo3O3M6Njoic3RyaW5nIjtzOjU6IjEuOS43Ijt9 #AVELSIEVE_CREATED1268148005 #AVELSIEVE_MODIFIED1268148005 require ["fileinto","envelope","reject","vacation","relational","comparator-i;ascii-numeric","regex","body"]; if #START_SIEVE_RULEYTo2OntzOjQ6ImNvbmQiO2E6MTp7aTowO2E6MTp7czo0OiJ0eXBlIjtzOjM6ImFsbCI7fX1zOjQ6InR5cGUiO3M6MToiMSI7czo2OiJhY3Rpb24iO3M6MToiNiI7czoxMzoidmFjX2FkZHJlc3NlcyI7czoyMToic2lrZXQuZ2Fib3JAY2VudGVyLmh1IjtzOjg6InZhY19kYXlzIjtzOjE6IjciO3M6MTE6InZhY19tZXNzYWdlIjtzOjUzNzoiMjAxMC4gbeFyY2l1cyA5IC0gMTkga%2FZ69nR0IHN6YWJhZHPhZ29uIHZhZ3lvaywgDQpsZXZlbGVpbWV0IGxlZ2hhbWFyYWJiIG3hcmNpdXMgMjIt6W4gZm9nb20gb2x2YXNuaS4NCg0KVOF2b2xs6XRlbSBhbGF0dCBr6XJlbSwgaG9neSBz%2FHJn9XMgZXNldGJlbiBrb2xs6WfhaW1hdCBrZXJlc3NlLg0KDQpDc%2BFraSBab2x04W4gLSD8emVtZWx0ZXTpcw0KdGVsOiAwNiAxIDk5OSA0NDYwDQplLW1haWw6IGNzYWtpLnpvbHRhbkBjZW50ZXIuaHUNCg0KS2VyZXNrZWRlbG1pIPxneWVrYmVuIGtvbGzpZ%2BFpbSBhIGtlcmVza2VkZWxlbUBjZW50ZXIuaHUgZW1haWwgY%2B1tZW4sDQppbGxldHZlIGEgMDYgMSA5OTkgNDQ2MCB0ZWxlZm9uc3rhbW9uIOFsbG5hayByZW5kZWxrZXrpc%2BlyZS4gDQoNCg0K3GR29npsZXR0ZWw6DQpTaWtldCBH4WJvcg0KDQpDRU5URVIuSFUgS2Z0Lg0KIA0KSXJvZGE6IDExMzMgQnVkYXBlc3QsIFbhZyB1LiAxMy4NClRlbDogKDM2MSkgOTk5IDQ0NjAsIDk5OSA0NDYxDQpGYXg6ICgzNjEpIDk5OSA0NDYyLCA5OTkgNDQ5OQ0KIjt9END_SIEVE_RULE true { vacation :days 7 :addresses ["user(a)domain.hu"] text: 2010. mA(A;rcius 9 - 19 kA(s'zA(s'tt szabadsA(A;gon vagyok, ^M leveleimet leghamarabb mA(A;rcius 22-A(n fogom olvasni.^M [...some more text...] ^M TA(A;vollA(tem alatt kA(rem, hogy sA(z'rgL's esetben kollA(gA(A;imat keresse.^M ^M ^M .^M ; } The text is in hungarian. The autoreply function is wors fine, but the email address are not saved, so when the sender sents more mail, an autoreply email is received again and again. Dovecot version is 1.2.10-0~auto+106 from http://xi.rename-it.nl/debian/ . Is this a bug or maybe I miss something? Is there any dovecot option about this? Thank you, tamas

4 4

[Dovecot] fchown error upgrading from 0.99 to 1.2
by Erick Perez 09 Mar '10

09 Mar '10

Hi there, after upgrading from 0.99 to 1.2, the following errors appear in the error log: Mar 09 15:50:23 POP3(cwchbodega): Error: fchown(/home/cwchbodega/mail/.imap/INBOX/dovecot.index.log.newlock, -1, 12(mail)) failed: Operation not permitted (egid=653(cwchbodega), group based on /var/mail/cwchbodega) Mar 09 15:50:53 POP3(cwchbodega): Error: fchown(/home/cwchbodega/mail/.imap/INBOX/dovecot.index.tmp, -1, 12(mail)) failed: Operation not permitted (egid=653(cwchbodega), group based on /var/mail/cwchbodega) Mar 09 15:50:53 POP3(cwchbodega): Error: fchown(/home/cwchbodega/mail/.imap/INBOX/dovecot.index.cache.lock, -1, 12(mail)) failed: Operation not permitted (egid=653(cwchbodega), group based on /var/mail/cwchbodega) As I check the perms on the files, i see that: /var/mail/cwchbodega -rw-rw---- 1 cwchbodega mail and /home/cwchbodega/mail/.imap/INBOX/ [root@mail INBOX]# ls -la /home/cwchbodega/mail/.imap/INBOX/ total 280 drwx------ 2 cwchbodega cwchbodega 4096 Mar 9 15:50 . drwx------ 3 cwchbodega cwchbodega 4096 Dec 4 00:40 .. -rw------- 1 cwchbodega cwchbodega 5 Dec 4 00:40 .customflags -rw------- 1 cwchbodega cwchbodega 17312 Mar 9 15:50 dovecot.index -rw------- 1 cwchbodega cwchbodega 26624 Mar 9 15:56 dovecot.index.cache -rw------- 1 cwchbodega cwchbodega 26128 Mar 9 15:56 dovecot.index.log -rw------- 1 cwchbodega cwchbodega 17016 Mar 9 08:43 .imap.index -rw------- 1 cwchbodega cwchbodega 143422 Mar 6 08:54 .imap.index.data -rw------- 1 cwchbodega cwchbodega 7696 Feb 22 08:26 .imap.index.log -rw------- 1 cwchbodega cwchbodega 13028 Mar 6 08:54 .imap.index.tree [root@mail INBOX]# Should I chown user:mail the contents of the INBOX folder? and use perms of 0600 ? thanks, -- ------------------------------------------------------------ Erick Perez Cel +(507) 6675-5083 ------------------------------------------------------------

2 1

[Dovecot] Timeout while waiting for release of fcntl() lock for mbox file /var/mail/mailbox
by Erick Perez 09 Mar '10

09 Mar '10

Hi there, im currently using dovecot 0.99.11 (dovecot-0.99.11-4.EL4) on Centos 4.4 i386 /var/log/maillog has started to produce several times per minute this entry: Mar 9 14:52:44 mail pop3(albrook): Timeout while waiting for release of fcntl() lock for mbox file /var/mail/albrook when this happens, users cannot receive email properly, but eventually (maybe a few minutes or a few hours) the email gets delivered. is this a problem in the 0.99 version? can i safely upgrade to 1.2? thanks,

2 2