- dovecot - dovecot.org

[Dovecot] RFC: Storing indexes in a (remote) database to increase dovecot scalability
by Mark Zealey 10 Aug '09

10 Aug '09

Hi all, This is just an idea I had over the weekend; I was wondering if instead of storing the indexes/caches on disk, it would be possible to have an option to store them in a remote database (eg mysql). There are several issues that we currently have with scaling dovecot, and I think that if it could store indexes in an external database we could alleviate most of these issues. In no particular order: 1) One major issue we have with dovecot is the number of iops it requires to keep its files in sync on our big disk arrays. If this were a mysql database, we'd have better ways of keeping these in sync, for example switching the speed at which it flushes the updates to disk, growth using partitioning/sharding, read-only slave instances etc, specific ssd storage for indexes/caches. I know that we could create a separate ssd disk array and export that over nfs too, but see issue (2) below. If indexes were stored in a mysql database we could choose to have a different reliability for the indexes (ie if the database server crashes we have 1 second of index changes lost, whereas for the email itself we can ensure it is always kept in sync on our other disk arrays). 2) nfs locking issues. No matter how hard we try, we always find there are some issues with concurrency over nfs. It really doesn't seem designed to handle these issues very well. I've not tried with nfsv4 and I believe there are some improvements there, but at the same time nfs wasn't really designed to store databases and have heavy concurrency. A proper database server (eg mysql, postgres) on the other hand thrives in this type of environment, especially when coupled with a transactional storage backend such as innodb which includes support for deadlock handling etc. 3) As we store all of our mail systems over nfs, effectively each export is a single-threaded connection to the server. No matter how fast the central nfs server you can realistically not go above 10k nfs ops/sec on the fastest of local networks due to latency issues. With mysql you could have multiple parallel connections to the central database store (1 per process?) which would mean more nfs ops available for actually serving the files. 4) Caching. Much more tuneable on a database server than in a filesystem. Eg in mysql there are loads of options to tune the various innodb/query/key caches, on nfs there are very few even on advanced netapps etc I know some of you store all your indexes on a single server locally & then store the emails on a remote central filestore. Whilst this is a good idea, it doesn't scale very well in that if the one server fails then everything has to be reindexed as clients login which causes incredible load as dovecot has to re-read most of the messages (in pop3 mode at least) in order to work out size counts etc. In my experience this reindexing would take out our email system for the best part of a day. I'm sure we could set something up with drbd mirroring or a central san array with ssd's for the index data, but this intrinsically does not scale with as much ease as a replicating database cluster could. There are also failover issues which can be very easily solved with a database system (eg mysql with an active/passive dual-master setup). Also, I know that most of these points refer to nfs; and I know that there are alternative methods (eg san + gfs or some other scalable filesystem) however most of these are relatively new technologies whereas nfs is pretty much the standard for remote file storage; certainly I'd consider it a lot more reliable when compared to the likes of gfs or lustre. Your typical large scale mail setup would be over nfs and this is the sort of environment that I've run into these scaling issues in. I've heard that gfs also doesn't work very well with 'hot spots' which indexes tend to be, although I've no first hand experience of this. So, although this post only really mentions nfs & mysql (the two technologies that I'm most familiar with) I think this is a much more general problem so I'm not advocating the use of either technology apart from them being most people's first choices for ease of use. What I'm really proposing is a way to separate the two different types of data that dovecot uses. Indexes&caches contain small frequent transactions; and whilst these are relatively important, dropping the past few seconds of these transactions doesn't seriously matter. On the other hand emails are larger, infrequently changed & it's important they are kept fully in sync - if we accept a message it should be guaranteed that we don't loose it. At the end of the day it seems to me that a filesystem is the ideal place to store emails, whereas a database server would be the ideal place to store indexes/caches. Obviously for speed of setup, simply storing everything on a filesystem is easy so I'm not suggesting that we abandon this route, but for truly large infrastructures I think being able to use a database system would have significant advantages in each of these four areas. Any thoughts? Mark -- Mark Zealey -- Platform Architect Product Development * Webfusion 123-reg.co.uk, webfusion.co.uk, donhost.co.uk, supanames.co.uk This mail is subject to http://www.gxn.net/disclaimer

2 1

[Dovecot] Dovecot-1.1.15 panics
by Mike Abbott 10 Aug '09

10 Aug '09

Dovecot-1.1.15 assert-fails frequently. Attached are some errors from the logs, and a stack trace from the common panic. Hope you can fix these soon. Thanks. 0 libSystem.B.dylib 0x00007fff8825e606 __kill + 10 1 libSystem.B.dylib 0x00007fff882e1c92 abort + 83 2 imap 0x0000000100069e1b default_fatal_finish + 70 3 imap 0x0000000100069e69 i_syslog_fatal_handler + 0 4 imap 0x00000001000695c8 i_info + 0 5 imap 0x000000010001913f maildir_uidlist_iter_next_rec + 210 6 imap 0x0000000100019177 maildir_uidlist_iter_next + 38 7 imap 0x0000000100018abb maildir_sync_index + 2019 8 imap 0x0000000100017788 maildir_sync_context + 662 9 imap 0x000000010001797c maildir_storage_sync_init + 104 10 imap 0x000000010000bbe3 imap_sync_init + 132 11 imap 0x000000010000be35 cmd_sync_delayed + 477 12 imap 0x0000000100006692 client_handle_input + 419 13 imap 0x000000010000676c client_input + 148 14 imap 0x0000000100070e60 io_loop_handler_run + 272 15 imap 0x0000000100070225 io_loop_run + 55 16 imap 0x000000010000d746 main + 1640 17 imap 0x0000000100000b84 start + 52

3 8

Re: [Dovecot] Mail archive
by fernando＠dfcom.com.br 10 Aug '09

10 Aug '09

I would like 'some process' to store old mails at a cheap storage. I know how to do it with symlinks, but I don´t know if it is the best option. So, I´m asking if dovecot improves it somehow. Fernando fernando at dfcom.com.br schrieb: > Hi, > > I´m using dovecot at our storages, accounts is distributed among many > storages (not entire domain), all of them with compression (when message > > 30Kb), nightly crontab script. > > Even though compression and many storages, we always have problems with > disk space and need to migrate accounts. > > Have you ever implemented any archive solution working with dovecot ? > > Regards, > Fernando > you might give more info what kind of "archive" you exactly mean meanwhile study http://wiki.dovecot.org/HowTo/ReadOnlyArchive http://wiki.dovecot.org/Plugins/Zlib http://wiki.dovecot.org/Plugins/Expire Alternative dbox directory expiration -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria

2 1

[Dovecot] Mail archive
by fernando＠dfcom.com.br 10 Aug '09

10 Aug '09

Hi, I´m using dovecot at our storages, accounts is distributed among many storages (not entire domain), all of them with compression (when message > 30Kb), nightly crontab script. Even though compression and many storages, we always have problems with disk space and need to migrate accounts. Have you ever implemented any archive solution working with dovecot ? Regards, Fernando

2 1

[Dovecot] inconsistency with expire-tool and expire dict
by LEVAI Daniel 10 Aug '09

10 Aug '09

Hi! Here is the problem: passdb: daniell:*::::::user=daniell2 userdb: daniell2::uid:gid:gecos:home:: dovecot.conf: plugin { expire = SA.* 1 # (There are SA.HAM and SA.SPAM directories) } When copying a message to eg. the SA.HAM directory, then dovecot inserts this into my expires table: dovecot=# select * from expires ; username | mailbox | expire_stamp ---------------------+---------+-------------- daniell | SA.HAM | 1249976454 ^^^^^^^ wrong username This causes an error when running expire-tool (after changing expire_stamp): # /usr/local/sbin/dovecot --exec-mail ext /usr/local/libexec/dovecot/expire- tool --test Info: User lookup failed: daniell Info: daniell/SA.HAM: no messages left If I change the username field in the expires table... : # UPDATE expires SET username = 'daniell2'; UPDATE 1 ... expire-tool is fine: # /usr/local/sbin/dovecot --exec-mail ext /usr/local/libexec/dovecot/expire- tool --test Info: daniell2/SA.HAM: timestamp 1249880093 (Mon Aug 10 06:54:53 2009) -> 1249976454 (Tue Aug 11 09:40:54 2009) Daniel -- LÉVAI Dániel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1

1 1

[Dovecot] Mail not begin proccessed
by Andre Labuschagne 10 Aug '09

10 Aug '09

Good day all, [Dovecot version 1.2.1] I've set up postfix and dovecot on Centos. I have a problem with the mail, it is begin received but not "processed" by dovecot. I've looked in the maillog file and found two problems; the first is that private/dovecot directory (I think it should be a directory) did not exist and the second problem is "mail transport unavailable". I've create a directory called /var/spool/postfix/private/dovecot but I get the following error in the log file: Connection refused (Is this an ownership issue? I've made the owner postfix) As for the second problem, I've spend the most of the past 3 days googling the error and found very little "useful" information. Do I need to supply more info? Best regards, Andre

3 4

[Dovecot] Released Sieve v0.1.11 and ManageSieve v0.11.8 for Dovecot v1.2.3
by Stephan Bosch 10 Aug '09

10 Aug '09

Hello Dovecot users, Apart from unfinished development of the date extension, this is a set of bug-fix releases. A few portability issues were found, a few stupidities were fixed and the ManageSieve proxy now also works with TLS. Most notably, the include extension had an issue regarding the expansion of $HOME in the personal include path. I've built a project site at: http://pigeonhole.dovecot.org Changelog Sieve v0.1.11: + Built skeleton implementation for the date extension (RFC5260). It compiles, but it does not do anything useful yet. Therefore, it is not part of the default compilation. - Fixed ARM portability issues caused by char type not being signed on that platform. Reading optional operands from a binary would fail for action side effects. Also, an accidental mixup of an int return type with bool caused the interpreter to continue on ARM even though an error occured. - Removed direct stdint.h includes to prevent portability issues. - Fixed segfault bug in the handling of script open failures. - Include: improved user error messages and system log messages. - Fixed copy-paste mixup between sieve_after and sieve_before settings in the LDA Sieve plugin. If only a sieve_after script was active, nothing would have been executed. Patch by Mike Abbott. - Include: fixed a bug in HOME substitution in the sieve_dir path. Surfaced in ManageSieve. Changelog ManageSieve v0.11.8: - Fixed TLS support for proxying ManageSieve. The protocol state machine was incorrect. Also added a check that disables ssl when 'starttls' is not enabled for the user. This produces a proper warning in the log file. There is no such thing as a managesieveS protocol which has SSL from the start. The releases are available as follows: Sieve: http://www.rename-it.nl/dovecot/1.2/dovecot-1.2-sieve-0.1.11.tar.gz http://www.rename-it.nl/dovecot/1.2/dovecot-1.2-sieve-0.1.11.tar.gz.sig ManageSieve (package + patch): http://www.rename-it.nl/dovecot/1.2/dovecot-1.2-managesieve-0.11.8.tar.gz http://www.rename-it.nl/dovecot/1.2/dovecot-1.2-managesieve-0.11.8.tar.gz.s… http://www.rename-it.nl/dovecot/1.2/dovecot-1.2.3-managesieve-0.11.8.diff.gz http://www.rename-it.nl/dovecot/1.2/dovecot-1.2.3-managesieve-0.11.8.diff.g… Have fun testing the new releases and don't hesitate to notify me when there are problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

3 3

[Dovecot] virtual plugin and ACL
by Nikita Koshikov 10 Aug '09

10 Aug '09

Hello list, We are using virtual plugin for to provide users two additional folders: Company/all - it contains all messages in user's mailbox Company/favorites - here is flagged messages. Here is namespace part of config file: namespace private { prefix = Company/ separator = / location = virtual:/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++ subscriptions = no } Everything works fine, but problem arises when someone rename or delete virtual mailbox. With above config, 1 user make changes to his mailbox, but changes distribute for entire domain. First, I have tried to add :CONTROL=/data/mail/%d/%n/data/virtual to location part of virtual namespace, but I have no luck with such settings. Directory /data/mail/%d/%n/data/virtual not even been created. Then I tried to setup ACL for virtual mailbox. Adding "acl" to mail_plugins in imap and lda section and acl=vfile to plugins config. Under /var/mail/virtual in each mailbox I create dovecot-acl file contaning: user=koshikov.n lrwstiekxa authenticated lrwstipe But this didn't work. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN AUTH=LOGIN] Server ready. a login test qwe a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH QUOTA] Logged in a list "" * * LIST (\HasNoChildren) "/" "Sent" * LIST (\HasNoChildren) "/" "Drafts" * LIST (\HasNoChildren) "/" "Spam" * LIST (\HasNoChildren) "/" "Trash" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Company" * LIST (\HasNoChildren) "/" "Company/all" * LIST (\HasNoChildren) "/" "Company/favorites" a OK List completed. a select Company/all * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk \*)] Flags permitted. * 13 EXISTS * 0 RECENT * OK [UIDVALIDITY 1249459221] UIDs valid * OK [UIDNEXT 14] Predicted next UID * OK [NOMODSEQ] No permanent modsequences a OK [READ-WRITE] Select completed. a getacl a BAD Error in IMAP command GETACL: Unknown command. a rename Company/all Company/test a OK Rename completed. After "rename completed" in /var/mail/virtual new file dovecot-acl-list is appear with content: 1249457074 test How can I get workaround for this situation ? I just need to restrict my users renaming and deleting virtualboxes.

2 2

[Dovecot] how secure is Dovecot when exposed to the Internet?
by Florin Andrei 10 Aug '09

10 Aug '09

$ dovecot -n # 1.1.11: /etc/dovecot/dovecot.conf # OS: Linux 2.6.28-11-server x86_64 Ubuntu 9.04 protocols: imap imaps managesieve I need to make an IMAP (actually imaps) server available over the Internet. Unfortunately, VPN is not available (not all clients support VPN), so I will have to expose the imaps port to the Internet. My question is: how reliable is Dovecot in such a setup? I am not talking about encryption (protecting the traffic between server and client). I am talking about having the daemon exposed to anything coming in from the Internet, buffer overflows and stuff like that. What's the security history of this software in situations like this? -- Florin Andrei http://florin.myip.org/

2 2

[Dovecot] http://wiki.dovecot.org/LDA typo
by LEVAI Daniel 09 Aug '09

09 Aug '09

-Note that dovecot.conf file must be world readable to enable deliver process read it, while running with user priviledges. +Note that dovecot.conf file must be world readable to enable deliver process read it, while running with user privileges. -- LÉVAI Dániel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1

2 2