October 2012 - dovecot

[Dovecot] some questions on dovecot or rather a mail system setup
by Christoph Anton Mitterer 09 Oct '12

09 Oct '12

Hi folks. Perhaps you find some time to look into this,... if you think I should better direct this do some dovecot mailing list,.. just tell :) I'm trying to plan my mail system and would have some questions. The overall idea is about the following: - There is a (internet) server, which is the MTA (which will be postfix) and imap and/or pop3 server (which shall be, guess, dovecot!). - Any spam filtering, virus-scanning, mail filtering happens on that server. - I have not yet decided on whether to use maildrop for this or Sieve. Maildrop is nice, but has one major deficiency which I don't know how to work around. - A few clients (that means I won't serve 100 of users) connect to that server via imap and should see all mails, etc. already in some fancy sorted hierarchy (that means filtered into different directories). So far nothing complicated. But now... - Apart from spam, I never delete mail; and because I'm subscribed to many lists, I get a lot of mail. - Storage on my server is limited and it's located somewhere at my ISP, so I generally do not trust it with respect to safety... For both reasons, I want the canonical archive of all mail to be at home at some local server. - The local server should also be an imap server, so that I can access the archive from may computers at home. - The local server won't be available from the internet. - The local archive should have the same folder hierarchy as the internet server (I'd prefer not to filter twice). Mail Flow: - I generally want to have _all_ mail (which is not sorted out because of being spam) to be archived at the local server. - But(!) I want to selectively keep (in addition) mail at the internet server. For example I may want to select the folder that contains all mail form some friend to be kept online completely. But I may want to decide that mailinglists keep only the last 10 days and/or 1000 messages of mail. - The idea is, that the local server regularly (when it is online/running) catches new mail from the internet server... and stores it in the archive. - So apart from new mail that has not yet been read, that local archive contains always all mails that are also on the internet server... the later may contain (for specific directories) the same, or just parts of. - The MUAs will then have two imap accounts, one to the internet server and one to the local archive,... each one being usable, depending on where I am. 1) This is where my first problem arises: How can I implement that mail flow, especially: - How do I secure that all mail is read from the internet server (i.e. that nothing is "forgotten"? - How do I make sure that no mails are retrieved twice (or more)? A problem which I often had with pop, when the mail client crashed during sync? - Further it must be secured, that when I delete something on the internet server, it is NOT deleted on the local server (on the next mail-fetching).... this is why I don't use the word "sync". a) One stupid solution would be, that I duplicate all mail on the online server,... one part is for staying online, one part is for being fetched to the local archive. As soon as it was fetched... that copy gets removed (always). That solution would give a clean and secured separation of both? b) I don't think offlineimap or any other caching-like solution is the right thing... especially as one must always fear that such a cache may be accidentally wiped. Are there better solutions than (a)? 2) Problem would be already a refinement of a working solution for (1) (but obviously not when using (1).(a) ). When e.g. reply to or forward a mail using the online server,... and that mail had already been fetched,... can I make the flag synced? 3) Is dovecot suitable for the local server? - Ideally of course, I would use dovecot there, too... because that would mean one piece of software less to understand. - I couldn't sue maildir locally, because I loose just to much space to the block fragmentation. - I'd prefer not to use dbox (the thing that the indices are crucial scares me a bit off). a) When using mbox... is dovecot able to manage a really big folder hierarchy that basically ever keeps growing... with easily several 100k mails per folder... and that is in total already over 100GB? - I would prefer to have fast full text search. Does dovecot provide this? I was looking into database backed mail systems (again,... just for the local archive)... namely dbmail and archiveopteryx (are there other open source solutions?)... Not sure which of the two... or whether it's a good idea at all. I remember some dovecot wiki page that showed a comparison which said that both do not perfectly implement imap. Any suggestions with respect to that? Or is there even some SQL backend planned for dovecot? Thanks a lot, Chris.

3 3

[Dovecot] Master User Question
by Marc Perkel 08 Oct '12

08 Oct '12

OK - I'm trying to do weird stuff so rather not answer why I'm doing this. Trying to log in using a master user: user@example.com*master@master.com When debugging the master authenticates - but then it tries to authenticate user(a)example.com and it's not found. And - it is true that the user doesn't actually exist. What I want to do is allow it to log in without checking if the user exists, just on the credentials of the master. I'm not quite understanding what the login attempt for user(a)example.com is trying to do. Is it looking for more necessary information? Hope I asked this clearly enough. Thanks in advance.

2 5

[Dovecot] Namespace Prefix Tutorial?
by Marc Perkel 08 Oct '12

08 Oct '12

I'm trying to grasp the namespace and prefix stuff and looking for a good tutorial page that explains what it is and how to use it. Thanks in advance

3 6

[Dovecot] another mysql quota problem
by Tibby 08 Oct '12

08 Oct '12

Hello! I am using dovecot 1.2.15 on Debian 6 I have Postfix configured with mysql, and also dovecot is auth-ing form mysql users table. The users table has username password quota stored. When I set qouta to whatever number, it doesnt even show in dovecot. telnet localhost 143 a login username(a)domain.tld password a getquotaroot inbox * QUOTAROOT "INBOX" a OK Getquotaroot completed. Basicly it doesn't show any kind of quota. in the /etc/dovecot/dovecot-sql.conf: driver = mysql connect = host=127.0.0.1 dbname=mail user=USERNAME password=PASSWORD default_pass_scheme = CRYPT password_query = SELECT email as user, password FROM users WHERE email='%u'; user_query = SELECT CONCAT(('/home/vmail/'), SUBSTRING_INDEX(email,'@',-1),'/',SUBSTRING_INDEX(email,'@',1),'/') AS home, 5000 AS uid, 5000 AS gid, CONCAT('maildir:storage=', floor(quota/1024)) AS quota FROM users WHERE email='%u'; Why it doesn't get the quota out from the mail DB's users table?

1 0

[Dovecot] Large subjects increase memory-usage and enlarge index-files
by Peer Heinlein 08 Oct '12

08 Oct '12

Several times we already had the problems, that accounts with more the 1.3 or 1.7 billion e-mails in one folder run out-of-memory, even if vsize_limit of 750 MB is set. In this case, the lmtpd-process haven't been able to allocate more memory to read/write/update the index-files and crashed (and the index-files become corrupted at the end.) [Please -- don't discuss about the need of INBOXes with 1.7 million (unread) e-mails (don't discuss that with ME. Personally, I agree, that there's NO need for that...).] But: We also noticed accounts with ~ 300.000 e-Mails running out of memory in the same situations. This happends, if the subject is very large (subject or some other header attributes). And: We've been able to reproduce out-of-memory-Problems with just 13.000 e-mails with VERY long subjects (e.g.: network monitoring status informations), even with a vsize_limit of 750 MB (which is already very much). 13.000 e-mails isn't very much. And it's easy to inject several thousands of prepared e-mails. Having many mails for accounts with huge (and broken) index-files slows down the delivery rate VERY much and increases the need for memory and cpu resources and I/O very much. So: This could be used for a very easy to do denial-of-service attac against Dovecot-based mailservers. I don't have a clear solution for that, Dovecot needs the subject information in its index files. But it looks like, it isn't a good idea to put the whole subject into the index. Maybe it's better/necessary to use just the first 50-70 characters for that and to keep the rest away from the index? I think I would prefer that even if that means, that accessing those folders with "special" e-mails will become slower because Dovecot has to get those informations directly from the e-mail. This performance issue is just a problem for the user. But crashing lmtpd-processes and lowering the delivery rate is a *real* problem for the whole IMAP-cluster. Peer -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-42 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 4

[Dovecot] [OT] How do I convert maildir to bsmtp format?
by Marc Perkel 08 Oct '12

08 Oct '12

Here's what I'm trying to do. I have a spam filtering operation as a front end for other servers. I've created a virtual server for spam storage where the user will be able to log in using squirrelmail/dovecot to review and release their spam. The email is stored in maildir format. Piecing it together I can use squirrelmail to pipe the email into something so that if a use finds a false positive they can hit the "release" button and the message is sent on to the recipient. Squirrelmail sends the message as you would receive it as stored in maildir format. I need to take this format and translate it to send it on to the user. Wondering what is the best way to do that. Something that translated it into bsmtp format would be great. I'm sure thare must be something out there. I just haven't found it yet. Thanks in advance.

3 2

[Dovecot] Dovecot configuration and question about IP trusted
by Mik J 07 Oct '12

07 Oct '12

Hello list, I've just finished to install Dovecot and things seems to work so far. After some little efforts though. My version is 2.0.20 Question 1: I'm trying to tighten the security a little bit and added in dovecot.conf login_trusted_networks = 192.168.1.0/30 Then restarted Dovecot My client has the IP 192.168.1.20 and it's still able to retrieve emails. I expected it to be forbidden. Am I missing something ? # dovecot -n | grep trust login_trusted_networks = 192.168.1.0/30 Question 2: I feel that Dovecot is slow. I'm doing my test with my iphone as an imap client. Test 1: I retrieve a mail on a remote server provided by a hosting company, it takes 2 seconds Test 2: I retrieve a mail on my server which is on my LAN, the mail includes a few letters in the subject and a few letters in the body. The action takes about 8 seconds. It's quite subtule to measure so first I would like to know if Dovecot tries to do a dns reverse lookup or something like that. And it would explain the overhead. Thank you

2 4

[Dovecot] Getting rid of the subscription file
by Marc Perkel 06 Oct '12

06 Oct '12

I'd like to eliminate the subscription file and have it just list the folders that are there. How do I do that? Thanks in advance.

4 5

[Dovecot] doveadm purge -A via doveadm-proxy director fails after some users
by Daniel Parthey 06 Oct '12

06 Oct '12

Hi, we have configured userdb and passdb in the director and try to iterate all users and pass the "purge" command via doveadm proxy to port 19000 on the correct director backend host. A single purge -u username(a)example.org via doveadm-proxy works correctly, but iterating over some users with -A fails. Note: users/domains have been anonymized in output: ------------------------------------------------------------------------ mail04:~# /usr/bin/doveadm -c /etc/dovecot-director/dovecot-director.conf -D purge -A 2>&1 doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user01(a)domain1.example.org): Debug: auth input: user=user01(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user02(a)domain1.example.org): Debug: auth input: user=user02(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user03(a)domain1.example.org): Debug: auth input: user=user03(a)domain1.example.org proxy host=10.129.3.192 proxy_refresh=86400 doveadm(user04(a)domain1.example.org): Debug: auth input: user=user04(a)domain1.example.org proxy host=10.129.3.192 proxy_refresh=86400 doveadm(user05(a)domain1.example.org): Debug: auth input: user=user05(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 doveadm(user06(a)domain1.example.org): Debug: auth input: user=user06(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user07(a)domain1.example.org): Debug: auth input: user=user07(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 doveadm(user08(a)domain1.example.org): Debug: auth input: user=user08(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user01(a)domain2.example.org): Debug: auth input: user=user01(a)domain2.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user09(a)domain1.example.org): Debug: auth input: user=user09(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 10 / 94doveadm(user10(a)domain1.example.org): Debug: auth input: user=user10(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 doveadm(user11(a)domain1.example.org): Debug: auth input: user=user11(a)domain1.example.org proxy host=10.129.3.191 proxy_refresh=86400 doveadm(user12(a)domain1.example.org): Debug: auth input: user=user12(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user13(a)domain1.example.org): Debug: auth input: user=user13(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 doveadm(user14(a)domain1.example.org): Debug: auth input: user=user14(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user15(a)domain1.example.org): Debug: auth input: user=user15(a)domain1.example.org proxy host=10.129.3.191 proxy_refresh=86400 doveadm(user16(a)domain1.example.org): Debug: auth input: user=user16(a)domain1.example.org proxy host=10.129.3.191 proxy_refresh=86400 doveadm(user17(a)domain1.example.org): Debug: auth input: user=user17(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user18(a)domain1.example.org): Debug: auth input: user=user18(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user19(a)domain1.example.org): Debug: auth input: user=user19(a)domain1.example.org proxy host=10.129.3.192 proxy_refresh=86400 20 / 94doveadm(user20(a)domain1.example.org): Debug: auth input: user=user20(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user21(a)domain1.example.org): Debug: auth input: user=user21(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user22(a)domain1.example.org): Debug: auth input: user=user22(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user02(a)domain2.example.org): Debug: auth input: user=user02(a)domain2.example.org proxy host=10.129.3.190 proxy_refresh=86400 doveadm(user23(a)domain1.example.org): Debug: auth input: user=user23(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user24(a)domain1.example.org): Debug: auth input: user=user24(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user01(a)domain3.example.org): Debug: auth input: user=user01(a)domain3.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user25(a)domain1.example.org): Debug: auth input: user=user25(a)domain1.example.org proxy host=10.129.3.192 proxy_refresh=86400 doveadm(user26(a)domain1.example.org): Debug: auth input: user=user26(a)domain1.example.org proxy host=10.129.3.191 proxy_refresh=86400 doveadm(user27(a)domain1.example.org): Debug: auth input: user=user27(a)domain1.example.org proxy host=10.129.3.190 proxy_refresh=86400 30 / 94doveadm(user28(a)domain1.example.org): Debug: auth input: user=user28(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user29(a)domain1.example.org): Debug: auth input: user=user29(a)domain1.example.org proxy host=10.129.3.191 proxy_refresh=86400 doveadm(user30(a)domain1.example.org): Debug: auth input: user=user30(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user31(a)domain1.example.org): Debug: auth input: user=user31(a)domain1.example.org proxy host=10.129.3.193 proxy_refresh=86400 doveadm(user31(a)domain1.example.org): Error: doveadm server failure doveadm: Error: Failed to iterate through some users ------------------------------------------------------------------------ The user "user31(a)domain1.example.org" is proxied to the correct backend host according to director status, but the dovecot.log on the doveadm service backend host shows the following error: Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31(a)domain1.example.org): Error: user user31(a)domain1.example.org: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied Jun 29 15:40:31 10.129.3.249 dovecot: doveadm(user31(a)domain1.example.org): Error: purge: User lookup failed: Internal error occurred. Refer to server log for more information. The wiki http://wiki2.dovecot.org/Services#doveadm states that the privileges are (temporarily) dropped to the mail user's privileges after userdb lookup. It seems that from the second purge on which is passed over a single doveadm connection, the user lookup fails. It also seems a bit strange, that the "-A" parameter can be observed in the doveadm tcp stream to the backend, since iteration should be already done in the director and the backend should purge only a single user: D username(a)example.org purge -A Is there a bug or have I misconfigured/overlooked something? Configs of mailbox backend and director are attached. Kind regards Daniel

2 8

[Dovecot] IPv6 & SSL
by Luigi Rosa 06 Oct '12

06 Oct '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I have a dual stack server with Dovecot 2.1.10 listening on v4 and v6 Dovecot has a Comodo SSL certificate issued via NameCheap that works as expected with IPv4 in 10-ssl.conf I have enabled these configuraction directives: ssl = yes ssl_cert = < /path/to/file.crt ssl_key = < /path/to/file.key ssl_parameters_regenerate = 202 hours If I connect to Dovecot using the IPv6 address of the server with Thunderbird 15.0.1 uising CRAM-MD5 averything is ok. If I enable SSL _and_ IPv6 on Thunderbird I get this error: Oct 5 20:05:04 mail dovecot: imap-login: Disconnected (no auth attempts in 1 secs): user=<>, rip=2001:470:1f09:203:fdbf:508e:4a29:56c5, lip=2001:470:1f09:203::badd:ecaf, TLS: SSL_read() failed: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca: SSL alert number 48, session=<ZcMRtlPLqgAgAQRwHwkCA/2/UI5KKVbF> Ciao, luigi - -- / +--[Luigi Rosa]-- \ I will tell you a great secret, Captain. Perhaps the greatest of all time. The molecules of your body are the same molecules that make up this station and the nebula outside, that burn inside the stars themselves. We are star stuff, we are the universe made manifest, trying to figure itself out. As we have both learned, sometimes the universe requires a change of perspective." --Delenn, "Distant Star", Babylon 5 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlBvI50ACgkQ3kWu7Tfl6ZRBSACfRkp4FYpWaEZUQhIh0t6Vfs/I JbcAoKGZ769yogYS7faCXKvPTuhQiHA8 =jxCB -----END PGP SIGNATURE-----

5 8