February 2008 - dovecot

[Dovecot] imap memory footprint rather large
by martin f krafft 12 May '08

12 May '08

Dear list, I am experimenting with a new mail handling setup and it involves a single IMAP folder with just under 70'000 messages. When OfflineIMAP connects to the server, the imap process starts to eat up a lot of memory: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15607 madduck 35 19 283m 244m 239m D 16.9 49.3 0:09.96 imap On the contrary, when "online" client, such as Thunderbird connect, memory usage is around 10m, which is entirely acceptable. The way offlineimap reads may is by FETCHing metadata, then APPENDing new local mail, SEARCHing for the UIDs of each uploaded mail, and finally FETCHing new remote mail. Memory use seems to be O(n) in the size of the folder. On the folder with 70k messages, dovecot seems to allocate 280m of memory, which it then fills to about 70% during the metadata FETCH, and then keeps growing while APPEND/SEARCHing the new local messages. The 70k mailbox is just short of 600Mb in size on disk. Dovecot uses 280Mb to serve it. Is it possible that dovecot is reading too much into memory, or over-optimising? Can I somehow tweak this to lower the memory footprint? Cheers, -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck "the unexamined life is not worth living" -- platon spamtraps: madduck.bogus(a)madduck.net

2 9

[Dovecot] Another numptie question - making maildirs for shared mailboxes.
by Stroller 12 May '08

12 May '08

Hi there, Sorry to have so much trouble with my Dovecot setup & to ask so many questions, but I'm having some problems with shared folders. I've read the thread at <http://dovecot.org/pipermail/dovecot/2005-January/006079.html> and added the following lines to /etc/dovecot.conf: namespace private { separator = / prefix = inbox = yes } namespace public { separator = / prefix = General/ location = maildir:/tmp/dovecot-test/ inbox = no hidden = no } So I create a directory /tmp/dovecot-test/ and an empty file /tmp/dovecot-test/dovecot-shared with liberal (777 & 666 respectively) permissions and then try to subscribe to the "General" folder in Outlook - I get a "failed to subscribe to the folder. The server responded `the mailbox doesn't exist: General`" error message, but cur, new & tmp are created in /tmp/dovecot-test. These subfolders are owned by the user that attempted to subscribe & have 700 permissions; chmod'ing them 777 makes no difference. Since "default_mail_env = maildir:%h/.maildir" in dovecot.conf I make a directory /tmp/dovecot-test/.maildir also containing cur, new, tmp & dovecot-shared. When I select "IMAP folders" in Outlook I am still unable to subscribe to "General" but am now able to subscribe to "General/maildir". The subscribed folder is shown in Outlook underneath the users' Inbox as a sub-folder of general, screenshot at <http://linux.stroller.uk.eu.org/DovecotSharedFolders.png> - see how I can't subscribe to "General" alone, but it is "brought in" by subscribing to "General/maildir"? If I try to copy a test message into General then, after a delay of a minute or so, I get "Can't move the items - this IMAP command could not be sent to the server before the connect was terminated"; if I try to copy the test message into General/maildir then I immediately get "Can't copy the items. The server responded: `Internal error occurred. Refer to server log for more information [2005-12-20 12:05:40]". I've tried tinkering with other directory structures under /tmp/dovecot-test/ with no success - can anyone suggest what I'm doing wrong, please? It's quite important to me that the "General" folder is a fairly high-level one - only one additional folder icon should show up for the users' shared mail. At one point I managed to create a folder General/General, but that's undesirable - the users wouldn't understand the concept of having a folder showing in Outlook that they can't write to, or which is only used as a container for another folder. (In any case, I believe I was still unable to write to it). I was about to say, "I know I could resolve this with the use of a symlink-type shared folder, but that's undesirable because I'd have to manually add the symlink for each user that's added to the system", but writing this it occurs to me that I could resolve that by adding the symlink to the the etc/skel/. Nevertheless, I see the namespaces approach as more elegant, as it allows for different user groups having different shared folders. I'd appreciate any advice - I suspect I'm just getting the directory hierarchies wrong, or the syntax of dovecot.conf wrong AGAIN! If anyone with a working shared folders setup could provide a copy of the appropriate section of dovecot.conf & a listing of the appropriate directories then I'd be EXTREMELY grateful. Stroller.

7 12

[Dovecot] imap fs quota (rpc) won't work?
by Adam McDougall 06 May '08

06 May '08

In the past I dabbled with the imap quota plugin with the fs backend because I wanted to report usage to my users (not limit them). At the time, the quota plugin would make dovecot crash when trying to write to a folder (I can bring up this report if needed). However, in a later beta of dovecot 1.1 I tried quota again but I cannot get it to report any results. I've been wanting to use fs:noenforcing but also tried just quota = fs to simplify. When I right click a folder stored in my homedir, thunderbird tells me 'the server does not support quotas' in the folder properties. What else can I do? Thanks! IModule loaded: /usr/local/lib/dovecot/imap/lib10_quota_plugin.so IModule loaded: /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota add storage dir = /home/mcdouga9/Maildir Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota block device = cypher:/vol/mail/home Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota mount point = /home gribble# quota -v mcdouga9 | grep home /home 10766672 20480000 20480000 963778 4294967295 4294967295 # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/de list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat

1 1

[Dovecot] Users w/o acl access appear to be subscribed to public folders (1.1b8)
by Adam McDougall 05 May '08

05 May '08

I noticed this today, I had a user outside of our department test out dovecot. They were using squirrelmail and I noticed that dovecot thinks this user is subscribed to ALL public folders even though a dovecot ACL prevents all access. I'm pretty sure access is still denied. I was able to reproduce this with a guest account I added: l lsub "" "#shared/decs/%" * LSUB (\Noselect) "/" "#shared/decs/linuxadmin" * LSUB (\Noselect) "/" "#shared/decs/jbossadmin" * LSUB () "/" "#shared/decs/support" * LSUB () "/" "#shared/decs/receipts" * LSUB (\Noselect) "/" "#shared/decs/pcadmin" * LSUB () "/" "#shared/decs/network" * LSUB (\Noselect) "/" "#shared/decs/printmaster" * LSUB () "/" "#shared/decs/postmaster" * LSUB (\Noselect) "/" "#shared/decs/unixadmin" * LSUB () "/" "#shared/decs/security" * LSUB (\Noselect) "/" "#shared/decs/webmaster" l OK Lsub completed. This only seems to happen when the acl plugin is enabled. Without the acl plugin, these are not listed as subscriptions. After deleting /egr/mail/shared/decs/dovecot-acl-list and re-enabling the acl plugin, I get this: l lsub "" "#shared/decs/%" * LSUB () "/" "#shared/decs/unixadmin" * LSUB () "/" "#shared/decs/support" * LSUB () "/" "#shared/decs/security" * LSUB () "/" "#shared/decs/printmaster" * LSUB () "/" "#shared/decs/postmaster" * LSUB () "/" "#shared/decs/pcadmin" * LSUB () "/" "#shared/decs/network" * LSUB () "/" "#shared/decs/linuxadmin" * LSUB () "/" "#shared/decs/webmaster" * LSUB () "/" "#shared/decs/jbossadmin" l OK Lsub completed. Is it related, or is it different just because a new dovecot-acl-list got created by another user already (but is mode 700?)

2 3

[Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
by Adam McDougall 04 May '08

04 May '08

I was trying to debug this with gdb, but I'm not sure how to make env MAIL=maildir:~/Maildir MAIL_PLUGINS=acl ACL=vfile gdb /tmp/imap understand the #shared/decs namespace from below so I can SELECT it. I have a number of maildirs under #ahared/decs, I tried opening at least 2 of them but dovecot gives sig 11 without a direct error about the problem. I don't think this was exactly the same problem in 1.1b3, but unfortunately I deleted all the patches I was testing with so it would be a pain to test the same situation with 1.1b3 again (although if needed, I can). If dovecot-shared exists in the folder I try to open, dovecot says: Oct 27 12:57:38 gribble dovecot: IMAP(mcdouga9): fchown() failed with file /home/mcdouga9/Maildir/dovecot11testing/public/indexes/decs/.unixadmin/dovecot.index.log: Operation not permitted Oct 27 12:57:38 gribble dovecot: child 55470 (imap) killed with signal 11 I am used to getting the first message because dovecot tries to apply perms from dovecot-shared to the index files which are elsewhere. If I open a public folder without dovecot-shared, I see no contents but it does not sig11. I am NOT using the acl plugin right now, it didn't seem to make a difference if it was loaded. When I left off with the dovecot 1.1b3 + 8 patches, I believe I was getting an assertion. Please let me know if I goofed up something, I have a strange feeling about that here, or if you need more info. I'd like to go ahead and get a gdb trace. # 1.1.beta4: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/cert.pem ssl_key_file: /usr/local/etc/apache2/ssl/cert.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 0 mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot11testing/public/control/decs:INDEX=%h/Maildir/dovecot11testing/public/indexes/decs list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes hidden: yes subscriptions: yes namespace: type: private separator: / inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls

2 18

[Dovecot] Embedded From_ lines breaking Content-Length (and Dovecot)
by Lennart Lövstrand 04 May '08

04 May '08

I feel like I'm going totally crazy. Is it just me, or have embedded From_ lines really been breaking mbox messages since (at least) dovecot 1.0? I found a whole lot of broken messages in an old mailbox of mine and when I looked closer at it, it seemed like Dovecot was ignoring the Content-Length header and truncating the messages at the first embedded From_ line instead. Figuring that this probably just was a fluke / old bug, I erased the index and upgraded Dovecot, first to 1.0.10, then to 1.1.beta14, but I'm still seeing the bug -- and it's even gotten worse! Earlier versions would just quietly split the message at the From_ line, but 1.1.beta14 is more diligent and will complain with: > dovecot[44795]: IMAP(lennart): FETCH for mailbox Drafts UID 196 got > too little data: 590 vs 719 before dropping the connection and exiting... which cause Thunderbird to reopen the connection and reissue the same request... which will cause Dovecot to complain and exit again... etc ad nauseam... It's trivial to reproduce too -- just mail yourself a message with a valid From_ line in it (assuming that your delivery system isn't doing >From-escaping), or put it in a draft plain text message and save it. Then go and look in your Drafts folder... (Like this one: "From someone Tue Jan 29 03:15:52 2008") Just be sure that you can stop your spinning mail client and/or server if you're using 1.1.beta14... --- OK, I am definitely not very familiar with the Dovecot source, but I spent some time trying to track this down today and it looks to me like i_stream_raw_mbox_read is being asked to find the end of the message (at the From_ line) before mbox_sync_parse_next_mail has had a chance to parse the Content-Length header and find out how big it is. Maybe someone who knows the code better can take a look at it and tell me if I'm barking up the totally wrong tree... Thanks, --Lennart

3 5

[Dovecot] Sieve 1.1.2 -> special chars
by mikkel＠euro123.dk 04 May '08

04 May '08

I'm unable to use local chars in vacation/auto replies with sieve. .dovecot.sieve.err gives the same error as when there's a parse error. Example of error: line 25: string 'æ ø å ö ä ü é á. ': My dovecot is v. 1.1b5 and sieve is v. 1.1.2. This issue was also present with dovecot 1.0.x Is there a way around this? - Mikkel

3 2

[Dovecot] 1.0.5: many pop3-login processes?
by Alan Ferrency 25 Apr '08

25 Apr '08

Hello, We are running dovecot 1.0.5 on a test server, with FreeBSD 6.2 (though I have noticed the same problem since dovecot versions in the 0.99 range). We don't have very many simultaneous pop/imap users, but we have a proliferation of pop3-login processes. Currently we have 128 such processes. We have 11 imap-login processes, but only a few actual imap processes running. Is this normal? Can we stop it? Is anyone else successfully running dovecot on FreeBSD 6.2? Thanks for your help. Alan Ferrency pair Networks, Inc. alan(a)pair.com

4 14

[Dovecot] alert function, V2
by Stewart Dean 25 Apr '08

25 Apr '08

I guess the alert function is not part of the spec, because it appears to be unimplemented in Dovecot, even though most clients implement it. Timo, I would hope this would be easy to implement, and it's functionality that will be valuable to every installation. Here's why. America recently had the tragedy of a shooting rampage on a large university campus (yes, America should do something about the violence and free access to guns in our culture/society). What was worse was that, due to the lack of an emergency notification process, some 20+ people died needlessly. Now all college and university administration are scrambling to set up emergency notification processes. Our first take on this is a massive all-hands mailing list, but a) best case, it would take 15 minutes (and as much as an hour) to get to everybody's mailbox. b) users would have to read through their mail and stumble on the message What's needed is an "in-your-face" unavoidable alert, and the UWIMAP alert functionality provides this. As I recall, you touch just about anything in your IMAP client and there's a popup message in the middle of the client IMAP window...and it happens repeatedly. We tried it back 3-4 years ago and found it too annoying to use. For this purpose, it's perfect. The wy UWIMAP did it, the presence of the file /etc/imapd.alert triggers it, and the text in that file is transmitted to the client. The message has to be short, a one-liner for a small popup window, but this could be a pointer to a shared system message folder. Of course, DC could sense the activation of the alert in some other fashion that the presence of a file. Ideally, whatever activates the alert function should be able to be turned on remotely; I had in mind to do it using a mail aliases that is a pipe to a executable... What say you? -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

6 9

Re: [Dovecot] sa learning from an imap spam folder
by frank.kintrup＠fkware.de 11 Apr '08

11 Apr '08

This is my approach, stored in /etc/cron.hourly. It's very new, so I'm still testing it. The goal is to learn HAM massages only if they are a day old, so that I can manually remove SPAM that slipped through. Mails tagged from Spamassassin are sorted automatically into the "Junk.Spam" mailbox and learned after 12 hours. I manually move any missed SPAM into the .Junk mailbox, so I'm sure there are no wrongly tagged messages there, so I can learn those messages as soon as they are found by cron. /etc/cron.hourly/sa-learn: #!/bin/sh umask 022 # Learn HAM messages which were roughly received between 24 and 25 hours ago find \ /var/vmail/fkware.de/frank.kintrup/Maildir/ \ -path '/var/vmail/fkware.de/frank.kintrup/Maildir/.Junk*' -prune -o \ -path '/var/vmail/fkware.de/frank.kintrup/Maildir/.Sent' -prune -o \ -path '/var/vmail/fkware.de/frank.kintrup/Maildir/.Trash' -prune -o \ -path '/var/vmail/fkware.de/frank.kintrup/Maildir/.Draft' -prune -o \ -iname '*server.fkware.de*' -type f -mmin +1435 -mmin -1505 \ -execdir sa-learn --username=vmail --no-sync --ham {} \; \ /dev/nul 2>/dev/nul # Learn and delete SPAM messages which were manually moved to the Junk folder find \ /var/vmail/fkware.de/frank.kintrup/Maildir/.Junk/ \ -iname '*server.fkware.de*' -type f \ -execdir sa-learn --username=vmail --no-sync --spam {} \; \ -execdir rm {} \; \ >/dev/nul 2>/dev/nul # Learn and delete SPAM messages which were received more than 12 hours ago # and automatically put into the Junk.Spam folder find \ /var/vmail/fkware.de/frank.kintrup/Maildir/.Junk.Spam/ \ -iname '*server.fkware.de*' -type f -mmin +720 \ -execdir sa-learn --username=vmail --no-sync --spam {} \; \ -execdir rm {} \; \ >/dev/nul 2>/dev/nul sa-learn --username=vmail --sync >/dev/nul exit 0

8 8