April 2008 - dovecot - dovecot.org

[Dovecot] imap memory footprint rather large
by martin f krafft 12 May '08

12 May '08

Dear list, I am experimenting with a new mail handling setup and it involves a single IMAP folder with just under 70'000 messages. When OfflineIMAP connects to the server, the imap process starts to eat up a lot of memory: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15607 madduck 35 19 283m 244m 239m D 16.9 49.3 0:09.96 imap On the contrary, when "online" client, such as Thunderbird connect, memory usage is around 10m, which is entirely acceptable. The way offlineimap reads may is by FETCHing metadata, then APPENDing new local mail, SEARCHing for the UIDs of each uploaded mail, and finally FETCHing new remote mail. Memory use seems to be O(n) in the size of the folder. On the folder with 70k messages, dovecot seems to allocate 280m of memory, which it then fills to about 70% during the metadata FETCH, and then keeps growing while APPEND/SEARCHing the new local messages. The 70k mailbox is just short of 600Mb in size on disk. Dovecot uses 280Mb to serve it. Is it possible that dovecot is reading too much into memory, or over-optimising? Can I somehow tweak this to lower the memory footprint? Cheers, -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck "the unexamined life is not worth living" -- platon spamtraps: madduck.bogus(a)madduck.net

2 9

[Dovecot] Another numptie question - making maildirs for shared mailboxes.
by Stroller 12 May '08

12 May '08

Hi there, Sorry to have so much trouble with my Dovecot setup & to ask so many questions, but I'm having some problems with shared folders. I've read the thread at <http://dovecot.org/pipermail/dovecot/2005-January/006079.html> and added the following lines to /etc/dovecot.conf: namespace private { separator = / prefix = inbox = yes } namespace public { separator = / prefix = General/ location = maildir:/tmp/dovecot-test/ inbox = no hidden = no } So I create a directory /tmp/dovecot-test/ and an empty file /tmp/dovecot-test/dovecot-shared with liberal (777 & 666 respectively) permissions and then try to subscribe to the "General" folder in Outlook - I get a "failed to subscribe to the folder. The server responded `the mailbox doesn't exist: General`" error message, but cur, new & tmp are created in /tmp/dovecot-test. These subfolders are owned by the user that attempted to subscribe & have 700 permissions; chmod'ing them 777 makes no difference. Since "default_mail_env = maildir:%h/.maildir" in dovecot.conf I make a directory /tmp/dovecot-test/.maildir also containing cur, new, tmp & dovecot-shared. When I select "IMAP folders" in Outlook I am still unable to subscribe to "General" but am now able to subscribe to "General/maildir". The subscribed folder is shown in Outlook underneath the users' Inbox as a sub-folder of general, screenshot at <http://linux.stroller.uk.eu.org/DovecotSharedFolders.png> - see how I can't subscribe to "General" alone, but it is "brought in" by subscribing to "General/maildir"? If I try to copy a test message into General then, after a delay of a minute or so, I get "Can't move the items - this IMAP command could not be sent to the server before the connect was terminated"; if I try to copy the test message into General/maildir then I immediately get "Can't copy the items. The server responded: `Internal error occurred. Refer to server log for more information [2005-12-20 12:05:40]". I've tried tinkering with other directory structures under /tmp/dovecot-test/ with no success - can anyone suggest what I'm doing wrong, please? It's quite important to me that the "General" folder is a fairly high-level one - only one additional folder icon should show up for the users' shared mail. At one point I managed to create a folder General/General, but that's undesirable - the users wouldn't understand the concept of having a folder showing in Outlook that they can't write to, or which is only used as a container for another folder. (In any case, I believe I was still unable to write to it). I was about to say, "I know I could resolve this with the use of a symlink-type shared folder, but that's undesirable because I'd have to manually add the symlink for each user that's added to the system", but writing this it occurs to me that I could resolve that by adding the symlink to the the etc/skel/. Nevertheless, I see the namespaces approach as more elegant, as it allows for different user groups having different shared folders. I'd appreciate any advice - I suspect I'm just getting the directory hierarchies wrong, or the syntax of dovecot.conf wrong AGAIN! If anyone with a working shared folders setup could provide a copy of the appropriate section of dovecot.conf & a listing of the appropriate directories then I'd be EXTREMELY grateful. Stroller.

7 12

[Dovecot] imap fs quota (rpc) won't work?
by Adam McDougall 06 May '08

06 May '08

In the past I dabbled with the imap quota plugin with the fs backend because I wanted to report usage to my users (not limit them). At the time, the quota plugin would make dovecot crash when trying to write to a folder (I can bring up this report if needed). However, in a later beta of dovecot 1.1 I tried quota again but I cannot get it to report any results. I've been wanting to use fs:noenforcing but also tried just quota = fs to simplify. When I right click a folder stored in my homedir, thunderbird tells me 'the server does not support quotas' in the folder properties. What else can I do? Thanks! IModule loaded: /usr/local/lib/dovecot/imap/lib10_quota_plugin.so IModule loaded: /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota add storage dir = /home/mcdouga9/Maildir Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota block device = cypher:/vol/mail/home Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota mount point = /home gribble# quota -v mcdouga9 | grep home /home 10766672 20480000 20480000 963778 4294967295 4294967295 # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/de list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat

1 1

[Dovecot] mail_chroot problems / dovecot 1.1rc4
by Michal Soltys 05 May '08

05 May '08

Hello I've noticed a little strange behaviour of that option. For example with following settings: system user: admin, with its home directory as /home/admin dovecot options: as reported by dovecot -n (in attachment) dovecot will try to chroot into /home/home/admin with the following message in logs, in my case: Fatal: chdir(/home/home/admin) failed with uid 1999: No such file or directory The same happens if I use per-user chroot= option in userdb, f.e. in passwd-file I've noticed it's just now, as I've always used explicit chroot dirs specified through passwd-file with /./ (which works perfectly fine, except harmless double slashes in logs), and didn't bother with single system user (I made some chroot/lock related bugreports in distant 1.0rcXX past). Cheers Michal # 1.1.rc4: /etc/dovecot.conf base_dir: /var/dovecot/ protocols: imap imaps pop3 pop3s ssl_listen: * ssl_ca_file: /etc/ssl/cert_bundle.pem ssl_cert_file: /etc/ssl/ca_ppgk/certs/fetch_crt.pem ssl_key_file: /etc/ssl/ca_ppgk/private/fetch_key.pem verbose_ssl: yes login_dir: /var/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_greeting: Dovecot IMAP server ready. login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_process_size: 32 valid_chroot_dirs: /home mail_chroot: /home verbose_proctitle: yes first_valid_uid: 1999 first_valid_gid: 10 mail_location: maildir:~/mspace/maildir:INDEX=~/mspace/index:INBOX=~/mspace/inbox:CONTROL=~/mspace/control mail_debug: yes fsync_disable: yes lock_method: flock mbox_read_locks: flock mbox_write_locks: flock mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail tb-extra-mailbox-sep imap_client_workarounds(imap): outlook-idle delay-newmail tb-extra-mailbox-sep imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh namespace: type: private separator: / inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login verbose: yes debug: yes passdb: driver: passwd-file args: /etc/dovecot.z1.passwd passdb: driver: passwd-file args: /etc/dovecot.passwd passdb: driver: passwd userdb: driver: passwd-file args: /etc/dovecot.z1.passwd userdb: driver: passwd-file args: /etc/dovecot.passwd userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: quota: maildir

2 4

[Dovecot] dovecot 1.1.rc3 assertion failed at index_mailbox_set_recent_uid while deleting message with thunderbird.
by Diego Liziero 05 May '08

05 May '08

To some users happens this assertion failure while deleting a message. dovecot: Mar 10 08:40:44 Panic: IMAP(user): file index-sync.c: line 39 (index_mailbox_set_recent_uid): assertion failed: (seq_range_exists (&ibox->recent_flags, uid)) dovecot: Mar 10 08:40:44 Error: IMAP(user): Raw backtrace: [see bleow] dovecot: Mar 10 08:40:44 Error: child 17683 (imap) killed with signal 6 And the message doesn't get deleted. Here one of the backtraces (gdb) bt #0 0x00ae9402 in __kernel_vsyscall () #1 0x00725ba0 in raise () from /lib/libc.so.6 #2 0x007274b1 in abort () from /lib/libc.so.6 #3 0x080ce02d in i_internal_fatal_handler (type=LOG_TYPE_PANIC, status=0, fmt=0x80e1750 "file %s: line %d (%s): assertion failed: (%s)", args=0xbfd8c6a4 "=???\016\b'") at failures.c:424 #4 0x080cdc3c in i_panic (format=0x80e1750 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:187 #5 0x0809bb4a in index_mailbox_set_recent_uid (ibox=0x88ca768, uid=1041) at index-sync.c:39 #6 0x0808085f in mbox_sync_loop (sync_ctx=0xbfd8c8d0, mail_ctx=0xbfd8cac4, partial=true) at mbox-sync.c:457 #7 0x08081361 in mbox_sync (mbox=0x88ca768, flags=18) at mbox-sync.c:1504 #8 0x08079cf9 in mbox_transaction_commit (t=0x88c5550, log_file_seq_r=0xbfd8cbc8, log_file_offset_r=0xbfd8cbc0) at mbox-transaction.c:45 #9 0x0809c69e in index_transaction_commit (_t=0x88cb1c0, uid_validity_r=0xbfd8cc48, first_saved_uid_r=0xbfd8cc44, last_saved_uid_r=0xbfd8cc40) at index-transaction.c:105 #10 0x0805b0cf in cmd_copy (cmd=0x88b75c0) at cmd-copy.c:141 #11 0x0805f079 in cmd_uid (cmd=0x88b75c0) at cmd-uid.c:26 #12 0x0805f9a9 in client_command_input (cmd=0x88b75c0) at client.c:546 #13 0x0805fa3e in client_command_input (cmd=0x88b75c0) at client.c:595 #14 0x080601c5 in client_handle_input (client=0x88b7368) at client.c:636 #15 0x080603de in client_input (client=0x88b7368) at client.c:691 #16 0x080d5ba0 in io_loop_handler_run (ioloop=0x88b59b0) at ioloop-epoll.c:201 #17 0x080d4e28 in io_loop_run (ioloop=0x88b59b0) at ioloop.c:301 #18 0x08067dbc in main (argc=Cannot access memory at address 0x413a ) at main.c:293 (gdb) bt full #0 0x00ae9402 in __kernel_vsyscall () No symbol table info available. #1 0x00725ba0 in raise () from /lib/libc.so.6 No symbol table info available. #2 0x007274b1 in abort () from /lib/libc.so.6 No symbol table info available. #3 0x080ce02d in i_internal_fatal_handler (type=LOG_TYPE_PANIC, status=0, fmt=0x80e1750 "file %s: line %d (%s): assertion failed: (%s)", args=0xbfd8c6a4 "=???\016\b'") at failures.c:424 backtrace = 0x88add50 "/usr/libexec/dovecot/imap [0x80ce024] -> /usr/libexec/dovecot/imap [0x80cdc3c] -> /usr/libexec/dovecot/imap [0x809bb4a] -> /usr/libexec/dovecot/imap [0x808085f] -> /usr/libexec/dovecot/imap(mbox_sync+"... #4 0x080cdc3c in i_panic (format=0x80e1750 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:187 args = 0xbfd8c6a4 "=???\016\b'" #5 0x0809bb4a in index_mailbox_set_recent_uid (ibox=0x88ca768, uid=1041) at index-sync.c:39 __PRETTY_FUNCTION__ = "index_mailbox_set_recent_uid" #6 0x0808085f in mbox_sync_loop (sync_ctx=0xbfd8c8d0, mail_ctx=0xbfd8cac4, partial=true) at mbox-sync.c:457 mail = {uid = 268, idx_seq = 133265, keywords = {arr = {buffer = 0x88d3770, element_size = 3218655176}, v = 0x88d3770, v_modifiable = 0x88d3770}, flags = 172 '???', uid_broken = 1, expunged = 0, pseudo = 1, from_offset = 143457960, body_size = 579466361970685888, offset = 143456608, space = 578778797692682360} rec = (const struct mail_index_record *) 0xb7e6f008 uid = 1041 messages_count = 3518 offset = 563 ret = <value optimized out> expunged = false skipped_mails = false uids_broken = false #7 0x08081361 in mbox_sync (mbox=0x88ca768, flags=18) at mbox-sync.c:1504 ret = <value optimized out> #8 0x08079cf9 in mbox_transaction_commit (t=0x88c5550, log_file_seq_r=0xbfd8cbc8, log_file_offset_r=0xbfd8cbc0) at mbox-transaction.c:45 mt = (struct mbox_transaction_context *) 0x88cb1c0 mbox = (struct mbox_mailbox *) 0x88ca768 lock_id = 3 ret = 0 #9 0x0809c69e in index_transaction_commit (_t=0x88cb1c0, uid_validity_r=0xbfd8cc48, first_saved_uid_r=0xbfd8cc44, last_saved_uid_r=0xbfd8cc40) at index-transaction.c:105 itrans = (struct mail_index_transaction *) 0x0 seq = 7 offset = 20544 #10 0x0805b0cf in cmd_copy (cmd=0x88b75c0) at cmd-copy.c:141 client = (struct client *) 0x88b7368 storage = (struct mail_storage *) 0x88b6c80 destbox = (struct mailbox *) 0x88ca768 t = (struct mailbox_transaction_context *) 0x0 search_arg = <value optimized out> messageset = 0x88bb6c0 "224" mailbox = 0x88bb6c8 "Trash" msg = <value optimized out> sync_flags = <value optimized out> imap_flags = <value optimized out> copy_count = 1 uid_validity = <value optimized out> uid1 = <value optimized out> uid2 = <value optimized out> ret = 1 __PRETTY_FUNCTION__ = "cmd_copy" #11 0x0805f079 in cmd_uid (cmd=0x88b75c0) at cmd-uid.c:26 command = (struct command *) 0x6 cmd_name = 0x88bb6b8 "copy" #12 0x0805f9a9 in client_command_input (cmd=0x88b75c0) at client.c:546 client = (struct client *) 0x88b7368 command = <value optimized out> __PRETTY_FUNCTION__ = "client_command_input" #13 0x0805fa3e in client_command_input (cmd=0x88b75c0) at client.c:595 client = (struct client *) 0x88b7368 command = (struct command *) 0x6 __PRETTY_FUNCTION__ = "client_command_input" #14 0x080601c5 in client_handle_input (client=0x88b7368) at client.c:636 _data_stack_cur_id = 3 ret = <value optimized out> remove_io = <value optimized out> handled_commands = false #15 0x080603de in client_input (client=0x88b7368) at client.c:691 cmd = <value optimized out> output = (struct ostream *) 0x88b74ec bytes = 24 __PRETTY_FUNCTION__ = "client_input" #16 0x080d5ba0 in io_loop_handler_run (ioloop=0x88b59b0) at ioloop-epoll.c:201 ctx = <value optimized out> event = (const struct epoll_event *) 0x88b5ae8 list = (struct io_list *) 0x88b7568 io = (struct io_file *) 0x88c0dd0 tv = {tv_sec = 1799, tv_usec = 999737} t_id = 2 msecs = <value optimized out> ret = 1 i = 0 j = 0 call = <value optimized out> #17 0x080d4e28 in io_loop_run (ioloop=0x88b59b0) at ioloop.c:301 No locals. #18 0x08067dbc in main (argc=Cannot access memory at address 0x413a ) at main.c:293 No locals. Regards, Diego.

2 6

[Dovecot] Users w/o acl access appear to be subscribed to public folders (1.1b8)
by Adam McDougall 05 May '08

05 May '08

I noticed this today, I had a user outside of our department test out dovecot. They were using squirrelmail and I noticed that dovecot thinks this user is subscribed to ALL public folders even though a dovecot ACL prevents all access. I'm pretty sure access is still denied. I was able to reproduce this with a guest account I added: l lsub "" "#shared/decs/%" * LSUB (\Noselect) "/" "#shared/decs/linuxadmin" * LSUB (\Noselect) "/" "#shared/decs/jbossadmin" * LSUB () "/" "#shared/decs/support" * LSUB () "/" "#shared/decs/receipts" * LSUB (\Noselect) "/" "#shared/decs/pcadmin" * LSUB () "/" "#shared/decs/network" * LSUB (\Noselect) "/" "#shared/decs/printmaster" * LSUB () "/" "#shared/decs/postmaster" * LSUB (\Noselect) "/" "#shared/decs/unixadmin" * LSUB () "/" "#shared/decs/security" * LSUB (\Noselect) "/" "#shared/decs/webmaster" l OK Lsub completed. This only seems to happen when the acl plugin is enabled. Without the acl plugin, these are not listed as subscriptions. After deleting /egr/mail/shared/decs/dovecot-acl-list and re-enabling the acl plugin, I get this: l lsub "" "#shared/decs/%" * LSUB () "/" "#shared/decs/unixadmin" * LSUB () "/" "#shared/decs/support" * LSUB () "/" "#shared/decs/security" * LSUB () "/" "#shared/decs/printmaster" * LSUB () "/" "#shared/decs/postmaster" * LSUB () "/" "#shared/decs/pcadmin" * LSUB () "/" "#shared/decs/network" * LSUB () "/" "#shared/decs/linuxadmin" * LSUB () "/" "#shared/decs/webmaster" * LSUB () "/" "#shared/decs/jbossadmin" l OK Lsub completed. Is it related, or is it different just because a new dovecot-acl-list got created by another user already (but is mode 700?)

2 3

[Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
by Adam McDougall 04 May '08

04 May '08

I was trying to debug this with gdb, but I'm not sure how to make env MAIL=maildir:~/Maildir MAIL_PLUGINS=acl ACL=vfile gdb /tmp/imap understand the #shared/decs namespace from below so I can SELECT it. I have a number of maildirs under #ahared/decs, I tried opening at least 2 of them but dovecot gives sig 11 without a direct error about the problem. I don't think this was exactly the same problem in 1.1b3, but unfortunately I deleted all the patches I was testing with so it would be a pain to test the same situation with 1.1b3 again (although if needed, I can). If dovecot-shared exists in the folder I try to open, dovecot says: Oct 27 12:57:38 gribble dovecot: IMAP(mcdouga9): fchown() failed with file /home/mcdouga9/Maildir/dovecot11testing/public/indexes/decs/.unixadmin/dovecot.index.log: Operation not permitted Oct 27 12:57:38 gribble dovecot: child 55470 (imap) killed with signal 11 I am used to getting the first message because dovecot tries to apply perms from dovecot-shared to the index files which are elsewhere. If I open a public folder without dovecot-shared, I see no contents but it does not sig11. I am NOT using the acl plugin right now, it didn't seem to make a difference if it was loaded. When I left off with the dovecot 1.1b3 + 8 patches, I believe I was getting an assertion. Please let me know if I goofed up something, I have a strange feeling about that here, or if you need more info. I'd like to go ahead and get a gdb trace. # 1.1.beta4: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/cert.pem ssl_key_file: /usr/local/etc/apache2/ssl/cert.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 0 mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot11testing/public/control/decs:INDEX=%h/Maildir/dovecot11testing/public/indexes/decs list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes hidden: yes subscriptions: yes namespace: type: private separator: / inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls

2 18

[Dovecot] UID inserted in the middle of mailbox errors
by Dean Brooks 04 May '08

04 May '08

Hi, We are still getting, on a daily basis, users who cannot move messages to Trash or expunge Trash due to these errors: Mar 28 10:43:57 tm2 dovecot: [ID 107833 mail.error] IMAP(kss021): mbox sync: UID inserted in the middle of mailbox /home/kss021/Mail/Trash (3137 > 536, seq=2, idx_msgs=76) Our users have no other access to their mailbox other than Dovecot, and we are using mbox format and 1.1rc3. Deleting the indexes fixes the problem, so it seems to be a problem with index corruption and not something inherently wrong with the mailbox. That is, once I delete the index files, the problem does not come back immediately. I've seen others on the list with the same problem, but I wasn't able to find a workaround. It's happening more and more frequently. I guess what I don't understand is why Dovecot cant just mark the index invalid and purge it instead of failing over and over. I thought the whole point behind Dovecot's indexing is that it was able to detect index inconsistencies and mark the index invalid. Any help is much appreciated. Even a quick stopgap fix would be a lifesaver, short of a cronjob that just purges everyone's index files every 2 hours (which sort of defeats the point of using Dovecot). -- Dean Brooks dean(a)iglou.com

3 4

[Dovecot] Embedded From_ lines breaking Content-Length (and Dovecot)
by Lennart Lövstrand 04 May '08

04 May '08

I feel like I'm going totally crazy. Is it just me, or have embedded From_ lines really been breaking mbox messages since (at least) dovecot 1.0? I found a whole lot of broken messages in an old mailbox of mine and when I looked closer at it, it seemed like Dovecot was ignoring the Content-Length header and truncating the messages at the first embedded From_ line instead. Figuring that this probably just was a fluke / old bug, I erased the index and upgraded Dovecot, first to 1.0.10, then to 1.1.beta14, but I'm still seeing the bug -- and it's even gotten worse! Earlier versions would just quietly split the message at the From_ line, but 1.1.beta14 is more diligent and will complain with: > dovecot[44795]: IMAP(lennart): FETCH for mailbox Drafts UID 196 got > too little data: 590 vs 719 before dropping the connection and exiting... which cause Thunderbird to reopen the connection and reissue the same request... which will cause Dovecot to complain and exit again... etc ad nauseam... It's trivial to reproduce too -- just mail yourself a message with a valid From_ line in it (assuming that your delivery system isn't doing >From-escaping), or put it in a draft plain text message and save it. Then go and look in your Drafts folder... (Like this one: "From someone Tue Jan 29 03:15:52 2008") Just be sure that you can stop your spinning mail client and/or server if you're using 1.1.beta14... --- OK, I am definitely not very familiar with the Dovecot source, but I spent some time trying to track this down today and it looks to me like i_stream_raw_mbox_read is being asked to find the end of the message (at the From_ line) before mbox_sync_parse_next_mail has had a chance to parse the Content-Length header and find out how big it is. Maybe someone who knows the code better can take a look at it and tell me if I'm barking up the totally wrong tree... Thanks, --Lennart

3 5

[Dovecot] Mailbox locking issue?
by Mark Sapiro 04 May '08

04 May '08

Here's my configuration. dovecot -n # 1.0.7: /usr/local/etc/dovecot.conf protocols: pop3 ssl_disable: yes disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/pop3-login mail_extra_groups: mail mail_location: /var/spool/mail/%u mbox_write_locks: fcntl dotlock The above setting was just added in an attempt to resolve the issue. It was default before. mail_executable: /usr/local/libexec/dovecot/pop3 mail_plugin_dir: /usr/local/lib/dovecot/pop3 pop3_uidl_format: %08Xu%08Xv auth default: mechanisms: plain apop login worker_max_count: 5 passdb: driver: passwd-file args: /usr/local/etc/dovecot.passwd passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix Postfix 2.3.3 mailbox_delivery_lock = fcntl, dotlock The problem is the following. A user uses fetchmail via an ssh tunnel to download mail via pop3 every 5 minutes. This normally works fine. perhaps once a week or so, maillog shows that Postfix delivers a message while fetchmail is retrieving. Here is one such. Apr 8 19:07:20 sbh16 dovecot: pop3-login: Login: user=<xxx>, method=PLAIN, rip=72.52.113.36, lip=72.52.113.36, secured Apr 8 19:07:22 sbh16 postfix/local[13745]: D959169046B: to=xxx(a)example.com>, orig_to=<xxx(a)example.org>, relay=local, delay=16, delays=15/0/0/1.1, dsn=2.0.0, status=sent (delivered to mailbox) Apr 8 19:07:22 sbh16 dovecot: POP3(xxx): Disconnected: Logged out top=2/15995, retr=0/0, del=2/2, size=15985 Five minutes later when fetchmail comes back, we see Apr 8 19:12:42 sbh16 dovecot: POP3(xxx): mbox sync: Expunged message reappeared in mailbox /var/spool/mail/xxx (UID 4683 < 4684, seq=2, idx_msgs=0) Apr 8 19:12:42 sbh16 dovecot: pop3-login: Login: user=<xxx>, method=PLAIN, rip=72.52.113.36, lip=72.52.113.36, secured Apr 8 19:12:46 sbh16 dovecot: POP3(xxx): mbox sync: UID inserted in the middle of mailbox /var/spool/mail/xxx (4684 > 4683, seq=2, idx_msgs=1) Apr 8 19:12:46 sbh16 dovecot: POP3(xxx): Disconnected: Logged out top=1/46375, retr=0/0, del=1/1, size=46365 >From now on, as messages are added to the mailbox by Postfix, fetchmail keeps retrieving both old and new messages. i.e. even though dovecot's log says del 1/1 (or del n/n) no messages are actually deleted from the mailbox. This situation continues until someone accesses the mailbox locally (e.g. via mutt through a local shell) and deletes the old messages. As noted above, this has been observed a few times with dovecot's default mbox_write_locks. I have since changed that to mbox_write_locks: fcntl dotlock to be consistent with Postfix's mailbox_delivery_lock = fcntl, dotlock I don't know if that will fix the problem or not, but it seems the issue with default dovecot mbox_write_locks should have been deadlock and time out rather than what we see. Also it seems that dovecot should recover from the 'corrupt' mailbox and fix it rather than just continuing to encounter the same problem. Any ideas or advice? -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 4