- dovecot - dovecot.org

[Dovecot] Problems with zlib
by Andre 06 Sep '10

06 Sep '10

Sometimes I see lines like these in my log: 20100906 11:34:32 imap(username(a)domain.com): Panic: file istream-zlib.c: line 173 (i_stream_zlib_read): assertion failed: (zstream->high_pos == 0) 20100906 11:34:32 imap(username(a)domain.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0x7f328d2244a2] -> /usr/lib/dovecot/libdovecot.so.0 [0x7f328d22450a] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f328d2248b3] -> /usr/lib/dovecot/modules/lib20_zlib_plugin.so [0x7f328be444fc] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x6b) [0x7f328d22a10b] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x4d) [0x7f328d22a3fd] -> /usr/lib/dovecot/libdovecot.so.0 [0x7f328d21476e] -> /usr/lib/dovecot/libdovecot.so.0 [0x7f328d214cbe] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x29) [0x7f328d2148c9] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x23) [0x7f328d214a43] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x7f328d4cb72f] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x7f328d4cb84a] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_special+0x3dc) [0x7f328d4cc99c] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x7f328d4dd86d] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID] [0x41256a] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID](imap_fetch_more+0x16c) [0x412b0c] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID](cmd_fetch+0x3fa) [0x40acea] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID] [0x41031c] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID] [0x4103cd] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID](client_handle_input+0x45) [0x410545] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID](client_input+0x62) [0x410f72] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xcb) [0x7f328d22f3ab] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x18) [0x7f328d22e4c8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f328d21dd63] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID](main+0x281) [0x419201] -> /lib/libc.so.6(__libc_start_main+0xe6) [0x7f328caa81a6] -> dovecot/imap [username(a)domain.com 192.168.4.253 UID] [0x408bd9] Here is the backtrace: [New process 11474] #0 0x00007f328cabbed5 in raise () from /lib/libc.so.6 (gdb) bt #0 0x00007f328cabbed5 in raise () from /lib/libc.so.6 #1 0x00007f328cabd3f3 in abort () from /lib/libc.so.6 #2 0x00007f328d2244b5 in ?? () from /usr/lib/dovecot/libdovecot.so.0 #3 0x00007f328d22450a in ?? () from /usr/lib/dovecot/libdovecot.so.0 #4 0x00007f328d2248b3 in i_panic () from /usr/lib/dovecot/libdovecot.so.0 #5 0x00007f328be444fc in ?? () from /usr/lib/dovecot/modules/lib20_zlib_plugin.so #6 0x00007f328d22a10b in i_stream_read () from /usr/lib/dovecot/libdovecot.so.0 #7 0x00007f328d22a3fd in i_stream_read_data () from /usr/lib/dovecot/libdovecot.so.0 #8 0x00007f328d21476e in ?? () from /usr/lib/dovecot/libdovecot.so.0 #9 0x00007f328d214cbe in ?? () from /usr/lib/dovecot/libdovecot.so.0 #10 0x00007f328d2148c9 in message_parser_parse_next_block () from /usr/lib/dovecot/libdovecot.so.0 #11 0x00007f328d214a43 in message_parser_parse_body () from /usr/lib/dovecot/libdovecot.so.0 #12 0x00007f328d4cb72f in ?? () from /usr/lib/dovecot/libdovecot-storage.so.0 #13 0x00007f328d4cb84a in ?? () from /usr/lib/dovecot/libdovecot-storage.so.0 #14 0x00007f328d4cc99c in index_mail_get_special () from /usr/lib/dovecot/libdovecot-storage.so.0 #15 0x00007f328d4dd86d in ?? () from /usr/lib/dovecot/libdovecot-storage.so.0 #16 0x000000000041256a in ?? () #17 0x0000000000412b0c in imap_fetch_more () #18 0x000000000040acea in cmd_fetch () #19 0x000000000041031c in ?? () #20 0x00000000004103cd in ?? () #21 0x0000000000410545 in client_handle_input () #22 0x0000000000410f72 in client_input () #23 0x00007f328d22f3ab in io_loop_handler_run () from /usr/lib/dovecot/libdovecot.so.0 #24 0x00007f328d22e4c8 in io_loop_run () from /usr/lib/dovecot/libdovecot.so.0 #25 0x00007f328d21dd63 in master_service_run () from /usr/lib/dovecot/libdovecot.so.0 #26 0x0000000000419201 in main () here is the relevant plugin section: plugin { zlib_save = gz zlib_save_level = 6 } What can I do to investigate deeper? TIA Andre

2 1

Re: [Dovecot] dovecot Digest, Vol 89, Issue 27
by Alan Brown 06 Sep '10

06 Sep '10

> > Timo Sirainen <tss(a)iki.fi> Wrote: > > > > > On Mon, 2010-09-06 at 14:26 +0100, Alan Brown wrote: > > > > Is there any way of enforcing a Maildir per-folder message limit in > > > > Dovecot? > > > > > No. > > > > Would you consider it as a feature request? > What if you just create a script that automatically archives older mails to <box>/archive/yyyy-mm mailboxes when number of messages in a mailbox increases too high for too long? Or something like that. I'm looking at doing that anyway, however it would be much easier to train the users if the server says "folder full"

1 0

Re: [Dovecot] dovecot Digest, Vol 89, Issue 25
by Alan Brown 06 Sep '10

06 Sep '10

Timo Sirainen <tss(a)iki.fi> Wrote: > On Mon, 2010-09-06 at 14:26 +0100, Alan Brown wrote: > > Is there any way of enforcing a Maildir per-folder message limit in > > Dovecot? > No. Would you consider it as a feature request? > > We're finding major performance hits once a threshold of files/directory > > is exceeded. This applies across all common filesystems with the > > threshold differing depending on the FS. GFS2 pretty much _STOPS_ for 10 > > minutes when users put 65k+ messages in one folder... > mdbox would probably work nicely. :) Yes, it probably would, however I have several layers of resistance to get past in order to implement that (ie, up to 2 years to get approval to switch to dbox) > > I'm currently running 1.1.20 but looking to update (RHEL5 cluster > > installation) > With v1.2+ you could also try maildir_very_dirty_syncs=yes That won't help much for initial directory opens, which is where most of the pain is. Past that point, the index of ~/Maildir/.folder/cur is mostly cached until you hit about 16,000 messages(files) on GFS2 and some people are insisting on 32k+ files in one folder with the odd few trying 65k or more (I'm surprised their clients don't break). For info sake: we have Dovecot running on a pair of dedicated dual E5530 (quad-core X86 1300MHZ FSB cpu) servers (RH cluster+GFS2) with 24Gb RAM apiece and ~2TB of 8Gb/s san-attached storage available. There are only ~250 accounts but there's around 600Gb in the maildir folder areas and another 300Gb in a separate dedicated inbox area - and I believe there's another 1TB of stuff in client local folders we're trying to pull back to the servers in order to simplify backups. Even with this kind of horsepower available, over-large folders can render the setup completely unresponsive for minutes at a time. I know this is a case of "work smarter, not harder", but I'm facing major delays until we're able to change the on-disk storage format. Alan

2 1

Re: [Dovecot] dovecot Digest, Vol 89, Issue 26
by Alan Brown 06 Sep '10

06 Sep '10

Timo Sirainen <tss(a)iki.fi> Wrote: >On Mon, 2010-09-06 at 12:37 +0100, William Blunn wrote: > > Whilst documenting LAYOUT=maildir++ under dbox, that got me thinking: > > > > Can we specify :LAYOUT=maildir++ with mbox? > Yes, ever since LAYOUT was added. > > If I have it right, this should then remove the problem of not being > > able to have messages and mail subfolders in the same mail folder. > Yes. Also it might be possible to use DIRNAME with LAYOUT=fs to solve > that problem, but I'm less sure about that. I anyway doubt most mbox > users want to switch to a whole new layout. Mbox users might not, but Maildir or dbox users might want to, especially if users have a lot of folders (One of mine has ~2500) Directories definitely get slower to list with more files/subdirectories in them and there are knee points beyond which they get substantially slower (Where the knee is depends on the type of FS but it's usually a power of 2)

1 0

[Dovecot] mdbox: Cannot create subfolder called "dbox-Mails" (2.0beta5)
by William Blunn 06 Sep '10

06 Sep '10

I am trying out "mdbox" under Dovecot 2.0beta5. Looking in the "mailboxes" directory under the mdbox storage root ("~/dbox" in my case), I can see that the mail folders are mapped into filesystem directories. But Dovecot seems to put all the message list information ("dovecot.index.cache", "dovecot.index.log") for any given mail folder into a directory called "dbox-Mails"... But this directory name exists in the same namespace as that used for mail subfolders... So this means we cannot create a mail subfolder whose name is "dbox-Mails"! I concede that this eventuality may be unlikely, but when designing in a green field, surely we should avoid such obvious pitfalls? Surely we would want to make it so that these objects exist in different namespaces, so that mail folders can be created with any name? So perhaps the directory "dbox-Mails" should use a character in its name which could not be used in mail folder names. Or mail folder names could have a non-identity mapping into directory names such that the directory name can never be "dbox-Mails". Regards, Bill

4 14

[Dovecot] Specify auth service for protocols
by Lukas Garberg 06 Sep '10

06 Sep '10

Hi all, I'm building a mail system where two machines are functioning both as POP3/IMAP-proxy (using dovecot 1.2.12) and SMTP-server with authentication for outgoing mail. The SMTP-server is postfix with dovecot SASL as authentication backend. User credentials for SMTP auth as well as the mapping from account to final server for the proxy is stored in a MySQL-database which dovecot queries. As I want the POP3/IMAP-proxy to accept any username and password (as the final POP3/IMAP server takes care of the authentication) the passdb SQL query has the statements "NULL AS password, 'Y' AS nopassword, 'Y' AS proxy" added to the query. As these additions cannot be used in the SQL query for the auth service used by postfix (as users would be able to authenticate with any password) I have defined a second auth service (auth postfix { ... }) in dovecot.conf which is slightly different; it returns the password hash instead of NULL as password and skips the proxy and nopasswd part. This works fine on one server, but on the other one the auth services get mixed up and the service defined for postfix (missing password and proxy stuff) is used by the POP3 and IMAP services and vice versa, which breaks the proxying. I've been trying my best to find any difference between the configuration of the two machines, but without result. The configurations are simply copies of each other. So, to finally arrive at my question: Is there any way to specify which authentication service is to be used by each protocol in dovecot.conf? Or is the solution instead to rewrite the passdb-query to take the service into account and return 'Y' AS proxy when the service is postfix and 'N' AS proxy when the service is POP3 or IMAP as I then would be able to use only one authentication backend? Sorry for the lengthy post and thank you in advance! Regards, Lukas Garberg

2 2

[Dovecot] notification plugin
by Wolfgang Rosenauer 06 Sep '10

06 Sep '10

Hi, I'm pretty new to dovecot's codebase and trying to write a plugin which performs certain plugins if new mail arrives in a user's mailbox. I saw the mail-log plugin and was playing around with it but it seems to not create events if new mails arrive but only if mail are copied or saved from an IMAP connection. Are there any hooks I can base my plugin on to get events in the above case? Any pointers? Thanks, Wolfgang

2 3

[Dovecot] Quotas - per folder message limits?
by Alan Brown 06 Sep '10

06 Sep '10

Is there any way of enforcing a Maildir per-folder message limit in Dovecot? We're finding major performance hits once a threshold of files/directory is exceeded. This applies across all common filesystems with the threshold differing depending on the FS. GFS2 pretty much _STOPS_ for 10 minutes when users put 65k+ messages in one folder... I'm currently running 1.1.20 but looking to update (RHEL5 cluster installation)

2 1

[Dovecot] Documentation for "altpath" / "altmove" / ?"Alternate storage"
by William Blunn 06 Sep '10

06 Sep '10

I was thinking about documentation for alternate storage. We have a few mentions at: http://wiki2.dovecot.org/MailboxFormat/dbox http://wiki2.dovecot.org/Tools/Doveadm http://wiki2.dovecot.org/Tools/Doveadm/Altmove But I was thinking it would be helpful if there could be a page about alternate storage. I could write the page, but I currently have so many unknowns that I think it would be better to gather some more information before writing the first draft. Note that I don't necessarily fully understand all the surrounding concepts, so I could have written something below which is based on a (vague) belief. Please correct me if I am wrong, or even if it's sort-of right but doesn't quite hit the point squarely. We might want to have a name for the wiki page. Perhaps "AlternateStorage"? Of course if it only applies to dbox, then another idea is to make it a section on the page "MailboxFormat/dbox". All suggestions gratefully received. I would be interested in an overview of what alternate storage is. It seems to be a way of transparently moving message data to alternare storage, with the idea that the alternate storage may be on a different filesystem which may be cheaper and/or slower than the main storage, which may be an economic way to store messages where the fastest access to them is less important. It is transparent in as much as IMAP/POP users will not normally be able to tell if any given message has been moved to alternate storage. A single mail folder can end up containing a mixture of messages stored in main storage and alternate storage. (Another question begged here of what consistent term should be used to refer to the main storage. Could be "main storage", "normal storage", "ordinary storage".) Also I would be interested to know the development/stability status of alternate storage: experimental / alpha / beta / stable. Also I would be interested to know the applicability against mailbox formats: mbox / maildir / sdbox / mdbox / cydir. Also I would be interested to know which data gets moved to the alternate storage, and which data stays in the main storage: message-data / control-data / index-data / combined-control-and-index-data. Also I would be interested to know how data can be moved to alternate storage. Seems to be that is only done by invocation of "doveadm altmove". There is a page for that already so we can link to that. Also it would be interesting to have some idea of how it works. For example there might be an "alternate storage" flag in the indexes, or Dovecot tries the main location first and if not found there then it tries the alternate storage. Just a brief overview, and any pertinent ramifications of that. Bill

2 4

Re: [Dovecot] \" character in folder name results in strange LIST
by Samuel Kvasnica 06 Sep '10

06 Sep '10

Hi Timo, >> * LIST (\HasNoChildren) "/" "folder1" >> * LIST (\HasNoChildren) "/" "folder2" >> * LIST (\HasNoChildren) "/" {9} >> six"wafer >> . OK List completed. >> >> Note the {9} length of the following real folder name. Is this normal >> handling of special folder names ? >Yes. Well, I did try other servers and e.g. on scalix I'm getting this: * LIST (\X-DirectRef=000d7f94a4731510 \X-ModDate=20100814200222) "/" "six\"wafer" I cannot find anything relevant in RFC3501 >> Mail clients do not seem to like this response. >What clients? They're buggy then.. Well, of course, way too many clients are :( at least Thunderbird will go totally crazy when getting this kind response. (My old python scripts expecting string, not a tuple go crazy as well... :-< )

5 26