- dovecot - dovecot.org

File locking issues
by Oliver Welter 16 Jan '15

16 Jan '15

Hi All, I upgrade my mail server from doveot 1.1.7 to 2.2.13 and encounter problems with file locking issues. The server has around 400 clients using IMAP and I get tons of "Warning: fscking index file .. dovecot.index" and "Error: Corrupted transaction log". Sometimes the transaction log problem is followed by "Panic: file mail-transaction-log.c: line 350 (mail_transaction_log_set_mailbox_sync_pos): assertion failed: (file_offset >= log->head->saved_tail_offset)" I tried all offered locking methods (fcntl, flock, dotlock) without success. The server runs inside a virtual guest (linux-vserver which is similar to LXC) using an overlay filesystem. I suspect the overlayfs but have no idea how to test this, besides - as I understand - dotlock should not rely on the fs locking at all :( I even wonder why the old dovecot worked for nearly 10 years without any problems, even under higher loads. The config is pretty straightforward: # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.4.57-vs2.3.3.9-g67e6589-dirty x86_64 Gentoo Base System release 2.2 auth_cache_negative_ttl = 0 auth_cache_size = 50 M auth_mechanisms = plain cram-md5 debug_log_path = /var/log/dovecot-debug.log default_client_limit = 900 default_process_limit = 250 disable_plaintext_auth = no first_valid_gid = 89 first_valid_uid = 89 info_log_path = /var/log/dovecot.log last_valid_gid = 89 last_valid_uid = 89 listen = * lock_method = dotlock login_greeting = Serverpilot.net Mail Service mail_gid = 89 mail_location = maildir:~/.maildir mail_uid = 89 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = imap pop3 service imap-login { process_min_avail = 5 service_count = 0 } ssl_cert = </var/qmail/control/servercert.pem ssl_cipher_list = ALL:!EXPORT:!LOW:!MEDIUM:!aNULL:+RC4:@STRENGTH ssl_dh_parameters_length = 2048 ssl_key = </var/qmail/control/servercert.pem ssl_protocols = !SSLv2 !TLSv1.2 userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol imap { mail_max_userip_connections = 25 } Any ideas are welcome. Oliver -- Protect your environment - close windows and adopt a penguin!

1 0

Questions regarding imaptest
by rub zorghy 15 Jan '15

15 Jan '15

Hi, The measurements were created under the following conditions : • operating system : Red Hat Enterprise Linux Server release 6.6 (Santiago) kernel in version 2.6.32-504.el6.x86_64 • virtual server (VMware) with an Intel(R) Xeon(R) 4vCPU E5649 x 2,526 GHz (2 cores per virtual socket) and 4 GB RAM • 7,200 RPM SATA 1TB (FC SAN IBM System Storage N3400) • all file systems had been formated in ext4 • Dovecot v.2.0.9 • imaptest v.1.0.0 • LDAP 389 Directory Server v.1.2.2 • free pagecache, dentries and inodes: echo 3 > /proc/sys/vm/drop_caches and services restarted before the test "Tuning" based upon the book « *Dovecot: POP3/IMAP servers for enterprises and ISPs* » Peer Heinlein : ---%<------------------------------------------ imap {} (10-master.conf). service imap { [...] process_limit = 8192 [...] } auth {} service auth { [...] client_limit = 9516 [...] } process_min_avail = 16 ---%<------------------------------------------ mail_max_userip_connections = 100 The test results shows : Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5% 324 142 128 263 299 405 92 153 320 336 335 800/800 212 92 92 182 166 233 59 87 178 201 202 800/800 - 4051 stalled for 16 secs in command: 8 APPEND "INBOX" () {3251+} 315 160 134 300 275 400 85 136 300 317 317 800/800 - 4051 stalled for 17 secs in command: 8 APPEND "INBOX" () {3251+} 247 130 163 292 251 366 99 115 239 240 240 800/800 222 122 117 234 279 383 112 147 263 222 222 800/800 188 93 94 177 168 232 65 87 162 196 195 800/800 290 126 119 260 262 383 102 126 283 282 283 800/800 240 125 107 243 259 353 102 130 253 256 256 800/800 276 158 144 291 280 400 101 149 266 261 261 800/800 254 95 89 189 235 313 78 118 243 253 253 800/800 82 829 926 861 722 593 455 628 1695 2252 4 ms/cmd avg Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5% 148 59 55 126 119 174 43 64 130 147 147 800/800 - 8770 stalled for 16 secs in command: 4 FETCH 1:2 (INTERNALDATE UID INTERNALDATE) 261 128 139 247 252 369 101 125 258 269 269 800/800 302 159 158 307 282 398 109 126 296 298 297 800/800 234 96 118 239 270 369 98 138 357 428 427 800/800 361 213 220 450 383 596 151 197 240 165 166 800/800 192 93 92 206 256 352 104 118 297 289 290 800/800 - 9806 stalled for 16 secs in command: 4 FETCH 1 (RFC822.SIZE) 245 85 72 162 128 190 52 73 157 174 173 800/800 - 9806 stalled for 17 secs in command: 4 FETCH 1 (RFC822.SIZE) 294 152 165 348 360 526 136 179 313 308 309 800/800 272 87 79 174 164 229 54 85 218 230 230 800/800 218 152 152 310 325 467 117 152 265 236 221 798/800 119 830 852 859 707 566 511 702 1672 2161 6 ms/cmd avg I have three questions : 1) how to interpret each row except the last one ? 2) how to explain the stalled commands ? 3) why benchmark shows too few read disk I/O ? Thanks so much, rub

1 1

Re: dovecot.org slow & imap search now blazing fast
by Joseph Tam 15 Jan '15

15 Jan '15

From: Laeeth Isharc <laeeth(a)laeeth.com> > In case you were not aware, I wanted to let you know that dovecot.org > has been very slow (often almost unusable) for at least several days now. Not for me as of the writring of this Email. Seems OK to me. Check your browser, network, etc. Joseph Tam <jtam.home(a)gmail.com>

1 0

HG e3640ccaa76d crashes upon IMAP SEARCH
by Thomas Leuxner 14 Jan '15

14 Jan '15

$ doveconf -n | head # 2.2.15 (e3640ccaa76d): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 $ /usr/lib/dovecot/imap -u tlx(a)leuxner.net * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL RIGHTS=texk] Logged in as tlx(a)leuxner.net a select "Shops/Amazon" * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk) * OK [PERMANENTFLAGS (\Answered \Flagged \Seen \Draft $NotJunk NotJunk \*)] Flags permitted. * 491 EXISTS * 0 RECENT * OK [UNSEEN 491] First unseen. * OK [UIDVALIDITY 1213986801] UIDs valid * OK [UIDNEXT 568] Predicted next UID * OK [HIGHESTMODSEQ 1029] Highest a OK [READ-WRITE] Select completed (0.003 secs). b search charset utf-8 body "Sleeve" imap(tlx(a)leuxner.net): Panic: file message-decoder.c: line 346 (message_decode_body): assertion failed: (ctx->translation_size <= sizeof(ctx->translation_buf)) imap(tlx(a)leuxner.net): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x7749f) [0x7ffe68e6d49f] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7ffe68e6d57a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffe68e1df65] -> /usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x688) [0x7ffe68e5c858] -> /usr/lib/dovecot/libdovecot.so.0(message_search_more+0xa1) [0x7ffe68e60881] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0x5b) [0x7ffe68e60a4b] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa9b7b) [0x7ffe69187b7b] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) [0x7ffe6915d675] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa951a) [0x7ffe6918751a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xab76a) [0x7ffe6918976a] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0xe2) [0x7ffe69189e32] -> /usr/lib/dovecot/modules/lib90_stats_plugin.so(+0x2daa) [0x7ffe66dc3daa] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x27) [0x7ffe691627a7] -> /usr/lib/dovecot/imap() [0x41da2f] -> /usr/lib/dovecot/imap() [0x41e0a2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xaa) [0x7ffe68e7f6ca] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x69) [0x7ffe68e804e9] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7ffe68e7f549] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7ffe68e7f5c8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ffe68e23433] -> /usr/lib/dovecot/imap(main+0x2a1) [0x40c591] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7ffe68a88ead] -> /usr/lib/dovecot/imap() [0x40c6f5] I tried to make it dump a core but failed... Regards Thomas

2 2

attn rick van rein (midget)
by Laeeth Isharc 14 Jan '15

14 Jan '15

From 11 Oct 2013 (Rick van Rein) "Hello, I love Dovecot, but when developing a small IMAP tool, I ran into searching behaviour can easily be optimised. Please forgive a rather detailed suggestion. This was on Dovecot 1.2.15 on Debian Squeeze. My tool? It's called "midget" and retrieves documents from an IMAP box based on their mid: or cid: identifier, as per RFC 2392. I thought this would be useful to retrieve email attachments into a remote shell environment. When using Kerberos, the credentials and a strong hint of one's mail address are already present anyway, DNS SRV does the rest. If you want to see the early code, I'll be happy to share it here." Hi Rick. Thanks for the suggested optimisation - I am not sure if it is you who I have to think for contributing towards making IMAP search fast again. In any case, if you see this I would be interested to see the source for midget if you are willing to share. If you don't want to post to the list then my email is laeeth at laeeth.com Thanks. Laeeth

1 0

dovecot.org slow & imap search now blazing fast
by Laeeth Isharc 14 Jan '15

14 Jan '15

Hi. In case you were not aware, I wanted to let you know that dovecot.org has been very slow (often almost unusable) for at least several days now. Also, I upgraded my dovecot from 2.2.10 to 2.2.15. Thank you for fixing IMAP search. I have c. 2mm messages, and it had become unusable before but it is now very fast. (I saw a reference to an optimisation on the mailing list from back in Oct 2013, so maybe that was it). Regards, Laeeth

1 0

pigeonhole ereject vs reject
by Robert Blayzor 13 Jan '15

13 Jan '15

Currently pigeonhole supports reject which would generate a NDR for each message. (If I understand the current documentation) Using Dovecot LMTP it would be more optimal to kick a 5xx back to the primary MTA to reject the delivery rather than generating more back scatter NDRs. Anyone know if this is possible now (to forgo the NDR)'s or do we need to wait for ereject ? If so, how much work is involved? -- Robert inoc.net!rblayzor http://inoc.net/

2 7

blowfish crypt
by Edgar Pettijohn 13 Jan '15

13 Jan '15

I'm setting up a new server and am planning on using blowfish crypt. This page from the wiki http://wiki2.dovecot.org/Authentication/PasswordSchemes says the password will begin with $2a$, however mine are starting $2b$. Is this wrong on the wiki or am I doing something wrong? Just want to catch this before it becomes a problem. Thanks, Edgar

1 1

dovecot and glusterfs
by Michael Schwartzkopff 13 Jan '15

13 Jan '15

Hi, I did some experiments with dovecot on a glusterfs on the active nodes without a director. So I had concurrent access to the files. With the help of the available documentation about NFS and fcntl locks I managed to find out the following: With the plain mbox format dovecot seems to apply and to honor the fcntl locks. But since this format is not used any more in real setups, it is useless. With mdbox and maildir format I could reliably crash my mail storage just by delivering mails to the both dovecots via LMTP to the same user. In maildir dovecot seems not the set / respect the fnctl locks of the index file. dotlocks do not seems to work either with mdbox. So I think the only solution os to use a director in a real world setup. Or is there any non-obvious trick that I did not check? I think it would be really nice if dovecot could use glusterfs for realy scalable postbox servers. Mit freundlichen Grüßen, Michael Schwartzkopff -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

4 6

Proxying when src and dest on non standard ports
by Thomas HUMMEL 13 Jan '15

13 Jan '15

Hello, I want to test dovecot proxying, on a user basis, in a simple set up (proxy, no proxy_maybe and forward the passwd to the remote server). This seem pretty simple except that, in order to test it alongside the production server, I set up 3 servers : . the proxy, listening to pop and imap on non standard ports 9143, 9993, 9110, 9995 . a proxy destination for some users, listening on 10143, 10993, 109110, 10995 . another proxy destination for some other users, listening on 20145, 20993, 209110, 20995 But there seems to be only one 'port' extra_field, so how can I handle the proxying of both IMAP and POP3 (especially for the same user who can use both) ? Thanks. -- Thomas Hummel | Institut Pasteur <hummel(a)pasteur.fr> | Groupe Exploitation et Infrastructure

1 0