- dovecot - dovecot.org

Segmentation fault on doveadm search -A with a huge user base
by Heiko Schlittermann 12 Feb '16

12 Feb '16

Hi, I'm using dovecot 2.2.9 with a director/backend setup. The user base is about 4711 users currently. If I start at one of the directors doveadm search -A all savedbefore 5000d it terminates with doveadm(1rrissma): Error: doveadm server disconnected before handshake: EOF doveadm(1rrissma): Error: 2001:638:913:f33::5:ff:24245: Command search failed for 1phaaman: EOF doveadm(1rrissma): Error: doveadm server failure doveadm: Error: Failed to iterate through some users The backend logs about that ... Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- 'd' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Fatal: master: service(doveadm): child 11235 killed with signal 11 (core dumped) Is this a known behaviour and fixed already, or do I need to do more investigation? (PS: Running the same command on one of the backends works w/o failure) Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
by Charles Marcus 12 Feb '16

12 Feb '16

Hello, I have an el-cheapo shared hosting account on Dreamhost, and have had it for a very long time. For the most part everything usually works fairly well, considering I do keep a lot of folders, and mail, on some of my accounts. They are running dovecot, but still don't have a response as to the version, or doveconf -n output yet. My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir. There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine. The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). Anyone ever seen this before? Would running: doveadm index -u myuser * or doveadm force-resync -u myuser * be appropriate commands to try to repair the damage (whatever it is)? Any other commands I could suggest running? Thanks. I know I haven't given much to go on. Charles /* */

2 2

Multiple quota rules from LDAP userdb?
by Heiko Schlittermann 12 Feb '16

12 Feb '16

Hello, I'm just asking myself, how I get multiple quota rules into the LDAP userdb. Dovecot is 2.2.9 userdb_attrs = … fooQuotaRule=quota_rule=%$ which allows me to have an user db entry as fooQuotaRule: *:storage=1000:messages=50 fooQuotaRule: Trash:storage=500 But, if I'd like to have another quota rule for a submailbox of that user? doveadm user returns the first field only field value uid 500 gid 500 home /volumes/dovecot/home/4b/heiko mail maildir:~:INBOX=/volumes/dovecot/inbox/4b/heiko:INDEX=/volumes/dovecot/cache/4b/heiko quota_rule *:storage=1000:messages=50 Inventing additional LDAP attributs fooQuotaRule2, ... doesn't scale well. Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Autoexpunge on INBOX? And disabling autoexpunge for a mailbox via userdb_import?
by Tom Johnson 12 Feb '16

12 Feb '16

I have autoexpunge working for various special_use mailboxes, but I'd like it to optionally be used for users for their inbox as well. I'd also like to be able to disable autoexpunge for some users for certain directories. Is it possible to specify an autoexpunge period for the main inbox? And can autoexpunge be turned off via userdb_import? Here's some detail behind my questions: I have the following in 15-mailboxes.conf, and all is working great for the autoexpunge feature for the Junk and Trash mailboxes: namespace inbox { separator = . mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk autoexpunge = 15d } mailbox Trash { auto = subscribe special_use = \Trash autoexpunge = 15d } mailbox Sent { auto = subscribe special_use = \Sent } } And I can change the autoexpunge field for those mailboxes via userdb_import in a SQL query - that's working great. But I would like for some users to expire the main inbox and the Sent mailbox, and for others for there to be no expiration. Imagine a scenario where there are users who get a maximum of 15 days of storage - everything older than that should expire out. Other users are standard email users - their inboxes, personal folders, Sent mail should never expire. Adding autoexpunge to the main mailbox gives an error: namespace inbox { separator = . autoexpunge = 999d ... ... } doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/15-mailboxes.conf line 48: Unknown setting: autoexpunge So is there a way to add an autoexpunge setting to the main inbox? And can I set autoexpunge to 0 via userdb_import to disable autoexpunge for a particular mailbox? That doesn't give any error, but does it properly disable autoexpunge for that mailbox? Thanks- Tom

2 1

Re: Question on setting up dovecot
by Jeffery Small 12 Feb '16

12 Feb '16

Many thanks to Joseph Tam and Steffen Kaiser for pointing me to the logging information that I should have been using in the first place. Now that I have separate log files and debugging information, I'm still at a loss to see why the mail is not being pulled over from the Linux system to the Windows 7 machine. To recap, here is the output from doveconf -n: ---------------------------------------------------------------------- # 2.2.18: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 4.2.0-27-generic x86_64 Ubuntu 15.10 auth_debug = yes auth_verbose = yes disable_plaintext_auth = no info_log_path = /var/log/dovecot.info log_path = /var/log/dovecot mail_debug = yes mail_location = mbox:~/lib/Mail:INBOX=/var/mail/%u passdb { driver = pam } protocols = imap pop3 ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = passwd } userdb { driver = passwd } verbose_ssl = yes ---------------------------------------------------------------------- Here is the configuration logging/debug info: ---------------------------------------------------------------------- log_path = /var/log/dovecot info_log_path = /var/log/dovecot.info auth_verbose = yes auth_debug = yes mail_debug = yes verbose_ssl = yes ---------------------------------------------------------------------- And here is the complete set of Debug/Info information logged when Outlook on the Win7 system attempts a Send/Receive operation. Long lines broken for readability: ---------------------------------------------------------------------- Feb 11 08:47:49 auth: Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth: Debug: Read auth token secret from \ /var/run/dovecot/auth-token-secret.dat Feb 11 08:47:49 auth: Debug: auth client connected (pid=7861) Feb 11 08:47:49 auth: Debug: client in: AUTH 1 PLAIN \ service=pop3 session=dF2ST4ErtQDMa1sC \ lip=<server_IP> rip=<client_IP> \ lport=110 rport=54453 resp=<hidden> Feb 11 08:47:49 auth-worker(7864): Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ lookup service=dovecot Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ #1/1 style=1 msg=Password: Feb 11 08:47:49 auth: Debug: client passdb out: OK 1 user=sherril Feb 11 08:47:49 auth: Debug: master in: REQUEST 1975517185 7861 1 \ b48910c85f39498e2fe83277d7e94028 session_pid=7866 Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): lookup Feb 11 08:47:49 auth: Debug: master userdb out: USER 1975517185 \ sherril system_groups_user=sherril uid=1001 gid=1001 \ home=/u/sherril Feb 11 08:47:49 pop3-login: Info: Login: user=<sherril>, method=PLAIN, \ rip=<client_IP>, lip=<server_IP>, mpid=7866, \ session=<dF2ST4ErtQDMa1sC> Feb 11 08:47:49 pop3(sherril): Debug: Effective uid=1001, gid=1001, \ home=/u/sherril Feb 11 08:47:49 pop3(sherril): Debug: fs: root=/u/sherril/lib/Mail, \ index=, indexpvt=, control=, inbox=/var/mail/sherril, alt= Feb 11 08:47:49 pop3(sherril): Info: Disconnected: Logged out top=0/0, \ retr=0/0, del=0/3, size=36928 ---------------------------------------------------------------------- There are no errors. We have a pop3 connection. The user's name, uid, gid, password, home directory are all correct. The inbox is listed as /var/mail/sherril which is correct, and this file has the correct ownership and permissions: # ls -l /var/mail/sherril -rw-rw---- 1 sherril mail 194900 Feb 11 09:00 /var/mail/sherril Yet, no mail is pulled over. Are there any clues here? Thanks, -- Jeff

5 6

Re: RFC: HTTP based storage API
by David Niklas 12 Feb '16

12 Feb '16

On 08 Feb 2016, at 11:01, Nagy, Attila <bra(a)fsn.hu> wrote: > Nearly every popular programming language has an LMTP/POP/IMAP > implementation, most of them suck in many different ways. > I don't know any server or library which provides a well-established, > compatible protocol frontend with an open backend API, which could be > used to easily make a custom storage backend for the LMTP/POP/IMAP > frontend in any language, in any programming paradigm, irregardless > of the frontend implementation. Sorry to come in late, but as a curl lover I'd like to know what is wrong with their implementation of pop3 and imap (project homepage curl.haxx.se). Thanks, David

2 1

[BUG] Dovecot 2.2.21 crashing with assertion failed: (!have_wait_unfinished || unfinished_count > 0)
by Gary Stanley 12 Feb '16

12 Feb '16

Hello Dovecot's IMAP process appears to be crashing at random times for a few users, but one is crashing at least 3 to 5 times an hour. I applied these patches per http://dovecot.org/list/dovecot/2016-January/102933.html /https://github.com/dovecot/core/commit/1ddf959a750f3860feff4ab3f0e908f32740978f.diff //https://github.com/dovecot/core/commit/c8e9fa2ffa2566e75f0500808b1bc9bf5d9db024.diff //https://github.com/dovecot/core/commit/15307c2c91854e766bd9fb095d611a29b3f7572e.diff //https://github.com/dovecot/core/commit/c7801f830c7d2e7d340065cdd5a5c795b1726223.diff //https://github.com/dovecot/core/commit/f136b0050b3125b466af73984177250b7ed1a3be.diff / But it's still crashing. # 2.2.21 (5345f22): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-604.30.3.lve1.3.63.el6.x86_64 x86_64 CloudLinux Server release 6.7 (Aleksei Gubarev) auth_cache_size = 1 k auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%" disable_plaintext_auth = no first_valid_uid = 201 mail_plugins = " zlib" maildir_very_dirty_syncs = yes namespace { inbox = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Archives { auto = no special_use = \Archive } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = create special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. type = private } passdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } plugin { acl = vfile quota = maildir quota_rule = INBOX.Trash:ignore } protocols = imap pop3 service auth { unix_listener auth-client { mode = 0666 } } service imap-login { client_limit = 500 inet_listener imap { address = * } inet_listener imaps { address = * } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service imap { process_limit = 512 vsz_limit = 256 M } service managesieve-login { client_limit = 500 process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service managesieve { process_limit = 512 vsz_limit = 256 M } service pop3-login { client_limit = 500 inet_listener pop3 { address = * } inet_listener pop3s { address = * } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service pop3 { process_limit = 512 vsz_limit = 256 M } ssl_cert = </etc/dovecot/ssl/dovecot.crt ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_key = </etc/dovecot/ssl/dovecot.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } protocol imap { imap_capability = +NAMESPACE imap_idle_notify_interval = 24 mins imap_logout_format = in=%i, out=%o, bytes=%i/%o mail_max_userip_connections = 20 mail_plugins = acl quota imap_quota imap_zlib } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o pop3_uidl_format = UID%u-%v } protocol lda { postmaster_address = postmaster(a)example.com } 1 18:23:54 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Panic: file imap-client.c: line 849 (client_check_command_hangs): assertion failed: (!have_wait_unfinished || unfinished_count > 0) Feb 11 18:23:54 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8096a) [0x7f996c47e96a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x809d6) [0x7f996c47e9d6]$ Feb 11 18:23:55 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Fatal: master: service(imap): child 329519 killed with signal 6 (core dumped) Here's the backtrace: (gdb) bt full #0 0x00007f046eb2e625 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = 98454 selftid = 98454 #1 0x00007f046eb2fe05 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x2, sa_sigaction = 0x2}, sa_mask = {__val = {139657018134932, 1, 18488448, 0, 139657017886537, 139657017552035, 139657017530872, 206158430224, 140725557583888, 140725557583680, 4362456, 140725557583788, 18440536, 0, 0, 6717455502603727992}}, sa_flags = 1635021663, sa_restorer = 0x7ffd38dee060} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f046ef10a60 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:201 backtrace = 0x1196158 "/usr/lib64/dovecot/libdovecot.so.0(+0x80a6a) [0x7f046ef10a6a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x80ad6) [0x7f046ef10ad6] -> /usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x7f046eeb994f] -> dove"... #3 0x00007f046ef10ad6 in i_internal_fatal_handler (ctx=0x7ffd38dee080, format=<value optimized out>, args=<value optimized out>) at failures.c:670 status = 0 #4 0x00007f046eeb994f in i_panic (format=0x18096 <Address 0x18096 out of bounds>) at failures.c:275 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, timestamp_usecs = 0} args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7ffd38dee160, reg_save_area = 0x7ffd38dee0a0}} #5 0x0000000000418247 in client_continue_pending_input () No symbol table info available. #6 0x00007f046ef22fdb in io_loop_call_io (io=0x11c5860) at ioloop.c:559 ioloop = 0x119e740 t_id = 2 __FUNCTION__ = "io_loop_call_io" #7 0x00007f046ef24577 in io_loop_handler_run_internal (ioloop=<value optimized out>) at ioloop-epoll.c:220 ctx = 0x11a0250 events = <value optimized out> event = 0x11a10c0 list = 0x11a1c80 io = <value optimized out> tv = {tv_sec = 29, tv_usec = 999480} events_count = <value optimized out> msecs = <value optimized out> ret = 1 i = <value optimized out> call = <value optimized out> ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "io_loop_handler_run_internal" #8 0x00007f046ef23085 in io_loop_handler_run (ioloop=0x119e740) at ioloop.c:607 No locals. #9 0x00007f046ef23228 in io_loop_run (ioloop=0x119e740) at ioloop.c:583 __FUNCTION__ = "io_loop_run" #10 0x00007f046eebeca3 in master_service_run (service=0x119e5e0, callback=<value optimized out>) at master-service.c:640 No locals. #11 0x0000000000424c5c in main () No symbol table info available.

2 2

Re: Question on setting up dovecot
by Joseph Tam 11 Feb '16

11 Feb '16

Jeffery Small writes: > There are no errors. We have a pop3 connection. The user's name, > uid, gid, password, home directory are all correct. The inbox is > listed as /var/mail/sherril which is correct, and this file has the > correct ownership and permissions: > > # ls -l /var/mail/sherril > -rw-rw---- 1 sherril mail 194900 Feb 11 09:00 /var/mail/sherril > > Yet, no mail is pulled over. Are there any clues here? OK, the next step might be to try talk POP3 directly to your dovecot server, and see if it responds to some common POP3 requests. You might have to use one of telnet server 110 nc server 110 (or "ncat" or "netcat") openssl s_client -connect server:995 (if using SSL) then test with these commands USER someuser PASS their-password (Should list messages and their sizes) LIST (Should list messages and their UIDLs) UIDL (Retrieve the first message) RETR 1 QUIT If these work, then I would look at the client. If not, check the logs why it failed. > Feb 11 08:47:49 pop3(sherril): Info: Disconnected: Logged out top=0/0, \ > retr=0/0, del=0/3, size=36928 If I read this right, the client never tried to download any messages (no top, no retr). Maybe the client downloaded the UIDLs and figured it already had local copies of those messages. Joseph Tam <jtam.home(a)gmail.com>

1 0

Config file syntax in gory detail
by Timo Sirainen 11 Feb '16

11 Feb '16

I finally wrote this: http://wiki2.dovecot.org/ConfigFile Looks like it ended up having a lot more content than I initially assumed.

5 4

Cache unseen count
by Jorge Carling 11 Feb '16

11 Feb '16

Dear all, Lately I am working on PHP mail account management application. It has, among others, Unread messages count column. To get the number of unread messages, I execute "doveadm search" command for each database row. The search time for single account is about 0.03 seconds. Our system has around 750 accounts, so the total loading time is currently around 20 seconds. When setting up Dovecot, I made the user_query, which is also used to store the message quota, and total number of messages stored in particular mailbox. Is there a keyword that can be included in the user_query, or some other way to maintain the unread messages count in the database? Maybe there is a way to execute a script every time a message is received, or becomes read, or unread. Kind regards,JC

2 1