February 2011 - dovecot

[Dovecot] Dovecot + Exim + Maildir quota problem
by Matteo Tinazzi 17 Feb '11

17 Feb '11

Hi all i'm a new user of this list, i got a server with DirectAdmin which has exim as MUA, dovecot as POP/IMAP server, recently i added the quota to exim and pointed the maildirsize file in the same location if the dovecot's one so both services can interact with it. This has been done to let exim update the quotas while doing localdelivery while dovecot can show quotas through roundcube and update them when users delete mails. The issue that i noticed is that some users get wrong quota informations... after a little investigation i think it could be a concurrent access to this file so dovecot tries to remove the deleted mails fomr maildirsize and exim tries to put new ones inside. In a specific case i got a user with a quota of 50M seeing that 75% of his quota filled while the real weight of the mail files is about 4% ... any help would be very appreciated

1 1

[Dovecot] Evolution IMAP with Dovecot as proxy
by Antonio Perez-Aranda 17 Feb '11

17 Feb '11

We have some users who use Evolution as mail client. During migration process we are run Dovecot as proxy of Courier-imap until any user get migrated depend of ldap attribute. At present, the problem is that with Thunderbird and other clients, The client run as normal. But Evolution can't get new messages or Suscription list. Capabilities on Courier and Dovecot are differents, but when a connection is proxied to Courier, Courier print the Capabilitites. Any body has any idea of this? -- Antonio Pérez-Aranda Alcaide aperezaranda(a)yaco.es Yaco Sistemas S.L. http://www.yaco.es/ C/ Rioja 5, 41001 Sevilla Teléfono +34 954 50 00 57 Fax +34 954 50 09 29

3 11

[Dovecot] Dovecot 2.0.9/1.2.11: crash in driver-pgsql.c
by dovecot＠excellency.fr 17 Feb '11

17 Feb '11

Hello, I finally took time to isolate and reproduce a long time Dovecot crash that is bugging me since at least Dovecot 1.0.x. It's still there in Dovecot 2.0.9, which I just tested. Here is the backtrace from my logs: Feb 16 16:56:57 dovecot: auth: Error: pgsql: Query timed out, aborting Feb 16 17:25:16 dovecot: auth: Panic: file driver-pgsql.c: line 349 (result_finish): assertion failed: (db->io == NULL) Feb 16 17:25:16 dovecot: auth: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0x7fdbfe6d4272] -> /usr/lib/dovecot/libdovecot.so.0 [0x7fdbfe6d42d6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7fdbfe6d46c8] -> dovecot/auth [0x42f555] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xdd) [0x7fdbfe6df9bd] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x5e) [0x7fdbfe6e05be] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7fdbfe6df618] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fdbfe6cd883] -> dovecot/auth(main+0x2d4) [0x419474] -> /lib/libc.so.6(__libc_start_main+0xe6) [0x7fdbfc4f91a6] -> dovecot/auth [0x40da09] Feb 16 17:26:35 dovecot: master: Error: service(auth): child 140263 killed with signal 6 (core not dumped) Both my userdb and passdb use a PostgreSQL database. The bug happens when the PostgreSQL host suddenly becomes unavailable. I can easily reproduce it by launching Dovecot with my host being up, and then null routing its IP address (to simulate an outage): route add -host 10.0.0.2 reject (10.0.0.2 being the IP address of the PostgreSQL host.) >From that point, if I try to authenticate (using a POP3 connection, for instance), dovecot-auth will crash one or two minutes later. In Dovecot 2.0.9, despite the crash, authentication will resume once the PostgreSQL is back up (which can be simulated with: route delete -host 10.0.0.2 reject). In Dovecot 1.x however (latest version I tested is 1.2.11), it will stay in this state forever (logs are flooded with Password query failed: Not connected to database.) Core dump is available on http://paste.excellency.fr/dovecot.core # dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.37 x86_64 Debian 5.0.6 auth_mechanisms = plain login auth_username_chars = disable_plaintext_auth = no first_valid_gid = 2000 first_valid_uid = 2000 listen = *, [::] log_timestamp = "%Y-%m-%d %H:%M:%S " mail_fsync = always mail_location = maildir:~/ mail_max_userip_connections = 100 mail_nfs_index = yes mail_nfs_storage = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mmap_disable = yes passdb { args = /etc/dovecot/dovecot-pgsql.conf driver = sql } plugin { quota = maildir sieve = filter.sieve sieve_max_actions = 64 sieve_max_redirects = 100 } protocols = imap pop3 service auth { unix_listener auth-client { mode = 0666 } } service imap-login { service_count = 0 } service imap { process_limit = 1024 } service managesieve-login { service_count = 0 } service pop3-login { service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert = </etc/ssl/certs/my.pem ssl_key = </etc/ssl/private/my.key userdb { args = /etc/dovecot/dovecot-pgsql.conf driver = sql } protocol imap { imap_client_workarounds = delay-newmail mail_plugins = quota imap_quota } protocol pop3 { mail_plugins = quota } # cat /etc/dovecot/dovecot-pgsql.conf driver = pgsql connect = host=10.0.0.2 dbname=foobar user=dovecot password=foobar default_pass_scheme = CRYPT password_query = SELECT password FROM mailbox WHERE domain = '%d' AND local_part = '%n' user_query = SELECT home, uid, gid, system_user, quota_rule FROM mailbox WHERE domain = '%d' AND local_part = '%n' -- Cyril

3 2

[Dovecot] Cannot subscribe to shared folders
by Lukas Haase 17 Feb '11

17 Feb '11

Hi, I am unable to subscribe to shared folders with Thunderbird although everything seems to be correct (dovecot 1.2). What does not work? I click "Subscribe..." in Thunderbird. The shared mailboxes are listed, all with [ ] (no checkbox). Therefore I mark them [x], click "subscribe" and "OK". However, the folders do not appear in the folder tree. Furthermore, when I again click "Subscribe..." the folders do NOT have the mark [x] any more. This happens on different computers with *new* profiles. My configuration is the following (maybe that is the problem?!): namespace private { separator = . prefix =. inbox = no list = no hidden = yes location = maildir:~/Maildir-root subscriptions = yes } namespace private { separator = . prefix = INBOX. inbox = yes subscriptions = yes } namespace public { separator = . prefix = Shared. location = maildir:/var/mail/shared:INDEX=~/Maildir/shared:LAYOUT=fs subscriptions = no } The folder /var/mail/shared has appropriate permissions/form: * Owner: $user:mailuser (all users are member of "mailuser") * All files 0660, all folders 0770 * There exists a dovecot-shared in every folder * There exists a dovecot-acl in each folder where each user has the permissions "lrwstie" I switched on "mail_debug = yes" but no messages. However, the file ~/Maildir-root/subscriptions contains the subscribed folders: Shared.Group1 Shared.Group2 Shared.Group1/Test Furthermore, I switched on rawlog. Everything seems to be OK as well: .in-file: 2 subscribe "Shared.Group1" 3 lsub "" "INBOX.*" 5 lsub "" "Shared.*" 6 list "" "INBOX" .out-file: 2 OK Subscribe completed. [...] I am pretty sure it already worked. But it stopped working. Does anybody have a hint what could be the problem? Regards, Luke

2 4

[Dovecot] dsync conversion error
by Frank Bonnet 17 Feb '11

17 Feb '11

Hello When converting mailboxes I get that kind of error messages : dsync(xxx): Error: Next message unexpectedly lost from mbox file /user/xxx/mail/Sent at 9580036 (cached) the dsync command I run is dsync -u xxx mirror mbox:/user/xxx/mail:INBOX=/var/mail/xxx Dovecot mail_location configuration is : mail_location = mbox:~/MailSpace/Mailboxes:DIRNAME=Mbox-Messages:INDEX=~/Index:CONTROL=~/Kontroll Note that ONLY happen on users's "Sent" folder, never on any other folder Thanks for any advices.

2 2

[Dovecot] Some questions about mdbox
by Maria Arrea 16 Feb '11

16 Feb '11

Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn… These are my questions (a long mail): *1º How should I restore from backup with mdbox & separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ "" savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2º Disaster recovery: indexes & mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3º Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size & mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a "variant" of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4º Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am starting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5º Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1º a user gets her mail via pop3 (or delete a message via imap and empties trash) 2º the message is still stored in my mail server, but has refcount=0 3º nightly backup is performed 4º nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria

3 2

Re: [Dovecot] Some questions about mdbox
by Maria Arrea 16 Feb '11

16 Feb '11

Hello everybody I have found that with bzip compression level 6 I have a better balance between compression and decompression speed with dovecot+mdbox, but still too slow compared with zlib. Does anybody have any other hint about my original questions (see below) Regards Maria Arrea ----- Original Message ----- From: Maria Arrea Sent: 02/15/11 09:16 PM To: dovecot(a)dovecot.org Subject: [Dovecot] Some questions about mdbox Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn… These are my questions (a long mail): *1º How should I restore from backup with mdbox & separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ "" savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2º Disaster recovery: indexes & mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3º Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size & mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a "variant" of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4º Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am starting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5º Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1º a user gets her mail via pop3 (or delete a message via imap and empties trash) 2º the message is still stored in my mail server, but has refcount=0 3º nightly backup is performed 4º nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria

1 0

[Dovecot] Auth processes/login processes count
by Thomas Hummel 16 Feb '11

16 Feb '11

Hello Timo, I'm running dovecot-1.2.12/Maildir/LDAP-passdb/LDAP-prefetched-userdb and I'm experiencing "random" latency (which even seem to make squirrelmail timeout sometimes) at login without noticing particular load on the server. LDAP backends are running fine as far as I know so I was investigating maybe mistuned limits : Dovecot is configured (for ~3000 users) with . each login processed in its own process : #login_process_per_connection = yes #login_max_connections = 256 . counts as : #login_processes_count = 3 login_max_processes_count = 1024 auth default { [...] count = 5 } I've got several questions : 1) how can I count more than login_max_processes_count processes ? # ps aux | egrep -i '(pop3|imap)-login' | wc -l # 1031 2) could be my auth process count the bottleneck ? is there a rule of thumb to deduce it from the login process count ? 3) on dovecot login/master/auth design, is my understanding correct ? - dovecot-1.2.x : . passdb and userdb lookup are done by the auth process, on behalf of either the login process which wants to know if the client's password is correct or of the master process which wants to check if it was actually correct and which wants to have the userdb info. . client connection fd is sent to the master which creates the mail process and transfers the fd to it. - dovecot-2.x : . client connection fd does not go through the master anymore but through a socket in the login/ directory . it's the mail process which ask the auth process for the userdb lookup 4) with LDAP backend using {CRYPT} passwords dans with plaintext auth enabled in dovecot, is the clear password crypted before comparing it to the LDAP crypted password or does dovecot try to simply bind to LDAP with the clear password ? Thanks -- Thomas Hummel | Institut Pasteur <hummel(a)pasteur.fr> | Groupe Exploitation et Infrastructure

2 1

[Dovecot] TLS with autosigned certs
by Antonio Perez-Aranda 16 Feb '11

16 Feb '11

Is normal this info/error with autosigned certs? Feb 14 20:25:47 pop3-login: Info: Disconnected (no auth attempts):ip=[127.0.0.1], port=[51243], protocol=pop3, TLS handshaking: SSL_accept() failed: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca: SSL alert number 48 I set on config a specific certs for pop3 protocol including key, cert and ca, protocol pop3 { ssl_cert = </etc/ssl/pop3d.cert ssl_key = </etc/ssl/pop3d.key ssl_ca = </etc/ssl/ca.pem } If I check as true confirmed on Thunderbird client, then, the error is no reported any more. -- Antonio Pérez-Aranda Alcaide aperezaranda(a)yaco.es Yaco Sistemas S.L. http://www.yaco.es/ C/ Rioja 5, 41001 Sevilla Teléfono +34 954 50 00 57 Fax +34 954 50 09 29

3 3

[Dovecot] test emails did not arrive at SMTP server : after dovecot installation
by sunhux G 16 Feb '11

16 Feb '11

Just set up postfix & it's running on my RHES 4.2 box. Immediately after postfix is up, I test sending emails from a permitted domain (ahhh, on this postfix server's domain firewall, we even have a firewall rule which permits Tcp25 from those few sending domains' SMTP servers) using an email client to sender_id@[IP_address_of_the_postfix_server] & the /var/log/maillog on the postfix server indicated the email arrives at the postfix server (with some errors though) : # grep recipient_id /var/log/maillog* maillog:Feb 15 11:41:52 hostname postfix/smtpd[6891]: NOQUEUE: reject: RCPT from gate1.mds.com.sg[203.126.130.157]: 554 5.7.1 <recipient_id(a)[202.6.163.31]>: Relay access denied; from=<prvs=020cae8c4=recipient_id(a)mds.com.sg> to=<recipient_id(a)[202.6.163.31]> proto=ESMTP helo=<gate1.mds.com.sg> maillog:Feb 15 13:43:20 hostname sendmail[7688]: NOQUEUE: SYSERR(recipient_id): can not chdir(/var/spool/mqueue/): Permission denied Then I installed dovecot rpm on my RHES box : uninstall it as it's an old version & reinstall with a newer version & start up dovecot as well. I did not test sending to sender_id@domain_name at that time because the domain I purchased from a domain provider/registrar has yet to be registered in our ISP's DNS. Subsequently I registered the following A, MX & NS records with our ISP : A: myportaltech.com. IN A 202.6.163.31 A: smtp.myportaltech.com. IN A 202.6.163.31 PTR: 31.163.6.202.in-addr.arpa. IN PTR smtp.myportaltech.com. MX: myportaltech.com. IN MX 10 smtp.myportaltech.com. NS: myportaltech.com. IN NS ns1.businessexprezz.com. NS: myportaltech.com. IN NS ns2.businessexprezz.com. The above myportaltech is just a fictitious name of my domain but I can provide the actual domain name if needed. After the above records have been propagated to all other DNSes, I test sending email from the same permitted domain, this time using domain name & the email never arrives & I did not receive a 'bounced mail' notification too. Then I test sending from the same domain to recipient_id(a)[202.6.163.31] & this time round, the test email never show up in /var/log/maillog* anymore. The network/security guys confirmed that the firewall logs did not show any denied SMTP records. So how do I go about troubleshooting this? Is this a DNS record entries issue, firewall/network issue, related to dovecot or something within my postfix server? Thanks Sun

2 1