- dovecot - dovecot.org

Re: Corruption & migration
by Daniel Miller 21 Mar '17

21 Mar '17

Now I'm seeing a bunch of these: 02:28 bubba dovecot: dsync-local(user(a)mydomain.com): Error: bzlib.read(/var/mail/mydomain.com/user/mdbox/storage/m.20): corrupted data at 24123649 Mar 21 11:02:28 bubba dovecot: dsync-local(user(a)mydomain.com): Error: dsync(oldserver.host): read(zlib(/var/mail/mydomain.com/user/mdbox/storage/m.20)) failed: read(/var/mail/mydomain.com/user/mdbox/storage/m.20) failed: bzlib.read(/var/mail/mydomain.com/user/mdbox/storage/m.20): corrupted data at 24123649 (last sent=mail, last recv=mail_request (EOL)) Mar 21 11:02:28 bubba dovecot: doveadm: Error: dsync-remote(user(a)mydomain.com): Error: dsync(localhost): read() failed: read((fd)) failed: dot-input stream ends without '.' line (last sent=mail_request (EOL), last recv=mail) Daniel On 3/20/2017 11:30 PM, Daniel Miller wrote: > I'm getting close - I believe I have one-way replication processing > now. It's amazing watching the mails magically appear in the new > server! I'll tender some updates to the wiki in the hopes it will > help others. > > At this time, I have two errors I haven't been able to eliminate with > config changes: > > dovecot: doveadm: Error: dsync-remote(user(a)mydomain.com): Warning: > Transaction log file > /var/mail/mydomain.com/user/sdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log > was locked for 109 seconds (rotating while syncing) > Mar 20 23:22:04 bubba dovecot: dsync-local(user(a)mydomain.com): Error: > Remote command returned error 75: ssh -lvmail 192.168.0.4 doveadm > dsync-server -uuser(a)mydomain.com > > I don't know if the transaction lock issue will fix itself via > repeated replication runs. But I have no idea what the "error 75" is. > > Daniel > > > On 3/20/2017 10:05 PM, Daniel Miller wrote: >> I have an existing server using mdbox & sis. There is a quantity of >> corruption - I have no idea where it came in. I'd had a previous >> rebuild some years ago and I thought I had a clean datastore. By >> "corruption" I mean missing attachments and either missing or >> mixed-up emails. >> >> I've setup a new server on a virtual machine. Instead of copying the >> old configs I hand-adjusted individual settings using the current >> distribution example. The new server is running...but empty. >> >> So now my intent is to migrate the mail to the new machine. A few >> items to note: >> 1. The existing mail server is the virtual host. The mails are >> stored locally, in mdbox format, with sis enabled. And possibly zlib. >> 2. The new server, via VirtualBox, is presently configured for >> Maildir. I may change to sdbox - I'm now a little reserved about mdbox. >> 3. Instead of storing the mails within the virtual machine, I'm >> using the host's drive. This is via an exported NFS4 folder from the >> host, mounted on the guest, using the options: >> "auto,noatime,hard,intr,async,nordirplus". >> 4. My intent is to have the new virtual server become the primary >> mail server. The mail store will continue to be accessed by NFS4. No >> other servers or processes (besides backup) should access the mail >> store. >> >> So...my question is what is the best method for the migration? >> >> Reading the docs on Dovecot replication I'm still very unsure of how >> to implement it - particularly the "master-master" or "master-client" >> relationships. The docs are not 100% clear on which server gets >> which options. I can guess...which I'd really rather not do. But it >> seems to me getting one-way replication working would be the most >> "elegant" solution here. >> >> Manual dsync is also an option - but again I'm not on which machine I >> should execute which options. >> >> Imapsync is a possibility and the syntax is clear - but very slow in >> execution for 100k mails. >> >> Any suggestions or pointers would be welcome. >>

1 0

New Member
by Ger Hooton Scoil Chroí Íosa Blarney 21 Mar '17

21 Mar '17

Hi All, Before I start posting where /how can I check to see if the question I wish to ask about has already been covered? __________________________________________________ Please consider the environment before printing this email. Ger

3 2

Re: Dovecot can't connect to openldap over starttls [REQUEST OF OPENLDAP]
by info＠gwarband.de 21 Mar '17

21 Mar '17

Can sombody say something about this request? This is an email from the openldap-technical mailinglist from openldap. Systemdetails are mention in the other email. -------- Originalnachricht -------- Betreff: Re: Dovecot can't connect to openldap over starttls Datum: 2017-03-20 16:18 Absender: Dan White <dwhite(a)cafedemocracy.org> Empfänger: info(a)gwarband.de Kopie: openldap-technical(a)openldap.org On 03/20/17 16:06 +0100, info(a)gwarband.de wrote: >> Debug Dovecot's implementation of ldap_start_tls_s(). > I don't have any idea how to set a higher debug level to dovecot. In > my opinion I have the highest. So I can't deliver a greater log. I recommend consulting Dovecot's advice on how to run a debugger, or dig into the code which calls libldap.

2 8

doveadm-sync stateful
by Cédric ML 21 Mar '17

21 Mar '17

Hello, I'm trying to migrate mail accounts from an old server to a new one. As I need to migrate dozens of accounts which take about 1G each, I need to do stateful sync to make my migration in two times : 1 - I run a : doveadm -D -o mail_fsync=never -o imapc_user=user1(a)olddomain.fr sync -s "" -R -1 -u user1(a)newdomain.fr imapc: > /tmp/firstsync.log 2>&1 my accounts are synced, and I get a "state" string (AQAAAHm4+Jk=) as the output of doveadm sync command. 2 - Changes are made on the old server, and I expect next dsync will sync only the changes / new mails. So I run a : doveadm -D -o mail_fsync=never -o imapc_user=user1(a)olddomain.fr sync -s "AQAAAHm4+Jk=" -R -1 -u user1(a)newdomain.fr imapc: > /tmp/secondsync.log 2>&1 My problem is that state of messages are not synchronized (mails marked as read on old server are still unread on new server, mails moved on a folder on the old server are still in the INBOX of new server). The full debug of ran commands is available here : http://fpaste.scsys.co.uk/557551 The result of "doveconf -n" is available here : http://fpaste.scsys.co.uk/557552 I'm using version 2.2.27 Can anyone tell me if I'm doing something wrong ? How can I debug more ? Regards, Cédric

2 2

Re: Corruption & migration
by Daniel Miller 21 Mar '17

21 Mar '17

I'm getting close - I believe I have one-way replication processing now. It's amazing watching the mails magically appear in the new server! I'll tender some updates to the wiki in the hopes it will help others. At this time, I have two errors I haven't been able to eliminate with config changes: dovecot: doveadm: Error: dsync-remote(user(a)mydomain.com): Warning: Transaction log file /var/mail/mydomain.com/user/sdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log was locked for 109 seconds (rotating while syncing) Mar 20 23:22:04 bubba dovecot: dsync-local(user(a)mydomain.com): Error: Remote command returned error 75: ssh -lvmail 192.168.0.4 doveadm dsync-server -uuser(a)mydomain.com I don't know if the transaction lock issue will fix itself via repeated replication runs. But I have no idea what the "error 75" is. Daniel On 3/20/2017 10:05 PM, Daniel Miller wrote: > I have an existing server using mdbox & sis. There is a quantity of > corruption - I have no idea where it came in. I'd had a previous > rebuild some years ago and I thought I had a clean datastore. By > "corruption" I mean missing attachments and either missing or mixed-up > emails. > > I've setup a new server on a virtual machine. Instead of copying the > old configs I hand-adjusted individual settings using the current > distribution example. The new server is running...but empty. > > So now my intent is to migrate the mail to the new machine. A few > items to note: > 1. The existing mail server is the virtual host. The mails are > stored locally, in mdbox format, with sis enabled. And possibly zlib. > 2. The new server, via VirtualBox, is presently configured for > Maildir. I may change to sdbox - I'm now a little reserved about mdbox. > 3. Instead of storing the mails within the virtual machine, I'm using > the host's drive. This is via an exported NFS4 folder from the host, > mounted on the guest, using the options: > "auto,noatime,hard,intr,async,nordirplus". > 4. My intent is to have the new virtual server become the primary > mail server. The mail store will continue to be accessed by NFS4. No > other servers or processes (besides backup) should access the mail store. > > So...my question is what is the best method for the migration? > > Reading the docs on Dovecot replication I'm still very unsure of how > to implement it - particularly the "master-master" or "master-client" > relationships. The docs are not 100% clear on which server gets which > options. I can guess...which I'd really rather not do. But it seems > to me getting one-way replication working would be the most "elegant" > solution here. > > Manual dsync is also an option - but again I'm not on which machine I > should execute which options. > > Imapsync is a possibility and the syntax is clear - but very slow in > execution for 100k mails. > > Any suggestions or pointers would be welcome. >

1 0

Corruption & migration
by Daniel Miller 21 Mar '17

21 Mar '17

I have an existing server using mdbox & sis. There is a quantity of corruption - I have no idea where it came in. I'd had a previous rebuild some years ago and I thought I had a clean datastore. By "corruption" I mean missing attachments and either missing or mixed-up emails. I've setup a new server on a virtual machine. Instead of copying the old configs I hand-adjusted individual settings using the current distribution example. The new server is running...but empty. So now my intent is to migrate the mail to the new machine. A few items to note: 1. The existing mail server is the virtual host. The mails are stored locally, in mdbox format, with sis enabled. And possibly zlib. 2. The new server, via VirtualBox, is presently configured for Maildir. I may change to sdbox - I'm now a little reserved about mdbox. 3. Instead of storing the mails within the virtual machine, I'm using the host's drive. This is via an exported NFS4 folder from the host, mounted on the guest, using the options: "auto,noatime,hard,intr,async,nordirplus". 4. My intent is to have the new virtual server become the primary mail server. The mail store will continue to be accessed by NFS4. No other servers or processes (besides backup) should access the mail store. So...my question is what is the best method for the migration? Reading the docs on Dovecot replication I'm still very unsure of how to implement it - particularly the "master-master" or "master-client" relationships. The docs are not 100% clear on which server gets which options. I can guess...which I'd really rather not do. But it seems to me getting one-way replication working would be the most "elegant" solution here. Manual dsync is also an option - but again I'm not on which machine I should execute which options. Imapsync is a possibility and the syntax is clear - but very slow in execution for 100k mails. Any suggestions or pointers would be welcome. -- Daniel

1 0

Re: Dovecot 2.2.27 proxy - enforcing per client IP connection limits
by Joseph Tam 21 Mar '17

21 Mar '17

Sami Ketola writes: >> Can anyone with Solr installed confirm/refute this: does installing >> Solr keep iOS clients from roofing the connection count? > > I doubt it, but since IMAP SEARCH goes all the way down to the backends > mail_max_userip_connections can be used to limit the number of > connections. Understood -- that's the current situation I'm in now. Our iOS users would launch a search resulting in a connection burst, hit the connection cap, log out all IMAP sessions out, then start the cycle again. This sometimes lasts for 10's of minutes. I'm not sure what the users sees. Sample logs entries: Mar 19 01:21:30 server dovecot: imap-login: Login: user=<user> ... [... 14 similar logins removed ...] Mar 19 01:21:41 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:42 server dovecot: imap-login: Maximum number of connections from user+IP exceeded (mail_max_userip_connections=16) : user=<user> ... Mar 19 01:21:42 server dovecot: imap(user): Logged out in=425 out=1107 [... 14 similar logouts removed ...] Mar 19 01:21:42 server dovecot: imap(user): Logged out in=382 out=1107 Mar 19 01:21:42 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:42 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:43 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:44 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:44 server dovecot: imap(user): Logged out in=442 out=1173 Mar 19 01:21:44 server dovecot: imap(user): Logged out in=442 out=1155 Mar 19 01:21:44 server dovecot: imap(user): Logged out in=442 out=1166 Mar 19 01:21:44 server dovecot: imap(user): Logged out in=442 out=1174 Mar 19 01:21:44 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:47 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:47 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:48 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:48 server dovecot: imap-login: Login: user=<user> ... Mar 19 01:21:49 server dovecot: imap-login: Login: user=<user> ... { ... and on and on for the next 10 minutes ... } However, there is a pause between each login that might be long enough to squeeze the search results in if given quickly enough. From the I/O stats, most of these searches have empty results. It probably won't prevent the connection cap problem, but it might minimize the length and severity of these connection storms. Of course, the real fix is for iOS mail-app developers to stop assuming the IMAP server is owned exclusively by the user by configuring some reasonable connection throttles. Joseph Tam <jtam.home(a)gmail.com>

2 1

Dovecot can't connect to openldap over starttls
by info＠gwarband.de 20 Mar '17

20 Mar '17

Hello guys, actually I'm trying to configure dovecot to access openldap for passwordcheck. My openldap is only allow access over "secure ldap". The dovecot can communicate with the openldap server but there is maybe a failure in the sslhandshake. Additional information you can find in the logs or in the dump below. Also I have my ldap config from dovecot in the links below. I have already created an bug reporting in the system of openldap but the answer was to get support from her. All datalinks: https://gwarband.de/openldap/dovecot.log https://gwarband.de/openldap/dovecot-ldap.conf https://gwarband.de/openldap/openldap.log https://gwarband.de/openldap/trace.dump The bugreportinglink from openldap: http://www.openldap.org/its/index.cgi/Incoming?id=8615 I hope you can help me. Regards. Tobias Warband

2 10

Can't verify remote server certs without trusted CAs (ssl_client_ca_* settings)
by Ralf Hildebrandt 20 Mar '17

20 Mar '17

Hi! I upgraded the 2.2 packages today (from 2:2.2.28-1~auto+5 to 2:2.2.28-1~auto+8) I now I'm getting an error: Mar 20 13:25:58 mproxy dovecot: auth: Error: imapc(email.charite.de:993): Couldn't initialize SSL context: Can't verify remote server certs without trusted CAs (ssl_client_ca_* settings) I checked, and alas, I had ssl_client_ca_dir = ssl_client_ca_file = So I set: ssl_client_ca_file = </etc/ssl/certs/ca-certificates.crt But I'm still getting the error above. I addition, dovecot is crashing with SIGSEGV: Mar 20 13:28:23 mproxy dovecot: auth: Error: imapc(email.charite.de:993): Couldn't initialize SSL context: Can't verify remote server certs without trusted CAs (ssl_client_ca_* settings) Mar 20 13:28:23 mproxy dovecot: auth: Error: imapc(email.charite.de:993): No SSL context Mar 20 13:28:23 mproxy dovecot: auth: Error: imap(la***sch,87.77.180.61): Disconnected from server Mar 20 13:28:23 mproxy postfix/submission/smtpd[32682]: warning: zb43d.pia.fu-berlin.de[87.77.180.61]: SASL PLAIN authentication failed: Connection lost to authentication server Mar 20 13:28:23 mproxy dovecot: auth: Fatal: master: service(auth): child 32685 killed with signal 11 (core dumped) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 5

doveadm proxy password
by Angel L. Mateo 20 Mar '17

20 Mar '17

Hi, I'm configuring a proxy host to connect to backend servers. As proxy is done based on an LDAP attribute of the user, I'm not using director. In the proxy server I have configured: doveadm_port = 24245 doveadm_password = secret And in the backend: service doveadm { inet_listener { port = 24245 } } local <mynetwork> { doveadm_password = secret } But when I run a doveadm command in the proxy I get: amateo_adm@musio10:/etc/dovecot/conf.d$ sudo doveadm quota get -u amateo(a)um.es doveadm(amateo(a)um.es): Error: doveadm authentication failed () doveadm(amateo(a)um.es): Error: myotis50.um.es:24245: Command quota get failed for amateo(a)um.es: EOF Quota name Type Value Limit % And in the backed I get: Mar 20 15:52:01 myotis50 dovecot: doveadm: Error: doveadm client authenticated with wrong password I have checked with tcpdump and in the command is sending the base64 encoding of "doveadmsecret" Any help? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

1 0