- dovecot - dovecot.org

Re: Reappearing emails - IMAP trace
by Ron Cleven 26 Mar '16

26 Mar '16

>> > >> >A few days back, I sent an overview of this problem, but received no responses. Since then, I have run dozens of traces to isolate the problem, difficult because there are timing issues involved. I have finally nailed it down. If this is not the proper place to report such bugs or if someone knows that this bug has been fixed, please let me know. As I noted in my earlier post, we have been running Dovecot 2.2.10 with a pair of CentOS 7 boxes with replications for the past year. We have been quite happy with the performance and reliability. >> > >> >Recently we received a report that emails could reappear in the INBOX after being deleted. After running a pile of traces, I determined that the problem was strangely related to replications. For the purposes of this discussion, I will refer to the two symmetric replicating servers as A and B. Further, let us assume that during "normal" operation, all the emails are delivered to A via SMTP and are replicated to B. Under those assumptions, if the IMAP user connects to A (where the messages were originally delivered), there is no problem, at least no problem I was able to find. The problem I am describing only arises if the IMAP user connects to B. Connecting to B has never presented any other problems that I am aware of. >> > >> >The test for which I have provided the trace starts with a test mailbox containing only 3 unread messages in the INBOX. Moving 1 of the unread messages to Trash is all that is needed to reproduce the problem. Remember this is ONLY a problem if the IMAP sessions do not connect to the server to which the messages were originally delivered. Also, I found that there is a timing window. The critical IMAP commands are: >> > >> > UID STORE xxx +FLAGS.SILENT (\Seen) >> > UID MOVE xxx Trash >> > >> >If you introduce a large enough delay (I arbitrarily chose 5 seconds) between those two commands, there is no problem. Presumably this has to do with the two boxes syncing up some critical data structure. > What mailbox format do you use? Are you able to reproduce this by running doveadm sync commands manually instead of letting replication do it? For example: > > - doveadm sync -s "" -d -u user@domain > state > - Run the UID STORE & UID MOVE > - doveadm sync -s "`cat state`" -d -u user@domain > > There have been some fixes, especially recentlyhttps://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b… sounds a bit similar although I don't really see how it would apply here. Would be a good idea to try anyway with v2.2.22.rc1 (which seems to be stable enough that I'll make v2.2.22 release soon). > > Anyway, I attempted a few times to reproduce it with your test but wasn't able to. I was out when you were kind enough to reply. To answer your question, we are using Maildir format. The trace I provided was based upon IMAP interactions with Roundcube (though the problem was reproducable with several mail clients). I left in a few more steps to make the trace look less contrived. However, I reduced it further to just a couple of connection sessions. What I found in that exercise that was not apparent to me in my prior posting was that the "STATUS INBOX" command that ultimately reveals the problem (it shows the message reappearing) only becomes "wrong" when it is done in a subsequent session. That is, even if I inject an artificially large delay after the "UID STORE" / "UID MOVE" commands before the "STATUS INBOX" command in the same session, that result is never "wrong". But, as soon as I open a subsequent IMAP session, the "STATUS INBOX" command then shows the problematic results. I have never dug into the Dovecot code base, but I assume this relates to how the session data is cached and how the replications update it. None of this is relevant if the problem has already been fixed, so I will endeavor to set up a couple of test boxes with the current version to verify. The link you provided does look quite hopeful. Thanks so much.

1 0

Re: IMAP Idle
by Joseph Tam 26 Mar '16

26 Mar '16

Joy <pj.netfilter(a)gmail.com> wrote: > We have implement imap idle in web mail built by us to have > push mail feature. IMAP idle working perfectly with browser notification > and we are happy with it but having one issue with users who close the > browser directly and never logout in that case there are number of idle > connection which are not in use and users are unable to login once IP wise > connection limit is reached. > > Dovecot is not closing connection which are not in use, is there any > setting available which can help me to resolve this issue. I had much the same situation where a user signed up with a roaming wireless carrier that assigned a new IP to the client whenever it got passed from one access point to another. Good fun when this person took a bus ride through the city, leaving orphaned connections in its wake. The IDLE disconnection timeout is hardwired in the Dovecot code http://wiki.dovecot.org/Timeouts It's set to the RFC minimum of 30min. You'll have to recompile Dovecot to lower this to a non-RFC compliant value. I'm not sure how this this will affect clients, but 30min seems to be overly generous. Joseph Tam <jtam.home(a)gmail.com>

2 1

NetApp NFS vs. ZFS and NFS for Maildir
by Alessio Cecchi 26 Mar '16

26 Mar '16

Hi, I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan). Our setup is Dovecot, Maildir for email storage and NFS to share mailboxes (more than 30k users) across POP/IMAP and MX servers. NetApp NFS works fine also under high load but have some limitation for inode numbers per Volume and is expensive (but recently their prices have dropped). ZFS, I read, suggest to create many small Raid Group to increase IOPS, but this configuration (N Raid instead of one RAID-DP like NetApp) is more complex to manage, or not? Someone has experiences with ZFS and NFS(v3) in high load environments? Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

6 16

State for dsync not working.
by Simon Pierre Desrosiers 26 Mar '16

26 Mar '16

Hello, Since replication does not seem to work on Mac OSX, I will run doveadm sync by hand every few minutes. In order to improve efficiency, I would like to use state. I have tried a few ways, but I always get the following error doveadm(user): Error: Saved sync state is invalid, falling back to full sync: Invalid base64 data KBtMOd0181bQ/wAAoQsxE90181YBAAAAAQAAAAAAAAAAAAAAAAAAABtI+mg= Here is the code I use to sync : for user in $userlist ; do doveadm sync -u $user -s "/Users/_dovecot/.bin/syncStates/syncstate$user" tcp:xxx.xxx.xxx.xxx > "/Users/_dovecot/.bin/syncStates/syncstate$user" done Any idea on how I can correct the encoding error ? Thank you

1 0

Dovecot on FreeBSD 10.1
by Søren Andersen 25 Mar '16

25 Mar '16

I'm experiencing the same problem as mentioned earlier in the mailing list. - Is there some kind of workaround for this issue? http://dovecot.org/pipermail/dovecot/2015-April/100549.html /Søren

2 3

pigeonhole
by ferdinando 25 Mar '16

25 Mar '16

hi all im trying to install pigeonhole with dovecot2 but it stops during "make install" with this error: ========================================================================================== .... /bin/sh ../../libtool --mode=install /usr/bin/install -c sievec sieve-dump sieve-test sieve-filter '/usr/local/cluster/fs-cl00/mail/sieve/bin' libtool: install: /usr/bin/install -c .libs/sievec /usr/local/cluster/fs-cl00/mail/sieve/bin/sievec /usr/bin/install: setting permissions for '/usr/local/cluster/fs-cl00/mail/sieve/bin/sievec': No data available Makefile:450: recipe for target 'install-binPROGRAMS' failed make[3]: *** [install-binPROGRAMS] Error 1 make[3]: Leaving directory '/usr/local/src/pigeonhole/src/sieve-tools' Makefile:648: recipe for target 'install-am' failed make[2]: *** [install-am] Error 2 make[2]: Leaving directory '/usr/local/src/pigeonhole/src/sieve-tools' Makefile:417: recipe for target 'install-recursive' failed make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory '/usr/local/src/pigeonhole/src' Makefile:732: recipe for target 'install-recursive' failed make: *** [install-recursive] Error 1 [root@mail02 pigeonhole]# [root@mail02 pigeonhole]# ls -l /usr/local/cluster/fs-cl00/mail/sieve/bin/sievec -rw------- 1 root root 118504 Mar 23 19:31 /usr/local/cluster/fs-cl00/mail/sieve/bin/sievec [root@mail02 pigeonhole]# file /usr/local/cluster/fs-cl00/mail/sieve/bin/sievec /usr/local/cluster/fs-cl00/mail/sieve/bin/sievec: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=586da03004b77848d5e1c66f9bb33259a984ba48, not stripped [root@mail02 pigeonhole]# ... ========================================================================================== configure is ok, no particular options are required, only ldap module enabled by plugin. make is ok, the job exits with no errors. dovecot itself seems ok, it's up and running what do i miss? need some particular uids/gids to accomplish the installation? here's my specs: pigeonhole commit 1d36af65473016aa8339a3671d8b555adc71cef3 dovecot commit 4a769606eb49926c6dfb9a532186e9e737683283 Fedora release 23 Linux version 4.4.5-300.fc23.x86_64 gcc version 5.3.1 thanks a lot in advance for any help =) Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo. E chiami gratis anche i numeri fissi e mobili nel mondo! Scarica subito l’app Vai su https://www.indoona.com/

1 1

IMAP Idle
by Joy 25 Mar '16

25 Mar '16

Hi guys, We have implement imap idle in web mail built by us to have push mail feature. IMAP idle working perfectly with browser notification and we are happy with it but having one issue with users who close the browser directly and never logout in that case there are number of idle connection which are not in use and users are unable to login once IP wise connection limit is reached. Dovecot is not closing connection which are not in use, is there any setting available which can help me to resolve this issue.

1 0

Trouble handling encrypted passwords
by Dan Wilson 25 Mar '16

25 Mar '16

I am running into trouble when I try to use encrypted passwords. I set my dovecot 2.2.10-5 installation back to plain text passwords and was then able to login and retrieve emails. When I set the default_pass_scheme to MD5-CRYPT in my sql config file, and store an MD5-CRYPT password in the sql database, I invariably get password mismatch errors on login: Mar 24 19:40:30 server1 dovecot: auth: Debug: client in: AUTH#0113#011PLAIN#011service=imap#011secured#011session=25HcztQuwQBHCDBi#011lip=xx.x.xx.xxx#011rip=71.8.48.98#011lport=993#011rport=60609#011resp=<hidden> Mar 24 19:40:34 server1 dovecot: auth-worker(9443): Debug: sql(test1(a)xxxxxx.com,xx.x.xx.xx): query: SELECT email as user, password FROM virtual_users WHERE email='test1(a)xxxxxx.com'; Mar 24 19:40:34 server1 dovecot: auth-worker(9443): sql(test1(a)xxxxxx,xx.x.xx.xx): Password mismatch Mar 24 19:40:36 server1 dovecot: auth: Debug: client passdb out: FAIL#0113#011user=test1(a)xxxxxx.com The password in the database was created using doveadm pw command and is stored as: {MD5-CRYPT}$1$5QAeG70c$dP8y7XJVd.VxZBF/DvcAa0 I tried removing the {MD5-CRYPT} from the password string, but that didn't help. I have to be doing something stupid because this should just work! Any ideas from anyone? Thanks much, Dan

1 0

Fw: new important message
by dovecot 25 Mar '16

25 Mar '16

Hello! New message, please read <http://myomtedavi.com/ask.php?397> dovecot

1 0

Replication issues master <-> master nfs backend
by William L. Thomson Jr. 24 Mar '16

24 Mar '16

I keep having some replication issues and not sure what can be done to resolve or correct. It does not seem to happen all the time, though for the last ~30 or so minutes and many messages seems to be happening consistent for me. I have 2 mail servers, basically clones, and thus master master replication. Most of the time things work fine. But many times an email or several will arrive on one, and never replicate to the other. I am not as concerned on the never replicating, as I am that the user never gets notified. Mail arrives on say server 1, users are checking mail on server 2, and they never see the email on server 2. This is not always the case, but its happening enough daily. I then log into one and run sync manually. Which usually syncs the mail on both servers, and then it arrives in the inbox. Here is an example, mail is on mail2, but not mail1. I am checking email on mail1 so I am not seeing the 1 email. Mail1 /home/wlt-ml/.maildir/new: total 0 Mail2 /home/wlt-ml/.maildir/new: total 12 -rw------- 1 wlt-ml site1 8502 Mar 22 14:57 1458673024.7643.mail2 Then I manually log into mail2 and run this command, though usually I can run it from either side, and just change the name to the other server. doveadm sync -u "*" remote:mail1 And then I end up with the missing email on mail1, and it arrives in my email client shortly there after Mail1 /home/wlt-ml/.maildir/new: total 12 -rw------- 1 wlt-ml site1 8502 Mar 22 14:57 1458673051.M838843P26735.mail1,S=8502,W=8678:2,T I have no idea why it does this. It seems to happen when when a full sync has taken place per doveadm replicator status wlt-ml. There does not seem to be any settings to force a full vs fast sync more often. No clue if this is even a full vs fast issue or other. I think it tends to happen more when people stay connected to the imap server. I had a theory that closing the email client and opening it again will get dovecot to sync. I believe this is still the case, but not able to confirm 100%. Also users are reporting closing Thunderbird. I can see them logging out and back in in the logs, but email does not replicate or show till I run doveadm sync manually. Tempted to have cron invoke that on the regular, but seems very hackish and likely will have its own issues doing that. Since its not the right way or how things were designed. Not sure if this is a bug or what. Hopefully miss-configuration on my end. Open to any feedback, advice, etc. I can provide replicator configuration but its pretty straight forward and mostly copy/paste from the replication page. Replication works, just seems it is not triggered to replicate at times or something. dovecot --version 2.2.22 (fe789d2) -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com

2 3