- dovecot - dovecot.org

[Dovecot] Dovecot Max Connections & mbox vs. maildir format - Recommendations?
by V S Rao 25 May '09

25 May '09

>> I had big issues with timeouts until I realize the the imap and mbox format >> isnt a good choice if your users have huge naiboxes. >> At my server there were users with mailboxes arount 700MB to over 1GB. each >> time one of those users checked their email >> the imap server had to open this huge file and parse it, causing I/O issues. >> The only solution I found it was switch from mbox format to maildir format. I am changing the subject line to reflect the discussion. I too have a similar case with large mailboxes. My user mix is 70% POP3 vs. 30% IMAP. Would maildir still be the best recommended format? What is the opinion of the experts here? The reason I am asking is that I am preparing to upgrade my mail server to RHEL 5.3. That would mean my Dovecot will be atleast 1.0.7, which should not be bad. If the recommendation is that I go from mbox format to Maildir, I want to plan that as well. However I don't want to land up in a situation where, I have to move back from Maildir to mbox again. So please do advice me on the best. BTW one question: Assuming that my server has enough resources, ( 8 CPU @ 3.0GHz, 8GB RAM, 2Gbps FC Storage directly attached to the server & Gigabit NIC) what is the maximum number of concurrent connections that Dovecot can handle POP3 + IMAP combined? And I am asking for all the processes combined: POP3-login processes IMAP-login processes POP3 sessions IMAP sessions Any TCP/IP kernel parameters to be tuned? or maybe in dovecot.conf? Any advice will be immensely valuable to me while doing the migration. Thanks and Regards V S Rao

2 1

[Dovecot] How can I put a file back in Maildir folder?
by Allan Cassaro 23 May '09

23 May '09

Hi, I have a problem: today, when I ran "fsck" some file was linked to "lost+find" folder... After some hours I has all files mapped (File / User). So... how can I put this files back in Maildir folder of this users? I tried with: cat $file | deliver -d username but without success... Any help will be appreciated! Thanks!

2 2

[Dovecot] How to add new features to sieve?
by Ed W 23 May '09

23 May '09

Hi, I like the idea of switching from maildrop to dovecot deliver+sieve, however, we use a couple of features that I'm not sure how to migrate: - We query the userdb and make some user specific params available to the filtering script (invented example, but say number of days to not repeat for the vacation script) - We have a fairly advanced filter (perl) which mangles the email in very specific ways to support users over low bandwidth connections. The email is delivered twice, once to an "Archive" folder and again to the inbox after being mangled. How to integrate this "mangling" script? - We have some custom routes for certain types of SMS messages, ie we have a free email to SMS option for certain networks. Could this be integrated and caught in "alert" functionality (how is the sms backend actually configured anyway?) I think the answer is to write one or more new sieve functions to provide this functionality, but can you give me a quickstart guide on how to do this? Certainly the first requirement is quite an interesting and generic plugin "lookup_db" type function which must have wider utility. The second requirement is a bit more specific, kind of like an xfilter wrapper which could be setup by the administrator I guess? Thanks Ed W

1 0

Re: [Dovecot] How to run Dovecot as IMAP Proxy?
by V S Rao 22 May '09

22 May '09

> > You will not get any support for 0.99 here, unless Timo can provide it. > It is *that* old, and your decision to use it - regardless of your > claimed reason - is baffling. I admit I am running a pretty old version and frankly speaking I was not hoping for any support. Let RH earn their money we paid as support. As you would have noticed I was looking for help in running Dovecot as a IMAP Proxy without the need to setup SQL etc. I only went into details to give clarity. I really appreciate all the responses and suggestions. >> We use recent Dovecot from atrpms.net with RHEL 5.3. Userbase is 25.000. >> Dovecot runs on three machines - as master (proxy) and with two slaves. >> The system is stable. We hardly need to touch it. I would be very much interested in knowing the details of your setup as I am suffering here with only 6000+ users. And that too with a config, which I frankly think is an overkill for our requirement. However maybe I should start a separate thread for that? Thanks and Regards V S Rao.

1 0

Re: [Dovecot] How to run Dovecot as IMAP Proxy?
by V S Rao 22 May '09

22 May '09

So I guess I will take the problem to imapproxy group and find out if I can get a clue to the issue I am facing. BTW, just to confirm once again, are there any known issues with Dovecot & imapproxyd? Because I find it strange that there should be around 8000 established connections between Squirrelmail and imapproxyd & around 400 connections between imapproxyd and Dovecot while on the mail server I see only a couple of connections (typically around 40). Or is it some configuration in Dovecot which is causing this kind of behavior, as I did not notice this while running uw-imap. Regards V S Rao

4 3

[Dovecot] v1.2.rc4 released
by Timo Sirainen 22 May '09

22 May '09

http://dovecot.org/releases/1.2/rc/dovecot-1.2.rc4.tar.gz http://dovecot.org/releases/1.2/rc/dovecot-1.2.rc4.tar.gz.sig I'm pretty hopeful that virtual mailbox code is finally crash-free. If people don't find many problems from this release, I'm thinking that rc5 could come in a couple of weeks and that would be renamed to v1.2.0 after a week or so if no one has reported bugs. v1.2.rc4 2009-05-17 Timo Sirainen <tss(a)iki.fi> * If /dev/arandom exists, use it instead of /dev/urandom (OpenBSD). * When logging to a file, the lines now start with a timestamp instead of "dovecot: " prefix. + IMAP: When multiple commands are pipelined, try harder to combine their mailbox syncing together. For example with Maildir pipelining STORE 1:* +FLAGS \Deleted and EXPUNGE commands the files won't be unnecessarily rename()d before being unlink()ed. + imap-proxy: Send backend's CAPABILITY if it's different from what was sent to client before. + IMAP: struct mail now keeps track of all kinds of statistics, such as number of open()s, stat()s, bytes read, etc. These fields could be exported by some kind of a statistics plugin (not included yet). + IMAP: SEARCH command now dynamically figures out how to run about 0.20 .. 0.25 seconds before seeing if there's other work to do. This makes the SEARCH performance much better. - Fixes to shared mailbox handling. - Fixes to virtual mailboxes. - THREAD command could have crashed. - Fixes to expire-tool. - mbox: Don't break if From_-line is preceded by CRLF (instead of LF). - dict process wasn't restarted after SIGHUP was sent to master.

2 1

[Dovecot] v1.1.15 released
by Timo Sirainen 22 May '09

22 May '09

http://dovecot.org/releases/1.1/dovecot-1.1.15.tar.gz http://dovecot.org/releases/1.1/dovecot-1.1.15.tar.gz.sig The command pipelining fix is pretty interesting. Wonder how much it improves performance, especially when deleting lots of messages from a large mailbox. v1.1.15 2009-05-17 Timo Sirainen <tss(a)iki.fi> + IMAP: When multiple commands are pipelined, try harder to combine their mailbox syncing together. For example with Maildir pipelining STORE 1:* +FLAGS \Deleted and EXPUNGE commands the files won't be unnecessarily rename()d before being unlink()ed. - IMAP: SEARCH command was slower than necessary in a large mailbox. - deliver: When forwarding messages, if -f parameter was given, it should have been used as the Return-Path. - Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. - mbox: Don't crash when expunging all messages and file doesn't end with [CR]LF. - expire-tool: Use mail_uid and mail_gid settings if userdb doesn't return uid/gid. - Berkeley DB dict: Transaction rollbacking was implemented wrong. - zlib plugin: Some email / FETCH command combinations could have disconnected the client or sent compressed garbage.

2 1

Re: [Dovecot] Authenticating with /etc/passwd
by June Qiu 22 May '09

22 May '09

Apologies for the confusion.. But I finally got what you meant. Yes, the version exists, but was omitted in the extract of the email. --- On Thu, 5/21/09, Charles Marcus <CMarcus(a)Media-Brokers.com> wrote: > From: Charles Marcus <CMarcus(a)Media-Brokers.com> > Subject: Re: [Dovecot] Authenticating with /etc/passwd > To: dovecot(a)dovecot.org > Date: Thursday, May 21, 2009, 10:17 PM > On 5/21/2009 10:11 AM, June Qiu > wrote: > > it is as displayed > > So there's some weird dovecot -n bug that doesn't show the > dovecot > version if the 'Warning: fd limit...' warning is output? > > > --- On Thu, 5/21/09, Charles Marcus <CMarcus(a)Media-Brokers.com> > wrote: > > > >> From: Charles Marcus <CMarcus(a)Media-Brokers.com> > >> Subject: Re: [Dovecot] Authenticating with > /etc/passwd > >> To: "June Qiu" <qljune(a)yahoo.com> > >> Cc: dovecot(a)dovecot.org > >> Date: Thursday, May 21, 2009, 6:39 PM > >> On 5/20/2009, June Qiu (qljune(a)yahoo.com) > >> wrote: > >>> dovecot -n: > >>> > >>> Warning: fd limit 256 is lower than what > Dovecot can > >> use under full load (more than 768). Either grow > the limit > >> or change login_max_processes_count and > max_mail_processes > >> settings > >>> # OS: SunOS 5.10 sun4u > >>> info_log_path: /var/log/dovecot.log > >> Is the missing version line a copy/paste error? Or > did the > >> warning > >> obscure it somehow? > > > -- > > Best regards, > > Charles >

1 0

Re: [Dovecot] How to run Dovecot as IMAP Proxy?
by V S Rao 22 May '09

22 May '09

Thanks for the many responses and views. I have taken RH support for my mail server only and so have to ensure they support me. So am going with their recommended version which is 0.99.x. Now on the webmail I have migrated to RHEL 5.3 and the dovecot being used there is 1.0.7.x. So that should support proxying, right? BTW, originally I did not go to Redhat. I moved from uw-imap to Dovecot 1.1.14 on the mail server for POP3. The problem of POP3 timeouts continued and I could not find any reason why POP3 was timing out when 400+ concurrent IMAP sessions were working fine. Moreover other network services such as SMTP, Telnet, SSH etc were working fine. Had the box installed behind a IPS box to see if there were any DoS attacks, but had to rule that out. As a last resort, paid RH for support and they made me downgrade Dovecot from 1.1.14 to 0.99.x. Surprisingly the moment I downgraded dovecot, the problem shifted to IMAP & remained ever since. So naturally they told me uw-imap was the problem and made me move to Dovecot 0.99.x for IMAP as well, but no improvement. But the surprising thing, as I mentioned in my original post, IMAP works fine on command line, with clients such as Outlook, THunderbird etc., but Webmail is very slow that its almost not available. BTW webmail was also running on RHEL 4.0. Seeing that the problem existed in webmail alone I upgraded webmail to RHEL 5.3 with all the latest RH supported packages on a new hardware. Any ideas to help me debug this issue would be of great help as I am stuck on this issue for almost a month now. Oh, btw regarding stats, I don't have a measure, but when webmail was working the login was well within 5 seconds. Now it takes forever (more than a min) or timeouts most of the time. Even after login, any request such as opening a mail or changing to a mail folder was well within 5 seconds. Now again it takes forever or timeouts most of the time. Thanks and Regards V S Rao ----- Original Message ----- From: "Timo Sirainen" <tss(a)iki.fi> To: "Dovecot Mailing List" <dovecot(a)dovecot.org> Sent: Friday, May 22, 2009 12:32:24 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: Re: [Dovecot] How to run Dovecot as IMAP Proxy? On Thu, 2009-05-21 at 11:47 -0700, Seth Mattinen wrote: > Yes, that too. My point was not to judge Dovecot's performance based on > a 0.99 version. ;) I've successfully used several versions of dovecot > 1.1.x with imapproxyd and IMAP-only webmail clients. Do you have any numbers about how much faster it works with imapproxyd vs. without?

4 3

Re: [Dovecot] How to run Dovecot as IMAP Proxy?
by V S Rao 22 May '09

22 May '09

Hi Timo, Thanks for the response. Apologize, but my responses are going to be a bit lengthy. > I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run > webmail using squirrelmail. When running uw-imap I used to run > up.imapproxyd on the webmail server for faster responses. After > migrating to Dovecot, I find that up.imapproxyd does not work well > with dovecot. > Why not? Here are my observations. I have around 6000+ mailboxes and roughly the same number of users. Earlier the mail server (running sendmail + uw-imap for POP3 & IMAP) would have around 80 pop3 connections (peak) and around 300 IMAP connections, concurrent. (ps -aef | grep imap | wc -l or ps -aef | grep pop3 | wc -l). There have been cases where I have observed upto 500 concurrent IMAP sessions. The IMAP connections are from a webmail server running Apache with Squirrelmail. The observation was that response was slow & so based on the suggestion on Squirrelmail for performance improvement we have installed up-imapproxyd on the squirrelmail machine. There was a significant improvement in the response times for the users, because of caching. After that I encountered some strange problems of POP3 timing out for users (earlier I did post that problem in this forum). I opened a ticket with Redhat and naturally they refused to support me with uw-imap running. So switched to Dovecot 0.99.x (I run the server on RHEL 4.0 and that is the max version supported by Redhat for that version). Ever since I did that POP3 works fine but now webmail is almost not available to the users. People usually get "connection dropped by IMAP server". However the IMAP server seems to work fine. Checked through manual "telnet <ip.address.of.mailserver> 143 & also through other client such as outlook & Thunderbird. I noticed that while the number of established connections for the webmail to mail server was shown as around 40 to 50 on the mail server, the same is shown as around 700 on webmail server. { netstat -an | grep :143 | grep ESTABLISHED | wc -l }. imapproxyd is configured to listen on port 9143 on the localhost and communicates to the mail server on std port 143. What was really bizzare is that when I check for the number of connections on port 9143 it shows around 8000! netstat -an | grep :9143 | grep ESTABLISHED | wc -l So I disabled the imapproxyd on the webmail & now at least the 50 connections are consistent. However the response is extremely slow and so was contemplating on the idea of trying out using another IMAP Proxy server and hence the question. > I don't understand Databases and don't want to install mysql etc for a > simple thing like IMAP PRoxy. > Do you just want it to proxy everyone to a single destination or to > different servers based on username? If you want everyone to the same > destination, why do you even want to use a proxy? I want to proxy all user to the same server. The idea being as explained above benefit from caching. > My question is can Dovecot be run as a IMAP PRoxy without the hassle > of setting up mysql etc? I use /etc/passwd for auth on the actual > dovecot server running IMAP & POP3. > Using checkpassword or SQLite would probably be the easiest solution. Or > generation of a passwd-file, but you'd have to keep it updated if users > change. If I used a passwd file on the webmail machine, running dovecot as a Proxy how would I configure dovecot on the webmail server? It has to accept connections from the squirrelmail and talk to the mailserver. I can do a rsync of the passwd file from the mail server to the webmail server using a small script whenever the file changes on the mail server. Of course the fundamental question from your response, I guess is, without a proxy why is the webmail slow? If proxy is not my problem, can you pls throw some light on why I am facing performance issues on my webmail alone? Thanks and Regards V S Rao

4 6