Hi Eduardo,

Since I few day I try to setup this config 2, but I've really problem with the NFS.

We come from Google Apps :) I several people on the hospital have large mailbox. For example 96000 mails in the INBOX folder, when I do "ls" that is quick for display mail, but with dovecot... I need wait 1 minuts the first time and after display the amount of mail it's quick.

We don't have NetAPP, we've Dell Equallogic with SATA2 7200rpm, gigabits ports. The Equallogic it's just for store email, and index file.

My config:

Dell Equallogic |-> VMWare ESXI (iscsi mount in VMware with VMWare VMFS 5 format 4To) |--> Mounted as slave disk on a VM Ubuntu and exported with this options "(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,async)" and formated as ReiserFS (I've read it's the best for many small files). |---> My 2 Dovecot Backend mount the NFS (fstab) to my previous VM with this options "nfs defaults 0 0" |----> Mail it's stored in MailDir format "mail_location = maildir:%h"

When I go on Roundcube with a mailbox who I've 96000 in the INBOX "cur" folder, the first time I need 1 minutes waiting. The second time it's a little more quick 10 seconds... But when I go back again (after a few hours) it's slow again... And we'are just 5 users at the moment for test...

I've not setup director for the moment I try first dovecot on my lab befor add director and move mailbox from Google...

Have you recommandation for tune or make NFS more quickly, because with Google it's really fast (ok it's google) but if we move to a more slow system, the user on the hospital would no be happy. I've try many config, but no really success for have a fast NFS, I don't know if the problem it's VMWARE or if the problem it's just my NFS options.

I'm not pettry good with linux, I'm most good with Windows server, I'm not a good man Google, Microsoft, I know it's devil for every linux sysadmin :)

Thanks, Nathan

----- Mail original ----- De: "Eduardo Ramos" eduardo@freedominterface.org À: dovecot@dovecot.org Envoyé: Jeudi 24 Juillet 2014 14:01:33 Objet: Re: Multiple servers and NFS

Hi Richard,

You can use one or more instances of Dovecot on the same machine, as you can see here (http://wiki2.dovecot.org/RunningDovecot) "Running Multiple Invocations of Dovecot".

The problem with DNS round-robind is that if you server goes down, DNS continues resolving for it. I would recommend use some balancer like LVS+keepalived.

Consider that multi layer solution:

| LVS + keepalived | | LVS + keepalived |

      |                                |

-------------- -------------- | Director 1 | | Director 2 |

      |                                |

----------------- ----------------- | IMAP/POP/LMTP | | IMAP/POP/LMTP | | Backend | | Backend |

      \                                /
       \                              /
        \       ------------         /
         -------|  NetAPP  |---------
                ------------

Remember, directors and backends could run on the same machine. I have a lab running that way. Maybe I can help you deploy. On 07/23/2014 07:23 PM, Richard Hector wrote:

Hi all,

For some reason, I didn't go to http://wiki2.dovecot.org/NFS until now, and I'm starting to get worried ...

The plan was to have multple servers (MXes) receiving mail, and delivering via LMTP to multiple backend dovecot servers (with amavis in front of dovecot; LMTP both sides). Then we'd have multiple servers for clients to use IMAP or POP3.

This is more or less how the system already works, except with Courier IMAP, and postfix on the backends, delivering to maildirs with procmail.

But with the recommendation to use the Director for both IMAP/POP3 and LMTP - that starts to sound like I need a whole bunch more servers to run Directors and proxies, and even then it might not be a good idea to have different servers running lmtp and imap/pop.

One possible mitigating point is that our 'load balancing' is DNS round-robin, so a given client will probably stick with a single imap/pop server anyway, but if the user has multiple clients (desktop/mobile etc) then they may still hit different servers.

Can someone clarify best practice for a setup needing multiple servers for load balancing and redundancy?

Is Courier already likely to have been suffering these problems?

Oh, the NFS server is a NetApp Filer, if that matters.

I'm using dovecot 2.2.9 from debian wheezy backports, in order to get the quota policy daemon support.

Thanks, Richard

Hello Eduardo,

Thank for your reply.

2 days ago I've try to setup the "INDEX" and "CONTROL" file on my local Harddrive (outside the NFS who the maildir it's stored).

That was my config line: maildir:%h:INDEX=/opt/dovecot/indexes/%u:CONTROL=/opt/dovecot/indexes/%u

But that was allway slow on the first access, I've think the "cache" expire after 1h for example and Dovecot need make a new "cache" after and that's the reason why the first access need 60 sec. and the next access need just 10 sec.

I've read the dovecot doku, she's really more clear that postfix doku... really :) It's easy to read it and understand the doc because we learn step by step and when we're in one step we learn everything about it and don't need to go in another section and come back again for finish the step...

Now I think my config (NFS mount/options) it's just a little poor and that's the reason why it's slow.

Bests Regards, Nathan

----- Mail original ----- De: "Eduardo Ramos" eduardo@freedominterface.org À: dovecot@dovecot.org Envoyé: Jeudi 24 Juillet 2014 19:27:50 Objet: Re: Multiple servers and NFS

Hi Nathan!

I think you can consider about your index files. By default dovecot stores index files in mailbox, but you can define another location. Local disk could be better choice. Take I look at this:

http://wiki2.dovecot.org/MailLocation http://wiki2.dovecot.org/MailLocation/SharedDisk http://wiki2.dovecot.org/IndexFiles

I think it will help you.

On 07/24/2014 02:09 PM, Nathan Schultheiss wrote:

Hi Eduardo,

Since I few day I try to setup this config 2, but I've really problem with the NFS.

We come from Google Apps :) I several people on the hospital have large mailbox. For example 96000 mails in the INBOX folder, when I do "ls" that is quick for display mail, but with dovecot... I need wait 1 minuts the first time and after display the amount of mail it's quick.

We don't have NetAPP, we've Dell Equallogic with SATA2 7200rpm, gigabits ports. The Equallogic it's just for store email, and index file.

My config:

Dell Equallogic |-> VMWare ESXI (iscsi mount in VMware with VMWare VMFS 5 format 4To) |--> Mounted as slave disk on a VM Ubuntu and exported with this options "(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,async)" and formated as ReiserFS (I've read it's the best for many small files). |---> My 2 Dovecot Backend mount the NFS (fstab) to my previous VM with this options "nfs defaults 0 0" |----> Mail it's stored in MailDir format "mail_location = maildir:%h"

When I go on Roundcube with a mailbox who I've 96000 in the INBOX "cur" folder, the first time I need 1 minutes waiting. The second time it's a little more quick 10 seconds... But when I go back again (after a few hours) it's slow again... And we'are just 5 users at the moment for test...

I've not setup director for the moment I try first dovecot on my lab befor add director and move mailbox from Google...

Have you recommandation for tune or make NFS more quickly, because with Google it's really fast (ok it's google) but if we move to a more slow system, the user on the hospital would no be happy. I've try many config, but no really success for have a fast NFS, I don't know if the problem it's VMWARE or if the problem it's just my NFS options.

I'm not pettry good with linux, I'm most good with Windows server, I'm not a good man Google, Microsoft, I know it's devil for every linux sysadmin :)

Thanks, Nathan

----- Mail original ----- De: "Eduardo Ramos" eduardo@freedominterface.org À: dovecot@dovecot.org Envoyé: Jeudi 24 Juillet 2014 14:01:33 Objet: Re: Multiple servers and NFS

Hi Richard,

You can use one or more instances of Dovecot on the same machine, as you can see here (http://wiki2.dovecot.org/RunningDovecot) "Running Multiple Invocations of Dovecot".

The problem with DNS round-robind is that if you server goes down, DNS continues resolving for it. I would recommend use some balancer like LVS+keepalived.

Consider that multi layer solution:

---

| LVS + keepalived | | LVS + keepalived |

---

       |                                |
--------------                   --------------
| Director 1 |                   | Director 2 |
--------------                   --------------
       |                                |

----------------- ----------------- | IMAP/POP/LMTP | | IMAP/POP/LMTP | | Backend | | Backend |

---

       \                                /
        \                              /
         \       ------------         /
          -------|  NetAPP  |---------
                 ------------

Remember, directors and backends could run on the same machine. I have a lab running that way. Maybe I can help you deploy. On 07/23/2014 07:23 PM, Richard Hector wrote:

...

Hi all,

For some reason, I didn't go to http://wiki2.dovecot.org/NFS until now, and I'm starting to get worried ...

The plan was to have multple servers (MXes) receiving mail, and delivering via LMTP to multiple backend dovecot servers (with amavis in front of dovecot; LMTP both sides). Then we'd have multiple servers for clients to use IMAP or POP3.

This is more or less how the system already works, except with Courier IMAP, and postfix on the backends, delivering to maildirs with procmail.

But with the recommendation to use the Director for both IMAP/POP3 and LMTP - that starts to sound like I need a whole bunch more servers to run Directors and proxies, and even then it might not be a good idea to have different servers running lmtp and imap/pop.

One possible mitigating point is that our 'load balancing' is DNS round-robin, so a given client will probably stick with a single imap/pop server anyway, but if the user has multiple clients (desktop/mobile etc) then they may still hit different servers.

Can someone clarify best practice for a setup needing multiple servers for load balancing and redundancy?

Is Courier already likely to have been suffering these problems?

Oh, the NFS server is a NetApp Filer, if that matters.

I'm using dovecot 2.2.9 from debian wheezy backports, in order to get the quota policy daemon support.

Thanks, Richard

Hallo Patrick,

I've a Dell Equallogic, I make one "volume" with 4 To capacity. This volume have a iscsi key, I had this key to VMWare for link VMware with this "volume".

After on VMware I format this volume with the VMFS5 format, VMware add it "Datastore2".

I make a first VM (storage VM) with 2 Hard Drive: First Hardrive (located in Datastore1) with 20Go capacity, I setup on it the ubuntu system. Second Hardrive (located in Datastore2) with 4To capacity, That's my slave hard drive formated in ReiserFS and mounted has /home/vmail I setup nfs-server service with this options in my "export" file (rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,async), the /home/vmail it's the export point.

I setup now 2 new VM (Dovecot Backend) with just one local Hard Drive (20Go). After the setup, I install nfs-client and mount /home/vmail to my "storage VMAIL".

Dovecot would store mail in /home/vmail/user I've just 1 domain, it's why I store direct in user directore (ex: /home/vmail/nathan.schultheiss ).

The storage VM it's here because I can't mount the ISCSI session on the 2 Backend server. I mount this "volume" first on VMWare because it's more easy for manage it and setup it on a Virtual Server (my storage server in this case).

It's not impossible that I make a total wrong archi... But I've think, easy, I mount the ISCSI session on my 2 backend in /home/vmail, that work for 1 but not for the second, and I receive many error. I think Equallogic don't like that 2 Virtual Server wrote on the same time to the same ISCSI session.

Freundliche Grüße aus Frankreich, Nathan

----- Mail original ----- De: "Patrick Westenberg" pw@wk-serv.de À: dovecot@dovecot.org Envoyé: Jeudi 24 Juillet 2014 19:56:26 Objet: Re: Multiple servers and NFS

Nathan Schultheiss schrieb:

My config:

Dell Equallogic |-> VMWare ESXI (iscsi mount in VMware with VMWare VMFS 5 format 4To) |--> Mounted as slave disk on a VM Ubuntu and exported with this options "(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,async)" and formated as ReiserFS (I've read it's the best for many small files). |---> My 2 Dovecot Backend mount the NFS (fstab) to my previous VM with this options "nfs defaults 0 0" |----> Mail it's stored in MailDir format "mail_location = maildir:%h"

I don't understand your setup esp. that your backend servers mount something from the "previous VM"?

Patrick

On 24 Jul 2014, at 20:09, Nathan Schultheiss nathan@schultheiss.fr wrote:

When I go on Roundcube with a mailbox who I've 96000 in the INBOX "cur" folder, the first time I need 1 minutes waiting. The second time it's a little more quick 10 seconds... But when I go back again (after a few hours) it's slow again... And we'are just 5 users at the moment for test...

See if maildir_very_dirty_syncs=yes helps.

On 25/07/14 09:12, Richard Hector wrote:

Rather than trying to draw increasingly complex diagrams in ASCII, I've put some here (without the LVS layer): https://walnut.gen.nz/mail-architectures.png

I've come up with a revised plan - I think we can do without LVS; SMTP should just work with multiple MX records, and IMAP/POP should be fine as well with RRDNS - the machines should be up most of the time, and if a customer has to click to reconnect every now and then on the rare occasions when they're not it's not a huge deal. Otherwise, we could also do load balancing on our routers.

Anyway - any comments on the sanity of this diagram most welcome :-)

https://walnut.gen.nz/mail-architecture-2.png

Richard

On 25/07/14 15:30, Eduardo Ramos wrote:

Hi Richard,

In fact I thought it a little confusing. I had some bad experience with DNS RR when one of my IMAP server got down. Clients continued trying connect to broken server and it caused some problems. But when everything is ok, it works well.

I drew a diagram with my idea. What do you think?

https://dl.dropboxusercontent.com/u/41373531/mail.png

Interesting, thanks. I'd forgotten to draw in the director ring.

As I said, if we need load balancing we can do that on the router, which as I understand it will do more or less the same thing as LVS. It might be Cisco SLB, but I'm not sure; I'm not the router guy :-)

But what interests me most is that your diagram shows the mx servers connecting directly to the backend servers, rather than going through the proxy director - I thought that was a no-no. Oh, and I don't think we want to load down our front-end MX servers with amavis, either.

Thanks for your input :-)

Richard

Am 25.07.2014 um 16:12 schrieb Eduardo Ramos:

I did not understand what the advantage of use dovecot LMTP with director too.

in "very short" words... with nfs ,the director should avoid concurrent events which may happen with lmtp too, depending to multiple server setup

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Am 28.07.2014 um 13:09 schrieb Nick Edwards:

On 7/26/14, Robert Schetterer rs@sys4.de wrote:

...

Am 25.07.2014 um 16:12 schrieb Eduardo Ramos:

...

I did not understand what the advantage of use dovecot LMTP with director too.

in "very short" words... with nfs ,the director should avoid concurrent events which may happen with lmtp too, depending to multiple server setup

A few of us run large NFS based systems without director, however mostly 99% pop3, not using director on imap has little impact either from our tests, remember, director is only a couple years at most old, people have been doing NFS mailstorage for decades, and with relation to dovecot, ten years or so, the sky never collapsed back then, it hasnt now either thus far :-> using director was considered in risk assessment as its another point of failure, and weighed against its claimed benefit, the decision was made its not justified.

note: we dont use lmtp, each mx mounts/stores directly to EMC storgage with dovecot-lda, 14 front ends = 14 direct storages, sure, means dovecot needs to be installed on each mx (but not listening), but it eliminates the need for dedicated back ends to send to, each mx is that backend.

12 pop3 servers, of note however, we use index:memory on pop3 and smtp's

mail_location = maildir:/mail/%1n/%1.1n/%2.1n/%n/Maildir:INDEX=MEMORY

only 3 imaps one of which is webmail, and of course we do not use index:memory on them, these are behind real (serveriron's) load balancers, so if using pretend load balancers :-> YMMV

That looks fine but now whats the problem ? For sure there are many ways to goal. do what you like. I also have no director setup using cluster file systems with loadbalancers working fine.

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

On 7/29/14, Daniel Parthey kabelpada@kabelmail.de wrote:

Nick Edwards wrote:

...

On 7/26/14, Robert Schetterer rs@sys4.de wrote:

...

Am 25.07.2014 um 16:12 schrieb Eduardo Ramos:

...

I did not understand what the advantage of use dovecot LMTP with director too.

in "very short" words... with nfs ,the director should avoid concurrent events which may happen with lmtp too, depending to multiple server setup

using director was considered in risk assessment as its another point of failure, and weighed against its claimed benefit, the decision was made its not justified.

mail_location = maildir:/mail/%1n/%1.1n/%2.1n/%n/Maildir:INDEX=MEMORY

With maildir you won't have data-loss without the director, since the index files are auto-regenerated without any problem.

disagree, if we'd had data loss we would have a case to use director, we also had none when we were using qmail and vpopmail, if dovecot did, and as said we are yet to see it, but if it did have data loss, than thats dovecots design issue, but I have no doubt it is that much of an issue.

and from memory the only difference is some messages that just arrive may or may not appear immediately, this is only a problem with imap, and of all the users, we have a some total of about 200 that bother with imap, the other 100K plus use pop3

With mdbox on NFS and no director, you will have data-loss sooner or later:

irrelevant, we use Maildir, it is time proved.