[Dovecot] lmtp problem with wrong index path

[dalevizo]

On Mon 25/02/2013 16:39, Timo Sirainen wrote:
> Since you're using directors, you can use mail_nfs_index=no and mail_nfs_storage=no. The performance will be better.

Ok but that'll have to wait until we've fully migrated from our current
MX servers with direct access to the mailboxes to lmtp.

> The main problem here is that you create indexes using %u, but there are 3 different valid %us: uid, mail and mailAlternateAddress. So you're most likely hitting problems for users whose mails are being delivered using mailAlternateAddress. Those alias expansions should normally be handled by your MTA.

Alas no. The MTA does handle alias expansions :

root at vader:lmtp $ ldapsearch -h deer -x mail=daleviz1 at otenet.gr uid uidNumber gidNumber mail mailAlternateAddress mailPath folderPath -LLL     
dn: uid=daleviz1,ou=people,dc=otenet,dc=gr
uid: daleviz1
uidNumber: 69757
gidNumber: 647
mail: daleviz1 at otenet.gr
mailAlternateAddress: dovecottest at otenet.gr
mailPath: /var/mail/U/V/H/daleviz1
folderPath: /var/mail/folders/U/V/H/daleviz1

root at vader:lmtp $ telnet deneb.otenet.gr 25
Trying 83.235.66.54...
Connected to deneb.otenet.gr (83.235.66.54).
Escape character is '^]'.
220 ESMTP
helo otenet.gr
250 deneb.otenet.gr
mail from:<dalevizo at otenet.gr>
250 2.1.0 Ok
rcpt to:<dovecottest at otenet.gr>
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
test mail
.
250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as E78418E0023
quit
221 2.0.0 Bye
Connection closed by foreign host.

Feb 26 00:19:07 deneb postfix/lmtp[19885]: E78418E0023: to=<daleviz1 at otenet.gr>, orig_to=<dovecottest at otenet.gr>, relay=pop08.otenet.gr[83.235.66.47]:24, delay=0.03, delays=0.01/0/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 <daleviz1 at otenet.gr> GgW/NcziK1GCKwAADehEhg Saved)

Feb 26 00:19:07 pop08 dovecot: lmtp(11138, daleviz1 at otenet.gr): GgW/NcziK1GCKwAADehEhg: msgid=unspecified: saved mail to INBOX

> With the above config you should be able to solve this by not using %u but the LDAP's mail field. So something like (with cleaning up the whole config):
> 
> user_attrs = \
>   =user=%{ldap:mail}, \
>   =home=%{ldap:folderPath}, \
>   =quota_rule=*:storage=%{ldap:mailQuota}M, \
>   =uid=%{ldap:uidNumber}, \
>   =gid=%{ldap:gidNumber}, \
>   =mail=mbox:~/:INBOX=%{ldap:mailPath}:INDEX=/indexes/%1M{ldap:mail}/%2.1M{ldap:mail}/%3.1M{ldap:mail}/%{ldap:mail}

I tried it but unfortunately the problem remains :

Feb 26 00:02:28 pop08 dovecot: lmtp(26796, dinos.mpalanikas at 3230.syzefxis.gov.gr): Error: stat(/var/index/dovecot/7/8/3/amalia at 3230.syzefxis.go
v.gr/.imap/INBOX/dovecot.index.log) failed: Permission denied (euid=10117261(<unknown>) egid=165(<unknown>) missing +x perm: /var/index/dovecot
/7/8/3/amalia at 3230.syzefxis.gov.gr, dir owned by 10117262:165 mode=0700)
Feb 26 00:02:28 pop08 dovecot: lmtp(26796, dinos.mpalanikas at 3230.syzefxis.gov.gr): Error: stat(/var/index/dovecot/7/8/3/amalia at 3230.syzefxis.go
v.gr/.imap/INBOX/dovecot.index) failed: Permission denied (euid=10117261(<unknown>) egid=165(<unknown>) missing +x perm: /var/index/dovecot/7/8
/3/amalia at 3230.syzefxis.gov.gr, dir owned by 10117262:165 mode=0700)
Feb 26 00:02:28 pop08 dovecot: lmtp(26796, dinos.mpalanikas at 3230.syzefxis.gov.gr): QYsqEkjfK1GsaAAADehEhg: sieve: msgid=<19dfa2f0f211951c56596c
9600194384 at otenet.gr>: stored mail into mailbox 'INBOX'

These two users, dinos.mpalanikas at 3230.syzefxis.gov.gr and
amalia at 3230.syzefxis.gov.gr have nothing in common apart from belonging
to the same domain.

root at vader:lmtp $ ldapsearch -h deer -x mail=dinos.mpalanikas at 3230.syzefxis.gov.gr uid uidNumber gidNumber mail mailAlternateAddress mailPath f
olderPath -LLL                                                                                                                                 
dn: uid=3230_syzefxis_gov_gr_user_003,ou=people,dc=otenet,dc=gr
uid: 3230_syzefxis_gov_gr_user_003
uidNumber: 10117261
gidNumber: 165
mail: dinos.mpalanikas at 3230.syzefxis.gov.gr
mailPath: /var/mail/C/3/6/3230_syzefxis_gov_gr_user_003
folderPath: /var/mail/folders/C/3/6/3230_syzefxis_gov_gr_user_003

root at vader:lmtp $ ldapsearch -h deer -x mail=amalia at 3230.syzefxis.gov.gr uid uidNumber gidNumber mail mailAlternateAddress mailPath folderPath 
-LLL
dn: uid=3230_syzefxis_gov_gr_user_002,ou=people,dc=otenet,dc=gr
uid: 3230_syzefxis_gov_gr_user_002
uidNumber: 10117262
gidNumber: 165
mail: amalia at 3230.syzefxis.gov.gr
mailPath: /var/mail/C/3/6/3230_syzefxis_gov_gr_user_002
folderPath: /var/mail/folders/C/3/6/3230_syzefxis_gov_gr_user_002

However as you can see it tries to write to amalia's index while delivering to dinos.mpalanikas.
Of course we're only talking about 1 failure in 578 successful deliveries however that server currently only serves less than 5% of our customres right now and we're worried that if we go full scale the problem will simply become greater.

> Easier of course would be if you could determine mailPath using a static template, then you could just use the global mail_location and %u directly, because the %u gets normalized to same as "mail" field.

Indeed but I think that would mean getting rid of our legacy 3-level hashing algorithm and using dovecot's own which in turn would mean having to move mailboxes around for hundrends of thousands of users.

Dimos Alevizos