More Dovecot 2.4 woes: LDAP cache bug and virtual autosubscribe
8 months ago I messaged this list communicating some of the issues I had with 2.4. Before I could even start testing 2.4 properly, 2 bugs had to be fixed—one that caused crashes on startup (https://gitlab.alpinelinux.org/alpine/aports/-/issues/17050), and one that caused passdb to use the same caching key for all connections and as such always have the same user log in. There was another bug, one that prevented me from using LDAP binds for passdb, but it had a workaround that mostly worked without side-eeffects. All of those have since been fixed, hurray!
So in my infinite naïveté, I decided to give 2.4 another shake. And it, again, does not work. Surprise!
I did manage to massage the configuration to allow me to at least run 2.4 without crashing and with users able to access (most of) their mailboxes, but there are still blockers to full functionality.
I face the following issues:
- Christian Pfeiffer back in November found a bug where auth caching with LDAP did not work, and due to how it's implemented, even turning caching off would still raise an error. I experienced the same issue for both userdb iteration and passdb bind logins. I had to comment out the userdb and use a static one instead (all my users, currently, use the same UID and easily composable path, so this wasn't a big deal) and start using a service account for LDAP binds.
- Virtual mailbox autosubscribe does not work anymore. It throws an error about trying to create the virtual mailbox. This worked in 2.3.
My virtual mailbox config: namespace virtual { type = private prefix = Search Folders. mail_driver = virtual mail_path = /etc/dovecot/virtual mail_index_path = ~/dovecot-virtual.cache mail_control_path = ~/dovecot-virtual.cache mail_volatile_path = ~/dovecot-virtual.cache hidden = yes list = children subscriptions = no
mailbox All { auto = subscribe special_use = \All comment = All messages, excluding Junk and Trash }
mailbox Unread { auto = subscribe special_use = \Important comment = All unread messages, excluding Junk and Trash }
mailbox Flagged { auto = subscribe special_use = \Flagged comment = All flagged messages } }
If there is something wrong with this config that might be responsible for my issue, please let me know. Otherwise, I believe this is a regression. I understand autosubscribe is documented to (and in other situations with non-virtual boxes) create mailboxes if missing, so if this is actually a fix to make the feature work as documented, we need a new option that allows for autosubscription without attempting to create the mailbox. Otherwise, my users will not know they even have these mailboxes—no one anymore knows or cares about mailbox (re: folder) subscription and won't know where to look to get them to show up in their client.
With autosubscribe enabled, I can see the mailboxes in my mail client without explicitly subscribing, but attempting to open them will throw a server error. Removing the setting will make the mailbox disappear without going into settings and explicitly subscribing.
Any ETA on the first bug being fixed, and any help with the second issue (bug or not) would be greatly appreciated.
Best regards.
P.S.: I wanted to spread this information that wasn't mentioned at all in the documentation. The shadow passdb was removed in the move to 2.4; why exactly is neither here nor there, though I will remark that I still don't know why it was done. I had believed at the time this necessitated enabling PAM for Dovecot, which it is not by default in my distro's package (Alpine), so if I wanted to continue allowing local users to log in, I would have to build Dovecot it myself and install PAM, neither of which I particularly wanted to do. I did play with passwd-file in the past, but there were issues with it that prevented me from using it—I don't remember what they were exactly, but I believe they have since been fixed, since in my testing now it does seem to work properly in the use-cases I tested.
Anyway, while going over my config and getting ready to re-install PAM, I had a thought. /etc/shadow uses the same format for the first two fields as /etc/password, and Dovecot still (supposedly) allows for passwords to be used when reading /etc/password, so wouldn't it be possible to point the driver at the shadow file? Turns out, yes, it works! I am now able to authenticate local users from /etc/shadow again. Thanks to this, I no longer need to build with PAM support and don't need to install PAM on a PAM-less system. I hope this isn't considered a bug or something and support for this use-case is removed, because I really don't want to bother once again with adding PAM when I don't have to. :)
On 18/02/2026 23:59 EET sev monster via dovecot <dovecot@dovecot.org> wrote:
8 months ago I messaged this list communicating some of the issues I had with 2.4. Before I could even start testing 2.4 properly, 2 bugs had to be fixed—one that caused crashes on startup (https://gitlab.alpinelinux.org/alpine/aports/-/issues/17050), and one that caused passdb to use the same caching key for all connections and as such always have the same user log in. There was another bug, one that prevented me from using LDAP binds for passdb, but it had a workaround that mostly worked without side-eeffects. All of those have since been fixed, hurray!
So in my infinite naïveté, I decided to give 2.4 another shake. And it, again, does not work. Surprise!
I did manage to massage the configuration to allow me to at least run 2.4 without crashing and with users able to access (most of) their mailboxes, but there are still blockers to full functionality.
Have you given https://dovecot.org/upgrader a try to see if it can massage your config better?
I face the following issues:
- Christian Pfeiffer back in November found a bug where auth caching with LDAP did not work, and due to how it's implemented, even turning caching off would still raise an error. I experienced the same issue for both userdb iteration and passdb bind logins. I had to comment out the userdb and use a static one instead (all my users, currently, use the same UID and easily composable path, so this wasn't a big deal) and start using a service account for LDAP binds.
- Virtual mailbox autosubscribe does not work anymore. It throws an error about trying to create the virtual mailbox. This worked in 2.3.
My virtual mailbox config: namespace virtual { type = private prefix = Search Folders. mail_driver = virtual mail_path = /etc/dovecot/virtual mail_index_path = ~/dovecot-virtual.cache mail_control_path = ~/dovecot-virtual.cache mail_volatile_path = ~/dovecot-virtual.cache hidden = yes list = children subscriptions = no
mailbox All { auto = subscribe special_use = \All comment = All messages, excluding Junk and Trash }
mailbox Unread { auto = subscribe special_use = \Important comment = All unread messages, excluding Junk and Trash }
mailbox Flagged { auto = subscribe special_use = \Flagged comment = All flagged messages } }
If there is something wrong with this config that might be responsible for my issue, please let me know. Otherwise, I believe this is a regression. I understand autosubscribe is documented to (and in other situations with non-virtual boxes) create mailboxes if missing, so if this is actually a fix to make the feature work as documented, we need a new option that allows for autosubscription without attempting to create the mailbox. Otherwise, my users will not know they even have these mailboxes—no one anymore knows or cares about mailbox (re: folder) subscription and won't know where to look to get them to show up in their client.
With autosubscribe enabled, I can see the mailboxes in my mail client without explicitly subscribing, but attempting to open them will throw a server error. Removing the setting will make the mailbox disappear without going into settings and explicitly subscribing.
Any ETA on the first bug being fixed, and any help with the second issue (bug or not) would be greatly appreciated.
Can you give us a hint what the error was?
Best regards.
P.S.: I wanted to spread this information that wasn't mentioned at all in the documentation. The shadow passdb was removed in the move to 2.4; why exactly is neither here nor there, though I will remark that I still don't know why it was done. I had believed at the time this necessitated enabling PAM for Dovecot, which it is not by default in my distro's package (Alpine), so if I wanted to continue allowing local users to log in, I would have to build Dovecot it myself and install PAM, neither of which I particularly wanted to do. I did play with passwd-file in the past, but there were issues with it that prevented me from using it—I don't remember what they were exactly, but I believe they have since been fixed, since in my testing now it does seem to work properly in the use-cases I tested.
Anyway, while going over my config and getting ready to re-install PAM, I had a thought. /etc/shadow uses the same format for the first two fields as /etc/password, and Dovecot still (supposedly) allows for passwords to be used when reading /etc/password, so wouldn't it be possible to point the driver at the shadow file? Turns out, yes, it works! I am now able to authenticate local users from /etc/shadow again. Thanks to this, I no longer need to build with PAM support and don't need to install PAM on a PAM-less system. I hope this isn't considered a bug or something and support for this use-case is removed, because I really don't want to bother once again with adding PAM when I don't have to. :)
No, passwd-file is rather essential so it's not going to go away any time soon. If it works as replacement for shadow, nice. Removal of "shadow" driver is mentioned at https://doc.dovecot.org/2.4.2/installation/upgrade/2.3-to-2.4.html#backends-...
Aki
Have you given https://dovecot.org/upgrader a try to see if it can massage your config better?
Yes, the config I had manually ported prior to the existence of the upgrader was almost 1:1, with a few more advanced settings I had configured missing from the upgrader. I did another try the output is about the same as last time; here's the output of the latest run:
namespace virtual {
hidden = yes
list = children
prefix = Search Folders.
subscriptions = no
type = private
mailbox All {
auto = subscribe
comment = All messages, excluding Junk and Trash
special_use = \All
}
mailbox Unread {
auto = subscribe
comment = All unread messages, excluding Junk and Trash
special_use = \Important
}
mailbox Flagged {
auto = subscribe
comment = All flagged messages
special_use = \Flagged
}
}
namespace virtual-user {
hidden = yes
list = children
prefix = User Search Folders.
subscriptions = no
type = private
}And this is the original from my last working 2.3 config:
namespace virtual {
type = private
separator = .
prefix = "Search Folders."
location = virtual:/etc/dovecot/virtual:INDEX=~/dovecot-virtual.cache:CONTROL=~/dovecot-virtual.cache:VOLATILEDIR=~/dovecot-virtual.cache
hidden = yes
list = children
subscriptions = no
mailbox All {
auto = subscribe
special_use = \All
comment = All messages, excluding Junk and Trash
}
mailbox Unread {
auto = subscribe
special_use = \Important
comment = All unread messages, excluding Junk and Trash
}
mailbox Flagged {
auto = subscribe
special_use = \Flagged
comment = All flagged messages
}
}
namespace virtual-user {
type = private
separator = .
prefix = "User Search Folders."
location = virtual:~/.virtual:INDEX=~/dovecot-virtual.user.cache:CONTROL=~/dovecot-virtual.user.cache:VOLATILEDIR=~/dovecot-virtual.user.cache:LAYOUT=maildir++
hidden = yes
list = children
subscriptions = no
}You can see the tool misses some entries I had to add back manually. As far as I can tell, the final 2.4 config send in my first message should be 1:1 to the 2.3 config in functionality.
Can you give us a hint what the error was?
For the LDAP caching issue, it's the same as he reported in his email:
auth: Fatal: auth-cache: ou=dudes,dc=mysite: Cache key must contain at least one variableI messed around with various configurations, and this one was the one that finally ended up working:
ldap_uris = ldapi://somesock
ldap_base = ou=dudes,dc=mysite
passdb ldap {
ldap_connection_group = login
bind = yes
bind_userdn = uid=%{user | username | lower },ou=dudes,dc=mysite
# LDAP FIX
filter = (&(objectClass=inetOrgPerson)(uid=%{user | username | lower }))
ldap_auth_dn = cn=dovecot,ou=serviceaccount,dc=mysite
ldap_auth_dn_password = secret
# fields {
# user = %{ldap:uid}
# }
}
# LDAP FIX
userdb static {
fields {
uid = mail
gid = mail
home = $SET:mail_home
}
}
# userdb ldap {
# ldap_auth_dn = cn=dovecot,ou=serviceaccount,dc=mysite
# ldap_auth_dn_password = secret
# filter = (&(objectClass=inetOrgPerson)(uid=%{user | username | lower }))
# fields {
# user = %{ldap:uid}
# uid = %{ldap:uidNumber | default('mail')}
# gid = %{ldap:gidNumber | default('mail')}
# home = %{ldap:homeDirectory | default($SET:mail_home)}
# }
# iterate_filter = (objectClass=inetOrgPerson)
# iterate_fields {
# user = %{ldap:uid}
# }
# }And for reference, here is the 2.3 config:
passdb {
driver = ldap
args = /etc/dovecot/ldap-passdb.conf.ext
}
userdb {
driver = ldap
default_fields = uid=mail gid=mail home=/var/mail/%Ln
args = /etc/dovecot/ldap-userdb.conf.ext
}ldap-passdb.conf.ext:
uris = ldapi://somesock
base = ou=dudes,dc=mysite
auth_bind = yes
auth_bind_userdn = uid=%Ln,ou=dudes,dc=mysiteldap-userdb.conf.ext:
uris = ldapi://somesock
dn = cn=dovecot,ou=serviceaccount,dc=mysite
dnpass = secret
base = ou=dudes,dc=mysite
iterate_filter = (objectClass=inetOrgPerson)
iterate_attrs = uid=username
user_filter = (&(objectClass=inetOrgPerson)(uid=%Ln))
user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gidSorry, I thought I had sent this in my original email. The commented out parts and the ones included after the LDAP FIX headers (the extra entries in the passdb and the static userdb) are the parts that don't work and trigger the cache bug. The thread "2.4.2 breaks user iterations for LDAP" has more information.
Regards.
participants (2)
-
Aki Tuomi
-
sev monster