[Dovecot] dirsize quota assertion problem
Our current virtual mailbox configuration is not compatible with one of the assertions in the dovecot quota plugin's assertions in quota-dirsize.c.
I believe the assertion is incorrect, but I would also be happy if I could get the same result with a better configuration setting.
Here is a sample passdb entry which causes the quota assertion to fail:
test@example.com:<snip>:3007:1000::/usr/boxes/username/example.com::userdb_mail=mbox:~/test^/.imap:INBOX=~/test userdb_uid=3007 userdb_gid=1000 userdb_home=/usr/boxes/username/example.com userdb_quota=dirsize:storage=100
The imap folder is "~/test^/.imap" and the INBOX is "~/test". As you can see, these are in different locations, but the INBOX string happens to be a substring of the imap folder directory.
An example of the failed assertion error:
Aug 31 16:44:13 www2 dovecot: IMAP(test@example.com): file quota-dirsize.c: line 180 (quota_count_path_add): assertion failed: (!is_file)
This particular error log line was generated by dovecot 1.0.3, but the code in quota-dirsize.c is the same in dovecot 1.1 alpha3's version of quota-dirsize.c:
131 if (strncmp(count_path[i].path, path, strlen(path)) == 0) { 132 /* the new path contains the existing path */ 133 i_assert(!is_file); 134 count_path += i; 135 break; 136 }
First, quota_count_path_add() is called for directory "~/test^/.imap" and it succeeds. When it is subsequently called for file "~/test":
- the strncmp matches because "~/test" is a substring of "~/test^/.imap"
- the assertion fails because "~/test" is a file and not a directory
It seems to me that instead of asserting !is_file, it would be better to simply skip the conditional in that case, since a path variable that is a file should never be tested to see whether it contains an existing path.
if (!is_file &&
strncmp(count_path[i].path, path, strlen(path)) == 0) {
/* the new path contains the existing path */
count_path += i;
break;
}But this also has a bug: if path is a directory which substring matches count_path[i].path, but the same path with a directory separator appended to it does not substring match count_path[i].path, then it is a different path and does not contain the existing path.
For example, if count_path[i] is the directory /usr/home/test/mailbox_folders, and path is the directory /usr/home/test/mailbox, then /usr/home/test/mailbox does not contain/usr/home/test/mailbox_folders, and this condition should fail.
If some other code were to enforce that /usr/home/test/mailbox be terminated with a directory separator, then this is not a problem and my patch should work.
I hope this makes sense.
Thanks,
Alan Ferrency pair Networks, Inc. alan@pair.com
On Fri, 2007-08-31 at 17:42 -0400, Alan Ferrency wrote:
test@example.com:<snip>:3007:1000::/usr/boxes/username/example.com::userdb_mail=mbox:~/test^/.imap:INBOX=~/test userdb_uid=3007 userdb_gid=1000 userdb_home=/usr/boxes/username/example.com userdb_quota=dirsize:storage=100
Those userdb_uid/gid/home aren't needed.
Aug 31 16:44:13 www2 dovecot: IMAP(test@example.com): file quota-dirsize.c: line 180 (quota_count_path_add): assertion failed: (!is_file)
This should fix it: http://hg.dovecot.org/dovecot-1.0/rev/a3f2f00cbd7c
Hello,
Thanks for the patch!
On Sun, 9 Sep 2007, Timo Sirainen wrote:
On Fri, 2007-08-31 at 17:42 -0400, Alan Ferrency wrote:
test@example.com:<snip>:3007:1000::/usr/boxes/username/example.com::userdb_mail=mbox:~/test^/.imap:INBOX=~/test userdb_uid=3007 userdb_gid=1000 userdb_home=/usr/boxes/username/example.com userdb_quota=dirsize:storage=100
Those userdb_uid/gid/home aren't needed.
This is a passdb file, used with userdb prefetch for pop and imap protocols (as well as being used directly as a userdb for the lda). They still aren't needed?
Aug 31 16:44:13 www2 dovecot: IMAP(test@example.com): file quota- dirsize.c: line 180 (quota_count_path_add): assertion failed: (!is_file)
This should fix it: http://hg.dovecot.org/dovecot-1.0/rev/a3f2f00cbd7c
Thanks! That works great, and does a much better job than my ad-hoc workaround.
An unrelated question:
I am looking for support in the deliver lda for a "maximum message size" quota feature, which is configurable per user, like the rest of the dovecot quota plugin. I haven't found such a feature documented anywhere, and I'm having trouble following the quota code enough to figure out how to add this feature.
Do you have any comments on whether you think this would be easy or difficult to add, and where it would be best to put it?
Thanks,
Alan Ferrency pair Networks, Inc. alan@pair.com
On Mon, 2007-09-10 at 10:55 -0400, Alan Ferrency wrote:
On Fri, 2007-08-31 at 17:42 -0400, Alan Ferrency wrote:
test@example.com:<snip>:3007:1000::/usr/boxes/username/example.com::userdb_mail=mbox:~/test^/.imap:INBOX=~/test userdb_uid=3007 userdb_gid=1000 userdb_home=/usr/boxes/username/example.com userdb_quota=dirsize:storage=100
Those userdb_uid/gid/home aren't needed.
This is a passdb file, used with userdb prefetch for pop and imap protocols (as well as being used directly as a userdb for the lda). They still aren't needed?
I think there isn't much point in using prefetch with passwd-file, because the information is already in memory and accessible with a fast hash table lookup. Having the information twice in the file eats more memory.
I am looking for support in the deliver lda for a "maximum message size" quota feature, which is configurable per user, like the rest of the dovecot quota plugin. I haven't found such a feature documented anywhere, and I'm having trouble following the quota code enough to figure out how to add this feature.
Do you have any comments on whether you think this would be easy or difficult to add, and where it would be best to put it?
Well, I think you should rather put this to deliver itself. Otherwise you'd be limiting also IMAP APPEND command.
I don't really see the point of having this setting though. Why not have one global limit (which already exists in your SMTP server)?
On Mon, 2007-09-10 at 10:55 -0400, Alan Ferrency wrote:
On Fri, 2007-08-31 at 17:42 -0400, Alan Ferrency wrote:
test@example.com:<snip>:3007:1000::/usr/boxes/username/example.com::userdb_mail=mbox:~/test^/.imap:INBOX=~/test userdb_uid=3007 userdb_gid=1000 userdb_home=/usr/boxes/username/example.com userdb_quota=dirsize:storage=100
Those userdb_uid/gid/home aren't needed.
This is a passdb file, used with userdb prefetch for pop and imap protocols (as well as being used directly as a userdb for the lda). They still aren't needed?
I think there isn't much point in using prefetch with passwd-file, because the information is already in memory and accessible with a fast hash table lookup. Having the information twice in the file eats more memory.
There was a reason I settled on using a prefetch userdb instead of a userdb passwd-file, but now I don't remember what it was. And looking at my conf, I am still using the userdb passwd-file later in the search path, to support deliver anyway, so it could most likely be improved.
I think this may have been related to file permissions: we didn't want to give all users read access to the userdb file after they've logged in, and prefetching allowed us to limit access to only the login/auth user. Running an smtpauth port for deliver kind of makes that a moot point anyway, though, so maybe we can clean things up a bit.
I am looking for support in the deliver lda for a "maximum message size" quota feature, which is configurable per user...
Well, I think you should rather put this to deliver itself. Otherwise you'd be limiting also IMAP APPEND command.
I don't really see the point of having this setting though. Why not have one global limit (which already exists in your SMTP server)?
Thanks.
Basically, we need this for backwards compatibility purposes, more than anything else.
We'd prefer to vendor-branch dovecot, so my current line of pursuit is using an external executable in the lda pipeline, only in cases when a per message limit is configured. I expect it would be far more efficient to put this inside dovecot, however.
Thanks, Alan Ferrency pair Networks, Inc. alan@pair.com
On Mon, 2007-09-10 at 11:42 -0400, Alan Ferrency wrote:
I think this may have been related to file permissions: we didn't want to give all users read access to the userdb file after they've logged in, and prefetching allowed us to limit access to only the login/auth user. Running an smtpauth port for deliver kind of makes that a moot point anyway, though, so maybe we can clean things up a bit.
Only dovecot-auth process accesses the userdb file.
I am looking for support in the deliver lda for a "maximum message size" quota feature, which is configurable per user... We'd prefer to vendor-branch dovecot, so my current line of pursuit is using an external executable in the lda pipeline, only in cases when a per message limit is configured. I expect it would be far more efficient to put this inside dovecot, however.
Maybe a whole new plugin checking this in mail_storage.save_init().. You could test this with quota plugin by adding the check in quota_save_init(). I don't remember if st->st_size is set with deliver though. If not, the check needs to be done afterwards in quota_save_finish().
On Mon, 10 Sep 2007, Timo Sirainen wrote:
On Mon, 2007-09-10 at 11:42 -0400, Alan Ferrency wrote:
(Snip: I use userdb prefetch and a bunch of userdb_* settings in the passdb file, and Timo suggested it was not necessary. I then recalled an incorrect explanation for why I might have done this...)
I think this may have been related to file permissions: we didn't want to give all users read access to the userdb file after they've logged in, and prefetching allowed us to limit access to only the login/auth user. Running an smtpauth port for deliver kind of makes that a moot point anyway, though, so maybe we can clean things up a bit.
Only dovecot-auth process accesses the userdb file.
I now remember (rediscovered) why I had to use userdb prefetch in our setup.
I'm using multiple passdb passwd-file files, one of which provides local IP based virtual mail hosting. Usernames are not guaranteed to be unique across both files.
If I use userdb passwd-file with the same two files, then sometimes the wrong userdb information will be used. If I log in to a username which is in both the first and second passdb file, and use the password for the second passdb's version of the username, it will log in correctly but use the username's userdb information in the _first_ userdb file that matches (instead of the second).
This is, needless to say, Very Bad. I fixed it using userdb prefetch, but I would prefer a better solution.
In this case, our preferred behavior is unambiguous, but I couldn't figure out how to configure it. If the username exists in the first passdb, but the incorrect password is used, login should be denied, instead of checking the second passdb file. The second passdb should only be used if the username doesn't exist in the first passdb.
This would avoid the possibility of accidentally using the second passdb for login, but the first userdb file for user information, and would remove ambiguity from setups such as ours.
Is the current behavior considered correct? Is this configurable to do what I need instead? I didn't find anything addressing it in the wiki.
Thanks,
Alan Ferrency pair Networks, Inc. alan@pair.com
I wrote:
If I use userdb passwd-file with the same two files, then sometimes the wrong userdb information will be used.
Shortly after reporting this issue, I came across two configuration solutions for my problem (please read below for details). I haven't tested the first one yet:
Assume I have the current passdb setup:
passdb passwd-file { args = <snip>/%l.passwd } passdb passwd-file { args = <snip>/virtual.passwd }
One way to make it impossible to log in to usernames found in %l.passwd if the password doesn't match %l.passwd is to add a "deny" passdb:
passdb passwd-file { args = <snip>/%l.passwd } passdb passwd-file { args = <snip>/%l.passwd deny = yes } passdb passwd-file { args = <snip>/virtual.passwd }
This seems like it should allow users in %l.passwd to log in, but if the password doesn't match, they'll be denied before an ambiguous username/password in virtual.passwd is tried.
The solution I settled on, and which I confirmed works for my setup, is to set the "user=" field in %l.passwd to change IP-based virtual usernames into fully qualified usernames, and then only use virtual.passwd for all userdb lookups.
For example, consider username "testguy" in %l.passwd, which also exists in virtual.passwd. In all cases, for virtual user testguy@example.com, I set up a %l.passwd entry for "testguy" with example.com's IP, and also an entry in virtual.passwd for "testguy@example.com".
My solution adds "user=testguy@example.com" to testguy's passwd-file entry in %l.passwd file.
Now, I can stop using %l.passwd as a userdb file, and all userdb info for virtual users will be looked up by the fully qualified username in virtual.passwd. There is no ambiguity when a non-virtual user logs in, because the per-IP names are always looked up with their canonical domain even if they match system usernames.
Thanks for your time, Alan Ferrency pair Networks, Inc. alan@pair.com
On Thu, 13 Sep 2007, Alan Ferrency wrote:
On Mon, 10 Sep 2007, Timo Sirainen wrote:
On Mon, 2007-09-10 at 11:42 -0400, Alan Ferrency wrote:
(Snip: I use userdb prefetch and a bunch of userdb_* settings in the passdb file, and Timo suggested it was not necessary. I then recalled an incorrect explanation for why I might have done this...)
I think this may have been related to file permissions: we didn't want to give all users read access to the userdb file after they've logged in, and prefetching allowed us to limit access to only the login/auth user. Running an smtpauth port for deliver kind of makes that a moot point anyway, though, so maybe we can clean things up a bit.
Only dovecot-auth process accesses the userdb file.
I now remember (rediscovered) why I had to use userdb prefetch in our setup.
I'm using multiple passdb passwd-file files, one of which provides local IP based virtual mail hosting. Usernames are not guaranteed to be unique across both files.
If I use userdb passwd-file with the same two files, then sometimes the wrong userdb information will be used. If I log in to a username which is in both the first and second passdb file, and use the password for the second passdb's version of the username, it will log in correctly but use the username's userdb information in the _first_ userdb file that matches (instead of the second).
This is, needless to say, Very Bad. I fixed it using userdb prefetch, but I would prefer a better solution.
In this case, our preferred behavior is unambiguous, but I couldn't figure out how to configure it. If the username exists in the first passdb, but the incorrect password is used, login should be denied, instead of checking the second passdb file. The second passdb should only be used if the username doesn't exist in the first passdb.
This would avoid the possibility of accidentally using the second passdb for login, but the first userdb file for user information, and would remove ambiguity from setups such as ours.
Is the current behavior considered correct? Is this configurable to do what I need instead? I didn't find anything addressing it in the wiki.
Thanks,
Alan Ferrency pair Networks, Inc. alan@pair.com
participants (2)
-
Alan Ferrency
-
Timo Sirainen