If anyone's interested about implementing quota code for Dovecot (there were a few of you before! :), now it's finally your chance to do it quite easily.
With the API changes committed today to CVS, it's now much easier to write plugins that modify mail-storage behavior. So, I finally fixed my half-working quota plugin that had been lying there for a while. It implements the RFC2087 quota commands for IMAP, a generic quota API that should allow pretty much any kind of quota implementation, and the actual quota limit checking for saving and copying mails. As an example there's an implementation which just scans through all files in a directory and sums their sizes.
Currently it can be made to work only when starting from command line, eg.:
export QUOTA=~/mail:storage=1024 export MODULE_DIR=/usr/local/lib/dovecot/imap ./imap
The QUOTA variable is passed to the quota implementation. In this case quota-dirsize reads it, figures out that ~/mail is the directory where to count the files, and 1024kB is the maximum for quota.
I think I'll change the master process code so that any unrecognized varibles that come from userdb are fed to imap/pop3 processes. That way you could simply make the userdb return the quota value (and set other user-specific settings as well).
Available from:
http://dovecot.org/patches/1.0/quota.tar.gz
Unpack under dovecot/src, compile command is at the beginning of quota-plugin.c
I'm still wondering where this code should be placed. Maybe the core implementation should be built into Dovecot, while the different quota implementations would be plugins? ..And even better would be if Dovecot's build process was modified so that it's possible to specify with configure script what to compile in and what to compile as plugins..
Anyway, comments welcome, Maildir++ implementation welcome, other implementations welcome as well :)
On Wed, Mar 16, 2005 at 12:15:43AM +0200, Timo Sirainen wrote:
If anyone's interested about implementing quota code for Dovecot (there were a few of you before! :), now it's finally your chance to do it quite easily.
Hah, excellent. I suspect my colleague Cor was one of the people interested in quota support, and I know I am. So I took the example and turned it into an rquotad (quota-over-nfs) implementation. It's fairly FreeBSD specific at the moment, as rquotad is an RPC mess and I had to be inspired by the source of the 'quota' command, but I suspect it's probably possible to port it to other operating systems. When it's done, that is.
The plugin seems a bit out-of-date, by the way; it passes the wrong number of arguments to 'mailbox_open', and uses a nonexistant function 'i_stream_get_size'. I had to fudge around that creatively to get the plugin to compile. The rquotad plugin is done, more or less, except for figuring out how to properly get at a mailbox's physical location, if it has one. I really don't want to maintain a static list in an environment variable; we currently use 10 different filesystems (all NFS) for mail storage, and they all have different quotas and usage. I don't particularly care about checking quotas before an attempted write, but I implemented that as best I could; I still need to figure out how to decode the mail_transaction_context to see where a mail is moved from/to in order to finish it.
The quota-plugin thing obviously isn't quite done yet (at least, not the tarball version from the patches/1.0 directory) -- it lacks some error checking, I think, and a capability hook from what I can tell. (There *is* a capability for quotas, I hope, in IMAP ?) If you lack time but have an idea as to what needs to be done, I can work on it and submit patches. Once I get the rquotad plugin *somewhat* working I'll put it up somewhere, too.
-- Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
On 20.4.2005, at 04:52, Thomas Wouters wrote:
The plugin seems a bit out-of-date, by the way; it passes the wrong number of arguments to 'mailbox_open', and uses a nonexistant function 'i_stream_get_size'.
Fixed and uploaded new quota.tar.gz to same place.
The rquotad plugin is done, more or less, except for figuring out how to properly get at a mailbox's physical location, if it has one. I really don't want to maintain a static list in an environment variable; we currently use 10 different filesystems (all NFS) for mail storage, and they all have different quotas and usage.
Well, that requires mbox/maildir-specific storage code then to get the path.. I think zlib plugin checks that it works only with mboxes, that might be helpful.
I don't particularly care about checking quotas before an attempted write, but I implemented that as best I could; I still need to figure out how to decode the mail_transaction_context to see where a mail is moved from/to in order to finish it.
Probably just by checking if it's mbox/maildir and then casting struct mailbox to struct maildir_mailbox and looking what it contains.. Or if you actually want to see the filenames it gets more difficult.
The quota-plugin thing obviously isn't quite done yet (at least, not the tarball version from the patches/1.0 directory) -- it lacks some error checking, I think,
I don't remember leaving anything intentionally?
and a capability hook from what I can tell. (There *is* a capability for quotas, I hope, in IMAP ?) If you lack time but have an idea as to what needs to be done, I can work on it and submit patches.
After login you should be able to modify capability_string directly. Just append QUOTA there. But I think you'll have to modify the pre-login capability as well because some clients don't check it afterwards. I haven't really figured out yet what to do about it. One possibility would be to add some plugin_capability() function into the plugin and let login processes get capabilities of all plugins. But maybe it's not such a good idea. Easiest way would be just to put the capability string into config file.
On Thu, Apr 21, 2005 at 10:12:43PM +0300, Timo Sirainen wrote:
On 20.4.2005, at 04:52, Thomas Wouters wrote:
Fixed and uploaded new quota.tar.gz to same place.
Thanks! I'll look at it tomorrow, and integrate it in my changes. quota-rquotad works, by the way; I'll clean it up a bit and provide a patch against quota.tar.gz. It still needs extensive testing, but I have eager coworkers waiting for just that :)
The rquotad plugin is done, more or less, except for figuring out how to properly get at a mailbox's physical location, if it has one. I really don't want to maintain a static list in an environment variable; we currently use 10 different filesystems (all NFS) for mail storage, and they all have different quotas and usage.
Well, that requires mbox/maildir-specific storage code then to get the path.. I think zlib plugin checks that it works only with mboxes, that might be helpful.
Hmm. I currently cheat, casting the struct_mailbox to a struct index_storage and using its 'dir' member. I don't even check that it is an index_storage struct; I know I should, but when wouldn't it be ?
I don't particularly care about checking quotas before an attempted write, but I implemented that as best I could; I still need to figure out how to decode the mail_transaction_context to see where a mail is moved from/to in order to finish it.
Probably just by checking if it's mbox/maildir and then casting struct mailbox to struct maildir_mailbox and looking what it contains.. Or if you actually want to see the filenames it gets more difficult.
Well, to properly check quotas, I'd have to compare quotaroots of the source and destination both (in case of a move), then do the proper calculation (taking into account blocksize of disk and such) -- I think I'll just skip that check. Quotas are enforced by the filesystem anyway.
The quota-plugin thing obviously isn't quite done yet (at least, not the tarball version from the patches/1.0 directory) -- it lacks some error checking, I think,
I don't remember leaving anything intentionally?
The one I ran into was cmd_getquotaroot not checking the return value of quota_root_iter_init(), and my quota_root_iter_init() was failing. At the time, I thought the reason of that failure was another part of the code, but it was 4AM, I don't quite remember what I thought it was and I don't think it really was that anyway.
Something else I noticed was that the storage quota name should (for Thunderbird, at least) be allcaps, or it won't recognize it as storage quota. Thunderbird doesn't do message quota (which I implemented as 'inode quota', as I'm maildir-centric) but I suspect that should probably be allcaps as well, for interoperability if nothing else.
And there's the bit where you need to set the MODULE_DIR and QUOTA environment variables and start imap by hand, rather than by dovecot. I hacked around that by hardcoding both in the source.
Easiest way would be just to put the capability string into config file.
That's what I ended up doing, after I reread RFC2087 and realized I'd missed the capability.
Thanks for the comments and the update,
Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
On 22.4.2005, at 02:26, Thomas Wouters wrote:
The rquotad plugin is done, more or less, except for figuring out how to properly get at a mailbox's physical location, if it has one. I really don't want to maintain a static list in an environment variable; we currently use 10 different filesystems (all NFS) for mail storage, and they all have different quotas and usage.
Well, that requires mbox/maildir-specific storage code then to get the path.. I think zlib plugin checks that it works only with mboxes, that might be helpful.
Hmm. I currently cheat, casting the struct_mailbox to a struct index_storage and using its 'dir' member. I don't even check that it is an index_storage struct; I know I should, but when wouldn't it be ?
For now, yes.. I thought I had moved "dir" already out of it to mbox_storage though. Guess I did it only with index_mailbox.
The quota-plugin thing obviously isn't quite done yet (at least, not the tarball version from the patches/1.0 directory) -- it lacks some error checking, I think,
I don't remember leaving anything intentionally?
The one I ran into was cmd_getquotaroot not checking the return value of quota_root_iter_init(), and my quota_root_iter_init() was failing.
It's not supposed to fail :) But I guess that iteration needs error checking as well. I changed quota_root_iter_deinit() return "int" where you can return -1 if it fails (just make init() return some dummy struct which causes next() to return NULL and deinit() to return -1).
Updated the plugin in web page.
Something else I noticed was that the storage quota name should (for Thunderbird, at least) be allcaps, or it won't recognize it as storage quota. Thunderbird doesn't do message quota (which I implemented as 'inode quota', as I'm maildir-centric) but I suspect that should probably be allcaps as well, for interoperability if nothing else.
It should be treated case-insensitive. Looks like Dovecot was using strcmp() to check its name, changed that. Changed also to use uppercase when sending the reply.
And there's the bit where you need to set the MODULE_DIR and QUOTA environment variables and start imap by hand, rather than by dovecot. I hacked around that by hardcoding both in the source.
module_dir can be set in configuration file? Quota environment is also now forwarded from userdb as well. So it'd work at least with SQL and static userdbs..
On Fri, Apr 22, 2005 at 01:26:23AM +0200, Thomas Wouters wrote:
quota-rquotad works, by the way; I'll clean it up a bit and provide a patch against quota.tar.gz.
Patch attached. Feel free to include it any which way. It probably requires extensive changes to make it portable, though; it's built for FreeBSD (which we use.) I've also only tested it with Netapp Filer backends (which we use :). It doesn't try to enforce the quotas (which would involve complicated guesses as to disk blocksize and such) but it does report them properly, so you can see your quota and usage in your mail client. It currently treats inode-quota as message-quota (we use maildir) but I only added that to see if it worked (we don't actually *have* inode or message quota, ourselves.) There's also a horrible cast-and-dereference in there to find the directory associated with a mailbox, but it seems to work for mbox and maildir in my tests. so I'm thinking it 'should be fine for now' ;)
Compile instructions at the top of quota-rquotad.c, usage similar to quota-dirsize; the QUOTA variable should be set to a comma-separated list of quotaroots (nfs mountpoints that use quota.) Perhaps (probably) quota-rquotad.c should be renamed quota-filesystem.c and use quotactl for local disks, but I haven't gotten to that part yet. (But quotactl is easy-peasy compared to rquotad.)
-- Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
On Monday 25 April 2005 09:58, Thomas Wouters wrote:
On Fri, Apr 22, 2005 at 01:26:23AM +0200, Thomas Wouters wrote:
quota-rquotad works, by the way; I'll clean it up a bit and provide a patch against quota.tar.gz.
Hi, Is rquotad_quota_try_alloc supposed to be commented out?
Patch attached. Feel free to include it any which way. It probably requires extensive changes to make it portable, though; it's built for FreeBSD (which we use.) I've also only tested it with Netapp Filer backends (which we use :). It doesn't try to enforce the quotas (which would involve complicated guesses as to disk blocksize and such) but it does report them properly, so you can see your quota and usage in your mail client. It currently treats inode-quota as message-quota (we use maildir) but I only added that to see if it worked (we don't actually *have* inode or message quota, ourselves.) There's also a horrible cast-and-dereference in there to find the directory associated with a mailbox, but it seems to work for mbox and maildir in my tests. so I'm thinking it 'should be fine for now' ;)
Compile instructions at the top of quota-rquotad.c, usage similar to quota-dirsize; the QUOTA variable should be set to a comma-separated list of quotaroots (nfs mountpoints that use quota.) Perhaps (probably) quota-rquotad.c should be renamed quota-filesystem.c and use quotactl for local disks, but I haven't gotten to that part yet. (But quotactl is easy-peasy compared to rquotad.)
Thanks,
Dominic GoodforBusiness.co.uk I.T. Services for SMEs in the UK.
On Mon, Apr 25, 2005 at 11:15:20AM +0100, Dominic Marks wrote:
Hi, Is rquotad_quota_try_alloc supposed to be commented out?
Yes, see:
It doesn't try to enforce the quotas (which would involve complicated guesses as to disk blocksize and such) but it does report them properly, so you can see your quota and usage in your mail client.
-- Thomas Wouters <thomas@xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
On Mon, 2005-04-25 at 10:58 +0200, Thomas Wouters wrote:
quota-rquotad works, by the way; I'll clean it up a bit and provide a patch against quota.tar.gz.
Just a few minor comments:
for (qr_idx = 0; qroots[qr_idx] != NULL; qr_idx++) {/* NOTHING */;}
Could be replaced with qr_idx = strarray_length(qroots);
root->root.quota = (struct quota *)quota;
I usually try to avoid casts whenever possible. If something is changed you'll then get a nice compile time error instead of a runtime crash :) So "a->quota would work instead..
if (statfs(((struct index_storage *)(mailbox_get_storage(box)))->dir,&statbuf) < 0) {return NULL;}
And like I said in previous mail, iter_init() shouldn't return NULL.. You could eg. set iter->iter.offset = quota->nroot (so iter_next() returns NULL) and add some iter->failure flag which iter_end() can then return (it returns int in new quota plugin code).
strncpy(iter->boxfs, statbuf.f_mntonname, MNAMELEN);
strncpy() is evil. It's not NUL-terminating strings. Dovecot has strocpy() which can be used I guess.. Or you could use dynamic string_t.
root->resources = p_new(root->pool, const char *, 1);root->resources[0] = NULL;
No real need to set [0] to NULL as p_new() always initializes the memory with zeros (and Dovecot relies on that in several places already).
participants (3)
-
Dominic Marks
-
Thomas Wouters
-
Timo Sirainen