Hi there,
I recently installed a plugin for SquirelMail (php imap client, squirrelmail.org) which adds up message sizes per folder to display a nice total size of each subscribed folder. After consulting the plugin maintainers at SquirrelMail in this thread:
http://thread.gmane.org/gmane.mail.squirrelmail.plugins/12468/focus=12468
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much. Could this be like Tomas Kuliavas wrote: "Maybe those message have some data that does not belong to message or their formating breaks dovecot's message parsing functions."? I'm using Dovecot-1.0.rc15, sorry if this has been fixed between this release and the current.
Kindest regards,
Tristan
-- Prism Mail Solutions Deployment, customization and hosting http://prism.nl
Tristan Woudenberg - Prism Mail Solutions schrieb:
Hi there,
I recently installed a plugin for SquirelMail (php imap client, squirrelmail.org) which adds up message sizes per folder to display a nice total size of each subscribed folder. After consulting the plugin maintainers at SquirrelMail in this thread:
http://thread.gmane.org/gmane.mail.squirrelmail.plugins/12468/focus=12468
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much. Could this be like Tomas Kuliavas wrote: "Maybe those message have some data that does not belong to message or their formating breaks dovecot's message parsing functions."? I'm using Dovecot-1.0.rc15, sorry if this has been fixed between this release and the current.
Kindest regards,
Tristan
Hi Tristan, i use quota plugin from squirrelmail 1.4.6-1.4.9a since dovcot 1.rc7 up to 1.rc17 message size is reported as well as folder size. You can also use the message size function included in squirrelmail this works nice too at my install, i guess you have some configure problem, which is not really related to a dovecots bug as far i remember you have to setup the right mutliplicator for kb in the plugin to get it work right, try the squirrel inbuild funktion to test , should be ok, be sure that you have imap server other in squirrel imap setup and follow instructions in the squirrel faq for dovecot with folders Regards Robert
-- Diese Nachricht wurde auf Viren und andere gefährliche Inhalte untersucht und ist - aktuelle Virenscanner vorausgesetzt - sauber.
On Tue, 2007-01-16 at 11:00 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much.
RFC822.SIZE reports message's "virtual size" as it is sent to IMAP clients, which means that if the maildir files contain LF-only linefeeds (which is usually the case), the RFC822.SIZE increases the size by the number of lines in the file (so each line ends with CRLF).
That however only increases the size reported by the Squirrelmail plugin. I don't know why it would report 1 MB less.
How does Squirrelmail plugin's INBOX usage compare with "du -sh cur new"? Do you have any files in tmp/ directories? Currently Dovecot never cleans up the tmp/ directory. I guess I should add the code for that..
On Tue, January 16, 2007 16:39, Timo Sirainen wrote:
On Tue, 2007-01-16 at 11:00 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much.
RFC822.SIZE reports message's "virtual size" as it is sent to IMAP clients, which means that if the maildir files contain LF-only linefeeds (which is usually the case), the RFC822.SIZE increases the size by the number of lines in the file (so each line ends with CRLF).
That however only increases the size reported by the Squirrelmail plugin. I don't know why it would report 1 MB less.
How does Squirrelmail plugin's INBOX usage compare with "du -sh cur new"? Do you have any files in tmp/ directories? Currently Dovecot never cleans up the tmp/ directory. I guess I should add the code for that..
I already posted this on the Squirrelmail list, but the folder sizes plugin outputs this:
Folder Sizes Folder Count Unread Size INBOX 260 19 3.2 M Drafts 0 0 0 k Sent 0 0 0 k Trash 0 0 0 k spam 0 0 0 k 1 Folder 0 0 0 k 5 Folders 260 19 3.2 M
while when I go to that maildir folder on my server:
$ du -hs useraccount 4.3M useraccount
Of course there's some overhead:
du -ah | grep dovecot 4.0K ./.INBOX.spam/dovecot.index.log 4.0K ./.INBOX.spam/dovecot.index 32K ./.INBOX.spam/dovecot.index.cache 4.0K ./.INBOX.spam/dovecot-uidlist 4.0K ./.Drafts/dovecot.index.log 4.0K ./.Drafts/dovecot.index 16K ./.Drafts/dovecot.index.cache 4.0K ./.Drafts/dovecot-uidlist 4.0K ./.Sent/dovecot.index.log 4.0K ./.Sent/dovecot.index 16K ./.Sent/dovecot.index.cache 4.0K ./.Sent/dovecot-uidlist 4.0K ./.Trash/dovecot.index.log 4.0K ./.Trash/dovecot.index 20K ./.Trash/dovecot.index.cache 4.0K ./.Trash/dovecot-uidlist 8.0K ./dovecot.index.log 4.0K ./dovecot.index 120K ./dovecot.index.cache 8.0K ./dovecot-uidlist
du -h subscriptions 4.0K subscriptions
276K in total, 4.4M - 276K isn't the reported 3.2M I would guess. Also if I do a FETCH 1:* (RFC822.SIZE) by hand on my imap server and sort the results for the 10 biggest emails:
(RFC822.SIZE 57151) (RFC822.SIZE 57352) (RFC822.SIZE 57477) (RFC822.SIZE 57498) (RFC822.SIZE 57640) (RFC822.SIZE 57776) (RFC822.SIZE 58842) (RFC822.SIZE 60778) (RFC822.SIZE 61320) (RFC822.SIZE 79539)
10 biggest files with du -ab
56526 ./cur/1165969700.640.mbox:2, 56754 ./cur/1166710093.P5305Q0M44492.host.domain.nl:2,S 56943 ./cur/1165969704.640.mbox:2, 57151 ./cur/1165969708.640.mbox:2, 57498 ./cur/1165969702.640.mbox:2, 57640 ./cur/1165969709.640.mbox:2, 57830 ./cur/1168348302.H160143P8248.host.domain.nl:2, 59710 ./cur/1166710169.P6010Q9M768683.host.domain.nl:2,S 61320 ./cur/1165969694.640.mbox:2, 79539 ./cur/1165969642.640.mbox:2,S
Quite strange don't you think? Some files do exactly match others don't, I doublechecked this by grepping the output for one of these mail sizes.
Kind regards,
Tristan
-- Prism Mail Solutions Deployment, customization and hosting http://prism.nl
On Tue, 2007-01-16 at 19:17 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
On Tue, January 16, 2007 16:39, Timo Sirainen wrote:
On Tue, 2007-01-16 at 11:00 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much.
RFC822.SIZE reports message's "virtual size" as it is sent to IMAP clients, which means that if the maildir files contain LF-only linefeeds (which is usually the case), the RFC822.SIZE increases the size by the number of lines in the file (so each line ends with CRLF).
That however only increases the size reported by the Squirrelmail plugin. I don't know why it would report 1 MB less.
How does Squirrelmail plugin's INBOX usage compare with "du -sh cur new"? Do you have any files in tmp/ directories? Currently Dovecot never cleans up the tmp/ directory. I guess I should add the code for that..
I already posted this on the Squirrelmail list, but the folder sizes plugin outputs this:
Yea, I noticed that. But what about "du -sh cur new", and what about files in tmp? I'm guessing you've some leftover files in tmp.
On Tue, January 16, 2007 19:49, Timo Sirainen wrote:
On Tue, 2007-01-16 at 19:17 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
On Tue, 2007-01-16 at 11:00 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much.
RFC822.SIZE reports message's "virtual size" as it is sent to IMAP clients, which means that if the maildir files contain LF-only
On Tue, January 16, 2007 16:39, Timo Sirainen wrote: linefeeds
(which is usually the case), the RFC822.SIZE increases the size by the number of lines in the file (so each line ends with CRLF).
That however only increases the size reported by the Squirrelmail plugin. I don't know why it would report 1 MB less.
How does Squirrelmail plugin's INBOX usage compare with "du -sh cur new"? Do you have any files in tmp/ directories? Currently Dovecot never cleans up the tmp/ directory. I guess I should add the code for that..
I already posted this on the Squirrelmail list, but the folder sizes plugin outputs this:
Yea, I noticed that. But what about "du -sh cur new", and what about files in tmp? I'm guessing you've some leftover files in tmp.
Nope sorry:
du -sh cur new 4.1M cur 12K new
tmp is completely empty.
Regards,
Tristan
-- Prism Mail Solutions Deployment, customization and hosting http://prism.nl
On Tue, 2007-01-16 at 21:48 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
du -sh cur new 4.1M cur
From the ls -l output that you sent me:
cat ls|awk '{print $5"+"}'|tr -d '\n'|sed 's/+$/\n/'|bc 3488542
So your filesystem has lost 0,7MB somehow. Are there any hidden files (ls -la)? If not, then I can think of two things:
There's a large file (or more) that have already been deleted, but some process (stuck maybe?) still keeps them open. But if you've rebooted since, this can't be the case.
Your filesystem has messed up something.
On Wed, 2007-01-17 at 01:05 +0200, Timo Sirainen wrote:
On Tue, 2007-01-16 at 21:48 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
du -sh cur new 4.1M cur
From the ls -l output that you sent me:
cat ls|awk '{print $5"+"}'|tr -d '\n'|sed 's/+$/\n/'|bc 3488542
So your filesystem has lost 0,7MB somehow. Are there any hidden files (ls -la)? If not, then I can think of two things:
There's a large file (or more) that have already been deleted, but some process (stuck maybe?) still keeps them open. But if you've rebooted since, this can't be the case.
Your filesystem has messed up something.
Actually the most obvious reason that I for some reason failed to think of:
- Your filesystem's block size is 4kB, and there's lots of wasted space because of that.
cat ls|awk '{print "("$5"+4095)/4096+"}'|tr -d '\n'|sed 's/^/(/'|sed 's/+$/)*4096\n/'|bc 4096000
So, no bugs anywhere.
On Wed, January 17, 2007 00:21, Timo Sirainen wrote:
On Wed, 2007-01-17 at 01:05 +0200, Timo Sirainen wrote:
On Tue, 2007-01-16 at 21:48 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
du -sh cur new 4.1M cur
From the ls -l output that you sent me:
cat ls|awk '{print $5"+"}'|tr -d '\n'|sed 's/+$/\n/'|bc 3488542
So your filesystem has lost 0,7MB somehow. Are there any hidden files (ls -la)? If not, then I can think of two things:
There's a large file (or more) that have already been deleted, but some process (stuck maybe?) still keeps them open. But if you've rebooted since, this can't be the case.
Your filesystem has messed up something.
Actually the most obvious reason that I for some reason failed to think of:
- Your filesystem's block size is 4kB, and there's lots of wasted space because of that.
cat ls|awk '{print "("$5"+4095)/4096+"}'|tr -d '\n'|sed 's/^/(/'|sed 's/+$/)*4096\n/'|bc 4096000
dumpe2fs /dev/md0 | grep -i 'block size' dumpe2fs 1.37 (21-Mar-2005) Block size: 4096
You were spot on!
So, no bugs anywhere.
Very nice! Now I just have to figure out a way to have my users understand why they're using more space on my server then the total of all their emails sugguests.
Regards,
Tristan
-- Prism Mail Solutions Deployment, customization and hosting http://prism.nl
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk.
One thing to keep in mind is that du reports on disk usage not file sizes. So du will report an answer based on the number of blocks being used. If you are using 1K blocks for example, messages smaller than 1K waste disk space. If you have a large number of messages the greater the discrepancy you might see.
Kenny Dail kend@amigo.net
Tristan Woudenberg - Prism Mail Solutions schrieb:
On Tue, January 16, 2007 16:39, Timo Sirainen wrote:
On Tue, 2007-01-16 at 11:00 +0100, Tristan Woudenberg - Prism Mail Solutions wrote:
There is quite a discrepancy in the rfc822.size reported by Dovecot and the actual space it takes up on my harddisk. I know there could be some overhead here from all the dovecot* and subscription list files in my imap folders, but like I stated in my email to the SquirrelMail plugin mailing list, about 1mb for a 4.3mb maildir mailbox might be a but much. RFC822.SIZE reports message's "virtual size" as it is sent to IMAP clients, which means that if the maildir files contain LF-only linefeeds (which is usually the case), the RFC822.SIZE increases the size by the number of lines in the file (so each line ends with CRLF).
That however only increases the size reported by the Squirrelmail plugin. I don't know why it would report 1 MB less.
How does Squirrelmail plugin's INBOX usage compare with "du -sh cur new"? Do you have any files in tmp/ directories? Currently Dovecot never cleans up the tmp/ directory. I guess I should add the code for that..
I already posted this on the Squirrelmail list, but the folder sizes plugin outputs this:
Folder Sizes Folder Count Unread Size INBOX 260 19 3.2 M Drafts 0 0 0 k Sent 0 0 0 k Trash 0 0 0 k spam 0 0 0 k 1 Folder 0 0 0 k 5 Folders 260 19 3.2 M
while when I go to that maildir folder on my server:
$ du -hs useraccount 4.3M useraccount
Of course there's some overhead:
du -ah | grep dovecot 4.0K ./.INBOX.spam/dovecot.index.log 4.0K ./.INBOX.spam/dovecot.index 32K ./.INBOX.spam/dovecot.index.cache 4.0K ./.INBOX.spam/dovecot-uidlist 4.0K ./.Drafts/dovecot.index.log 4.0K ./.Drafts/dovecot.index 16K ./.Drafts/dovecot.index.cache 4.0K ./.Drafts/dovecot-uidlist 4.0K ./.Sent/dovecot.index.log 4.0K ./.Sent/dovecot.index 16K ./.Sent/dovecot.index.cache 4.0K ./.Sent/dovecot-uidlist 4.0K ./.Trash/dovecot.index.log 4.0K ./.Trash/dovecot.index 20K ./.Trash/dovecot.index.cache 4.0K ./.Trash/dovecot-uidlist 8.0K ./dovecot.index.log 4.0K ./dovecot.index 120K ./dovecot.index.cache 8.0K ./dovecot-uidlist
du -h subscriptions 4.0K subscriptions
276K in total, 4.4M - 276K isn't the reported 3.2M I would guess. Also if I do a FETCH 1:* (RFC822.SIZE) by hand on my imap server and sort the results for the 10 biggest emails:
(RFC822.SIZE 57151) (RFC822.SIZE 57352) (RFC822.SIZE 57477) (RFC822.SIZE 57498) (RFC822.SIZE 57640) (RFC822.SIZE 57776) (RFC822.SIZE 58842) (RFC822.SIZE 60778) (RFC822.SIZE 61320) (RFC822.SIZE 79539)
10 biggest files with du -ab
56526 ./cur/1165969700.640.mbox:2, 56754 ./cur/1166710093.P5305Q0M44492.host.domain.nl:2,S 56943 ./cur/1165969704.640.mbox:2, 57151 ./cur/1165969708.640.mbox:2, 57498 ./cur/1165969702.640.mbox:2, 57640 ./cur/1165969709.640.mbox:2, 57830 ./cur/1168348302.H160143P8248.host.domain.nl:2, 59710 ./cur/1166710169.P6010Q9M768683.host.domain.nl:2,S 61320 ./cur/1165969694.640.mbox:2, 79539 ./cur/1165969642.640.mbox:2,S
Quite strange don't you think? Some files do exactly match others don't, I doublechecked this by grepping the output for one of these mail sizes.
Kind regards,
Tristan
Hi Tristan heres my output
ls -lh total 4.0K -rw------- 1 vmail vmail 1.4K Jan 14 14:25 1168781146.V811I2164003M702149.mail:2,RS
squirrel So, 14:25 A adsf 1.4 k sqplugin folder size INBOX 1 0 1.4 k
so i cant see any failure
Regards
-- Diese Nachricht wurde auf Viren und andere gefährliche Inhalte untersucht und ist - aktuelle Virenscanner vorausgesetzt - sauber.
participants (4)
-
Kenny Dail
-
Robert Schetterer
-
Timo Sirainen
-
Tristan Woudenberg - Prism Mail Solutions