[Dovecot] Purging a large folder... anything I need to be careful of?
I've got a large message folder (about 40G in 'cur') that I need to empty to a more reasonable size. Oddly, my Thunderbird e-mail client only sees about 2.4G of that, and has "looped" a few times. What I mean is, I thought someone was cleaning the folder out, but Thunderbird apparently was actually just looping back to 0 every 4 gigs and starting to count up again Ha. But that's a sidenote. So, since Thunderbird apparently is not seeing all of the messages, what is the safest way to delete the files straight from the 'cur' folder? IS that safe? Will I mess up any of the dovecot indexes in any way?? Dave
Dave schrieb:
I've got a large message folder (about 40G in 'cur') that I need to empty to a more reasonable size. Oddly, my Thunderbird e-mail client only sees about 2.4G of that, and has "looped" a few times. What I mean is, I thought someone was cleaning the folder out, but Thunderbird apparently was actually just looping back to 0 every 4 gigs and starting to count up again Ha. But that's a sidenote. So, since Thunderbird
But it's a strange one. I have some fairly big folders (>>10G) too but there is no problem at all with TB 2.0.0.23.
apparently is not seeing all of the messages, what is the safest way to delete the files straight from the 'cur' folder? IS that safe? Will I mess up any of the dovecot indexes in any way??
Yes it is safe. Dovecot has self healing abilities which will fix the index files.
Dave
On 2010-02-10 12:59 AM, Dennis Guhl wrote:
Dave schrieb:
I've got a large message folder (about 40G in 'cur') that I need to empty to a more reasonable size. Oddly, my Thunderbird e-mail client only sees about 2.4G of that, and has "looped" a few times. What I mean is, I thought someone was cleaning the folder out, but Thunderbird apparently was actually just looping back to 0 every 4 gigs and starting to count up again Ha. But that's a sidenote. So, since Thunderbird
But it's a strange one. I have some fairly big folders (>>10G) too but there is no problem at all with TB 2.0.0.23.
Not entirely accurate. TB does indeed have a big problem with mbox files larger than 4GB. But as long as you don't set an IMAP folder that is larger than 4GB to full offline mode, you should be fine, BUT... the calculated size will be displayed using a 32-bit value, so as it calculates the size, when it hits 4GB, it will roll over and start again
- so, a 6GB and a 10GB folder will both be displayed as 2GB...
apparently is not seeing all of the messages,
Yes it is - it is simply not calculating the folder size correctly. I don't know if this an underlying OS issue (32-bit vs 64-bit), or a TB bug...
what is the safest way to delete the files straight from the 'cur' folder? IS that safe? Will I mess up any of the dovecot indexes in any way??
Yes it is safe. Dovecot has self healing abilities which will fix the index files.
I don't think dovecots indexes are affected by this - it is merely a cosmetic TB issue.
--
Best regards,
Charles
Charles Marcus schrieb:
On 2010-02-10 12:59 AM, Dennis Guhl wrote:
Dave schrieb:
I've got a large message folder (about 40G in 'cur') that I need to empty to a more reasonable size. Oddly, my Thunderbird e-mail client only sees about 2.4G of that, and has "looped" a few times. What I mean is, I thought someone was cleaning the folder out, but Thunderbird apparently was actually just looping back to 0 every 4 gigs and starting to count up again Ha. But that's a sidenote. So, since Thunderbird But it's a strange one. I have some fairly big folders (>>10G) too but there is no problem at all with TB 2.0.0.23.
Not entirely accurate. TB does indeed have a big problem with mbox files
Oh, I think that should be a bug, since today files with more then 4GB are nor rare. But using only IMAP I never encountered this problem and wasn't aware of it.
larger than 4GB. But as long as you don't set an IMAP folder that is larger than 4GB to full offline mode, you should be fine, BUT... the calculated size will be displayed using a 32-bit value, so as it calculates the size, when it hits 4GB, it will roll over and start again
- so, a 6GB and a 10GB folder will both be displayed as 2GB...
OK, this I never observed. But there are no quotas on the folder so I never took a look at the size with TB.
apparently is not seeing all of the messages,
Yes it is - it is simply not calculating the folder size correctly. I don't know if this an underlying OS issue (32-bit vs 64-bit), or a TB bug...
And what is displayed as the message count if exceed 32 bit?
what is the safest way to delete the files straight from the 'cur' folder? IS that safe? Will I mess up any of the dovecot indexes in any way?? Yes it is safe. Dovecot has self healing abilities which will fix the index files.
I don't think dovecots indexes are affected by this - it is merely a cosmetic TB issue.
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
On Wed, 2010-02-10 at 13:40 +0100, Dennis Guhl wrote:
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
It shouldn't even log anything. Dovecot's Maildir code was built so that it expected there to be other non-Dovecot MUAs accessing and changing the Maildir. Only recently I added maildir_very_dirty_syncs=yes setting where it assumes it's the only thing accessing maildir to improve performance.
Timo Sirainen schrieb:
On Wed, 2010-02-10 at 13:40 +0100, Dennis Guhl wrote:
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
It shouldn't even log anything. Dovecot's Maildir code was built so that it expected there to be other non-Dovecot MUAs accessing and changing the Maildir. Only recently I added maildir_very_dirty_syncs=yes setting where it assumes it's the only thing accessing maildir to improve performance.
Am I wrong or did you remove the log entry? IIRC it was there in the past.
On Wed, 2010-02-10 at 14:45 +0100, Dennis Guhl wrote:
Timo Sirainen schrieb:
On Wed, 2010-02-10 at 13:40 +0100, Dennis Guhl wrote:
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
It shouldn't even log anything. Dovecot's Maildir code was built so that it expected there to be other non-Dovecot MUAs accessing and changing the Maildir. Only recently I added maildir_very_dirty_syncs=yes setting where it assumes it's the only thing accessing maildir to improve performance.
Am I wrong or did you remove the log entry? IIRC it was there in the past.
If you saw a log entry it happened because of something else, not simply because you deleted a file from cur/ or new/.
Timo Sirainen schrieb:
On Wed, 2010-02-10 at 14:45 +0100, Dennis Guhl wrote:
Timo Sirainen schrieb:
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log. It shouldn't even log anything. Dovecot's Maildir code was built so that it expected there to be other non-Dovecot MUAs accessing and changing
On Wed, 2010-02-10 at 13:40 +0100, Dennis Guhl wrote: the Maildir. Only recently I added maildir_very_dirty_syncs=yes setting where it assumes it's the only thing accessing maildir to improve performance. Am I wrong or did you remove the log entry? IIRC it was there in the past.
If you saw a log entry it happened because of something else, not simply because you deleted a file from cur/ or new/.
Oh, then I think I should have a closer look when I see something about the indexes in the logs.
Thanks.
On 2010-02-10 7:40 AM, Dennis Guhl wrote:
Charles Marcus schrieb:
On 2010-02-10 12:59 AM, Dennis Guhl wrote:
Dave schrieb: Not entirely accurate. TB does indeed have a big problem with mbox files
Oh, I think that should be a bug, since today files with more then 4GB are nor rare.
I agree... I've been considering opening a bug about this, but it hasn't been high on my list since I don't have my large folders set to full offline mode (just my Inbox mainly)...
But using only IMAP I never encountered this problem and wasn't aware of it.
You would if you had set any of the folders larger than 4GB to full offline mode and sync'd them... ;)
But there are no quotas on the folder so I never took a look at the size with TB.
I always enable extra columns in the folder pane and add the size column, so I can always see how big my folders are...
And what is displayed as the message count if exceed 32 bit?
Message count is correctly displayed, as far as I've been able to determine.
I don't think dovecots indexes are affected by this - it is merely a cosmetic TB issue.
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
Yes, of course the indexes will be rebuilt, but that is normal when messages are deleted/added. I meant they won't be affected in any abnormal way...
--
Best regards,
Charles
Charles Marcus schrieb:
On 2010-02-10 7:40 AM, Dennis Guhl wrote:
Charles Marcus schrieb:
Not entirely accurate. TB does indeed have a big problem with mbox files
Oh, I think that should be a bug, since today files with more then 4GB are nor rare.
I agree... I've been considering opening a bug about this, but it hasn't been high on my list since I don't have my large folders set to full offline mode (just my Inbox mainly)...
I have no folder at all set to offline mode. So it would make no sense for to file this bug since I can't give any hints what goes wrong. Maybe someone like Dave with a more urgent need will file it.
[..]
I don't think dovecots indexes are affected by this - it is merely a cosmetic TB issue.
When you go to ~/Maildir/cur/ to remove messages (as dave intend) dovecots indexes will indeed be affected. But it will cause no trouble beside throwing an entry in the log.
Yes, of course the indexes will be rebuilt, but that is normal when messages are deleted/added. I meant they won't be affected in any abnormal way...
Yes, of course.
Yes it is - it is simply not calculating the folder size correctly. I don't know if this an underlying OS issue (32-bit vs 64-bit), or a TB bug...
Ahh, you were right about this, I did see later that all the messages *were* there in fact, but it does "loop" the folder size as was stated (6G looks like 2G in TB, etc). I appreciate all the info from everyone, and I was able just to just move all of the 40G around using Thunderbird once I saw that all the messages were indeed there. Took a while :), but now we have some things archived and folders are down to a more "reasonable" size (reasonable for us anyway).
It is good to know that removing messages directly from the source folders won't mess up anything as far as Dovecot or its indexes is concerned though, so, thanks for that information everyone!! Dave
Yes it is - it is simply not calculating the folder size correctly. I don't know if this an underlying OS issue (32-bit vs 64-bit), or a TB bug...
Ahh, you were right about this, I did see later that all the messages *were* there in fact, but it does "loop" the folder size as was stated (6G looks like 2G in TB, etc). I appreciate all the info from everyone, and I was able just to just move all of the 40G around using Thunderbird once I saw that all the messages were indeed there. Took a while :), but now we have some things archived and folders are down to a more "reasonable" size (reasonable for us anyway).
It is good to know that removing messages directly from the source folders won't mess up anything as far as Dovecot or its indexes is concerned though, so, thanks for that information everyone!! Dave
participants (5)
-
Charles Marcus
-
Dave
-
David Salisbury
-
Dennis Guhl
-
Timo Sirainen