[Dovecot] More efficient Deleting of a whole folder (or Purging Trash)
Hi,
I've straced a few dovecot processes after hitting purge on a large Trash folder (35,000 messages). It looks like it is going through each message, one by one, and removing (unlinking) each one.
Is there not a more efficient way to do this? If Dovecot knows the whole folder is being deleted (ie a Trash purge), could it do something clever with the filesystem to just remove the whole folder?
Incidently, the command that initially deleted those 35k messages to Trash took about 1 hour to complete on my server and effectively locked my account out for all that time. There were also an awlful lot of these output from my strace:
nanosleep({0, 131475000}, NULL) = 0 lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
Not sure if there's much one can do about speeding up deleting a lot of messages though. My webmail client has a 'delete all messages' button for each folder. If there was an efficient IMAP command for deleting a whole folder that would be useful in those circumstances.
Interested to know if there is anything that can be improved.
Thanks, Daniel
Dovecot 1.0.9 XFS Filesystem Maildir Directories
At 4:03 PM +0000 1/29/08, Daniel Watts wrote:
Hi,
I've straced a few dovecot processes after hitting purge on a large Trash folder (35,000 messages). It looks like it is going through each message, one by one, and removing (unlinking) each one.
Is there not a more efficient way to do this? If Dovecot knows the whole folder is being deleted (ie a Trash purge), could it do something clever with the filesystem to just remove the whole folder?
Not with most filesystems. Generally you need to unlink each file in a directory first before removing the directory. I'm not familiar with the details of XFS, but with most filesystems the directory is just a list of the names and inodes of the items in the directory, so deleting the directory node itself leaves all of those inodes and all of the data blocks they point to untouched: all you've done is lost track of them.
For Dovecot, the mailbox index also has to be updated for each deletion, and while that in itself is not terribly expensive per message, it has to be done one message at a time to maintain consistency between the index and the real mail.
Incidently, the command that initially deleted those 35k messages to Trash took about 1 hour to complete on my server and effectively locked my account out for all that time. There were also an awlful lot of these output from my strace:
nanosleep({0, 131475000}, NULL) = 0 lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
Not sure if there's much one can do about speeding up deleting a lot of messages though. My webmail client has a 'delete all messages' button for each folder. If there was an efficient IMAP command for deleting a whole folder that would be useful in those circumstances.
Interested to know if there is anything that can be improved.
As a general issue, I doubt it.
However:
Dovecot 1.0.9 XFS Filesystem Maildir Directories
XFS historically has had very poor delete performance unless specially tuned.
Bill Cole
bill@scconsult.com
On Tue, 29 Jan 2008, Bill Cole wrote:
For Dovecot, the mailbox index also has to be updated for each deletion, and while that in itself is not terribly expensive per message, it has to be done one message at a time to maintain consistency between the index and the real mail.
In general, this isn't really true - because Dovecot can deal with other programs accessing the Maildir, it can't treat the index as canonical.
Therefore if Dovecot knows it is deleting a whole folder, Dovecot could just as easily skip over the index updates or equivalently "decide to do them all at once at the end", at which point the index is empty.
-- Asheesh.
-- Any sufficiently advanced technology is indistinguishable from a rigged demo. -- Andy Finkel, computer guy
At 6:32 PM -0800 1/29/08, Asheesh Laroia imposed structure on a stream of electrons, yielding:
On Tue, 29 Jan 2008, Bill Cole wrote:
For Dovecot, the mailbox index also has to be updated for each deletion, and while that in itself is not terribly expensive per message, it has to be done one message at a time to maintain consistency between the index and the real mail.
In general, this isn't really true - because Dovecot can deal with other programs accessing the Maildir, it can't treat the index as canonical.
I was speaking specifically of what Dovecot might be able to do to speed up user experience, which would be to update (i.e. clear) the index, then spawn off something to actually do the slow task of deleting the files. To keep that behavior from cascading into pointless index rebuilds during the cleanup time, there would need to be some big changes in how Dovecot keeps index files in synch.
Or put another way: the consistency problem is not that Dovecot can't regenerate its index files from the mailbox, but rather that it can and will do so whenever it sees a directory change, and it always trusts the data over the index (that's GOOD.) If the index were to be updated ahead of a slow change of the data in order to provide the user the appearance of speed, it violates the design of the index system.
Therefore if Dovecot knows it is deleting a whole folder, Dovecot could just as easily skip over the index updates or equivalently "decide to do them all at once at the end", at which point the index is empty.
Except that it wouldn't really help. The index updates are not the expensive part, the changes to the filesystem are. In this case it looks like an issue with XFS, a filesystem that is not normally recommended for usage that involves the creation and deletion of a lot of small files. Changing a little bit of data in an index file is very fast compared to the deletion of a file from a directory with 35k entries.
--
Bill Cole
bill@scconsult.com
On Tue, 29 Jan 2008, Bill Cole wrote:
At 6:32 PM -0800 1/29/08, Asheesh Laroia imposed structure on a stream of electrons, yielding:
On Tue, 29 Jan 2008, Bill Cole wrote:
For Dovecot, the mailbox index also has to be updated for each deletion, and while that in itself is not terribly expensive per message, it has to be done one message at a time to maintain consistency between the index and the real mail.
In general, this isn't really true - because Dovecot can deal with other programs accessing the Maildir, it can't treat the index as canonical.
I was speaking specifically of what Dovecot might be able to do to speed up user experience, which would be to update (i.e. clear) the index, then spawn off something to actually do the slow task of deleting the files. To keep that behavior from cascading into pointless index rebuilds during the cleanup time, there would need to be some big changes in how Dovecot keeps index files in synch.
Ah, I see.
Or put another way: the consistency problem is not that Dovecot can't regenerate its index files from the mailbox, but rather that it can and will do so whenever it sees a directory change, and it always trusts the data over the index (that's GOOD.) If the index were to be updated ahead of a slow change of the data in order to provide the user the appearance of speed, it violates the design of the index system.
Therefore if Dovecot knows it is deleting a whole folder, Dovecot could just as easily skip over the index updates or equivalently "decide to do them all at once at the end", at which point the index is empty.
Except that it wouldn't really help. The index updates are not the expensive part, the changes to the filesystem are. In this case it looks like an issue with XFS, a filesystem that is not normally recommended for usage that involves the creation and deletion of a lot of small files. Changing a little bit of data in an index file is very fast compared to the deletion of a file from a directory with 35k entries.
I misunderstood the cause of the slowness then, if it is really true that updating the index is not an important factor. I don't remember seeing profiling, but since I don't have any myself I'll take your word for it.
-- Asheesh.
-- "Atomic batteries to power, turbines to speed." -- Robin, The Boy Wonder
I misunderstood the cause of the slowness then, if it is really true that updating the index is not an important factor. I don't remember seeing profiling, but since I don't have any myself I'll take your word for it.
It always frustrates me to see people speculate on stuff instead of getting their stop watch out and trying it
Just get a small shell script to run touch ${rand_file} 35,000 times
and time it. Then drop into your shell and do rm -rf on the folder
(for bonus marks you could purge the disk cache before doing this).
This gives you a benchmark on your hardware separately to what dovecot
offers. Take that number and lets work from there... Shouldn't take
too long to do something like this?
Ed W
On Wed, 30 Jan 2008, Ed W wrote:
I misunderstood the cause of the slowness then, if it is really true that updating the index is not an important factor. I don't remember seeing profiling, but since I don't have any myself I'll take your word for it.
It always frustrates me to see people speculate on stuff instead of getting their stop watch out and trying it
Just get a small shell script to run
touch ${rand_file}35,000 times and time it. Then drop into your shell and do rm -rf on the folder (for bonus marks you could purge the disk cache before doing this). This gives you a benchmark on your hardware separately to what dovecot offers. Take that number and lets work from there... Shouldn't take too long to do something like this?
Since the problem is supposedly file-system specific, you're assuming Asheesh uses the XFS file system. And seeing as how he's not the one with the original problem, it's also assuming he cares enough to run a test like that. Since it was put forth already in the thread that the file system is the problem, there's not much incentive to test that hypothesis if it involves non-trivial work.
I know I've definitely wasted a lot of time testing out some of the various problems that have come up on this list by setting up weird dovecot installations. But, not everyone wants to do that for every single case.
Plus, in this case, you can just google: xfs delete.
The first result[1] points out the importance of tuning, but the third[2] shows exactly what was said on the list: XFS shows very poor delete performance on directories full of small files. (Deleting a directory with ~8000 small files, times in seconds: JFS 3, reiser 1, XFS 30, ext3 2)
[1] http://everything2.com/index.pl?node_id=1479435 [2] http://sidux.com/PNphpBB2-viewtopic-t-5275.html
Best, Ben
P.S. Rereading this it sounds more confrontational than I'd meant to be. No offense intended - just a general: running your own benchmarks isn't always worthwhile or necessary, especially when it's not your problem.
On 1/30/08, Ed W lists@wildgooses.com wrote:
Just get a small shell script to run
touch ${rand_file}35,000 times and time it. Then drop into your shell and do rm -rf on the folder (for bonus marks you could purge the disk cache before doing this). This gives you a benchmark on your hardware separately to what dovecot offers. Take that number and lets work from there... Shouldn't take too long to do something like this?
Plus indexing work (or quota ... ewww), I wouldn't know if Dovecot updates and index after every action or what have you. I think Unix picks up most of the resource juggle here
Off topic, in the end Dovecot is insanely fast, especially with the LDA to write to indexes on the delivery side and using the custom file name hints for pop3 and what have you. If the index is not known and imap is out of the blue told to Examine() and Delete(...) and Expunge() there might be a significant difference.
Making a perl script to insert 35k emails using dovecot deliver doesn't impact my under resourced machines and I work with spam/(false|ham|queue) system with mail::imapclient automation and the script flies through thousands of messages on the hour without notice.
-- Gabriel Millerd
On Wed, 30 Jan 2008, Gabriel Millerd wrote:
On 1/30/08, Ed W lists@wildgooses.com wrote:
Just get a small shell script to run
touch ${rand_file}35,000 times and time it. Then drop into your shell and do rm -rf on the folder (for bonus marks you could purge the disk cache before doing this). This gives you a benchmark on your hardware separately to what dovecot offers. Take that number and lets work from there... Shouldn't take too long to do something like this?Plus indexing work (or quota ... ewww), I wouldn't know if Dovecot updates and index after every action or what have you. I think Unix picks up most of the resource juggle here
Off topic, in the end Dovecot is insanely fast, especially with the LDA to write to indexes on the delivery side and using the custom file name hints for pop3 and what have you. If the index is not known and imap is out of the blue told to Examine() and Delete(...) and Expunge() there might be a significant difference.
Making a perl script to insert 35k emails using dovecot deliver doesn't impact my under resourced machines and I work with spam/(false|ham|queue) system with mail::imapclient automation and the script flies through thousands of messages on the hour without notice.
Did you then delete those 35K messages in one fell swoop? I wouldn't be surprised if XFS's performance were significantly different between these exaggerrated extremes:
repeat 35000 times: { create a file, delete a file }
and
create 35000 files, delete 35000 files
Best, Ben
On Wed, 30 Jan 2008, Benjamin R. Haskell wrote:
On Wed, 30 Jan 2008, Gabriel Millerd wrote:
Making a perl script to insert 35k emails using dovecot deliver doesn't impact my under resourced machines and I work with spam/(false|ham|queue) system with mail::imapclient automation and the script flies through thousands of messages on the hour without notice.
Did you then delete those 35K messages in one fell swoop? I wouldn't be surprised if XFS's performance were significantly different between these exaggerrated extremes:
repeat 35000 times: { create a file, delete a file }
and
create 35000 files, delete 35000 files
Best, Ben
Forgot to ask: Are you using XFS?
On Jan 29, 2008, at 6:03 PM, Daniel Watts wrote:
Is there not a more efficient way to do this? If Dovecot knows the
whole folder is being deleted (ie a Trash purge), could it do
something clever with the filesystem to just remove the whole folder?
If you use IMAP DELETE command, it renames the directory to ..DOVECOT-
TRASH (or something like it) and only after that it starts unlinking
the files. It wouldn't be too difficult to have this return success
immediately and then keep deleting the files on the background. I'm
not sure if this is worth the trouble though. It would also be
possible to do this as a plugin.
Incidently, the command that initially deleted those 35k messages to
Trash took about 1 hour to complete on my server and effectively
locked my account out for all that time. There were also an awlful
lot of these output from my strace:nanosleep({0, 131475000}, NULL) = 0 lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/ dovecot-uidlist.lock", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
You mean this was in the deleting process, or the other processes? The
deleting process should have kept the maildir locked while it was
deleting the files, so other processes were locked out. This is by
design and can't be changed, at least until I some day add inotify
support for maildir synchronization.
Hi Timo,
On Jan 29, 2008, at 6:03 PM, Daniel Watts wrote:
Is there not a more efficient way to do this? If Dovecot knows the whole folder is being deleted (ie a Trash purge), could it do something clever with the filesystem to just remove the whole folder?
If you use IMAP DELETE command, it renames the directory to ..DOVECOT-TRASH (or something like it) and only after that it starts unlinking the files. It wouldn't be too difficult to have this return success immediately and then keep deleting the files on the background. I'm not sure if this is worth the trouble though. It would also be possible to do this as a plugin. Something like that would be great. Would this actually then avoid the locking problem described below?
Incidently, the command that initially deleted those 35k messages to Trash took about 1 hour to complete on my server and effectively locked my account out for all that time. There were also an awlful lot of these output from my strace:
nanosleep({0, 131475000}, NULL) = 0 lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
You mean this was in the deleting process, or the other processes? The deleting process should have kept the maildir locked while it was deleting the files, so other processes were locked out. This is by design and can't be changed, at least until I some day add inotify support for maildir synchronization. You're probably right - another process is probably trying to access the folder that is currently being deleted and being prevented from doing so. There was an awlful lot of polling going on though - probably man tens per second. Not sure if this causes the system to thrash a bit and raise load unnecessarily.
If you renamed the folder to *.DOVECOT-DELETED or something you won't have another process trying to access the same folder and locking the account up.
IMHO this benefit is worth the effort. =)
Daniel
On Thu, 2008-01-31 at 10:01 +0000, Daniel Watts wrote:
Hi Timo,
On Jan 29, 2008, at 6:03 PM, Daniel Watts wrote:
Is there not a more efficient way to do this? If Dovecot knows the whole folder is being deleted (ie a Trash purge), could it do something clever with the filesystem to just remove the whole folder?
If you use IMAP DELETE command, it renames the directory to ..DOVECOT-TRASH (or something like it) and only after that it starts unlinking the files. It wouldn't be too difficult to have this return success immediately and then keep deleting the files on the background. I'm not sure if this is worth the trouble though. It would also be possible to do this as a plugin. Something like that would be great. Would this actually then avoid the locking problem described below?
Well, kind of, because the entire mailbox is deleted it can't even be opened by another session :) Although it would be possible to create it back immediately after deletion, but there's a small window during which it's possible that the mailbox selection fails.
IMHO this benefit is worth the effort. =)
There are two separate things here:
Making DELETE command work on background. This would require the user to specifically delete + create back the mailbox and not just expunge all messages. This wouldn't be too difficult.
Optimize expunging all messages by renaming Maildir/cur under Maildir/tmp/, mkdir Maildir/cur back and then start deleting Maildir/tmp/cur on background. This has a race condition between rename() + mkdir(), so I don't want to ever do this by default.
Anyway I'm not going to implement either one anytime soon, but feel free to write a plugin. :)
participants (7)
-
Asheesh Laroia
-
Benjamin R. Haskell
-
Bill Cole
-
Daniel Watts
-
Ed W
-
Gabriel Millerd
-
Timo Sirainen