[Dovecot] Something other than dotlock for uidlist locking?
If I read the code correctly, only dotlocks are supported for locking
of dovecot-uidlist. And I don't see any settings in the config file
for it.
As it turns out, dotlocks are very slow on my system, due to their
implementation via hardlinks.
Would it be possible to support other locking mechanisms for locking
of the uidlist file?
-jdb
On Thu, 2005-08-25 at 10:26 -0700, James Berry wrote:
If I read the code correctly, only dotlocks are supported for locking
of dovecot-uidlist. And I don't see any settings in the config file
for it.As it turns out, dotlocks are very slow on my system, due to their
implementation via hardlinks.Would it be possible to support other locking mechanisms for locking
of the uidlist file?
They are locked only when they are modified, and the modification works by creating a new file and renaming it over the dovecot-uidlist file.
Is it only hardlinks that are problematic? It shouldn't be too difficult to change the dotlock code to rely on working O_EXCL instead of hardlinks.
Hmm. Actually I just did that in CVS. I guess I should also put out some setting to enable it in config file. For now you could just change this:
ret = set->use_excl_lock ?
try_create_lock_excl(&lock_info, write_pid) :
try_create_lock_hardlink(&lock_info, write_pid);to
ret = try_create_lock_excl(&lock_info, write_pid);and see if it works faster.
Hi Timo, (By the way, this seems to relate to a performance issue on
Mac OS X with HFS+).
On Aug 28, 2005, at 12:30 PM, Timo Sirainen wrote:
On Thu, 2005-08-25 at 10:26 -0700, James Berry wrote:
If I read the code correctly, only dotlocks are supported for locking of dovecot-uidlist. And I don't see any settings in the config file for it.
As it turns out, dotlocks are very slow on my system, due to their implementation via hardlinks.
Would it be possible to support other locking mechanisms for locking of the uidlist file?
They are locked only when they are modified, and the modification
works by creating a new file and renaming it over the dovecot-uidlist file.Is it only hardlinks that are problematic? It shouldn't be too
difficult to change the dotlock code to rely on working O_EXCL instead of hardlinks.Hmm. Actually I just did that in CVS. I guess I should also put out
some setting to enable it in config file. For now you could just change
this:ret = set->use_excl_lock ? try_create_lock_excl(&lock_info, write_pid) : try_create_lock_hardlink(&lock_info, write_pid);
to
ret = try_create_lock_excl(&lock_info, write_pid);
and see if it works faster.
Well, a year later I finally got around to trying this out. On a Mac
system. And guess what? It seems much, much snappier. Sorry about the
delay in testing it. One of the symptoms I've been seeing on the
server is that performance gradually degrades over time, even if
dovecot is restarted. The problem seems better after a server
restart, which clears out an area on disk which tracks unlinked and
hardlinked files. I believe this option will fix that performance issue.
Any chance of getting a configuration option for this?
James
Yes, please! Our users are absolutely dying here... Since this problem didn't crop up until we were 90% moved onto Dovecot, we've now got ~100 users who have constant timeouts/locking issues/stale lock files/etc under OS X... Reboots help a very* small amount, but ideally we'd like to have a clean version of dovecot (using beta8 now, too scared to update to rc2) that doesn't need rebooting (though, due to this debacle, we have been migrating to another mail platform entirely - still, anything that improves performance will help with our move, as well).
Thanks,
-deano
James Berry wrote:
Hi Timo, (By the way, this seems to relate to a performance issue on Mac OS X with HFS+).
On Aug 28, 2005, at 12:30 PM, Timo Sirainen wrote:
On Thu, 2005-08-25 at 10:26 -0700, James Berry wrote:
If I read the code correctly, only dotlocks are supported for locking of dovecot-uidlist. And I don't see any settings in the config file for it.
As it turns out, dotlocks are very slow on my system, due to their implementation via hardlinks.
Would it be possible to support other locking mechanisms for locking of the uidlist file?
They are locked only when they are modified, and the modification works by creating a new file and renaming it over the dovecot-uidlist file.
Is it only hardlinks that are problematic? It shouldn't be too difficult to change the dotlock code to rely on working O_EXCL instead of hardlinks.
Hmm. Actually I just did that in CVS. I guess I should also put out some setting to enable it in config file. For now you could just change this:
ret = set->use_excl_lock ? try_create_lock_excl(&lock_info, write_pid) : try_create_lock_hardlink(&lock_info, write_pid);to
ret = try_create_lock_excl(&lock_info, write_pid);and see if it works faster.
Well, a year later I finally got around to trying this out. On a Mac system. And guess what? It seems much, much snappier. Sorry about the delay in testing it. One of the symptoms I've been seeing on the server is that performance gradually degrades over time, even if dovecot is restarted. The problem seems better after a server restart, which clears out an area on disk which tracks unlinked and hardlinked files. I believe this option will fix that performance issue.
Any chance of getting a configuration option for this?
James
On Aug 8, 2006, at 5:43 PM, James Berry wrote:
Hi Timo, (By the way, this seems to relate to a performance issue
on Mac OS X with HFS+).
I think the problem may be more general than this particular case.
This fix speeds up some accesses, but there are still bottlenecks, I
believe. I have a feeling the performance issues will manifest in any
cases where hardlinks are used heavily.
Can you tell me, Timo, in what other cases hard links are used (file
copy, seemingly being one) and whether there is any way to turn them
off or rework the code to avoid them?
Thanks!
James
On Aug 28, 2005, at 12:30 PM, Timo Sirainen wrote:
On Thu, 2005-08-25 at 10:26 -0700, James Berry wrote:
If I read the code correctly, only dotlocks are supported for
locking of dovecot-uidlist. And I don't see any settings in the config file for it.As it turns out, dotlocks are very slow on my system, due to their implementation via hardlinks.
Would it be possible to support other locking mechanisms for locking of the uidlist file?
They are locked only when they are modified, and the modification
works by creating a new file and renaming it over the dovecot-uidlist file.Is it only hardlinks that are problematic? It shouldn't be too
difficult to change the dotlock code to rely on working O_EXCL instead of hardlinks.Hmm. Actually I just did that in CVS. I guess I should also put
out some setting to enable it in config file. For now you could just change
this:ret = set->use_excl_lock ? try_create_lock_excl(&lock_info, write_pid) : try_create_lock_hardlink(&lock_info, write_pid);
to
ret = try_create_lock_excl(&lock_info, write_pid);
and see if it works faster.
Well, a year later I finally got around to trying this out. On a
Mac system. And guess what? It seems much, much snappier. Sorry
about the delay in testing it. One of the symptoms I've been seeing
on the server is that performance gradually degrades over time,
even if dovecot is restarted. The problem seems better after a
server restart, which clears out an area on disk which tracks
unlinked and hardlinked files. I believe this option will fix that
performance issue.Any chance of getting a configuration option for this?
James
Hey Timo,
Thanks again for all the work you do on DoveCot. I'd like to put
together a patch to avoid use of hardlinks on Mac OS X. It definitely
looks doable. (a couple of questions below...)
On Aug 8, 2006, at 6:16 PM, James Berry wrote:
On Aug 8, 2006, at 5:43 PM, James Berry wrote:
Hi Timo, (By the way, this seems to relate to a performance issue
on Mac OS X with HFS+).I think the problem may be more general than this particular case.
This fix speeds up some accesses, but there are still bottlenecks,
I believe. I have a feeling the performance issues will manifest in
any cases where hardlinks are used heavily.Can you tell me, Timo, in what other cases hard links are used
(file copy, seemingly being one) and whether there is any way to
turn them off or rework the code to avoid them?
In seaching the code for "link(", I find only four affected files/
cases. Here I look into possible implementations for a patch to
AVOID_HARDLINKS. Such a patch would dramatically improve performance
on Mac OS X with the HFS+ filesystem.
Timo, would you mind commenting on these to make sure my
understanding is correct? And for the last one, I need a suggestion
on how to proceed.
file-copy.c: [EASY]
Looks like it would be easy to take the already coded case for when try_hardlink is false.
file-dotlock.c: [EASY]
In dotlock_create, use try_create_lock_excl rather than try_create_lock_hardlink.
maildir-copy.c: [LOOKS EASY]
In maildir_copy(), take the fallback case using mail_storage_copy.maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallback case, and I'm unsure of the desired symantics, so I could use some suggestions
here: can we just fall back into something like the code in file-copy.c?
Thanks for your thoughts!
James
On 8/9/06, James Berry <james@jberry.us> wrote:
maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallback case, and I'munsure of the desired symantics, so I could use some suggestions here: can we just fall back into something like the code in file-copy.c?
James-
This probably has to do with the way that Maildir's work. Check out this links for details:
http://cr.yp.to/proto/maildir.html http://en.wikipedia.org/wiki/Maildir
The use of hardlinks give you a level of atomicity and efficiency that I'm not sure you're going to achieve using a standard copy, but I suppose if HFS+ doesn't handle hardlinks well then it's not doing much for you. Basically the hardlink solution gives you good atomicity even through a process crash/kill... something that a straight data copy and delete can't easily ensure.
It's really unfortunate that Apple made such an enormously poor decision in the implementation of HFS+. Any chance you could just use a UFS1 or UFS2 formatted drive for your mail storage?
Good luck, Ben
On Aug 9, 2006, at 2:06 PM, Ben Schumacher wrote:
On 8/9/06, James Berry <james@jberry.us> wrote:
maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallbackcase, and I'm unsure of the desired symantics, so I could use some suggestions here: can we just fall back into something like the code in file- copy.c?
James-
This probably has to do with the way that Maildir's work. Check out this links for details:
http://cr.yp.to/proto/maildir.html http://en.wikipedia.org/wiki/Maildir
The use of hardlinks give you a level of atomicity and efficiency that I'm not sure you're going to achieve using a standard copy, but I suppose if HFS+ doesn't handle hardlinks well then it's not doing much for you. Basically the hardlink solution gives you good atomicity even through a process crash/kill... something that a straight data copy and delete can't easily ensure.
Hmm looks to me like rename() would give the same level of atomicity
as link?
So instead of doing link/unlink, we could just do rename()?
It's really unfortunate that Apple made such an enormously poor decision in the implementation of HFS+. Any chance you could just use a UFS1 or UFS2 formatted drive for your mail storage?
Yes. Unfortunately hardlinks were grafted on to HFS+ after it was
designed. They are a hack. They work fine if there are not too many
of them. But they suffer if you create many, many of them.
Hardinked files on HFS+ live in a single special hidden directory.
Any file that becomes multiply linked is moved into that directory.
And the only way for it to leave that directory is for (a) the link
count to go to zero and (b) for the volume to be remounted. What this
means is that even just using link/unlink to move a file will cause
the file to accumulate in that hidden directory (with linkcount=1).
So in the MailDir case, _all_ mail files will ultimately end up in
that hidden directory. Making the matter worse, temporary files, such
as .lock files, will accumulate at linkcount=0 in the hidden
directory until the volume is remounted (usually at reboot). So as
you can image, the performance of hardlinks suffers due to the size
of this directory as soon as you get very many files, and grows worse
the longer the machine is up.
So, hardlinks were grafted on to HFS+ anticipating that unix-like
systems needed to hardlink _some_ files. The design did not
anticipate the sort of usage of hardlinks by dovecot. And it fails
miserably.
It certainly looks possible to me that we can make the needed changes
to get very reasonable performance without hardlinks, and without
having to use UFS, which has its own problems with many files such as
we get with MailDir.
James
On Wed, 2006-08-09 at 14:28 -0700, James Berry wrote:
Hardinked files on HFS+ live in a single special hidden directory.
Any file that becomes multiply linked is moved into that directory.
And the only way for it to leave that directory is for (a) the link
count to go to zero and (b) for the volume to be remounted. What this
means is that even just using link/unlink to move a file will cause
the file to accumulate in that hidden directory (with linkcount=1).
Wow... Rename() as a single step was added fairly late in the unix game. It's hard to imagine many programs working for long on a system that doesn't release the space after a link/unlink operation.
-- Les Mikesell lesmikesell@gmail.com
So in the MailDir case, _all_ mail files will ultimately end up in
that hidden directory. Making the matter worse, temporary files, such
as .lock files, will accumulate at linkcount=0 in the hidden
directory until the volume is remounted (usually at reboot). So as
you can image, the performance of hardlinks suffers due to the size
of this directory as soon as you get very many files, and grows worse
the longer the machine is up.So, hardlinks were grafted on to HFS+ anticipating that unix-like
systems needed to hardlink _some_ files. The design did not
anticipate the sort of usage of hardlinks by dovecot. And it fails
miserably.It certainly looks possible to me that we can make the needed changes
to get very reasonable performance without hardlinks, and without
having to use UFS, which has its own problems with many files such as
we get with MailDir.James
On 8/9/06, James Berry <james@jberry.us> wrote:
Hmm looks to me like rename() would give the same level of atomicity as link?
So instead of doing link/unlink, we could just do rename()?
This is not entirely true. The point of the link/unlink pairing is to ensure that the target file doesn't exist. There exists a race condition that the link/unlink pairing will protect against: link will not overwrite an existing file, rename will. The work around would be to do something hackish along the lines of creating a dotfile for the target of the rename before doing the actual rename... it's ugly, but I think it's probably the safest cross networked filesystems way of ensuring that the target file isn't being written at exactly the same time you're trying to do your rename.
Cheers. Ben
On 9.8.2006, at 23.44, James Berry wrote:
In seaching the code for "link(", I find only four affected files/ cases. Here I look into possible implementations for a patch to
AVOID_HARDLINKS. Such a patch would dramatically improve
performance on Mac OS X with the HFS+ filesystem.
Hmh. The only annoying problem with this is that it should be runtime
configurable, but there's no clean way to do that without larger
changes.
Timo, would you mind commenting on these to make sure my
understanding is correct? And for the last one, I need a suggestion
on how to proceed.file-copy.c: [EASY]
Looks like it would be easy to take the already coded case for
when try_hardlink is false.
Yes, easy. This case could even be #ifdefed. Although currently the
only user of file_copy() is copying ssl-parameters.dat from /var/lib/
dovecot to /var/run/dovecot, and I don't see it being used elsewhere
anytime soon.
file-dotlock.c: [EASY]
In dotlock_create, use try_create_lock_excl rather than
try_create_lock_hardlink.
This really should rather be configurable from dovecot.conf, but
there's no simple way to do that, except something like a getenv()
hack inside the file-dotlock.c code..
I don't think this should be #ifdefed since O_EXCL could break if
you're using NFS.
maildir-copy.c: [LOOKS EASY]
In maildir_copy(), take the fallback case using mail_storage_copy.
This is the default (maildir_copy_with_hardlinks=no) so no need to do
anything.
maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallback case, and
I'm unsure of the desired symantics, so I could use some
suggestions here: can we just fall back into something like the
code in file-copy.c?
Like you noticed, this can be changed to just rename(). The only
reason it's link()+unlink() is because maildir spec says it should be
done like that. And in case the maildir filenames aren't really
unique the link()+unlink() method makes sure that mails aren't
overwritten, but that practically can't happen with Dovecot. This
code could perhaps also be #ifdefed..
Timo,
Thanks for your comments and attention. I'm attaching a patch which
is a bit rough, but it's what I'm going to begin testing with.
Basically in all cases ifdef'd based on AVOID_HARDLINKS. Hopefully
you might be able to improve the configurability of the dotlock case.
Thanks for such a great product.
James
On Aug 9, 2006, at 3:05 PM, Timo Sirainen wrote:
On 9.8.2006, at 23.44, James Berry wrote:
Timo, would you mind commenting on these to make sure my
understanding is correct? And for the last one, I need a
suggestion on how to proceed.file-copy.c: [EASY]
Looks like it would be easy to take the already coded case for
when try_hardlink is false.Yes, easy. This case could even be #ifdefed. Although currently the
only user of file_copy() is copying ssl-parameters.dat from /var/ lib/dovecot to /var/run/dovecot, and I don't see it being used
elsewhere anytime soon.
#ifdef'd
file-dotlock.c: [EASY]
In dotlock_create, use try_create_lock_excl rather than
try_create_lock_hardlink.This really should rather be configurable from dovecot.conf, but
there's no simple way to do that, except something like a getenv()
hack inside the file-dotlock.c code..I don't think this should be #ifdefed since O_EXCL could break if
you're using NFS.
#ifdef'd to always take the O_EXCL case if AVOID_HARDLINKS is defined.
maildir-copy.c: [LOOKS EASY]
In maildir_copy(), take the fallback case using mail_storage_copy.
This is the default (maildir_copy_with_hardlinks=no) so no need to
do anything.
Okay, nothing done.
maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallback case, and
I'm unsure of the desired symantics, so I could use some
suggestions here: can we just fall back into something like the
code in file-copy.c?Like you noticed, this can be changed to just rename(). The only
reason it's link()+unlink() is because maildir spec says it should
be done like that. And in case the maildir filenames aren't really
unique the link()+unlink() method makes sure that mails aren't
overwritten, but that practically can't happen with Dovecot. This
code could perhaps also be #ifdefed..
#ifdef'd to always use rename() if AVOID_HARDLINKS is defined.
Patch attached:
Hi Timo,
Just to follow up, I've been running with these patches for the last
24 hours or so. With no problems on my end. And with great benefit
wrt performance. Many operations have improved by several orders of
magnitude.
Could we get this patch into rc7?
James
On Aug 9, 2006, at 4:03 PM, James Berry wrote:
Timo,
Thanks for your comments and attention. I'm attaching a patch which
is a bit rough, but it's what I'm going to begin testing with.
Basically in all cases ifdef'd based on AVOID_HARDLINKS. Hopefully
you might be able to improve the configurability of the dotlock case.Thanks for such a great product.
James
On Aug 9, 2006, at 3:05 PM, Timo Sirainen wrote:
On 9.8.2006, at 23.44, James Berry wrote:
Timo, would you mind commenting on these to make sure my
understanding is correct? And for the last one, I need a
suggestion on how to proceed.file-copy.c: [EASY]
Looks like it would be easy to take the already coded case for
when try_hardlink is false.Yes, easy. This case could even be #ifdefed. Although currently
the only user of file_copy() is copying ssl-parameters.dat from / var/lib/dovecot to /var/run/dovecot, and I don't see it being used
elsewhere anytime soon.#ifdef'd
file-dotlock.c: [EASY]
In dotlock_create, use try_create_lock_excl rather than
try_create_lock_hardlink.This really should rather be configurable from dovecot.conf, but
there's no simple way to do that, except something like a getenv()
hack inside the file-dotlock.c code..I don't think this should be #ifdefed since O_EXCL could break if
you're using NFS.#ifdef'd to always take the O_EXCL case if AVOID_HARDLINKS is defined.
maildir-copy.c: [LOOKS EASY]
In maildir_copy(), take the fallback case using mail_storage_copy.
This is the default (maildir_copy_with_hardlinks=no) so no need to
do anything.Okay, nothing done.
maildir-save.c: [NEED HELP]
In maildir_file_move(), there seems to be no fallback case, and
I'm unsure of the desired symantics, so I could use some
suggestions here: can we just fall back into something like the
code in file-copy.c?Like you noticed, this can be changed to just rename(). The only
reason it's link()+unlink() is because maildir spec says it should
be done like that. And in case the maildir filenames aren't really
unique the link()+unlink() method makes sure that mails aren't
overwritten, but that practically can't happen with Dovecot. This
code could perhaps also be #ifdefed..#ifdef'd to always use rename() if AVOID_HARDLINKS is defined.
Patch attached:
<patch-avoid-hardlinks.diff>
participants (5)
-
Ben Schumacher
-
Dean Blackburn
-
James Berry
-
Les Mikesell
-
Timo Sirainen