[Dovecot] Bug? 1.0.0-test28 NFS locking problems
Hi Slight problem - linux 2.4 running dovecot, Solaris 2.8 home directory server. Full lockd support etc. I had to make the following code patches (at end of mail for clarity) to get dovecot to read the users mbox files at all. 1) fcntl with F_SETLKW will not work against a Solaris 2.8 server as proved with a small test program. Other forms of fcntl (ie F_SETLK) are OK. I noticed there are some code witches, but setting mbox_lock_timeout=0 only seems to knock one of the fcntl's over to F_SETLK 2) O_CREAT | O_EXCL are documented not to work over NFS properly (see man open(2) on linux). So I knocked out the O_EXCL and combined with (1) dovecot will talk over NFS to remote mbox files. THESE ARE NOT VIABLE CODE PATCHES They illustrate a problem I found - they are hackish and totally wrong. I'm not sure of the best way to correct this problem cleanly, so I am just submitting it as a potential bug. Cheers Tim Southerwood ************************************************** --- dovecot-1.0-test28/src/lib/file-lock.c.orig 2004-07-20 14:29:47.000000000 +0100 +++ dovecot-1.0-test28/src/lib/file-lock.c 2004-07-20 14:30:15.000000000 +0100 @@ -38,7 +38,7 @@ fl.l_start = 0; fl.l_len = 0; - while (fcntl(fd, timeout != 0 ? F_SETLKW : F_SETLK, &fl) < 0) { + while (fcntl(fd, timeout != 0 ? F_SETLK : F_SETLK, &fl) < 0) { if (timeout == 0 && (errno == EACCES || errno == EAGAIN)) return 0; --- dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-lock.c.orig 2004-07-20 14:29:54.000000000 +0100 +++ dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-lock.c 2004-07-20 14:30:35.000000000 +0100 @@ -361,7 +361,7 @@ fl.l_start = 0; fl.l_len = 0; - wait_type = max_wait_time == 0 ? F_SETLK : F_SETLKW; + wait_type = max_wait_time == 0 ? F_SETLK : F_SETLK; while (fcntl(ctx->ibox->mbox_fd, wait_type, &fl) < 0) { if (errno != EINTR) { if (errno != EAGAIN && errno != EACCES) ************************************************** and ************************************************** diff -NaurbB dovecot-1.0-test28.orig/src/lib/file-dotlock.c dovecot-1.0-test28/src/lib/file-dotlock.c--- dovecot-1.0-test28.orig/src/lib/file-dotlock.c 2004-06-28 18:28:01.000000000 +0100+++ dovecot-1.0-test28/src/lib/file-dotlock.c 2004-07-20 14:50:33.000000000 +0100 @@ -192,7 +192,7 @@ return -1; } - fd = open(*path_r, O_RDWR | O_EXCL | O_CREAT, 0666); + fd = open(*path_r, O_RDWR | O_CREAT, 0666); if (fd != -1) return fd; diff -NaurbB dovecot-1.0-test28.orig/src/lib-storage/index/maildir/maildir-util.c dovecot-1.0-test28/src/lib-storage/index/maildir/maildir-util.c--- dovecot-1.0-test28.orig/src/lib-storage/index/maildir/maildir-util.c 2004-06-16 01:50:41.000000000 +0100+++ dovecot-1.0-test28/src/lib-storage/index/maildir/maildir-util.c 2004-07-20 14:51:25.000000000 +0100@@ -241,7 +241,7 @@ if (stat(path, &st) < 0 && errno == ENOENT) { /* doesn't exist */ mode_t old_mask = umask(0); - fd = open(path, O_WRONLY | O_CREAT | O_EXCL, mode); + fd = open(path, O_WRONLY | O_CREAT, mode); umask(old_mask); if (fd != -1 || errno != EEXIST) break; diff -NaurbB dovecot-1.0-test28.orig/src/lib-storage/index/mbox/mbox-storage.c dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-storage.c--- dovecot-1.0-test28.orig/src/lib-storage/index/mbox/mbox-storage.c 2004-07-20 14:45:55.000000000 +0100+++ dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-storage.c 2004-07-20 14:49:56.000000000 +0100@@ -386,7 +386,7 @@ int fd; /* make sure inbox file itself exists */ - fd = open(storage->inbox_path, O_RDWR | O_CREAT | O_EXCL, 0660); + fd = open(storage->inbox_path, O_RDWR | O_CREAT, 0660); if (fd != -1) (void)close(fd); @@ -579,7 +579,7 @@ } /* create the mailbox file */ - fd = open(path, O_RDWR | O_CREAT | O_EXCL, 0660); + fd = open(path, O_RDWR | O_CREAT, 0660); if (fd != -1) { (void)close(fd); return 0; diff -NaurbB dovecot-1.0-test28.orig/src/lib-storage/index/mbox/mbox-storage.c.mbox dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-storage.c.mbox--- dovecot-1.0-test28.orig/src/lib-storage/index/mbox/mbox-storage.c.mbox 2004-07-11 22:03:09.000000000 +0100+++ dovecot-1.0-test28/src/lib-storage/index/mbox/mbox-storage.c.mbox 2004-07-20 14:50:58.000000000 +0100@@ -386,7 +386,7 @@ int fd; /* make sure inbox file itself exists */ - fd = open(storage->inbox_path, O_RDWR | O_CREAT | O_EXCL, 0660); + fd = open(storage->inbox_path, O_RDWR | O_CREAT, 0660); if (fd != -1) (void)close(fd); @@ -579,7 +579,7 @@ } /* create the mailbox file */ - fd = open(path, O_RDWR | O_CREAT | O_EXCL, 0660); + fd = open(path, O_RDWR | O_CREAT, 0660); if (fd != -1) { (void)close(fd); return 0; ************************************************** -- Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London
On Tue, 2004-07-20 at 19:24, Tim Southerwood wrote:
Slight problem - linux 2.4 running dovecot, Solaris 2.8 home directory server. Full lockd support etc.
I had to make the following code patches (at end of mail for clarity) to get dovecot to read the users mbox files at all.
- fcntl with F_SETLKW will not work against a Solaris 2.8 server as proved with a small test program. Other forms of fcntl (ie F_SETLK) are OK.
Hmm. I guess I could make it optionally emulate F_SETLKW by looping and checking the lock a few times a second. Or you could just disable fcntl locks:
fcntl_locks_disable = yes mbox_read_locks = dotlock mbox_write_locks = dotlock
- O_CREAT | O_EXCL are documented not to work over NFS properly (see man open(2) on linux). So I knocked out the O_EXCL and combined with (1) dovecot will talk over NFS to remote mbox files.
I don't think it breaks even if it's used, it just doesn't do anything? Dovecot doesn't rely on it, it's just for extra safety in case it works.
Hi
On Tue, 20 Jul 2004 20:29:01 +0300 Timo Sirainen tss@iki.fi wrote:
On Tue, 2004-07-20 at 19:24, Tim Southerwood wrote:
Slight problem - linux 2.4 running dovecot, Solaris 2.8 home directory server. Full lockd support etc.
I had to make the following code patches (at end of mail for clarity) to get dovecot to read the users mbox files at all.
- fcntl with F_SETLKW will not work against a Solaris 2.8 server as proved with a small test program. Other forms of fcntl (ie F_SETLK) are OK.
Hmm. I guess I could make it optionally emulate F_SETLKW by looping and checking the lock a few times a second. Or you could just disable fcntl locks:
fcntl_locks_disable = yes mbox_read_locks = dotlock mbox_write_locks = dotlock
- O_CREAT | O_EXCL are documented not to work over NFS properly (see man open(2) on linux). So I knocked out the O_EXCL and combined with (1) dovecot will talk over NFS to remote mbox files.
I don't think it breaks even if it's used, it just doesn't do anything? Dovecot doesn't rely on it, it's just for extra safety in case it works.
Not sure - it does seem to do what it's supposed according to man open (fail), according to strace. But before I make those changes, the code has problems with my setup. After it works. The test28 works perfectly in this repsect at home where I don't use NFS for mail.
I think I should isolate exactly which opens need tweaking and come back with something more specific.
Put this on hold as unconfirmed.
Tim
-- Tim Southerwood Website: http://www.dionic.net/ email: ts@DIESPAMDIE.dionic.net (remove DIESPAMDIE. to get address)
On 21.7.2004, at 01:09, Tim Southerwood wrote:
- O_CREAT | O_EXCL are documented not to work over NFS properly (see man open(2) on linux). So I knocked out the O_EXCL and combined with (1) dovecot will talk over NFS to remote mbox files.
I don't think it breaks even if it's used, it just doesn't do anything? Dovecot doesn't rely on it, it's just for extra safety in case it works.
Not sure - it does seem to do what it's supposed according to man open (fail), according to strace. But before I make those changes, the code has problems with my setup. After it works. The test28 works perfectly in this repsect at home where I don't use NFS for mail.
What problems? Does it print error messages? I browsed google for a while and it looks like that the O_EXCL problem is gone with NFSv3, except with Linux.
Would be nice if there was a summary of different NFS problems with different NFS clients and servers..
Tim Southerwood ts@doc.ic.ac.uk writes:
- fcntl with F_SETLKW will not work against a Solaris 2.8 server as proved with a small test program. Other forms of fcntl (ie F_SETLK) are OK.
Why not? Can you show the test program?
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
On Thu, 22 Jul 2004 14:08:41 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
- fcntl with F_SETLKW will not work against a Solaris 2.8 server as proved with a small test program. Other forms of fcntl (ie F_SETLK) are OK.
Why not? Can you show the test program?
Don't know why yet - I assume it's unimplemented in the Solaris NFS dialect?
Here's the test program, fcntltest.c
//*********************************************************
#include
main(int argc, char* argv[]) { if (argc != 2) { fprintf (stderr, "Usage: %s filetolock\n", argv[0]); return(1); }
int result;
int fd;
struct flock lock;
char *filetolock;
int locked=0;
filetolock = argv[1];
fd = open(filetolock, O_RDWR);
if (fd == -1)
{
fprintf(stderr, filetolock);
perror (" - error opening file");
return(2);
}
lock.l_type = F_RDLCK; /* F_RDLCK, F_WRLCK, F_UNLCK */
lock.l_start = 0; /* byte offset relative to l_whence */
lock.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
lock.l_len = 0; /* #bytes (0 means to EOF) */
result = fcntl(fd, F_SETLKW, &lock);
if (result == -1)
{
perror("Error locking file with F_SETLKW");
}
else
{
printf("Locked with F_SETLKW\n");
locked=1;
}
if (!locked)
{
result = fcntl(fd, F_SETLK, &lock);
if (result == -1)
{
perror("Error locking file with F_SETLK");
}
else
{
printf ("Locked with F_SETLK\n");
locked=1;
}
}
if (locked)
{
printf("Sleeping - try to lock same file from secondprocess\n"); sleep(30); } close(fd); printf("File closed\n"); }
//*********************************************************
gcc -o fcntltest fcntltest.c
# The linux NFS server from linux client # ts@lenin > ./fcntltest /vol/csg/public/users/ts/wibble.blah Locked with F_SETLKW Sleeping - try to lock same file from second process File closed
# The local filesystem # ts@lenin > ./fcntltest /tmp/p.ps Locked with F_SETLKW Sleeping - try to lock same file from second process File closed
# The Solaris 2.8 server # ts@lenin > ./fcntltest /homes/ts/liquid-1.0.0.tar.gz Error locking file with F_SETLKW: No locks available Locked with F_SETLK Sleeping - try to lock same file from second process File closed
That seems to verify it in my setup.
Cheers
Tim
-- Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London
-- Mr Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London Email personal: ts@dionic.net work: ts@doc.ic.ac.uk Tel home: +44 (0)1892 824850 mobile: +44 (0)7739-902439 work: +44 (0)20-759-48392
Tim Southerwood ts@doc.ic.ac.uk writes:
Don't know why yet - I assume it's unimplemented in the Solaris NFS dialect?
I don't think so. I've run your program in these combinations, with two different hosts for each run:
Solaris 2.8 NFS client <-> Solaris 2.8 NFS server Linux 2.4 NFS client <-> Solaris 2.8 NFS server Solaris 2.8 NFS client <-> Linux 2.4 NFS server.
In all cases, the screen output was:
Locked with F_SETLKW Sleeping - try to lock same file from second process [half a minute paused] File closed
Is there anything special WRT share options in your /etc/dfs/dfstab?
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
On Fri, 23 Jul 2004 01:40:20 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
Don't know why yet - I assume it's unimplemented in the Solaris NFS dialect?
I don't think so. I've run your program in these combinations, with two different hosts for each run:
Solaris 2.8 NFS client <-> Solaris 2.8 NFS server Linux 2.4 NFS client <-> Solaris 2.8 NFS server Solaris 2.8 NFS client <-> Linux 2.4 NFS server.
In all cases, the screen output was:
Locked with F_SETLKW Sleeping - try to lock same file from second process [half a minute paused] File closed
Is there anything special WRT share options in your /etc/dfs/dfstab?
Now that's very interesting. I shall examine this in more detail. All our Solaris servers have the same setup and I didn't do it - so my initial thought was "Solaris must be weird"!
OK - thanks for the comparative test - I shall look into this.
Cheers
Tim
-- Tim J Southerwood Senior Programmer
On Fri, 23 Jul 2004 01:40:20 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
Don't know why yet - I assume it's unimplemented in the Solaris NFS dialect?
I don't think so. I've run your program in these combinations, with two different hosts for each run:
Solaris 2.8 NFS client <-> Solaris 2.8 NFS server Linux 2.4 NFS client <-> Solaris 2.8 NFS server Solaris 2.8 NFS client <-> Linux 2.4 NFS server.
In all cases, the screen output was:
Locked with F_SETLKW Sleeping - try to lock same file from second process [half a minute paused] File closed
Is there anything special WRT share options in your /etc/dfs/dfstab?
Hello Matthias,
Now that's very interesting. I shall examine this in more detail. All our Solaris servers have the same setup and I didn't do it - so my initial thought was "Solaris must be weird"!
OK - thanks for the comparative test - I shall look into this.
One thing: I can't initially see any odd options set in our Solaris dfstab - the only two options set are:
"nosuid" and "root" for a few backup servers
Do you have anything different in your 2.8 solaris server?
Cheers
Tim
-- Tim J Southerwood Senior Programmer
-- Mr Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London Email personal: ts@dionic.net work: ts@doc.ic.ac.uk Tel home: +44 (0)1892 824850 mobile: +44 (0)7739-902439 work: +44 (0)20-759-48392
Tim Southerwood ts@doc.ic.ac.uk writes:
Now that's very interesting. I shall examine this in more detail. All our Solaris servers have the same setup and I didn't do it - so my initial thought was "Solaris must be weird"!
Do you have anything different in your 2.8 solaris server?
Good question. Not that I know of - for the test was conducted in a LAN that isn't mine, I've masked the IP addresses and hostnames:
Sun Microsystems Inc. SunOS 5.8 Generic February 2000 bash-2.05$ cat /etc/dfs/dfstab [...comments snipped...] share -F nfs -o rw=@XXX.XXX.XXX.XXX/26 /space share -F nfs -o ro=@XXX.XXX.XXX.XXX/26 /opt
bash-2.05$ uname -a SunOS somehost 5.8 Generic_117350-04 sun4u sparc SUNW,Ultra-5_10
Are your nlockmgrs running? All relevant patches installed? The machines I tested on have been fully patched up two days before the test -- "pgrep lockd" and "rpcinfo -p" on client and server should tell you.
At any rate, I don't think Solaris fcntl-style file locking is broken.
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
Hi Matthias
On Fri, 23 Jul 2004 15:46:59 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
Now that's very interesting. I shall examine this in more detail. All our Solaris servers have the same setup and I didn't do it - so my initial thought was "Solaris must be weird"!
Do you have anything different in your 2.8 solaris server?
Good question. Not that I know of - for the test was conducted in a LAN that isn't mine, I've masked the IP addresses and hostnames:
Sun Microsystems Inc. SunOS 5.8 Generic February 2000 bash-2.05$ cat /etc/dfs/dfstab [...comments snipped...] share -F nfs -o rw=@XXX.XXX.XXX.XXX/26 /space share -F nfs -o ro=@XXX.XXX.XXX.XXX/26 /opt
bash-2.05$ uname -a SunOS somehost 5.8 Generic_117350-04 sun4u sparc SUNW,Ultra-5_10
Well ours are behind on some patches:
SunOS buzzard.doc.ic.ac.uk 5.8 Generic_108528-27 sun4u sparc SUNW,Sun-Fire-280R
Are your nlockmgrs running? All relevant patches installed? The machines I tested on have been fully patched up two days before the test-- "pgrep lockd" and "rpcinfo -p" on client and server should tell you.
Yes, they are all happy.
At any rate, I don't think Solaris fcntl-style file locking is broken.
I beg to differ - at least it looks like there *was* a bug which may have been cleared by a patch that you have that we don't. I'll pass this to our Solaris admin. Also, I have turned up quite a lot of problems with F_SETLKW on the samba lists - seems to be a common problem with Solaris servers. I'll have a look on the Sun lists and see if there is a patch specifically for this.
Anyrate, one of two conclusions:
It's not dovecot's fault, which is fair, and this bug should be dropped as not a bug.
It's not really a bug with dovecot, but if F_SETLKW fails, dovecot should fall back to something involving F_SETLK.
I prefer number 2) as a user as it's clear to me that there is some brokenness on NFS servers in the world and it would be to the greater good if dovecot were more forgiving.
Cheers
Tim
-- Tim J Southerwood Senior Programmer
Hi Matthias
On Fri, 23 Jul 2004 15:46:59 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
Now that's very interesting. I shall examine this in more detail. All our Solaris servers have the same setup and I didn't do it - so my initial thought was "Solaris must be weird"!
Do you have anything different in your 2.8 solaris server?
Good question. Not that I know of - for the test was conducted in a LAN that isn't mine, I've masked the IP addresses and hostnames:
Sun Microsystems Inc. SunOS 5.8 Generic February 2000 bash-2.05$ cat /etc/dfs/dfstab [...comments snipped...] share -F nfs -o rw=@XXX.XXX.XXX.XXX/26 /space share -F nfs -o ro=@XXX.XXX.XXX.XXX/26 /opt
bash-2.05$ uname -a SunOS somehost 5.8 Generic_117350-04 sun4u sparc SUNW,Ultra-5_10
Well ours are behind on some patches:
SunOS buzzard.doc.ic.ac.uk 5.8 Generic_108528-27 sun4u sparc SUNW,Sun-Fire-280R
Are your nlockmgrs running? All relevant patches installed? The machines I tested on have been fully patched up two days before the test-- "pgrep lockd" and "rpcinfo -p" on client and server should tell you.
Yes, they are all happy.
At any rate, I don't think Solaris fcntl-style file locking is broken.
I beg to differ - at least it looks like there *was* a bug which may have been cleared by a patch that you have that we don't. I'll pass this to our Solaris admin. Also, I have turned up quite a lot of problems with F_SETLKW on the samba lists - seems to be a common problem with Solaris servers. I'll have a look on the Sun lists and see if there is a patch specifically for this.
Anyrate, one of two conclusions:
It's not dovecot's fault, which is fair, and this bug should be dropped as not a bug.
It's not really a bug with dovecot, but if F_SETLKW fails, dovecot should fall back to something involving F_SETLK.
I prefer number 2) as a user as it's clear to me that there is some brokenness on NFS servers in the world and it would be to the greater good if dovecot were more forgiving.
Cheers
Tim
-- Tim J Southerwood Senior Programmer
-- Mr Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London Email personal: ts@dionic.net work: ts@doc.ic.ac.uk Tel home: +44 (0)1892 824850 mobile: +44 (0)7739-902439 work: +44 (0)20-759-48392
Tim Southerwood ts@doc.ic.ac.uk writes:
I beg to differ - at least it looks like there *was* a bug which may have been cleared by a patch that you have that we don't. I'll pass this to our Solaris admin. Also, I have turned up quite a lot of problems with F_SETLKW on the samba lists - seems to be a common problem with Solaris servers. I'll have a look on the Sun lists and see if there is a patch specifically for this.
OK, that site isn't running services such as Samba on Solaris - the boxen are somewhat oldish low-end Ultra 5 machines with single UltraSPARC IIi 333 and 400. Nothing you want to run servers on if you have a Xeon 2.8 or a Dual Athlon MP in the next rack. :)
It's not dovecot's fault, which is fair, and this bug should be dropped as not a bug.
It's not really a bug with dovecot, but if F_SETLKW fails, dovecot should fall back to something involving F_SETLK.
I prefer number 2) as a user as it's clear to me that there is some brokenness on NFS servers in the world and it would be to the greater good if dovecot were more forgiving.
Number 2 is a bit disputable, because the OS is supposed to behave, and it is more sensible to fix the cause rather than implement workarounds in 1000+1 applications.
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
On Fri, 23 Jul 2004 17:39:48 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
It's not dovecot's fault, which is fair, and this bug should be dropped as not a bug.
It's not really a bug with dovecot, but if F_SETLKW fails, dovecot should fall back to something involving F_SETLK.
I prefer number 2) as a user as it's clear to me that there is some brokenness on NFS servers in the world and it would be to the greater good if dovecot were more forgiving.
Number 2 is a bit disputable, because the OS is supposed to behave, and it is more sensible to fix the cause rather than implement workarounds in 1000+1 applications.
Well, I would agree with you in an ideal world. However, there may be sites where the linux admin has no control over the Solaris (or other broken) servers holding users' mail - so it would be more beneficial that the application have a failover mode (or at least some compile time configuration). Dovecot uses this locking in two places that matter so it's not IMHO a terrible disaster to add a small workaround.
Having said that, I must be respectful; it's not my code, and I'm grateful that it exists at all. It's a big improvement over WU-IMAP. Of course the main developers must decide what to do. But I've expressed my opinion which I feel has some merit.
Best wishes
Tim
Tim J Southerwood Senior Programmer CSG, Dept of Computing, Imperial College, London
On 23.7.2004, at 21:12, Tim Southerwood wrote:
Well, I would agree with you in an ideal world. However, there may be sites where the linux admin has no control over the Solaris (or other broken) servers holding users' mail - so it would be more beneficial that the application have a failover mode (or at least some compile time configuration). Dovecot uses this locking in two places that matter so it's not IMHO a terrible disaster to add a small workaround.
How well does NFS fcntl locking work in general? I've only heard of bad things with it, so adding a workaround for a combination which works badly in any case isn't really useful.
If I added a workaround, should it be a compile time workaround enabled with some #define, a config-file configurable workaround or automatic (errno == ENOLCK -> do it)?
Timo Sirainen tss@iki.fi writes:
How well does NFS fcntl locking work in general? I've only heard of bad things with it, so adding a workaround for a combination which works badly in any case isn't really useful.
fcntl-style file locking has been absolutely reliable for me for almost five years, with a mixed Linux/Solaris LAN.
I find it a bit scary that a Solaris NFS locking problem is reported, Sun are the guys that paved the way for NFS locking, after all, so I'd tend to believe it's a site issue not a general Solaris issue.
Is anyone else seeing fcntl locking problems on Solaris 8 for F_SETLKW ("block until lock obtained")?
If I added a workaround, should it be a compile time workaround enabled with some #define, a config-file configurable workaround or automatic (errno == ENOLCK -> do it)?
If I were Dovecot's maintainer, I'd chose fully automatic fall-back, to avoid bloating the configuration file and documentation with something that the user who wants the beast to "just work" doesn't care about, and I'd also avoid #ifdef - conditional compilation is ugly, more difficult to read and most difficult to test. The "just do it" is the least effort :)
But I'd wait until I knew the reason cannot be fixed within Solaris. After all, the kernel patch level they have is oldish, and so might be the lockd or other NFS related patches, and we don't yet know if there is a special configuration or tuning or kernel limit but might know next week.
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
On Friday, July 23, Matthias Andree wrote:
[snip]
I find it a bit scary that a Solaris NFS locking problem is reported, Sun are the guys that paved the way for NFS locking, after all, so I'd tend to believe it's a site issue not a general Solaris issue.
Is anyone else seeing fcntl locking problems on Solaris 8 for F_SETLKW ("block until lock obtained")?
I cannot reproduce this problem here. Using the supplied test program, the output is as expected:
Locked with F_SETLKW
Sleeping - try to lock same file from second process
File closedThis is with Solaris 8 on both ends.
Tim Southerwood ts@doc.ic.ac.uk writes:
Number 2 is a bit disputable, because the OS is supposed to behave, and it is more sensible to fix the cause rather than implement workarounds in 1000+1 applications.
Well, I would agree with you in an ideal world. However, there may be sites where the linux admin has no control over the Solaris (or other broken) servers holding users' mail - so it would be more beneficial that the application have a failover mode (or at least some compile time configuration).
This would constitute a violation of responsibilities or layers. What you are suggesting appears as though you would want the software to work around a problem when the administrator teams for Solaris and Linux aren't talking to each other.
Dovecot uses this locking in two places that matter so it's not IMHO a terrible disaster to add a small workaround.
I've often been tempted to add some special case to software and ultimately given in, only to find out weeks, months or even years later that the special case handler wasn't working properly -- such seldomly used code is a maintenance nightmare. And these special cases need rather lengthy comments because a few months later the maintainer will see the code and throw it out because it looks extraneous.
Maybe such a workaround should be kept as a separate patch and not become part of the baseline code.
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
On Fri, 23 Jul 2004 21:41:36 +0200 Matthias Andree matthias.andree@gmx.de wrote:
Tim Southerwood ts@doc.ic.ac.uk writes:
This would constitute a violation of responsibilities or layers. What you are suggesting appears as though you would want the software to work around a problem when the administrator teams for Solaris and Linux aren't talking to each other.
Well, yes, upto a point. We're OK *potentially*, because I'm one of two linux admins and the solaris guy sits behind me (but on leave right now).
But I've worked in places before where a small department might run a gateway server (eg IMAP) but have to deal with a centrally provided filestore on some uber-SAN provided by a totally different department (I'm citing universities here) - and worse, getting the central IT people to touch "their" fileserver can be practically impossible in any sensible timeframe.
It's a quite grim, but nonetheless, true reality that some people who may want to run dovecot are stuck in such a situation. Service level aggreements signed by upper tier managers usually don't include "fcntl/F_SETLKW must work". I speak from experience.
Dovecot uses this locking in two places that matter so it's not IMHO a terrible disaster to add a small workaround.
I've often been tempted to add some special case to software and ultimately given in, only to find out weeks, months or even years later that the special case handler wasn't working properly -- such seldomly used code is a maintenance nightmare. And these special cases need rather lengthy comments because a few months later the maintainer will see the code and throw it out because it looks extraneous.
Yes, I understand - I've seen the code to GNU/tar! (except for the "throwing out" bit - that program is *the* museum of cruft) I totally agree on the desire to keep stuff clean. Finding the balance is usually a matter of debate though.
Maybe such a workaround should be kept as a separate patch and not become part of the baseline code.
That would be a wise and perfectly helpful way to proceed if that is what you would prefer. Put a patch on the ftp site and note it in a FAQ along the lines: "so you've got a broken NFS server" or something.
Incidently, talking to another colleague, it seems that we also had this problem with sunsite.org.uk (which we operate) - that was exhibiting the same problem with F_SETLKW with one ftpd program running over the NFS share between it's four hosts (between solaris client and server NFS). That problem was "fixed" in a hurry by using a different ftpd.
Unfortunately, we don't usually have enough time to get to the bottom of every odd fault we get, but this time I'm being more tenacious because this is irritating me (Solaris, not dovecot).
I'm on leave for 1.5 weeks - but I'll mail through to our solaris chap and let him have a look at it.
Anyway - I still stand by my point of view that not all NFS implementations are perfect, and doing something which helps people around broken systems (which they may not control) is helpful - but in the way that buggers up the dovecot codebase the least.
I suppose that if 99.5% of dovecot's userbase don't have this issue, perhaps you should leave the code alone. I will eventually do my own patch, but I'll try and do it a right as possible and I'll mail it in here if anyone else needs it.
Best wishes,
Tim
-- Tim Southerwood
Tim Southerwood ts@doc.ic.ac.uk writes:
It's a quite grim, but nonetheless, true reality that some people who may want to run dovecot are stuck in such a situation. Service level aggreements signed by upper tier managers usually don't include "fcntl/F_SETLKW must work". I speak from experience.
For such a situation, another approach would be to split the stuff that needs file locks from the stuff that does not - for instance, leave the largish Maildirs on the NAS or in the SAN, and keep the indexes local and stuff plenty of RAM: it's faster than the network anyways. Dovecot 0.99.10.X allows me to split indexes from Maildirs, I haven't looked if this is a complete solution though.
Yes, I understand - I've seen the code to GNU/tar! (except for the "throwing out" bit - that program is *the* museum of cruft) I totally agree on the desire to keep stuff clean. Finding the balance is usually a matter of debate though.
And other software is spoiled by GNU tar. I've recently seen BSD tar (in FreeBSD -CURRENT) complain about the -o option that automake used to tell it "dereference symlinks"...
Unfortunately, we don't usually have enough time to get to the bottom of every odd fault we get, but this time I'm being more tenacious because this is irritating me (Solaris, not dovecot).
There are several bugs and if the solution is a documentation "install Sun patch 123456 level -07 or higher", it will also work.
The question is whether a pragmatic solution (switch ftpd, move indexes off the SAN as suggested above) or a thorough one is warranted. If the same problem haunts multiple applications, a closer inspection is usually justified.
-- Matthias Andree
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)
participants (4)
-
Matthias Andree
-
Tim Southerwood
-
Timo Sirainen
-
William Yodlowsky