[Dovecot] 0.99.11-test6
http://dovecot.procontrol.fi/test/
I think I've fixed the problems that were corrupting indexes with the new code. It happened every time you had expunged some messages and new messages arrived. I think this release might actually work :)
disable_plaintext_auth = yes will be default from now on. It allows plaintext authentication from localhost (127.*, ::1) however.
I just figured out that the lockless cache file reading prevents me from fixing the mbox code to work the same way it used to. I'm not sure actually how this would be best fixed.. Problem is that headers in mails change and move the messages around in the mailbox, so I have to keep track of them. Previously I just updated the header size in index files.. but without read locks this can't be done safely.
One possibility would be to parse the whole mbox file every time when the mailbox is opened (or when it needs to be read) and save the header sizes in memory. That'd make it slower than it is today though..
Or maybe it'd work if I just grabbed exclusive lock whenever reading header sizes. They could be then copied into memory to avoid locking more than once. But that still creates locking contention if lots of clients are reading the mailbox. Exactly in the situation where I'd mostly use mboxes..
I could also save it in .imap.index file which still currently uses read-locks. But I'd rather want to get rid of them in there too.
On Sunday, Aug 24, 2003, at 11:43 Europe/Helsinki, Timo Sirainen wrote:
I just figured out that the lockless cache file reading prevents me from fixing the mbox code to work the same way it used to. I'm not sure actually how this would be best fixed.. Problem is that headers in mails change and move the messages around in the mailbox, so I have to keep track of them. Previously I just updated the header size in index files.. but without read locks this can't be done safely.
Stupid stupid stupid.
Why did it take this long to figure out.
Write: lock(); data[0] = stuff; data[1] = stuff; unlock(); Read: do { copy = data; } while (copy[0] != copy[1]);
There's my lockless reading for data that changes. Solves mbox problems, allows easily changing .imap.index to be read-lockless and probably more. Doubles the space requirements, but that's still small.
On Mon, Aug 25, 2003 at 07:15:07AM +0300, Timo Sirainen wrote:
Why did it take this long to figure out.
Write: lock(); data[0] = stuff; data[1] = stuff; unlock(); Read: do { copy = data; } while (copy[0] != copy[1]);
It's above me Timo, but well done anyway :)
C.
charlie@rubberduck.com - Melbourne, Australia http://rubberduck.com/~yeled/ PGP: 0x14AA7941 || finger yeled@lazy.spodder.com
hi,
Stupid stupid stupid.
hehe, be patient with yourself ;)
Why did it take this long to figure out.
good solutions need time. that's it.
Write: lock(); data[0] = stuff; data[1] = stuff; unlock(); Read: do { copy = data; } while (copy[0] != copy[1]);
go ahead ;-)
thanks for your effort!
-- regards, TOM
Key fingerprint = 1862 006F B4A8 7C65 5704 63A3 663F 5F19 6E72 53CF
Timo Sirainen tss@iki.fi writes:
Or maybe it'd work if I just grabbed exclusive lock whenever reading header sizes. They could be then copied into memory to avoid locking more than once. But that still creates locking contention if lots of clients are reading the mailbox. Exactly in the situation where I'd mostly use mboxes..
Would not one rather use Maildir and assume the Kernel caches directory data? No locking contention...
-- Matthias Andree
On Monday, Aug 25, 2003, at 12:09 Europe/Helsinki, Matthias Andree wrote:
Or maybe it'd work if I just grabbed exclusive lock whenever reading header sizes. They could be then copied into memory to avoid locking more than once. But that still creates locking contention if lots of clients are reading the mailbox. Exactly in the situation where I'd mostly use mboxes..
Would not one rather use Maildir and assume the Kernel caches directory data? No locking contention...
For read-only mailboxes it's much faster to read a single mbox file than to keep opening multiple maildir files. And when you're only reading mboxes, there's no lock contention.
On Mon, 25 Aug 2003, Timo Sirainen wrote:
Would not one rather use Maildir and assume the Kernel caches directory data? No locking contention... For read-only mailboxes it's much faster to read a single mbox file
On Monday, Aug 25, 2003, at 12:09 Europe/Helsinki, Matthias Andree wrote: than to keep opening multiple maildir files. And when you're only reading mboxes, there's no lock contention.
If a truncation occurs while concurrent processes are reading, how is this handled?
Andy
-- Andreas Aardal Hanssen
On Monday, Aug 25, 2003, at 15:01 Europe/Helsinki, Andreas Aardal Hanssen wrote:
For read-only mailboxes it's much faster to read a single mbox file than to keep opening multiple maildir files. And when you're only reading mboxes, there's no lock contention.
If a truncation occurs while concurrent processes are reading, how is this handled?
There's no truncation with read-only mailboxes of course :)
Anyway, the mbox files themselves will still be read-locked so no-one can modify them (if they still do, it breaks of course). It's only the indexes that won't be read-locked. Indexes are never truncated, but they may be replaced with rename().
I'm actually still wondering a bit if this works. Have to ask from some people who really know. The problem is that if I write 123 over XXX, can a simultaneous read() return 1X3 in some situation? (1XX, 12X, XX3 and X23 are fine) That might break my plan..
On Mon, 2003-08-25 at 15:27, Timo Sirainen wrote:
I'm actually still wondering a bit if this works. Have to ask from some people who really know. The problem is that if I write 123 over XXX, can a simultaneous read() return 1X3 in some situation? (1XX, 12X, XX3 and X23 are fine) That might break my plan..
That would probably work, but just to make sure here's an addition to the plan. Can anyone think of a situation where this would break?
While MD5 is probably good enough, it doesn't _guarantee_ the consistency. I just thought of a simple algorithm that I think would. I'll go and use that unless someone proves it wrong :)
Except of course if there's 256 writes and the last one is non-ordered and it all happens while a read() is executing.. Less than unlikely I'd say.
I don't think there's any other potential problems with read() than that it may not have all data up to date? Such as it would never temporarily show the data as zero?
static int verify(const unsigned char *buf, size_t size) { const unsigned char *checksum = buf + size; unsigned char xor; size_t i;
xor = buf[0] ^ checksum[0];
for (i = 1; i < size; i++) {
if ((buf[i] ^ xor) != checksum[i])
return 0;
}
return 1;
}
void write_data(const void *data, size_t size) { unsigned char *checksum = buf + size; unsigned char xor; size_t i;
memcpy(buf, data, size);
checksum[0]++; // always different checksum
xor = buf[0] ^ checksum[0];
for (i = 1; i < size; i++)
checksum[i] = buf[i] ^ xor;
}
void read_data(void *data, size_t size) { unsigned char copy[size*2];
do {
memcpy(copy, buf, size*2);
} while (!verify(copy, size));
memcpy(data, copy, size);
}
On Thu, 2003-08-28 at 04:23, Timo Sirainen wrote:
That would probably work, but just to make sure here's an addition to the plan. Can anyone think of a situation where this would break?
Sure someone could :)
Yet another idea which looks simple enough to hopefully work perfectly:
checksum[n] = data[n-1] xor data[n]
And if it does work, I'm sure it's some widely known algorithm that I've just never heard of.. :)
It also just occured to me that this provides easy detection of corrupted data if system crashes or whatever.
On Sunday 24 August 2003 10:43, Timo Sirainen wrote:
messages arrived. I think this release might actually work :)
But not on my machine ;(
I got those compilation errors:
Making all in lib-auth
make[3]: Entering directory /home/jens/dovecot-0.99.11-test6/src/lib-auth' /opt/diet/bin/diet gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I../../src/lib -L/usr/local/lib -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -c auth-client.c auth-client.c:7: auth-server-connection.h: No such file or directory make[3]: *** [auth-client.o] Error 1 make[3]: Leaving directory
/home/jens/dovecot-0.99.11-test6/src/lib-auth'
Using auth-server-connection.h from CVS worked ok, until I got: auth-server-connection.c:11: auth-server-request.h: No such file or directory
looks like you've forgotten some files.
Anyway, I will test latest CVS on my machines :)
Jens
On Tue, 2003-08-26 at 04:40, Jens Gutzeit wrote:
messages arrived. I think this release might actually work :)
But not on my machine ;(
auth-client.c:7: auth-server-connection.h: No such file or directory
Right.. I always forget to add those .h files into Makefiles. automake should complain about them..
0.99.11-test7 fixes that.
Timo Sirainen tss@iki.fi writes:
On Tue, 2003-08-26 at 04:40, Jens Gutzeit wrote:
messages arrived. I think this release might actually work :)
But not on my machine ;(
auth-client.c:7: auth-server-connection.h: No such file or directory
Right.. I always forget to add those .h files into Makefiles. automake should complain about them..
Run "make distdir" and configure and compile from the newly-made distdir, then you'll see what's missing from the distro.
CVS is a different matter, you'd need to cvs export to a fresh directory first to find files that have to be added.
-- Matthias Andree
On Tue, 26 Aug 2003, Matthias Andree wrote:
Timo Sirainen tss@iki.fi writes:
On Tue, 2003-08-26 at 04:40, Jens Gutzeit wrote:
messages arrived. I think this release might actually work :) But not on my machine ;( auth-client.c:7: auth-server-connection.h: No such file or directory Right.. I always forget to add those .h files into Makefiles. automake should complain about them.. Run "make distdir" and configure and compile from the newly-made distdir, then you'll see what's missing from the distro.
"make distcheck" does it all for you, if I'm not mistaken.
Andy :-)
-- Andreas Aardal Hanssen
Andreas Aardal Hanssen dovecot@andreas.hanssen.name writes:
On Tue, 26 Aug 2003, Matthias Andree wrote:
Timo Sirainen tss@iki.fi writes:
On Tue, 2003-08-26 at 04:40, Jens Gutzeit wrote:
messages arrived. I think this release might actually work :) But not on my machine ;( auth-client.c:7: auth-server-connection.h: No such file or directory Right.. I always forget to add those .h files into Makefiles. automake should complain about them.. Run "make distdir" and configure and compile from the newly-made distdir, then you'll see what's missing from the distro.
"make distcheck" does it all for you, if I'm not mistaken.
make distcheck does more than just that, it'll also try make dist and do a VPATH build.
-- Matthias Andree
participants (6)
-
Andreas Aardal Hanssen
-
Charlie Allom
-
Jens Gutzeit
-
Matthias Andree
-
Timo Sirainen
-
tom hensel