[Dovecot] 1.0-test25 with new cache file code
Pretty much the only change since test24 is a redesign of how cache file writing works. Reading works as before - without locks.
Before (in 1.0-tests, 0.99 was worse) we used to just try to lock it. If it failed, we just didn't update cache. If it didn't fail, the lock could have been kept a long time. For example if a user was downloading a 10MB mail, the lock could have been kept the whole time and no-one else could have modified the cache.
The old behaviour wasn't actually too bad, as normally it didn't matter if something couldn't be written to cache. It would have been the next time, all it did was cost some I/O.
The new behaviour is more reliable as the locks are kept only a very short time. It works by first writing the updates only into internal 32kB buffer. Once the buffer gets full, or the transaction completes, it's written to disk. When buffer gets full, space is reserved from the cache file for the current transaction. The amount of space is more than is needed for writing the internal buffer, so we don't have to lock the cache file every time the buffer gets full (and so we don't need a huge buffer).
Once transaction completes, we lock the cache again and try to free the extra reserved space we have left. If multiple processes weren't updating the cache file at the same time, the freeing should be as simple as decreasing "used_file_size" field in header. Alternatively we'll update a linked list of "holes" in the file. When reserving space, we first try to use those holes instead of appending to end of file.
Pointers to new cache records are also written to transaction log at the end of cache transaction. They'll get written to index at next sync after which they become visible to other processes. Each cache record contains a pointer to previous record, so there can be multiple cache records for a single message.
The bad thing with the new logic is that nothing prevents two processes from writing the same cache records and so using twice as much space as necessary. But I don't think it's much of a problem. They'll go away at next cache compression.
These changes also make it quite simple to make cache file work with shared filesystems. I'd just need to implement some kind of internal caching buffer which remembers what parts of file are read into memory.
These changes also make it possible to easily and reliably store X-UIDL headers into cache file, where the POP3 UIDL command could quickly get them. Not in this release yet, though..
On Mon, Jul 05, 2004 at 01:47:28AM +0300, Timo Sirainen wrote:
Pretty much the only change since test24 is a redesign of how cache file writing works. Reading works as before - without locks.
Doesn't work for me here.
Starting test25 with a mailbox having indexes gererated with test24:
Dovecot v1.0-test25 starting up imap-login: Login: juergen [192.168.0.1] child 26007 (imap) killed with signal 11
Ok, that's not the point, because after deleting all indexes I can open small mboxes with around 50 email without problems.
But, if I try to open bigger ones (>200 emails), the imap process hangs with 100% cpu usage. This seems to happen after 81920 bytes written to dovecot.index.cache, this is the value I got for different mboxes ?
Back to test24 for now, which works really perfect for me ;)
Greetings Jürgen
-- juergen.daubert@t-online.de
On 187, 07 05, 2004 at 10:14:22PM +0200, Juergen Daubert wrote:
On Mon, Jul 05, 2004 at 01:47:28AM +0300, Timo Sirainen wrote:
Pretty much the only change since test24 is a redesign of how cache file writing works. Reading works as before - without locks.
Doesn't work for me here.
Starting test25 with a mailbox having indexes gererated with test24:
Dovecot v1.0-test25 starting up imap-login: Login: juergen [192.168.0.1] child 26007 (imap) killed with signal 11
Ok, that's not the point, because after deleting all indexes I can open small mboxes with around 50 email without problems.
But, if I try to open bigger ones (>200 emails), the imap process hangs with 100% cpu usage. This seems to happen after 81920 bytes written to dovecot.index.cache, this is the value I got for different mboxes ?
SImilar problem here, trying to open large maildir lead to pop3 process eating 100% cpu.
strace doesn't show any activity except this:
alarm(30) = 0 sigreturn() = ? (mask now [RTMIN]) --- SIGALRM (Alarm clock) @ 0 (0) ---
Attaching gdb to the offending process gives me this info:
smtp:~/COPM# gdb -p 22347 GNU gdb 6.0-debian Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux". Attaching to process 22347 Reading symbols from /usr/lib/dovecot/pop3...(no debugging symbols found)...done. Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 0x0806c04b in mail_cache_get_transaction () (gdb) where #0 0x0806c04b in mail_cache_get_transaction () #1 0x0806c93f in mail_cache_add () #2 0x08065e4d in index_mail_cache_add () #3 0x0806635b in index_mail_get_size () #4 0x080664cb in index_mail_init_stream () #5 0x08059e94 in maildir_mailbox_list_next () #6 0x08066182 in index_mail_get_size () #7 0x08059e30 in maildir_mailbox_list_next () #8 0x08052464 in _start () #9 0x080c293c in ?? ()
I didn't dig deeper yet.
-- Andrey Panin | Linux and UNIX system administrator pazke@donpac.ru | PGP key: wwwkeys.pgp.net
participants (3)
-
Andrey Panin
-
Juergen.Daubert＠t-online.de
-
Timo Sirainen