[Dovecot] Unexpected behaviour when deleteing a big mailbox

Wed Mar 21 16:28:39 EET 2007

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

my last tests left me with a mailbox with > 80'000 messages in 
Maildir. Dovecot served them fine. No problem. I usually use IMAP's "Mark 
messages a deleted" function and expunge later.

Because the "Move to Trash" is default in many MUAs, I decided to try it 
on this mailbox.

I do think that it is hard to detect and prevent such problem, but could 
it be implemented that when the client connection times out, the COPY 
process is terminated and rolled back? I mean, as long as the messages are 
copied/linked to */tmp nothing is done, yet, hence, a simple cleanup of 
*/tmp would result in a rollback. Due to the timeout, the EXPUNGE wasn't 
issued so far and the source mailbox is unchanged.
Or, is it possible to prevent two COPYs of an overlapping message range 
into the same destination folder simultaneously?
Or, can I prevent to COPY such a huge range of messages at once?

I know 80'000 messages are much, but I have users with an INBOX of at 
least 40'000, eventually moving all files into an archive mailbox.

I'm also surprised to find that I didn't get two full copies of the source 
mailbox.

==============

Start situation:
4.8G    .box1
(I didn't recorded the individual subdirs, unfortunately.)

# find .box1 -type f | wc -l
84273

Newly created .Trash.

===

I started Thunderbird, let sync the mailbox, selected them all, and 
pressed "Delete".
Some time later: Connection timed out.
I, wondered, and hit "Delete" a second time.
Again, connection timed out.

===

I was surprised, however, to see that the Dovecot log with still active 
telling me that mails are copied to Trash.

I ended up with two processes copying all messages to .Trash/tmp on the 
server. The process I traced a bit later linked any file after the other. 
What I wondered right from the beginning was, that

du -hs .??*

displayed about 3.5GB in .Trash, but still 4.8GB in .box1.
So, why are there 3.5GB additional files in .Trash, when I configured 
Dovecot to use hardlinks?

The end situation is that .box1 contains all messages still, but any 
Unseen messages are moved to .box1/cur, and .Trash contains:

7.4G    .Trash/cur
791M    .Trash/new
76K     .Trash/tmp

# find .Trash/ -type f | wc -l
121713

The size nearly doubled.
# du -hs .box1 ; du -sh .Trash/
4.8G    .box1
8.2G    .Trash/

====

Last lines in Dovecot log:
dovecot: Mar 21 11:46:15 Info: IMAP(dvtest) [11535]: copy -> Trash: 
uid=83964, 
msgid=<xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx at xx-xxxx-xxxxx-xxxx.xx>, 
box=box1
dovecot: Mar 21 11:46:15 Info: IMAP(dvtest) [11535]: copy -> Trash: 
uid=83965, msgid=<xxxxxxxx.xxxxxxx at xx-xxxx-xxxxx-xxxx.xx>, box=box1
dovecot: Mar 21 11:46:29 Info: IMAP(dvtest) [11535]: Disconnected: Mailbox 
is in inconsistent state, please relogin.

Last lines of strace:
fstat64(13, {st_mode=S_IFREG|0600, st_size=1341104, ...}) = 0
stat64("/var/cache/dovecot/31045/index/.box1/dovecot.index", 
{st_mode=S_IFREG|0600, st_size=1341104, ...}) = 0
alarm(120)                              = 0
fcntl64(13, F_SETLKW64, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}, 
0xbfe562d4) = 0
alarm(0)                                = 120
fstat64(13, {st_mode=S_IFREG|0600, st_size=1341104, ...}) = 0
mmap2(NULL, 1341104, PROT_READ|PROT_WRITE, MAP_SHARED, 13, 0) = 0xb3ffc000
munmap(0xb4285000, 1341104)             = 0
write(1, "* 7998 FETCH (FLAGS ($Label2 $La"..., 2067) = 2067
write(1, "* 8091 FETCH (FLAGS (\\Answered \\"..., 2056) = -1 EPIPE (Broken 
pipe)
- --- SIGPIPE (Broken pipe) @ 0 (0) ---
munmap(0xb7cbc000, 1341104)             = 0
fcntl64(13, F_SETLK64, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}, 
0xbfe56394) = 0
write(2, "\1IDisconnected: Mailbox is in in"..., 66) = 66

# /usr/local/dovecot.rc27/sbin/dovecot -n
# /usr/local/dovecot.rc27/etc/dovecot.conf
base_dir: /var/run/dovecot/
log_path: /var/log/dovecot/dovecot.log
protocols: imap imaps pop3 pop3s
ssl_ca_file: /etc/ssl/certs/ca.crt
ssl_cert_file(default): /etc/ssl/certs/imap.pem
ssl_cert_file(imap): /etc/ssl/certs/imap.pem
ssl_cert_file(pop3): /etc/ssl/certs/pop3.pem
ssl_key_file(default): /etc/ssl/private/imap.key
ssl_key_file(imap): /etc/ssl/private/imap.key
ssl_key_file(pop3): /etc/ssl/private/pop3.key
disable_plaintext_auth: no
verbose_ssl: yes
login_dir: /var/run/dovecot//login
login_executable(default): 
/usr/local/dovecot.rc27/libexec/dovecot/imap-login
login_executable(imap): /usr/local/dovecot.rc27/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/dovecot.rc27/libexec/dovecot/pop3-login
login_log_format_elements: %p: user=<%u> method=%m rip=%r lip=%l %c
verbose_proctitle: yes
first_valid_uid: 1000
mail_location: 
maildir:%h/MailDir:CONTROL=/var/cache/dovecot/%i/control:INDEX=/var/cache/dovecot/%i/index
mail_debug: yes
dotlock_use_excl: yes
maildir_copy_with_hardlinks: yes
maildir_copy_preserve_filename: yes
mail_drop_priv_before_exec: yes
mail_executable(default): /usr/local/dovecot.rc27/libexec/dovecot/imap
mail_executable(imap): /usr/local/dovecot.rc27/libexec/dovecot/imap
mail_executable(pop3): /usr/local/dovecot.rc27/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota mail_log zlib
mail_plugins(imap): quota imap_quota mail_log zlib
mail_plugins(pop3): quota mail_log
mail_plugin_dir(default): /usr/local/dovecot.rc27/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/dovecot.rc27/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/dovecot.rc27/lib/dovecot/pop3
mail_log_prefix: %Us(%u) [%p]:
mail_log_max_lines_per_sec: 30
pop3_uidl_format(default):
pop3_uidl_format(imap):
pop3_uidl_format(pop3): %u
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): oe-ns-eoh
namespace:
   type: private
   separator: .
   inbox: yes
   hidden: yes
auth default:
   mechanisms: plain login
   cache_size: 10
   username_chars: 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890
   verbose: yes
   debug: yes
   passdb:
     driver: ldap
     args: /usr/local/dovecot.rc26/etc/dovecot-ldap.conf
   passdb:
     driver: shadow
   userdb:
     driver: prefetch
   userdb:
     driver: ldap
     args: /usr/local/dovecot.rc26/etc/dovecot-ldap.conf
   userdb:
     driver: passwd-file
     args: /etc/passwd
   socket:
     type: listen
     client:
       path: /var/run/dovecot/auth-client
       mode: 432
     master:
       path: /var/run/dovecot/auth-master
       mode: 432
       group: mail
plugin:
   quota: fs

Bye,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBRgFBGy9SORjhbDpvAQLBcQf+OAqiufFlyPqJTyM3Ylkmiae2afXPe2wl
bNWg6AyDrdvFw+rtdc5vnYukQUSk2a8zQBggt3Rfb1zZQb0pszrzLdR+fOwnW2uE
kDyHRhyy88aVpM848oQKYnlWdNHFrdwbyp2myU4JxJhu0bXsAhyMgf9txshwe5cd
23O1x+gkWUp+1wuz84uTep3Kcs1wVXWANEN5RT4YMI3MOqMrcyB+P0lj0luy0HTH
HE9T6mLqzGt0Cw6yfm+1W8bU3Wnt3C5je15gT++OPd5YmDvpQs+fOjavDIITsuOf
jBck4N+KjwOM+FzP8Ea1WNWCvObJKdeiISkoFWFO9bafw5Yue9hFNA==
=yyP5
-----END PGP SIGNATURE-----