pread() failures when using mdbox on btrfs
(Note: I am not subscribed to this ML, so please be sure to CC me in replies.)
Hello,
I see a strange bug when running dovecot under btrfs with a mailbox in mdbox format. At some point, which can take a few minutes or more than a day, dovecot starts logging pread() failures, like so:
Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- Mails/dovecot.index.log: Input/output error
These would trigger when getmail runs (which in turn is configured to use dovecot-lda). It's only ever the *.index.log files, and it would usually happen to the same one many times in a row. To demonstrate, this is the full list of unique pread() failures on one host (my desktop PC):
journalctl -u dovecot -p 3 -o cat | sort -u | grep pread
imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/BTRFS/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Gentoo/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Newsletters/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/SCons/User/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error
and on the current host (a home server):
journalctl -u dovecot -p 3 -o cat | sort -u | grep pread
imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/Privat/dbox-Mails/dovecot.index.log: Input/output error imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/storage/dovecot.map.index.log: Input/output error
Rebooting the system would make it go away (without loss of mails), though remounting the file system should have worked, too. I tried a variety of options, but the only one that helped was to convert the mailbox from mdbox to maildir. I have not seen any pread() failures in the two weeks since.
I originally wrote to the btrfs ML, see the thread at [0], in particular my messages [1] and [2], for additional details, including references to what little I could find on the web. However, nothing much came of that thread, so I am now posting here.
My dovecot configuration:
doveconf -n
2.2.19: /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.9 (357ac0a0e68b+)
OS: Linux 4.4.8-gentoo x86_64 Gentoo Base System release 2.2
auth_mechanisms = plain login
mail_location = maildir:~/.maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = *
driver = pam
}
plugin {
sieve = file:~/.sieve;active=~/.dovecot.sieve
}
postmaster_address = root@diefledermaus
protocols = imap sieve
service imap-login {
inet_listener imap {
port = 10087
}
inet_listener imaps {
port = 10887
}
}
ssl = required
ssl_cert = </etc/ssl/dovecot/dovecot.pem
ssl_key = </etc/ssl/dovecot/dovecot.key
userdb {
driver = passwd
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
mail_plugins = sieve
}
protocol sieve {
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date
}
I used to use claws-mail, then switched to KMail in September 2015. So the errors happened with both clients.
[0] http://thread.gmane.org/gmane.comp.file-systems.btrfs/49637 [1] http://article.gmane.org/gmane.comp.file-systems.btrfs/49647 [2] http://article.gmane.org/gmane.comp.file-systems.btrfs/55552
Greetings
Marc Joliet
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup
On 05 May 2016, at 16:21, Marc Joliet <marcec@gmx.de> wrote:
(Note: I am not subscribed to this ML, so please be sure to CC me in replies.)
Hello,
I see a strange bug when running dovecot under btrfs with a mailbox in mdbox format. At some point, which can take a few minutes or more than a day, dovecot starts logging pread() failures, like so:
Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- Mails/dovecot.index.log: Input/output error
These would trigger when getmail runs (which in turn is configured to use dovecot-lda). It's only ever the *.index.log files, and it would usually happen to the same one many times in a row. To demonstrate, this is the full list of unique pread() failures on one host (my desktop PC):
*.index.log files are always appended to using O_APPEND flag. Maybe this is relevant.
Also when a new .log file is created it's opened without the O_APPEND flag and the O_APPEND is added later. This was causing a bug recently in unionfs, which ignored the flag change and caused log file corruption.
Rebooting the system would make it go away (without loss of mails), though remounting the file system should have worked, too. I tried a variety of options, but the only one that helped was to convert the mailbox from mdbox to maildir. I have not seen any pread() failures in the two weeks since.
I can't really think of why Maildir format would get rid of the problem, since the dovecot.index.log files still exist and they're accessed the same way. Maybe with Maildir there are enough locks that it prevents some race condition that triggers this.
You could try stress testing with imaptest to see if it's easy to reproduce: http://imapwiki.org/ImapTest
On Friday 06 May 2016 01:51:34 Timo Sirainen wrote:
On 05 May 2016, at 16:21, Marc Joliet <marcec@gmx.de> wrote:
(Note: I am not subscribed to this ML, so please be sure to CC me in replies.)
Hello,
I see a strange bug when running dovecot under btrfs with a mailbox in mdbox format. At some point, which can take a few minutes or more than a day, dovecot starts logging pread() failures, like so:
Apr 02 23:43:42 thetick dovecot[570]: imap(marcec): Error: pread() failed with file /home/marcec/.mdbox/mailboxes/LinuxAudio/User/dbox- Mails/dovecot.index.log: Input/output error
These would trigger when getmail runs (which in turn is configured to use dovecot-lda). It's only ever the *.index.log files, and it would usually happen to the same one many times in a row. To demonstrate, this is the full list of unique pread() failures on one host (my desktop PC): *.index.log files are always appended to using O_APPEND flag. Maybe this is relevant.
Also when a new .log file is created it's opened without the O_APPEND flag and the O_APPEND is added later. This was causing a bug recently in unionfs, which ignored the flag change and caused log file corruption.
I will mention that in the btrfs ML thread. Who knows, maybe it will lead to something.
Rebooting the system would make it go away (without loss of mails), though remounting the file system should have worked, too. I tried a variety of options, but the only one that helped was to convert the mailbox from mdbox to maildir. I have not seen any pread() failures in the two weeks since. I can't really think of why Maildir format would get rid of the problem, since the dovecot.index.log files still exist and they're accessed the same way. Maybe with Maildir there are enough locks that it prevents some race condition that triggers this.
You could try stress testing with imaptest to see if it's easy to reproduce: http://imapwiki.org/ImapTest
I will experiment with that when I have some more time. Thanks for the suggestions!
Greetings
Marc Joliet
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup
Am Friday 27 May 2016 schrieb Timo Sirainen <tss@iki.fi>
On 05 May 2016, at 16:21, Marc Joliet <marcec@gmx.de> wrote: [...]
Rebooting the system would make it go away (without loss of mails), though remounting the file system should have worked, too. I tried a variety of options, but the only one that helped was to convert the mailbox from mdbox to maildir. I have not seen any pread() failures in the two weeks since. I can't really think of why Maildir format would get rid of the problem, since the dovecot.index.log files still exist and they're accessed the same way. Maybe with Maildir there are enough locks that it prevents some race condition that triggers this.
Yeah, I was somewhat surprised even without knowing that. And what do you know: two days ago I discovered pread() failures in my system log again, but this time they went away by themselves without me noticing that they even occurred:
Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:03:45 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error Mai 23 18:13:57 diefledermaus dovecot[600]: imap(marcec): Error: pread() failed with file /home/marcec/.maildir/.LinuxAudio.Dev/dovecot.index.log: Input/output error
There were also a whole bunch affecting the same file on May 20th, but not before. I probably didn't notice those because they started shortly before I rebooted for a kernel upgrade (from gentoo-sources 4.4.10 to 4.5.5).
You could try stress testing with imaptest to see if it's easy to reproduce: http://imapwiki.org/ImapTest
I still haven't had time to do this, but haven't forgotten!
Greetings
Marc Joliet
"People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup
participants (2)
-
Marc Joliet
-
Timo Sirainen