Retrieving mail from read-only mdbox

[Mark Moseley]

>
> >
> > > I've tried using IMAP with mail_location pointed at the snapshot, but,
> > > though I can get a listing of emails in the mailbox, the fetch fails
> when
> > > dovecot can't write-lock dovecot.index.log.
> >
> > I've thought about doing this someday (adding snapshots to a user's
> > namespace) but never got around to doing it.  Snapshots get rotated
> > (e.g. hourly.1 -> hourly.2 -> etc.)  so every hour, so any indices
> > produced gets invalidated.  You would need to generate MEMORY indices
> > to create it on the fly.  Something like
> >
> >       namespace snapshots {
> >               location = ...:INDEX==MEMORY
> >               ...
> >       }
> >
> > I'm not sure how dovecot would react when NetApp pulls the rug out
> > from under one of the hourly snapshots and replace it with the next
> > hour's version.
> >
> > Joseph Tam <jtam.home at gmail.com>
>
> location=...:INDEX=MEMORY, actually.
>
> When the rug gets pulled, what happens, depends on whether the user has
> the snapshots location open or not, but it would probably kick the user out
> in the end and complain. But then the user would probably reconnect? =)
>
>

I didn't want to muddy the waters in my first message with all the stuff
I've tried, but that was one of them. On our existing maildir mailboxes, we
have INDEX pointed at a local SSD, so that was one of the first things I
tried. For this readonly mdbox, I tried pointing INDEX= at a local disk, as
well as MEMORY. I've also got CONTROL= set to local disk as well for this.

However (and I assumed that this was probably due to the nature of mdbox
and a known thing), if I set INDEX to anything (MEMORY or a path) on my
mdbox mailboxes, dovecot acts as if the mailbox is empty. I've tried every
permutation I can think of, but the end result is this: without INDEX=, I
can get a list of messages (still can't FETCH due to the index); if I add
INDEX=, I get no results. Debug output with INDEX shows that auth is ok,
the index is being picked up correctly and that the mail_location is still
correct.

I notice too that when I have INDEX set to a path, when I strace the imap
process, though it stat()s the mail_location, it never once tries to stat()
or open any of the storage files under the mail_location. It *does* stat()
the 'storage' directory inside the snapshot, but never walks that directory
nor stat()s the m.# files inside of it.

If I have INDEX set to MEMORY, as of 2.2.30 (I didn't see this with 2.2.27,
though I still got an empty mailbox result), I get a traceback in the logs:

Jun 01 14:10:06 imap-login: Info: Login: user=<mdbox at test.com>,
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=8542, secured,
session=<KoZO8ulQ3ul/AAAB>
Jun 01 14:10:06 imap(mdbox at test.com): Panic: file mailbox-list.c: line
1330: unreached
Jun 01 14:10:06 imap(mdbox at test.com): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x9d9e2) [0x6b2dd043f9e2] ->
/usr/lib/dovecot/libdovecot.so.0(+0x9dacd) [0x6b2dd043facd] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x6b2dd03d1821] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_list_get_root_forced+0x51)
[0x6b2dd0721891] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mdbox_map_init+0x28)
[0x6b2dd07376b8] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mdbox_storage_create+0xeb)
[0x6b2dd073e2cb] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_create_full+0x3d4)
[0x6b2dd0714884] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_create+0x2c)
[0x6b2dd0714c2c] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_namespaces_init_add+0x159)
[0x6b2dd070ac09] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_namespaces_init+0xd9)
[0x6b2dd070bd99] -> dovecot/imap [mdbox at test.com 127.0.0.1]() [0x426a10] ->
/usr/lib/dovecot/libdovecot.so.0(+0x37783) [0x6b2dd03d9783] ->
/usr/lib/dovecot/libdovecot.so.0(+0x37a4d) [0x6b2dd03d9a4d] ->
/usr/lib/dovecot/libdovecot.so.0(+0x383da) [0x6b2dd03da3da] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x6b2dd0454c42] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
[0x6b2dd04562b9] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x6b2dd0454cdc]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x6b2dd0454e88] ->
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x6b2dd03dbd93]
-> dovecot/imap [mdbox at test.com 127.0.0.1](main+0x302) [0x40caa2] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x6b2dcfff9830] ->
dovecot/imap [mdbox at test.com 127.0.0.1](_start+0x29) [0x40cc29]
Jun 01 14:10:06 imap(mdbox at test.com): Fatal: master: service(imap): child
8542 killed with signal 6 (core dumps disabled)

But again, I figured that maybe this was normal for mdbox (or that I've
goofed something else up). Also, the 'If I set index, the mailbox is empty
or I get a traceback' happens if I'm looking at a readonly mdbox or a
regular, writable mdbox (and the writable one works perfectly if you remove
INDEX=).

BTW, in my case, the 'snapshot' IMAP wouldn't be directly accessible to end
users, but would be used by automation tools (to recover accidentally
deleted messages -- and yeah, I'm looking at lazy expunge currently), so
I'm not worried about weirdness from snapshots changing. In fact, the
server can throw errors all day, as long as I can fetch messages :)


doveconf -n (there might be some oddities in here, since I've been goofing
with configs trying to make this work):

# 2.2.30.1 (eebd877): /etc/dovecot/dovecot/dovecot.conf
# Pigeonhole version 0.4.18 (29cc74d)
# OS: Linux 4.4.52 x86_64 Ubuntu 16.04.1 LTS
auth_cache_negative_ttl = 1 mins
auth_cache_size = 10 M
auth_cache_ttl = 10 mins
auth_debug = yes
auth_mechanisms = plain login
auth_username_chars =
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot/debug.log
default_client_limit = 10000
default_internal_user = doveauth
default_process_limit = 5000
deliver_log_format = M=%m, F=%f, S="%s" B="%p/%w" => %$
disable_plaintext_auth = no
first_valid_uid = 199
imap_capability = +UNSELECT
last_valid_uid = 201
listen = *
lock_method = dotlock
log_path = /var/log/dovecot/mail.log
mail_debug = yes
mail_location =
maildir:~/Maildir:INDEX=/var/cache/dovecot/Indexes/%2Mu/%2.2Mu/%u
mail_nfs_storage = yes
mail_privileged_group = mail
mail_uid = 200
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext
mmap_disable = yes
namespace {
  hidden = no
  inbox = yes
  list = yes
  location =
  prefix = INBOX.
  separator = .
  subscriptions = yes
  type = private
}
passdb {
  args = /etc/dovecot/include/sql.conf
  driver = sql
}
plugin {
  quota =
  quota_rule = *:bytes=25M
  quota_rule2 = INBOX.Trash:bytes=+10%%
  quota_rule3 = *:messages=3000
  stats_refresh = 30 secs
  stats_track_cmds = no
}
protocols = imap
service auth-worker {
  unix_listener auth-worker {
    mode = 0666
  }
  user = doveauth
}
service auth {
  client_limit = 0
  unix_listener login/auth {
    mode = 0666
  }
  user = doveauth
}
service doveadm {
  inet_listener {
    port = 24245
  }
  inet_listener http {
    port = 24246
  }
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  process_min_avail = 20
  service_count = 1000
  vsz_limit = 512 M
}
service imap {
  client_limit = 10
  idle_kill = 5 mins
  process_limit = 2000
  process_min_avail = 10
  service_count = 1000
  vsz_limit = 512 M
}
service stats {
  fifo_listener stats-mail {
    mode = 0600
    user = mailuser
  }
}
ssl_cert = </etc/dovecot/ssl/certs/server.crt
ssl_cipher_list =
HIGH:MEDIUM:!kEDH:!ADH:!aNULL:!NULL:!LOW:+SHA1:+MD5:!ECDHE-RSA-DES-CBC3-SHA:!DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA
ssl_key =  # hidden, use -P to show it
ssl_parameters_regenerate = 4 hours
ssl_protocols = !SSLv2 !SSLv3
stats_memory_limit = 128 M
stats_user_min_time = 12 hours
userdb {
  driver = prefetch
}
userdb {
  args = /etc/dovecot/include/sql.conf
  driver = sql
}
verbose_proctitle = yes
protocol imap {
  mail_max_userip_connections = 100
  mail_plugins = zlib quota imap_quota stats imap_stats
}
protocol lda {
  debug_log_path = /var/log/dovecot/lda/lda-quotawarning.log
  log_path = /var/log/dovecot/lda/lda-quotawarning.log
}
local 10.0.0.0/8 {
  doveadm_password =  # hidden, use -P to show it
}