Re: Retrieving mail from read-only mdbox
I've tried using IMAP with mail_location pointed at the snapshot, but, though I can get a listing of emails in the mailbox, the fetch fails when dovecot can't write-lock dovecot.index.log.
I've thought about doing this someday (adding snapshots to a user's namespace) but never got around to doing it. Snapshots get rotated (e.g. hourly.1 -> hourly.2 -> etc.) so every hour, so any indices produced gets invalidated. You would need to generate MEMORY indices to create it on the fly. Something like
namespace snapshots {
location = ...:INDEX==MEMORY
...
}I'm not sure how dovecot would react when NetApp pulls the rug out from under one of the hourly snapshots and replace it with the next hour's version.
Joseph Tam jtam.home@gmail.com
On June 1, 2017 at 4:05 AM Joseph Tam jtam.home@gmail.com wrote:
I've tried using IMAP with mail_location pointed at the snapshot, but, though I can get a listing of emails in the mailbox, the fetch fails when dovecot can't write-lock dovecot.index.log.
I've thought about doing this someday (adding snapshots to a user's namespace) but never got around to doing it. Snapshots get rotated (e.g. hourly.1 -> hourly.2 -> etc.) so every hour, so any indices produced gets invalidated. You would need to generate MEMORY indices to create it on the fly. Something like
namespace snapshots { location = ...:INDEX==MEMORY ... }
I'm not sure how dovecot would react when NetApp pulls the rug out from under one of the hourly snapshots and replace it with the next hour's version.
Joseph Tam jtam.home@gmail.com
location=...:INDEX=MEMORY, actually.
When the rug gets pulled, what happens, depends on whether the user has the snapshots location open or not, but it would probably kick the user out in the end and complain. But then the user would probably reconnect? =)
Aki
I've tried using IMAP with mail_location pointed at the snapshot, but, though I can get a listing of emails in the mailbox, the fetch fails
when
dovecot can't write-lock dovecot.index.log.
I've thought about doing this someday (adding snapshots to a user's namespace) but never got around to doing it. Snapshots get rotated (e.g. hourly.1 -> hourly.2 -> etc.) so every hour, so any indices produced gets invalidated. You would need to generate MEMORY indices to create it on the fly. Something like
namespace snapshots { location = ...:INDEX==MEMORY ... }I'm not sure how dovecot would react when NetApp pulls the rug out from under one of the hourly snapshots and replace it with the next hour's version.
Joseph Tam jtam.home@gmail.com
location=...:INDEX=MEMORY, actually.
When the rug gets pulled, what happens, depends on whether the user has the snapshots location open or not, but it would probably kick the user out in the end and complain. But then the user would probably reconnect? =)
I didn't want to muddy the waters in my first message with all the stuff I've tried, but that was one of them. On our existing maildir mailboxes, we have INDEX pointed at a local SSD, so that was one of the first things I tried. For this readonly mdbox, I tried pointing INDEX= at a local disk, as well as MEMORY. I've also got CONTROL= set to local disk as well for this.
However (and I assumed that this was probably due to the nature of mdbox and a known thing), if I set INDEX to anything (MEMORY or a path) on my mdbox mailboxes, dovecot acts as if the mailbox is empty. I've tried every permutation I can think of, but the end result is this: without INDEX=, I can get a list of messages (still can't FETCH due to the index); if I add INDEX=, I get no results. Debug output with INDEX shows that auth is ok, the index is being picked up correctly and that the mail_location is still correct.
I notice too that when I have INDEX set to a path, when I strace the imap process, though it stat()s the mail_location, it never once tries to stat() or open any of the storage files under the mail_location. It *does* stat() the 'storage' directory inside the snapshot, but never walks that directory nor stat()s the m.# files inside of it.
If I have INDEX set to MEMORY, as of 2.2.30 (I didn't see this with 2.2.27, though I still got an empty mailbox result), I get a traceback in the logs:
Jun 01 14:10:06 imap-login: Info: Login: user=mdbox@test.com,
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=8542, secured,
session=
But again, I figured that maybe this was normal for mdbox (or that I've goofed something else up). Also, the 'If I set index, the mailbox is empty or I get a traceback' happens if I'm looking at a readonly mdbox or a regular, writable mdbox (and the writable one works perfectly if you remove INDEX=).
BTW, in my case, the 'snapshot' IMAP wouldn't be directly accessible to end users, but would be used by automation tools (to recover accidentally deleted messages -- and yeah, I'm looking at lazy expunge currently), so I'm not worried about weirdness from snapshots changing. In fact, the server can throw errors all day, as long as I can fetch messages :)
doveconf -n (there might be some oddities in here, since I've been goofing with configs trying to make this work):
# 2.2.30.1 (eebd877): /etc/dovecot/dovecot/dovecot.conf # Pigeonhole version 0.4.18 (29cc74d) # OS: Linux 4.4.52 x86_64 Ubuntu 16.04.1 LTS auth_cache_negative_ttl = 1 mins auth_cache_size = 10 M auth_cache_ttl = 10 mins auth_debug = yes auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#" base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot/debug.log default_client_limit = 10000 default_internal_user = doveauth default_process_limit = 5000 deliver_log_format = M=%m, F=%f, S="%s" B="%p/%w" => %$ disable_plaintext_auth = no first_valid_uid = 199 imap_capability = +UNSELECT last_valid_uid = 201 listen = * lock_method = dotlock log_path = /var/log/dovecot/mail.log mail_debug = yes mail_location = maildir:~/Maildir:INDEX=/var/cache/dovecot/Indexes/%2Mu/%2.2Mu/%u mail_nfs_storage = yes mail_privileged_group = mail mail_uid = 200 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mmap_disable = yes namespace { hidden = no inbox = yes list = yes location = prefix = INBOX. separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/include/sql.conf driver = sql } plugin { quota = quota_rule = *:bytes=25M quota_rule2 = INBOX.Trash:bytes=+10%% quota_rule3 = *:messages=3000 stats_refresh = 30 secs stats_track_cmds = no } protocols = imap service auth-worker { unix_listener auth-worker { mode = 0666 } user = doveauth } service auth { client_limit = 0 unix_listener login/auth { mode = 0666 } user = doveauth } service doveadm { inet_listener { port = 24245 } inet_listener http { port = 24246 } } service imap-login { inet_listener imap { port = 143 } process_min_avail = 20 service_count = 1000 vsz_limit = 512 M } service imap { client_limit = 10 idle_kill = 5 mins process_limit = 2000 process_min_avail = 10 service_count = 1000 vsz_limit = 512 M } service stats { fifo_listener stats-mail { mode = 0600 user = mailuser } } ssl_cert =
participants (3)
-
Aki Tuomi
-
Joseph Tam
-
Mark Moseley