Dear Colleagues,
I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u"
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On 26.09.2018 12:22, Victor Sudakov wrote:
Dear Colleagues,
I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u"
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
Dovecot is tolerant to changes with mbox and maildir. It will reindex if it detects someone changing them outside.
Aki
On 26 Sep 2018, at 13.07, Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 26.09.2018 12:22, Victor Sudakov wrote:
Dear Colleagues,
I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u"
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
Dovecot is tolerant to changes with mbox and maildir. It will reindex if it detects someone changing them outside.
That is true but it however has performance impacts if the caching is impaired but still does works.
Sami
Sami Ketola wrote:
On 26 Sep 2018, at 13.07, Aki Tuomi aki.tuomi@open-xchange.com wrote:
I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u"
However, I often read and modify the mailboxes locally with Mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
Dovecot is tolerant to changes with mbox and maildir. It will reindex if it detects someone changing them outside.
That is good news! Thank you Aki!
That is true but it however has performance impacts if the caching is impaired but still does works.
Ah, is this perhaps the reason why my Android IMAP client sometimes takes so long to list a folder's contents? I must have touched the folder locally by Mutt and Dovecot is reindexing, is that right?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Thomas Leuxner wrote:
However, I often read and modify the mailboxes locally with Mutt (e.g. append and delete mails).
Why not use Mutt's IMAP capabilities and keep the indexes nice and clean?
Maybe the only reason is that I don't want to bother with authentication and don't want to keep my password in muttrc.
Anyway, my mail is sorted and delivered to ~/Mail/* by procmail, so the mboxes will be touched by a process outside of dovecot.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
What I do is keep my passwords in a file encrypted by GPG and have mutt do: source "gpg -q --textmode -d ~/.neomutt/passwords.gpg |"
that way they're not readable except by use of a passworded GPG key.
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Drive, Round Rock, TX 78665-2106
On 9/27/18, 9:50 AM, "dovecot on behalf of Victor Sudakov"
Thomas Leuxner wrote:
>
> > > >> However, I often read and modify the mailboxes locally with Mutt (e.g.
> > > >> append and delete mails).
>
> Why not use Mutt's IMAP capabilities and keep the indexes nice and clean?
Maybe the only reason is that I don't want to bother with
authentication and don't want to keep my password in muttrc.
Anyway, my mail is sorted and delivered to ~/Mail/* by procmail, so
the mboxes will be touched by a process outside of dovecot.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/Larry Rosenman wrote:
What I do is keep my passwords in a file encrypted by GPG and have mutt do: source "gpg -q --textmode -d ~/.neomutt/passwords.gpg |"
that way they're not readable except by use of a passworded GPG key.
Very clever, I must admit.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On Wed, 26 Sep 2018, Victor Sudakov wrote:
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
I do it all the time. Works fine.
As others have written, you may see performance degradation as Dovecot will have to rebuild indices, but if you have small mailboxes, this won't be too bad. The only reason I use direct file access rather than IMAP is that I'm too lazy to work out a passwordless access method. If this doesn't bother you or you can configure this (e.g. Kerberos, keyring, etc.), IMAP access is preferable since you won't pull the indices out from Dovecot's feet.
You'll also get a lot of innocuous griping in the log files about UIDVALIDITY and mailbox corruption, but they can be safely ignored.
Joseph Tam jtam.home@gmail.com
Joseph Tam wrote:
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
I do it all the time. Works fine.
Great, thank you!
As others have written, you may see performance degradation as Dovecot will have to rebuild indices, but if you have small mailboxes, this won't be too bad. The only reason I use direct file access rather than IMAP is that I'm too lazy to work out a passwordless access method. If this doesn't bother you or you can configure this (e.g. Kerberos, keyring, etc.), IMAP access is preferable since you won't pull the indices out from Dovecot's feet.
What passwordless access methods does Dovecot support? I'm more or less experienced with GSSAPI but for the present I would not like to set up a KDC somewhere on a box exposed to the Internet.
You'll also get a lot of innocuous griping in the log files about UIDVALIDITY and mailbox corruption, but they can be safely ignored.
My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
I have my dovecot running with TLS, so the passwords are NOT transmitted in the clear (starttls).
On Thu, Sep 27, 2018 at 10:42 AM Victor Sudakov vas@mpeks.tomsk.su wrote:
Joseph Tam wrote:
However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails).
Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me?
I do it all the time. Works fine.
Great, thank you!
As others have written, you may see performance degradation as Dovecot will have to rebuild indices, but if you have small mailboxes, this won't be too bad. The only reason I use direct file access rather than IMAP is that I'm too lazy to work out a passwordless access method. If this doesn't bother you or you can configure this (e.g. Kerberos, keyring, etc.), IMAP access is preferable since you won't pull the indices out from Dovecot's feet.
What passwordless access methods does Dovecot support? I'm more or less experienced with GSSAPI but for the present I would not like to set up a KDC somewhere on a box exposed to the Internet.
You'll also get a lot of innocuous griping in the log files about UIDVALIDITY and mailbox corruption, but they can be safely ignored.
My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Larry Rosenman wrote:
I have my dovecot running with TLS, so the passwords are NOT transmitted in the clear (starttls).
The problem is not transmitting them in the clear but keeping them in the clear in a local file like muttrc, or entering the password each time you launch mutt.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On 9/27/18, 10:57 AM, "Victor Sudakov"
Larry Rosenman wrote:
> I have my dovecot running with TLS, so the passwords are NOT transmitted in
> the clear (starttls).
The problem is not transmitting them in the clear but keeping them in
the clear in a local file like muttrc, or entering the password each
time you launch mutt.Hence my gpg text file trick. (
Larry Rosenman wrote:
Larry Rosenman wrote: > I have my dovecot running with TLS, so the passwords are NOT transmitted in > the clear (starttls). The problem is not transmitting them in the clear but keeping them in the clear in a local file like muttrc, or entering the password each time you launch mutt.Hence my gpg text file trick. (
Do you enter your gpg password each time you launch mutt, or do you use some kind of gpg agent?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On Thu, 27 Sep 2018, Victor Sudakov wrote:
As others have written, you may see performance degradation as Dovecot will have to rebuild indices, but if you have small mailboxes, this won't be too bad. The only reason I use direct file access rather than IMAP is that I'm too lazy to work out a passwordless access method. If this doesn't bother you or you can configure this (e.g. Kerberos, keyring, etc.), IMAP access is preferable since you won't pull the indices out from Dovecot's feet.
What passwordless access methods does Dovecot support? I'm more or less experienced with GSSAPI but for the present I would not like to set up a KDC somewhere on a box exposed to the Internet.
Yes, Kerberos is one way. Or some password broker as Larry has done but the client has to be able to work with it. You can also pipe to imap worker process via ssh/pubkey if your server will allow it. Perhaps dovecot supports SSL client certificate authentication as well -- I haven't invested a lot of time looking into the various schemes.
Whether you use any of them is up to you. As I said, direct mailbox file access will work. However, if you deal with Gb size mailboxes, re-indexing and general mail operations will be painful.
My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway.
Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
:0 ... pattern ...
| /path/to/dovecot-lda -d {user} -m {mailbox}
...
# End of file: deliver to inbox
:0 w
| /path/to/dovecot-lda -d {user}Joseph Tam jtam.home@gmail.com
Joseph Tam wrote:
[dd]
Whether you use any of them is up to you. As I said, direct mailbox file access will work. However, if you deal with Gb size mailboxes, re-indexing and general mail operations will be painful.
My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway.
Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
:0 ... pattern ... | /path/to/dovecot-lda -d {user} -m {mailbox}
...
# End of file: deliver to inbox :0 w | /path/to/dovecot-lda -d {user}
This will clutter up ~/.procmailrc, but may be I'll try someday.
Now I have encountered a different problem. My Android IMAP client (the Gmail map) shows lots of mails in my Inbox. However, I already deleted those mails locally from /var/mai/vas. They are being cached somewhere.
Is it not dovecot caching them?
Nor can I find a way to forcefully rescan an IMAP folder in the Gmail app.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Joseph Tam wrote:
[dd]
Whether you use any of them is up to you. As I said, direct mailbox file access will work. However, if you deal with Gb size mailboxes, re-indexing and general mail operations will be painful.
My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway.
Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
:0 ... pattern ... | /path/to/dovecot-lda -d {user} -m {mailbox}
...
# End of file: deliver to inbox :0 w | /path/to/dovecot-lda -d {user}
This will clutter up ~/.procmailrc, but may be I'll try someday.
Now I have encountered a different problem. My Android IMAP client (the Gmail map) shows lots of mails in my Inbox. However, I already deleted those mails locally from /var/mai/vas. They are being cached somewhere.
Is it not dovecot caching them?
Nor can I find a way to forcefully rescan an IMAP folder in the Gmail app.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
This time an Android IMAP client complains about the "unexpected end of the network stream" while opening a mailbox named "dss", and dovecot logs:
Sep 28 23:01:34 admin dovecot: imap-login: Login: user=<vas>, method=PLAIN, rip=78.140.19.131, lip=95.179.154.55, mpid=85536, TLS, session=<6ITSkfB2JfNOjBOD> Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>: Corrupted record in index cache file /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 56: Broken physical size in mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (8271 < 8282, box=dss, UID=56) Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>: Mailbox dss: UID=56: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (8271 < 8282, box=dss, UID=56) (FETCH BODY[2]) Sep 28 23:01:35 admin dovecot: imap(vas)<85536><6ITSkfB2JfNOjBOD>: FETCH read() failed in=896 out=54723 deleted=0 expunged=0 trashed=0 hdr_count=15 hdr_bytes=7777 body_count=14 body_bytes=37063 Sep 28 23:02:54 admin dovecot: imap-login: Login: user=<vas>, method=PLAIN, rip=78.140.19.131, lip=95.179.154.55, mpid=85560, TLS, session=<x6mSlvB2NN1OjBOD> Sep 28 23:02:55 admin dovecot: Error: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Corrupted record in index cache file /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 27: Broken physical size in mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (4867 < 4878, box=dss, UID=27) Sep 28 23:02:55 admin dovecot: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Panic: file istream.c: line 315 (i_stream_read_memarea): assertion failed: (old_size <= _stream->pos - _stream->skip) Sep 28 23:02:55 admin dovecot: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Fatal: master: service(imap): child 85560 killed with signal 6 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set service imap { drop_priv_before_exec=yes })
Is this that corrupted index case? It even causes the panic in the daemon. Can I fix the index or somehow reindex the mailbox?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On 28 September 2018 at 19:07 Victor Sudakov vas@mpeks.tomsk.su wrote:
This time an Android IMAP client complains about the "unexpected end of the network stream" while opening a mailbox named "dss", and dovecot logs:
Sep 28 23:01:34 admin dovecot: imap-login: Login: user=<vas>, method=PLAIN, rip=78.140.19.131, lip=95.179.154.55, mpid=85536, TLS, session=<6ITSkfB2JfNOjBOD> Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>: Corrupted record in index cache file /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 56: Broken physical size in mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (8271 < 8282, box=dss, UID=56) Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>: Mailbox dss: UID=56: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (8271 < 8282, box=dss, UID=56) (FETCH BODY[2]) Sep 28 23:01:35 admin dovecot: imap(vas)<85536><6ITSkfB2JfNOjBOD>: FETCH read() failed in=896 out=54723 deleted=0 expunged=0 trashed=0 hdr_count=15 hdr_bytes=7777 body_count=14 body_bytes=37063 Sep 28 23:02:54 admin dovecot: imap-login: Login: user=<vas>, method=PLAIN, rip=78.140.19.131, lip=95.179.154.55, mpid=85560, TLS, session=<x6mSlvB2NN1OjBOD> Sep 28 23:02:55 admin dovecot: Error: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Corrupted record in index cache file /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 27: Broken physical size in mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than expected (4867 < 4878, box=dss, UID=27) Sep 28 23:02:55 admin dovecot: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Panic: file istream.c: line 315 (i_stream_read_memarea): assertion failed: (old_size <= _stream->pos - _stream->skip) Sep 28 23:02:55 admin dovecot: imap(vas)<85560><x6mSlvB2NN1OjBOD>: Fatal: master: service(imap): child 85560 killed with signal 6 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set service imap { drop_priv_before_exec=yes })
Is this that corrupted index case? It even causes the panic in the daemon. Can I fix the index or somehow reindex the mailbox?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
You can use doveadm force-resync -u vas "*"
Aki
Aki Tuomi wrote:
Is this that corrupted index case? It even causes the panic in the daemon. Can I fix the index or somehow reindex the mailbox?
You can use doveadm force-resync -u vas "*"
Thanks!
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
participants (6)
-
Aki Tuomi
-
Joseph Tam
-
Larry Rosenman
-
Sami Ketola
-
Thomas Leuxner
-
Victor Sudakov