Re: Backups

11 Dec 2022


      --
John Tulp johntulp@tulpex.com
tulpex
On Sun, 2022-12-11 at 00:05 -0800, Doug Hardie wrote:
...
...
On Dec 10, 2022, at 11:09 AM, John Tulp johntulp@tulpholdings.com wrote:
On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote:
...
...
On Dec 4, 2022, at 1:42 PM, Doug Hardie bc979@lafn.org wrote:
...
On Dec 3, 2022, at 11:50 PM, Doug Hardie bc979@lafn.org wrote:
I started to investigate using doveadm backup to backup my mail
system.  I have a small number of users and the mail store is not
large.  It uses maildir format.  I setup a test system that is not
connected to the internet and started up dovecot.  I used the
following command to backup one user:
doveadm backup -u ben remote:test
ben is the user is in the mail store.  Test is the actual server
name.  That worked just fine.  The maildir was copied completely
(as best as I can tell with ls).  Then I tried the second user:
doveadm backup -u jean remote:test
This gives 2 error messages:
doveadm(jean)[]: Error: Mailbox INBOX:
Failed to get attribute
vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox
attributes not enabled
doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command
returned error 65: ssh test doveadm dsync-server -ujean -U
In addition, the maildir directories are created, but there are no
emails in any of them (e.g., cur).  What is the problem with the
2nd and why does it behave differently from the first?
I managed to resolve most of the issue.  I use pigeonhole on the
primary server.  Apparently not having pigeonhole installed on the
test machine caused the errors above.  The test machine was never
intended to receive mail hence, no need to install pigeonhole as the
LDA would never be used.  However, when the backup was running, it
choked on transferring the sieve file.  I have no idea where the
mentioned file resides as I couldn't find it anywhere on the primary
server.  Installing pigeonhole resolved the issue for all but one
user. With that user, I get the following error messages:
doveadm(doug)[]: Panic: file
istream-seekable.c: line 238 (read_from_buffer): assertion failed:
(*ret_r > 0)
Abort
doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed:
EOF (last sent=mail, last recv=mail_request (EOL))
doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command
returned error 134: ssh test doveadm dsync-server -udoug -U
In addition, only a few cur files are transferred in INBOX.
Repeating the backup generates the same errors and no additional
emails are transferred.  I am wondering if the problem is something
in the INBOX.
I have pretty much reached a dead end.  I am unable to determine the
cause of this abnormal termination.  The logged messages don't give
much help.  I can't tell if it is the primary server or test machine
that is terminating ssh.  I have setup a second test machine.  Both of
the test machines are raspberry pi 4s.  The real backup machine is
intel.  On both the test machines, 2 of the three users are backedup
properly with no errors.  Only one use has the issue.  It builds the
directory structure correctly then starts transferring the INBOX cur
files.  Eight files are transferred correctly before it stop.  It is
the same set of 8 on both test machines.  That leads me to believe
there is something funny in one of the messages, but which one.  There
are over 100 messages in that directory.  The order of file transfer
is not by date.
How do I get doveadm to tell me which file is being transferred when
the problem occurs?
if the problem is in one of the messages, perhaps change the test set.
for convenience, can temporarily rename/move things to get them out of
the way. divide test set in half, then in half again, etc., to zero in
on the one causing the issue, that'll show if a particular message is
problem or not.
once you find a problem message, if the why isn't obvious, i'd try
looking at it in hex, check permissions, etc.
I have found the cause for the errors:  Somehow I had overlooked the -D argument to doveadm.  That is really helpful in diagnosing backup issues.  I discovered that there was one email that was 130+ MB.  Doveadm cannot handle that.  It appears there is a message size limit somewhere.  I don't know if that is changeable or not.  Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly.  I don't get too many emails that large, but it does happen occasionally.  How do I go about telling dovecot to handle those?
-- Doug
sorry, no clue.  i did find this link quickly...
https://doc.dovecot.org/configuration_manual/quota/
looks like the following section near the bottom of page may apply ?
"Maximum Saved Mail Size"
again, sorry, email is not "my thing" (chuckle inserted here).