Backups

[John Tulp]

-- 
John Tulp <johntulp at tulpex.com>
tulpex

On Sun, 2022-12-11 at 00:05 -0800, Doug Hardie wrote:
> > On Dec 10, 2022, at 11:09 AM, John Tulp <johntulp at tulpholdings.com> wrote:
> > 
> > 
> > 
> > On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote:
> >>> On Dec 4, 2022, at 1:42 PM, Doug Hardie <bc979 at lafn.org> wrote:
> >>> 
> >>>> On Dec 3, 2022, at 11:50 PM, Doug Hardie <bc979 at lafn.org> wrote:
> >>>> 
> >>>> I started to investigate using doveadm backup to backup my mail
> >>>> system.  I have a small number of users and the mail store is not
> >>>> large.  It uses maildir format.  I setup a test system that is not
> >>>> connected to the internet and started up dovecot.  I used the
> >>>> following command to backup one user:
> >>>> 
> >>>> doveadm backup -u ben remote:test
> >>>> 
> >>>> ben is the user is in the mail store.  Test is the actual server
> >>>> name.  That worked just fine.  The maildir was copied completely
> >>>> (as best as I can tell with ls).  Then I tried the second user:
> >>>> 
> >>>> doveadm backup -u jean remote:test
> >>>> 
> >>>> This gives 2 error messages:
> >>>> 
> >>>> doveadm(jean)[]<McRYOFI0jGN1MwAAmC+70w>: Error: Mailbox INBOX:
> >>>> Failed to get attribute
> >>>> vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox
> >>>> attributes not enabled
> >>>> 
> >>>> doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command
> >>>> returned error 65: ssh test doveadm dsync-server -ujean -U
> >>>> 
> >>>> In addition, the maildir directories are created, but there are no
> >>>> emails in any of them (e.g., cur).  What is the problem with the
> >>>> 2nd and why does it behave differently from the first?
> >>> 
> >>> I managed to resolve most of the issue.  I use pigeonhole on the
> >>> primary server.  Apparently not having pigeonhole installed on the
> >>> test machine caused the errors above.  The test machine was never
> >>> intended to receive mail hence, no need to install pigeonhole as the
> >>> LDA would never be used.  However, when the backup was running, it
> >>> choked on transferring the sieve file.  I have no idea where the
> >>> mentioned file resides as I couldn't find it anywhere on the primary
> >>> server.  Installing pigeonhole resolved the issue for all but one
> >>> user. With that user, I get the following error messages:
> >>> 
> >>> doveadm(doug)[]<NAfSLTASjWNWTAAAmC+70w>: Panic: file
> >>> istream-seekable.c: line 238 (read_from_buffer): assertion failed:
> >>> (*ret_r > 0)
> >>> Abort
> >>> 
> >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed:
> >>> EOF (last sent=mail, last recv=mail_request (EOL))
> >>> 
> >>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command
> >>> returned error 134: ssh test doveadm dsync-server -udoug -U
> >>> 
> >>> In addition, only a few cur files are transferred in INBOX.
> >>> Repeating the backup generates the same errors and no additional
> >>> emails are transferred.  I am wondering if the problem is something
> >>> in the INBOX.
> >> 
> >> 
> >> 
> >> 
> >> I have pretty much reached a dead end.  I am unable to determine the
> >> cause of this abnormal termination.  The logged messages don't give
> >> much help.  I can't tell if it is the primary server or test machine
> >> that is terminating ssh.  I have setup a second test machine.  Both of
> >> the test machines are raspberry pi 4s.  The real backup machine is
> >> intel.  On both the test machines, 2 of the three users are backedup
> >> properly with no errors.  Only one use has the issue.  It builds the
> >> directory structure correctly then starts transferring the INBOX cur
> >> files.  Eight files are transferred correctly before it stop.  It is
> >> the same set of 8 on both test machines.  That leads me to believe
> >> there is something funny in one of the messages, but which one.  There
> >> are over 100 messages in that directory.  The order of file transfer
> >> is not by date.  
> >> 
> >> 
> >> How do I get doveadm to tell me which file is being transferred when
> >> the problem occurs?  
> > 
> > if the problem is in one of the messages, perhaps change the test set.
> > for convenience, can temporarily rename/move things to get them out of
> > the way. divide test set in half, then in half again, etc., to zero in
> > on the one causing the issue, that'll show if a particular message is
> > problem or not.
> > 
> > once you find a problem message, if the why isn't obvious, i'd try
> > looking at it in hex, check permissions, etc.
> 
> I have found the cause for the errors:  Somehow I had overlooked the -D argument to doveadm.  That is really helpful in diagnosing backup issues.  I discovered that there was one email that was 130+ MB.  Doveadm cannot handle that.  It appears there is a message size limit somewhere.  I don't know if that is changeable or not.  Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly.  I don't get too many emails that large, but it does happen occasionally.  How do I go about telling dovecot to handle those?
> 
> -- Doug
> 
> 

sorry, no clue.  i did find this link quickly...

https://doc.dovecot.org/configuration_manual/quota/

looks like the following section near the bottom of page may apply ?

"Maximum Saved Mail Size"

again, sorry, email is not "my thing" (chuckle inserted here).