Backups

[Doug Hardie]

> On Dec 11, 2022, at 12:42 AM, John Tulp <johntulp at tulpex.com> wrote:
> 
> 
> -- 
> John Tulp <johntulp at tulpex.com>
> tulpex
> 
> On Sun, 2022-12-11 at 00:05 -0800, Doug Hardie wrote:
>>> On Dec 10, 2022, at 11:09 AM, John Tulp <johntulp at tulpholdings.com> wrote:
>>> 
>>> 
>>> 
>>> On Fri, 2022-12-09 at 20:03 -0800, Doug Hardie wrote:
>>>>> On Dec 4, 2022, at 1:42 PM, Doug Hardie <bc979 at lafn.org> wrote:
>>>>> 
>>>>>> On Dec 3, 2022, at 11:50 PM, Doug Hardie <bc979 at lafn.org> wrote:
>>>>>> 
>>>>>> I started to investigate using doveadm backup to backup my mail
>>>>>> system.  I have a small number of users and the mail store is not
>>>>>> large.  It uses maildir format.  I setup a test system that is not
>>>>>> connected to the internet and started up dovecot.  I used the
>>>>>> following command to backup one user:
>>>>>> 
>>>>>> doveadm backup -u ben remote:test
>>>>>> 
>>>>>> ben is the user is in the mail store.  Test is the actual server
>>>>>> name.  That worked just fine.  The maildir was copied completely
>>>>>> (as best as I can tell with ls).  Then I tried the second user:
>>>>>> 
>>>>>> doveadm backup -u jean remote:test
>>>>>> 
>>>>>> This gives 2 error messages:
>>>>>> 
>>>>>> doveadm(jean)[]<McRYOFI0jGN1MwAAmC+70w>: Error: Mailbox INBOX:
>>>>>> Failed to get attribute
>>>>>> vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox
>>>>>> attributes not enabled
>>>>>> 
>>>>>> doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command
>>>>>> returned error 65: ssh test doveadm dsync-server -ujean -U
>>>>>> 
>>>>>> In addition, the maildir directories are created, but there are no
>>>>>> emails in any of them (e.g., cur).  What is the problem with the
>>>>>> 2nd and why does it behave differently from the first?
>>>>> 
>>>>> I managed to resolve most of the issue.  I use pigeonhole on the
>>>>> primary server.  Apparently not having pigeonhole installed on the
>>>>> test machine caused the errors above.  The test machine was never
>>>>> intended to receive mail hence, no need to install pigeonhole as the
>>>>> LDA would never be used.  However, when the backup was running, it
>>>>> choked on transferring the sieve file.  I have no idea where the
>>>>> mentioned file resides as I couldn't find it anywhere on the primary
>>>>> server.  Installing pigeonhole resolved the issue for all but one
>>>>> user. With that user, I get the following error messages:
>>>>> 
>>>>> doveadm(doug)[]<NAfSLTASjWNWTAAAmC+70w>: Panic: file
>>>>> istream-seekable.c: line 238 (read_from_buffer): assertion failed:
>>>>> (*ret_r > 0)
>>>>> Abort
>>>>> 
>>>>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: read(test) failed:
>>>>> EOF (last sent=mail, last recv=mail_request (EOL))
>>>>> 
>>>>> doveadm(doug)[]<C2AvHjASjWN2gQAAZU03Dg>: Error: Remote command
>>>>> returned error 134: ssh test doveadm dsync-server -udoug -U
>>>>> 
>>>>> In addition, only a few cur files are transferred in INBOX.
>>>>> Repeating the backup generates the same errors and no additional
>>>>> emails are transferred.  I am wondering if the problem is something
>>>>> in the INBOX.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> I have pretty much reached a dead end.  I am unable to determine the
>>>> cause of this abnormal termination.  The logged messages don't give
>>>> much help.  I can't tell if it is the primary server or test machine
>>>> that is terminating ssh.  I have setup a second test machine.  Both of
>>>> the test machines are raspberry pi 4s.  The real backup machine is
>>>> intel.  On both the test machines, 2 of the three users are backedup
>>>> properly with no errors.  Only one use has the issue.  It builds the
>>>> directory structure correctly then starts transferring the INBOX cur
>>>> files.  Eight files are transferred correctly before it stop.  It is
>>>> the same set of 8 on both test machines.  That leads me to believe
>>>> there is something funny in one of the messages, but which one.  There
>>>> are over 100 messages in that directory.  The order of file transfer
>>>> is not by date.  
>>>> 
>>>> 
>>>> How do I get doveadm to tell me which file is being transferred when
>>>> the problem occurs?  
>>> 
>>> if the problem is in one of the messages, perhaps change the test set.
>>> for convenience, can temporarily rename/move things to get them out of
>>> the way. divide test set in half, then in half again, etc., to zero in
>>> on the one causing the issue, that'll show if a particular message is
>>> problem or not.
>>> 
>>> once you find a problem message, if the why isn't obvious, i'd try
>>> looking at it in hex, check permissions, etc.
>> 
>> I have found the cause for the errors:  Somehow I had overlooked the -D argument to doveadm.  That is really helpful in diagnosing backup issues.  I discovered that there was one email that was 130+ MB.  Doveadm cannot handle that.  It appears there is a message size limit somewhere.  I don't know if that is changeable or not.  Anyway, removing that email from the mailbox, deleting it from the Trash, and purging the account enabled backup to work properly.  I don't get too many emails that large, but it does happen occasionally.  How do I go about telling dovecot to handle those?
>> 
>> -- Doug
>> 
>> 
> 
> sorry, no clue.  i did find this link quickly...
> 
> https://doc.dovecot.org/configuration_manual/quota/
> 
> looks like the following section near the bottom of page may apply ?
> 
> "Maximum Saved Mail Size"
> 
> again, sorry, email is not "my thing" (chuckle inserted here).

As best as I can tell, that requires quota to be setup as plugin which I do not. Also, it indicates that the default value is 0 which is unlimited.

-- Doug