Once you cache grows bigger than 0x4000000 you have problems

---Aki TuomiDovecot oy -------- Original message --------From: Ian Bobbitt <ibobbitt@globalnoc.iu.edu> Date: 23/02/2018 20:33 (GMT+02:00) To: dovecot@dovecot.org Subject: Re: Assertion during dsync receive

Thanks. I've had the user clear out that mailbox, and replication is
working fine for them again.



Is there a better way to catch this than watch for crashes and read
the backtrace to find what mailbox needs to be shrunk?



Where is the threshold for "too big"?



-- Ian



On 2/23/18 11:33 AM, Aki Tuomi wrote:


  
  The mailbox is too big.
  

  
  

  
  

  
  
    
    ---
    Aki Tuomi
    Dovecot oy
  
  

  
  
    -------- Original message --------
    From: Ian Bobbitt <ibobbitt@globalnoc.iu.edu> 
    Date: 23/02/2018 17:52 (GMT+02:00) 
    To: dovecot@dovecot.org 
    Subject: Assertion during dsync receive 
    

    
  
  Hi,

  

  I'm getting an assertion failed on the receiving side, causing
  syncs to fail for one user. The servers are setup so that

  only one is receiving any traffic other than replication at any
  time. The one that's only receiving replications is the

  one that's failing.

  

  I've tried deleting the user's home on the receiving server, but
  it still crashes during the sync. Oddly, the user's

  home is 7.4G on the sending server, but ends up at 42G on the
  receiving side, even after deleting and trying a fresh sync.

  

  The mailbox implicated in the backtrace ("Spam") does have a very
  large number of messages in it.

  On sender:

  Spam messages=1217764 recent=0 uidnext=1218103
  uidvalidity=1379509105 unseen=16 highestmodseq=744588
  vsize=34468460093

  guid=090ed93a7a055559abf10200fdf6807a firstsaved=1498744186

  On receiver:

  Spam messages=1217766 recent=352 uidnext=1218105
  uidvalidity=1379509105 unseen=16 highestmodseq=744589
  vsize=34468496809

  guid=090ed93a7a055559abf10200fdf6807a firstsaved=1519396172

  

  Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking
  index file

  /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index

  Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
  /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes

  Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file
  mail-index-util.c: line 10 (mail_index_uint32_to_offset):

  assertion failed: (offset < 0x40000000)

  Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw
  backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)

  [0x7feb584143de] ->
  /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) [0x7feb584144be]
  ->

  /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c]
  ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0) [0x7feb587906d0] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854) [0x7feb58774f34] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f) [0x7feb587884ff] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de) [0x7feb5870b3ae] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
  [0x7feb5870ccd8] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c)
  [0x7feb5870ce7c] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b) [0x7feb5870cf3b] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
  [0x7feb586f2834] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
  [0x7feb586f28d7] ->

  dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475)
  [0x445495] -> dovecot/doveadm-server() [0x43edc0] ->

  dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653]
  -> dovecot/doveadm-server(dsync_brain_run+0x541)

  [0x43acf1] -> dovecot/doveadm-server() [0x43b070] ->
  dovecot/doveadm-server() [0x44fe5f] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
  [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) [0x7feb5842b3bf] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
  [0x7feb58429d6c] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
  [0x7feb58429f28] -> dovecot/doveadm-server() [0x4209c5] ->

  dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server()
  [0x4377f4] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
  [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) [0x7feb5842b3bf] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
  [0x7feb58429d6c]

  Feb 23 14:57:33 dsync-local(bgeels): Fatal: master:
  service(doveadm): child 82098 killed with signal 6 (core dumped)

  

  I've attached the output of `doveconf -n` and the full backtrace
  from a core dump.

  

  Dovecot 2.2.33.2 (GhettoForge package)

  CentOS 7 x86_64

  XFS, no NFS.

  

  

  -- Ian

It is problem for any mailbox

---Aki TuomiDovecot oy -------- Original message --------From: Tanstaafl <tanstaafl@libertytrek.org> Date: 23/02/2018 21:36 (GMT+02:00) To: dovecot@dovecot.org Subject: Re: Assertion during dsync receive On Fri Feb 23 2018 13:53:27 GMT-0500 (Eastern Standard Time), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:

Once you cache grows bigger than 0x4000000 you have problems

This is for a single mailbox? IS this only a problem for mbox and maybe sdbox?

On 23 February 2018 at 23:12 Tanstaafl <tanstaafl@libertytrek.org> wrote:

On Fri Feb 23 2018 14:53:53 GMT-0500 (Eastern Standard Time), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:

...

It is problem for any mailbox

Ok, so, I'm still unclear...

what cache are we talking about? Are you saying that there is a limit to how many emails dovecot can store in a single... 'folder'?

Dovecot has cache for commonly fetched fields. This is called dovecot.index.cache. The maximum file size for the cache is about 4G, the maximum offset is 0x40000000 * 4, because all offsets are divided by 4. If the cache file becomes full, there are things you can do..

1. you can rm the cache file and hope it does not get full again too fast.
2. remove some mails

Aki

On 13.03.2018 13:40, Tanstaafl wrote:

On Sat Feb 24 2018 11:32:05 GMT-0500 (Eastern Daylight Time), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:

...

...

On 23 February 2018 at 23:12 Tanstaafl <tanstaafl@libertytrek.org> wrote: Ok, so, I'm still unclear...

what cache are we talking about? Are you saying that there is a limit to how many emails dovecot can store in a single... 'folder'? Dovecot has cache for commonly fetched fields. This is called dovecot.index.cache. The maximum file size for the cache is about 4G, the maximum offset is 0x40000000 * 4, because all offsets are divided by

4. If the cache file becomes full, there are things you can do..

5. you can rm the cache file and hope it does not get full again too fast. 2. remove some mails Ok, so... I'm *still* unclear, and please understand I'm not trying to be difficult, but I want/need to understand this. One thing not helping is I don't have a working dovecot setup I can access to go peek at the filesystem, which would probably let me answer this for myself.

When you say 'commonly fetched fields', can you confirm this means it is simply the number of emails that can cause the problem? Yes

Is this index file kept on a per *folder* basis? Or per user/account?

Yes

If it is per folder, then it is much less of a problem, and could be worked around simply by moving messages around (say, archiving). It can help. But if you are not accessing all the mails e.g. migrating them, you can usually just rm the cache file and it'll probably won't grow fast back.

Thanks again

Aki

On 13.03.2018 14:10, Tanstaafl wrote:

On Tue Mar 13 2018 07:42:31 GMT-0400 (Eastern Standard Time), Aki Tuomi <aki.tuomi@dovecot.fi> wrote:

...

...

Is this index file kept on a per *folder* basis? Or per user/account? Yes Yes... per folder? Or per user? Sorry, ment per folder. =)

...

...

If it is per folder, then it is much less of a problem, and could be worked around simply by moving messages around (say, archiving). It can help. But if you are not accessing all the mails e.g. migrating them, you can usually just rm the cache file and it'll probably won't grow fast back. So, how would doing this impact performance? If it doesn't much, then maybe a simple script to test the size and delete when it gets larger than (whatever you like)?

Thanks Aki!