doveadm sync out of memory

[Casey Stone]

On Feb 5, 2015, at 10:39 PM, Casey Stone <tcstone at caseystone.com> wrote:

> Hello:
> 
> I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. 
> 
> I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.
> 
> Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:
> 
> doveadm sync -R tcps:mainserver.example.com:12345
> 
> Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.
> 
> The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?
> 
> Thanks for your help.

No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run)
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$
Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$
Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<<many, many more brain messages>>>

(end of the run)
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb]
Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied.