doveadm sync out of memory

[Casey Stone]

On Feb 17, 2015, at 10:51 AM, FUSTE Emmanuel <emmanuel.fuste at thalesgroup.com> wrote:

> Le 16/02/2015 20:40, Casey Stone a écrit :
>> On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel <emmanuel.fuste at thalesgroup.com> wrote:
>> 
>>> Le 13/02/2015 16:19, Casey Stone a écrit :
>>>> On Feb 5, 2015, at 10:39 PM, Casey Stone <tcstone at caseystone.com> wrote:
>>>> 
>>>>> Hello:
>>>>> 
>>>>> I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.
>>>>> 
>>>>> I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.
>>>>> 
>>>>> Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:
>>>>> 
>>>>> doveadm sync -R tcps:mainserver.example.com:12345
>>>>> 
>>>>> Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.
>>>>> 
>>>>> The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?
>>>>> 
>>>>> Thanks for your help.
>>>> No repsonses :-(
>>>> 
>>>> Here is what it looks like when it crashes with an out of memory error:
>>>> 
>>>> (start of the run)
>>>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone
>>>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$
>>>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$
>>>> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$
>>>> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1
>>>> 
>>>> <<<many, many more brain messages>>>
>>>> 
>>>> (end of the run)
>>>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory
>>>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb]
>>>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)
>>>> 
>>>> I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied.
>>> To use dsync, use Dovecot 2.2.15 to avoid any trouble.
>>> Use apt.dovecot.fi repository.
>>> 
>>> Emmanuel.
>> At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running.
>> 
>> I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo?
>> 
>> By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying).
>> 
>> Thanks.
> Yes, it is safe. You dont need to remove dovecot-core etc before 
> installing  from the new repo. Apt will do it for you automaticaly.
> 
> Emmanuel.

I made the change to dovecot-ee from the official dovecot.fi repo. Here's the simulation I made before doing it:

# apt-get -s install dovecot-ee-core dovecot-ee-imapd dovecot-ee-pop3d dovecot-ee-lmtpd dovecot-ee-sieve dovecot-ee-managesieved
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  dovecot-ee-gssapi dovecot-ee-pgsql dovecot-ee-mysql dovecot-ee-sqlite
  dovecot-ee-ldap dovecot-ee-solr dovecot-ee-lucene
The following packages will be REMOVED
  dovecot-core dovecot-imapd dovecot-managesieved dovecot-pop3d dovecot-sieve
  mail-stack-delivery
The following NEW packages will be installed
  dovecot-ee-core dovecot-ee-imapd dovecot-ee-lmtpd dovecot-ee-managesieved
  dovecot-ee-pop3d dovecot-ee-sieve
0 to upgrade, 6 to newly install, 6 to remove and 10 not to upgrade.
Remv mail-stack-delivery [1:2.2.9-1ubuntu2.1]
Remv dovecot-managesieved [1:2.2.9-1ubuntu2.1]
Remv dovecot-sieve [1:2.2.9-1ubuntu2.1]
Remv dovecot-core [1:2.2.9-1ubuntu2.1] [dovecot-pop3d:amd64 dovecot-imapd:amd64 ]
Remv dovecot-imapd [1:2.2.9-1ubuntu2.1] [dovecot-pop3d:amd64 ]
Remv dovecot-pop3d [1:2.2.9-1ubuntu2.1]
Inst dovecot-ee-core (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Inst dovecot-ee-imapd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Inst dovecot-ee-lmtpd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Inst dovecot-ee-sieve (0.4.6-2 Dovecot Enterprise Edition:stable [amd64])
Inst dovecot-ee-managesieved (0.4.6-2 Dovecot Enterprise Edition:stable [amd64])
Inst dovecot-ee-pop3d (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-core (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-imapd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-lmtpd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-sieve (0.4.6-2 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-managesieved (0.4.6-2 Dovecot Enterprise Edition:stable [amd64])
Conf dovecot-ee-pop3d (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])

There was a problem though, Ubuntu-related I guess, maybe due to removing the package mail-stack-delivery. It changed my /etc/postfix/main.cf config! This rendered my server inoperable of course. It changed the SSL certs back to snakeoil, changed home_mailbox to blank, changed SASL type to Cryus among other changes. It did not completely replace my main.cf file though, just made some changes!

So, that's horrible. Be sure to backup your config files before doing this if you feel like trying it. 

I'll follow up on whether it fixes any memory or syncing issues for me.