doveadm sync out of memory

Casey Stone

6 Feb 2015 6 Feb '15

12:39 a.m.

Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help.

Show replies by date

Casey Stone

13 Feb 13 Feb

5:19 p.m.

On Feb 5, 2015, at 10:39 PM, Casey Stone tcstone@caseystone.com wrote:

...

Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help.

No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<>>

(end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied.

FUSTE Emmanuel

5:42 p.m.

Le 13/02/2015 16:19, Casey Stone a écrit :

...

On Feb 5, 2015, at 10:39 PM, Casey Stone tcstone@caseystone.com wrote:

...
Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help. No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<>>

(end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. To use dsync, use Dovecot 2.2.15 to avoid any trouble. Use apt.dovecot.fi repository.

Emmanuel.

Casey Stone

16 Feb 16 Feb

9:40 p.m.

On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fuste@thalesgroup.com wrote:

...

Le 13/02/2015 16:19, Casey Stone a écrit :

...
On Feb 5, 2015, at 10:39 PM, Casey Stone tcstone@caseystone.com wrote:

...
Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help. No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<>>

(end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. To use dsync, use Dovecot 2.2.15 to avoid any trouble. Use apt.dovecot.fi repository.

Emmanuel.

At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running.

I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo?

By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying).

Thanks.

FUSTE Emmanuel

17 Feb 17 Feb

12:51 p.m.

Le 16/02/2015 20:40, Casey Stone a écrit :

...

On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fuste@thalesgroup.com wrote:

...
Le 13/02/2015 16:19, Casey Stone a écrit :

...
On Feb 5, 2015, at 10:39 PM, Casey Stone tcstone@caseystone.com wrote:

...
Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help. No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<>>

(end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. To use dsync, use Dovecot 2.2.15 to avoid any trouble. Use apt.dovecot.fi repository.

Emmanuel. At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running.

I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo?

By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying).

Thanks. Yes, it is safe. You dont need to remove dovecot-core etc before installing from the new repo. Apt will do it for you automaticaly.

Emmanuel.

Casey Stone

18 Feb 18 Feb

11:07 p.m.

On Feb 17, 2015, at 10:51 AM, FUSTE Emmanuel emmanuel.fuste@thalesgroup.com wrote:

...

Le 16/02/2015 20:40, Casey Stone a écrit :

...
On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel emmanuel.fuste@thalesgroup.com wrote:

...
Le 13/02/2015 16:19, Casey Stone a écrit :

...
On Feb 5, 2015, at 10:39 PM, Casey Stone tcstone@caseystone.com wrote:

...
Hello:

I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync.

I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined.

Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command:

doveadm sync -R tcps:mainserver.example.com:12345

Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user.

The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug?

Thanks for your help. No repsonses :-(

Here is what it looks like when it crashes with an out of memory error:

(start of the run) Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1

<<>>

(end of the run) Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump)

I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. To use dsync, use Dovecot 2.2.15 to avoid any trouble. Use apt.dovecot.fi repository.

Emmanuel. At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running.

I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo?

By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying).

Thanks. Yes, it is safe. You dont need to remove dovecot-core etc before installing from the new repo. Apt will do it for you automaticaly.

Emmanuel.

I made the change to dovecot-ee from the official dovecot.fi repo. Here's the simulation I made before doing it:

# apt-get -s install dovecot-ee-core dovecot-ee-imapd dovecot-ee-pop3d dovecot-ee-lmtpd dovecot-ee-sieve dovecot-ee-managesieved Reading package lists... Done Building dependency tree
Reading state information... Done Suggested packages: dovecot-ee-gssapi dovecot-ee-pgsql dovecot-ee-mysql dovecot-ee-sqlite dovecot-ee-ldap dovecot-ee-solr dovecot-ee-lucene The following packages will be REMOVED dovecot-core dovecot-imapd dovecot-managesieved dovecot-pop3d dovecot-sieve mail-stack-delivery The following NEW packages will be installed dovecot-ee-core dovecot-ee-imapd dovecot-ee-lmtpd dovecot-ee-managesieved dovecot-ee-pop3d dovecot-ee-sieve 0 to upgrade, 6 to newly install, 6 to remove and 10 not to upgrade. Remv mail-stack-delivery [1:2.2.9-1ubuntu2.1] Remv dovecot-managesieved [1:2.2.9-1ubuntu2.1] Remv dovecot-sieve [1:2.2.9-1ubuntu2.1] Remv dovecot-core [1:2.2.9-1ubuntu2.1] [dovecot-pop3d:amd64 dovecot-imapd:amd64 ] Remv dovecot-imapd [1:2.2.9-1ubuntu2.1] [dovecot-pop3d:amd64 ] Remv dovecot-pop3d [1:2.2.9-1ubuntu2.1] Inst dovecot-ee-core (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Inst dovecot-ee-imapd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Inst dovecot-ee-lmtpd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Inst dovecot-ee-sieve (0.4.6-2 Dovecot Enterprise Edition:stable [amd64]) Inst dovecot-ee-managesieved (0.4.6-2 Dovecot Enterprise Edition:stable [amd64]) Inst dovecot-ee-pop3d (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-core (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-imapd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-lmtpd (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-sieve (0.4.6-2 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-managesieved (0.4.6-2 Dovecot Enterprise Edition:stable [amd64]) Conf dovecot-ee-pop3d (1:2.2.15.8-1 Dovecot Enterprise Edition:stable [amd64])

There was a problem though, Ubuntu-related I guess, maybe due to removing the package mail-stack-delivery. It changed my /etc/postfix/main.cf config! This rendered my server inoperable of course. It changed the SSL certs back to snakeoil, changed home_mailbox to blank, changed SASL type to Cryus among other changes. It did not completely replace my main.cf file though, just made some changes!

So, that's horrible. Be sure to backup your config files before doing this if you feel like trying it.

I'll follow up on whether it fixes any memory or syncing issues for me.

3524

Age (days ago)

3537

Last active (days ago)

List overview

5 comments

2 participants

participants (2)

Casey Stone
FUSTE Emmanuel