fail: doveadm backup -s "<state>" with a huuge number of mailboxes
Hello,
I'm using 2.2.13.
For syncing I'm using repetitions of about the following (simplified for readability):
state_file=.s
doveadm \
-o imapc_master_user=<master> \
-o imapc_password=<pw> \
-o imapc_host=<host> \
-o mail_fsync=never \
backup -s "$(<.s)" -u "<user>" -R imapc: >.s_new
mv .s_new .sThe user has about 4501 IMAP folders. This makes a huge state output of about 128 kB. This in turn leads to an overly long command line in the next iteration. (excuse: command line too long)
Is there any chance to solve this? I think, the only possibility would be to have doveadm-backup -s read the state from a *file*, wouldn't it?
(Apart from the fact that I do not see a manpage for this, except entries in the Wiki (but this may be due to poor packaging) of the Debian dovecot package.)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 12 Nov 2015, at 12:56, Heiko Schlittermann <hs@schlittermann.de> wrote:
Hello,
I'm using 2.2.13.
For syncing I'm using repetitions of about the following (simplified for readability):
state_file=.s doveadm
-o imapc_master_user=<master>
-o imapc_password=<pw>
-o imapc_host=<host>
-o mail_fsync=never
backup -s "$(<.s)" -u "<user>" -R imapc: >.s_new mv .s_new .sThe user has about 4501 IMAP folders. This makes a huge state output of about 128 kB. This in turn leads to an overly long command line in the next iteration. (excuse: command line too long)
Is there any chance to solve this? I think, the only possibility would be to have doveadm-backup -s read the state from a *file*, wouldn't it?
Right now no way to read it from a file. But is this Linux? I thought Linux had unlimited (or at least very large) command line length nowadays.
Timo Sirainen <tss@iki.fi> (Do 12 Nov 2015 23:12:21 CET): …
The user has about 4501 IMAP folders. This makes a huge state output of about 128 kB. This in turn leads to an overly long command line in the next iteration. (excuse: command line too long)
Is there any chance to solve this? I think, the only possibility would be to have doveadm-backup -s read the state from a *file*, wouldn't it?
Right now no way to read it from a file. But is this Linux? I thought Linux had unlimited (or at least very large) command line length nowadays.
Yes, the product of size and number of the arguments is limited:
*** 189961 args ( 10 chars each) with total length 1899610
*** 99503 args ( 20 chars each) with total length 1990060
*** 2087 args ( 1000 chars each) with total length 2087000
*** 16 args (128000 chars each) with total length 2048000The upper limit for the sum of parameters seems to be ~2 MB
But,
x127=$(perl -e 'print "x" x (127 * 2**10)') # 127 KB
x128=$(perl -e 'print "x" x (128 * 2**10)') # 128 KB
/bin/echo $x127 # works
/bin/echo $x127 $x127 # works
/bin/echo $x128 # breaks
/bin/echo $x128 $x128 # breaksIt seems that a *single* command line parameter is limited to somewhat near 128 kB. The complete amount of command line data is near 2 MB.
Conclusion: doveadm sync should be able to read a state *file*. IMHO
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
participants (2)
-
Heiko Schlittermann
-
Timo Sirainen