[Dovecot] pop3c migration?
Hi,
I'm running dsync migrations using imapc and the source IMAP server is just too slow. It has taken 8 hours to migrate a mailbox with 47,000 messages. It seems most of the mailboxes are never accessed with IMAP, so it would be just as good to migrate them via POP3 which should work faster in my case.
Is this actually meant to work?
doveadm -D -o pop3c_user=xxxxxx@example.com -o 'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u yyyyy@example.com -R pop3c:
gives:
doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_init (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(yyyyy@example.com): Debug: Added userdb setting: mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: Added userdb setting: plugin/quota_rule=*:storage=5120M doveadm(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy doveadm(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: doveadm(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= doveadm(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: maildir++: root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt= dsync(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy dsync(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= Segmentation fault
remembering something about that... :
# mkdir /tmp/pop3c # chown vmail:vmail /tmp/pop3c/ # doveadm -D -o pop3c_user=xxxxxx@example.com -o 'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u yyyyy@example.com -R pop3c:/tmp/pop3c doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_init (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(yyyyy@example.com): Debug: Added userdb setting: mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: Added userdb setting: plugin/quota_rule=*:storage=5120M doveadm(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy doveadm(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: doveadm(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= doveadm(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: maildir++: root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt= dsync(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy dsync(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c:/tmp/pop3c dsync(yyyyy@example.com): Debug: fs: root=/tmp/pop3c/., index=/tmp/pop3c, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /tmp/pop3c/.: mode=0755 gid=-1 dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /stor/real/domains/example.com/yy/yyyyy/Maildir: mode=0700 gid=-1 dsync(yyyyy@example.com): Debug: Namespace : /tmp/pop3c/./INBOX doesn't exist yet, using default permissions dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /tmp/pop3c/.: mode=0755 gid=-1 dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Looking up IP address dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Connecting to 120.121.122.123:110 dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Authenticating as xxxxxx@example.com dsync(yyyyy@example.com): Panic: file mail-storage.c: line 1311 (mailbox_get_metadata): assertion failed: ((items & MAILBOX_METADATA_GUID) == 0 || !guid_128_is_empty(metadata_r->guid)) dsync(yyyyy@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x45caa) [0xb761ccaa] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) [0xb761cdc7] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb761d647] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x120) [0xb76f3040] -> doveadm() [0x80740bc] -> doveadm(dsync_worker_mailbox_iter_next+0x21) [0x8071c11] -> doveadm() [0x806c1f7] -> doveadm(dsync_brain_sync+0x6c0) [0x806b890] -> doveadm(dsync_brain_sync_all+0x18) [0x806c4f8] -> doveadm() [0x806961b] -> doveadm() [0x8069bbf] -> doveadm() [0x80541ec] -> doveadm(doveadm_mail_try_run+0x257) [0x8054ee7] -> doveadm(main+0x391) [0x805d0c1] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb748ae46] -> doveadm() [0x8054001] Aborted
Now I'm lost... Am I in the right direction?
Thanks,
Gedalya
# dovecot -n # 2.1.17: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-686-pae i686 Debian 7.1 auth_default_realm = example.com auth_mechanisms = plain login cram-md5 auth_verbose = yes auth_verbose_passwords = plain default_client_limit = 8096 dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no imapc_host = mail01.example.com mail_gid = vmail mail_plugins = listescape mail_prefetch_count = 50 mail_uid = vmail namespace { hidden = yes list = no location = pop3c: prefix = POP3-MIGRATION-NS/ } namespace inbox { inbox = yes location = prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash expire2 = Trash.* expire3 = Junk expire_dict = proxy::expire pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX sieve = ~/.dovecot.sieve sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve sieve_dir = ~/sieve } pop3c_host = 120.121.122.123 postmaster_address = postmaster@example.com service auth-worker { user = $default_internal_user } service auth { client_limit = 20480 } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 10240 } service lmtp { inet_listener lmtp { address = 0.0.0.0 port = 7025 } } service pop3-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 4096 } ssl_cert =
/etc/dovecot/dovecot-sql.conf.ext : driver = mysql connect = ..... default_pass_scheme = PLAIN
password_query = SELECT password,
'/stor/real/domains/%d/%2n/%n' AS userdb_home,
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as userdb_mail,
concat('*:storage=', quota_mb, 'M') as userdb_quota_rule,
'vmail' AS userdb_uid, 'vmail' AS userdb_gid
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
user_query = SELECT '/stor/real/domains/%d/%2n/%n' AS home,
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as mail,
concat('*:storage=', quota_mb, 'M') as quota_rule,
'vmail' AS uid, 'vmail' AS gid
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
iterate_query = SELECT userid as user, domain FROM email where deleted=0
OK so now I tried a doveadm backup -R with pop3c using dovecot 2.2.4 (50117c22151e) (auto build) and it works fine, and indeed very fast.
Funny, I ran into a segmentation fault by mistakenly passing pop3c_password instead of imapc_password, thought it would be worth posting.
I'm using the cutting-edge dovecot over NFS storing to the production server running 2.1.7. I'll be seeing "Corrupted index cache file .. Invalid magic in hole header" when every account logs in for the first time post-migration. Is this OK? I'm using Maildir.
Segfault:
Starting program: /usr/bin/doveadm -D -o imapc_user=xxxx@example.com -o pop3c_password=******* -o imapc_host=mail01.example.com import -s -u xxxx@example.com imapc: '' NOT mailbox INBOX [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1". doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_backend_rescan (this is usually intentional, so just ignore this message) doveadm(root): Debug: Effective uid=0, gid=0, home=/root
Program received signal SIGSEGV, Segmentation fault.
imapc_storage_client_unref (_client=_client@entry=0x80c3130) at
imapc-storage.c:256
256 imapc-storage.c: No such file or directory.
(gdb) bt full
#0 imapc_storage_client_unref (_client=_client@entry=0x80c3130) at
imapc-storage.c:256
client = 0x0
cb = <optimized out>
__FUNCTION__ = "imapc_storage_client_unref"
#1 0xb7ed4660 in imapc_list_deinit (_list=0x80c3028) at imapc-list.c:106
list = 0x80c3028
#2 0xb7ef5c44 in mailbox_list_create (driver=driver@entry=0xb7f57ad2
"imapc", ns=ns@entry=0x80c2f78, set=set@entry=0xbffff854,
flags=MAILBOX_LIST_FLAG_NO_MAIL_FILES,
list_r=list_r@entry=0xbffff844, error_r=error_r@entry=0xbffff928)
at mailbox-list.c:184
list = 0x80c3028
idx = 6
__FUNCTION__ = "mailbox_list_create"
#3 0xb7eedadb in mail_storage_create_full (ns=ns@entry=0x80c2f78,
driver=<optimized out>, driver@entry=0x0, data=<optimized out>,
flags=flags@entry=0, storage_r=storage_r@entry=0xbffff8cc,
error_r=error_r@entry=0xbffff928) at mail-storage.c:356
storage_class = 0xb7f8d060
storage = 0x0
list = 0xbffff880
list_set = {layout = 0xb7f57ad2 "imapc", root_dir = 0x0,
index_dir = 0x0, index_pvt_dir = 0x0, control_dir = 0x0, alt_dir = 0x0,
inbox_path = 0x0, subscription_fname = 0x0,
maildir_name = 0xb7f52fbc "", mailbox_dir_name = 0xb7f52fbc
"", escape_char = 37 '%', broken_char = 0 '\000', utf8 = false,
alt_dir_nocheck = false}
list_flags = <optimized out>
p = <optimized out>
__FUNCTION__ = "mail_storage_create_full"
#4 0xb7eede35 in mail_storage_create (ns=ns@entry=0x80c2f78,
driver=driver@entry=0x0, flags=flags@entry=0,
error_r=error_r@entry=0xbffff928) at mail-storage.c:407
storage = 0xb7eeddf0
#5 0xb7ee788f in mail_namespaces_init_location (user=0x80bef90,
location=location@entry=0x80b12ab "imapc:",
error_r=error_r@entry=0xbffff978) at mail-namespace.c:460
inbox_set = 0x80bfd50
unexpanded_inbox_set = 0x80bfd80
ns = 0x80c2f78
mail_set = 0x80bf638
error = <optimized out>
driver = <optimized out>
location_source = <optimized out>
default_location = <optimized out>
__FUNCTION__ = "mail_namespaces_init_location"
#6 0x0805b072 in cmd_import_init (_ctx=0x80b4ed8, args=0x80b1200) at
doveadm-mail-import.c:179
ctx = 0x80b4ed8
input = {module = 0x808e8e5 "mail", service = 0x0, username =
0x8092f03 "doveadm", session_id = 0x0, local_ip = {family = 0, u = {ip6
= {__in6_u = {__u6_addr8 = '\000'
On 21.7.2013, at 14.06, Gedalya gedalya@gedalya.net wrote:
OK so now I tried a doveadm backup -R with pop3c using dovecot 2.2.4 (50117c22151e) (auto build) and it works fine, and indeed very fast.
Funny, I ran into a segmentation fault by mistakenly passing pop3c_password instead of imapc_password, thought it would be worth posting.
Fixed: http://hg.dovecot.org/dovecot-2.2/rev/1a5d92b8d3d5
I'm using the cutting-edge dovecot over NFS storing to the production server running 2.1.7. I'll be seeing "Corrupted index cache file .. Invalid magic in hole header" when every account logs in for the first time post-migration. Is this OK? I'm using Maildir.
You should run at least v2.1.11, which adds support for v2.2 index changes. Now when those errors are happening, the entire cache file gets deleted and the performance gets worse.
participants (2)
-
Gedalya
-
Timo Sirainen