[Dovecot] pop3c migration?
Hi,
I'm running dsync migrations using imapc and the source IMAP server is just too slow. It has taken 8 hours to migrate a mailbox with 47,000 messages. It seems most of the mailboxes are never accessed with IMAP, so it would be just as good to migrate them via POP3 which should work faster in my case.
Is this actually meant to work?
doveadm -D -o pop3c_user=xxxxxx@example.com -o 'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u yyyyy@example.com -R pop3c:
gives:
doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_init (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(yyyyy@example.com): Debug: Added userdb setting: mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: Added userdb setting: plugin/quota_rule=*:storage=5120M doveadm(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy doveadm(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: doveadm(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= doveadm(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: maildir++: root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt= dsync(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy dsync(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= Segmentation fault
remembering something about that... :
# mkdir /tmp/pop3c # chown vmail:vmail /tmp/pop3c/ # doveadm -D -o pop3c_user=xxxxxx@example.com -o 'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u yyyyy@example.com -R pop3c:/tmp/pop3c doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_init (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(yyyyy@example.com): Debug: Added userdb setting: mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: Added userdb setting: plugin/quota_rule=*:storage=5120M doveadm(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy doveadm(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: doveadm(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= doveadm(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir doveadm(yyyyy@example.com): Debug: maildir++: root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt= dsync(yyyyy@example.com): Debug: Effective uid=1001, gid=1001, home=/stor/real/domains/example.com/yy/yyyyy dsync(yyyyy@example.com): Debug: Namespace : type=private, prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, subscriptions=yes location=pop3c: dsync(yyyyy@example.com): Debug: fs: root=, index=, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c:/tmp/pop3c dsync(yyyyy@example.com): Debug: fs: root=/tmp/pop3c/., index=/tmp/pop3c, control=, inbox=, alt= dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /tmp/pop3c/.: mode=0755 gid=-1 dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /stor/real/domains/example.com/yy/yyyyy/Maildir: mode=0700 gid=-1 dsync(yyyyy@example.com): Debug: Namespace : /tmp/pop3c/./INBOX doesn't exist yet, using default permissions dsync(yyyyy@example.com): Debug: Namespace : Using permissions from /tmp/pop3c/.: mode=0755 gid=-1 dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Looking up IP address dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Connecting to 120.121.122.123:110 dsync(yyyyy@example.com): Debug: pop3c(120.121.122.123): Authenticating as xxxxxx@example.com dsync(yyyyy@example.com): Panic: file mail-storage.c: line 1311 (mailbox_get_metadata): assertion failed: ((items & MAILBOX_METADATA_GUID) == 0 || !guid_128_is_empty(metadata_r->guid)) dsync(yyyyy@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x45caa) [0xb761ccaa] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) [0xb761cdc7] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb761d647] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x120) [0xb76f3040] -> doveadm() [0x80740bc] -> doveadm(dsync_worker_mailbox_iter_next+0x21) [0x8071c11] -> doveadm() [0x806c1f7] -> doveadm(dsync_brain_sync+0x6c0) [0x806b890] -> doveadm(dsync_brain_sync_all+0x18) [0x806c4f8] -> doveadm() [0x806961b] -> doveadm() [0x8069bbf] -> doveadm() [0x80541ec] -> doveadm(doveadm_mail_try_run+0x257) [0x8054ee7] -> doveadm(main+0x391) [0x805d0c1] -> /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb748ae46] -> doveadm() [0x8054001] Aborted
Now I'm lost... Am I in the right direction?
Thanks,
Gedalya
# dovecot -n # 2.1.17: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-686-pae i686 Debian 7.1 auth_default_realm = example.com auth_mechanisms = plain login cram-md5 auth_verbose = yes auth_verbose_passwords = plain default_client_limit = 8096 dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no imapc_host = mail01.example.com mail_gid = vmail mail_plugins = listescape mail_prefetch_count = 50 mail_uid = vmail namespace { hidden = yes list = no location = pop3c: prefix = POP3-MIGRATION-NS/ } namespace inbox { inbox = yes location = prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash expire2 = Trash.* expire3 = Junk expire_dict = proxy::expire pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX sieve = ~/.dovecot.sieve sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve sieve_dir = ~/sieve } pop3c_host = 120.121.122.123 postmaster_address = postmaster@example.com service auth-worker { user = $default_internal_user } service auth { client_limit = 20480 } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 10240 } service lmtp { inet_listener lmtp { address = 0.0.0.0 port = 7025 } } service pop3-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 4096 } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } verbose_proctitle = yes protocol doveadm { mail_plugins = listescape pop3_migration } protocol lmtp { mail_plugins = listescape sieve } protocol imap { mail_max_userip_connections = 25 mail_plugins = listescape imap_quota ssl_cert = </etc/dovecot/imap.example.com.crt ssl_key = </etc/dovecot/imap.example.com.key } protocol pop3 { mail_max_userip_connections = 10 mail_plugins = listescape pop3_fast_size_lookups = yes pop3_no_flag_updates = yes pop3_reuse_xuidl = no pop3_save_uidl = yes pop3_uidl_format = %08Xu%08Xv ssl_cert = </etc/dovecot/pop.example.com.crt ssl_key = </etc/dovecot/pop.example.com.key }
/etc/dovecot/dovecot-sql.conf.ext : driver = mysql connect = ..... default_pass_scheme = PLAIN
password_query = SELECT password,
'/stor/real/domains/%d/%2n/%n' AS userdb_home,
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as userdb_mail,
concat('*:storage=', quota_mb, 'M') as userdb_quota_rule,
'vmail' AS userdb_uid, 'vmail' AS userdb_gid
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
user_query = SELECT '/stor/real/domains/%d/%2n/%n' AS home,
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as mail,
concat('*:storage=', quota_mb, 'M') as quota_rule,
'vmail' AS uid, 'vmail' AS gid
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
iterate_query = SELECT userid as user, domain FROM email where deleted=0
OK so now I tried a doveadm backup -R with pop3c using dovecot 2.2.4 (50117c22151e) (auto build) and it works fine, and indeed very fast.
Funny, I ran into a segmentation fault by mistakenly passing pop3c_password instead of imapc_password, thought it would be worth posting.
I'm using the cutting-edge dovecot over NFS storing to the production server running 2.1.7. I'll be seeing "Corrupted index cache file .. Invalid magic in hole header" when every account logs in for the first time post-migration. Is this OK? I'm using Maildir.
Segfault:
Starting program: /usr/bin/doveadm -D -o imapc_user=xxxx@example.com -o pop3c_password=******* -o imapc_host=mail01.example.com import -s -u xxxx@example.com imapc: '' NOT mailbox INBOX [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1". doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_backend_rescan (this is usually intentional, so just ignore this message) doveadm(root): Debug: Effective uid=0, gid=0, home=/root
Program received signal SIGSEGV, Segmentation fault. imapc_storage_client_unref (_client=_client@entry=0x80c3130) at imapc-storage.c:256 256 imapc-storage.c: No such file or directory. (gdb) bt full #0 imapc_storage_client_unref (_client=_client@entry=0x80c3130) at imapc-storage.c:256 client = 0x0 cb = <optimized out> __FUNCTION__ = "imapc_storage_client_unref" #1 0xb7ed4660 in imapc_list_deinit (_list=0x80c3028) at imapc-list.c:106 list = 0x80c3028 #2 0xb7ef5c44 in mailbox_list_create (driver=driver@entry=0xb7f57ad2 "imapc", ns=ns@entry=0x80c2f78, set=set@entry=0xbffff854, flags=MAILBOX_LIST_FLAG_NO_MAIL_FILES, list_r=list_r@entry=0xbffff844, error_r=error_r@entry=0xbffff928) at mailbox-list.c:184 list = 0x80c3028 idx = 6 __FUNCTION__ = "mailbox_list_create" #3 0xb7eedadb in mail_storage_create_full (ns=ns@entry=0x80c2f78, driver=<optimized out>, driver@entry=0x0, data=<optimized out>, flags=flags@entry=0, storage_r=storage_r@entry=0xbffff8cc, error_r=error_r@entry=0xbffff928) at mail-storage.c:356 storage_class = 0xb7f8d060 storage = 0x0 list = 0xbffff880 list_set = {layout = 0xb7f57ad2 "imapc", root_dir = 0x0, index_dir = 0x0, index_pvt_dir = 0x0, control_dir = 0x0, alt_dir = 0x0, inbox_path = 0x0, subscription_fname = 0x0, maildir_name = 0xb7f52fbc "", mailbox_dir_name = 0xb7f52fbc "", escape_char = 37 '%', broken_char = 0 '\000', utf8 = false, alt_dir_nocheck = false} list_flags = <optimized out> p = <optimized out> __FUNCTION__ = "mail_storage_create_full" #4 0xb7eede35 in mail_storage_create (ns=ns@entry=0x80c2f78, driver=driver@entry=0x0, flags=flags@entry=0, error_r=error_r@entry=0xbffff928) at mail-storage.c:407 storage = 0xb7eeddf0 #5 0xb7ee788f in mail_namespaces_init_location (user=0x80bef90, location=location@entry=0x80b12ab "imapc:", error_r=error_r@entry=0xbffff978) at mail-namespace.c:460 inbox_set = 0x80bfd50 unexpanded_inbox_set = 0x80bfd80 ns = 0x80c2f78 mail_set = 0x80bf638 error = <optimized out> driver = <optimized out> location_source = <optimized out> default_location = <optimized out> __FUNCTION__ = "mail_namespaces_init_location" #6 0x0805b072 in cmd_import_init (_ctx=0x80b4ed8, args=0x80b1200) at doveadm-mail-import.c:179 ctx = 0x80b4ed8 input = {module = 0x808e8e5 "mail", service = 0x0, username = 0x8092f03 "doveadm", session_id = 0x0, local_ip = {family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, local_port = 0, remote_port = 0, userdb_fields = 0x0, flags_override_add = 2056, flags_override_remove = MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP, no_userdb_lookup = 0} service_user = 0x80b8e90 user = 0x80bef90 ---Type <return> to continue, or q <return> to quit--- src_location = 0x80b12ab "imapc:" error = 0x4 <Address 0x4 out of bounds> #7 0x080575f8 in doveadm_mail_single_user (ctx=0x80b4ed8, input=0xbffffa6c, error_r=0xbffffa64) at doveadm-mail.c:328 __FUNCTION__ = "doveadm_mail_single_user" #8 0x08057c37 in doveadm_mail_cmd (argv=0x80b1200, argc=<optimized out>, cmd=0x80b4388) at doveadm-mail.c:516 input = {module = 0x0, service = 0x8092f03 "doveadm", username = 0x80b1299 "xxxx@example.com", session_id = 0x0, local_ip = {family = 0, u = {ip6 = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = { __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, local_port = 0, remote_port = 0, userdb_fields = 0x0, flags_override_add = 0, flags_override_remove = 0, no_userdb_lookup = 0} ctx = 0x80b4ed8 getopt_args = <optimized out> ret = <optimized out> c = <optimized out> wildcard_user = 0x0 error = <optimized out> #9 doveadm_mail_try_run (cmd_name=cmd_name@entry=0x80b128c "import", argc=argc@entry=9, argv=argv@entry=0x80b11f0) at doveadm-mail.c:608 cmd = 0x80b4388 #10 0x08056b26 in main (argc=9, argv=0x80b11f0) at doveadm.c:398 cmd_name = <optimized out> i = <optimized out> quick_init = false c = <optimized out>
On 21.7.2013, at 14.06, Gedalya <gedalya@gedalya.net> wrote:
OK so now I tried a doveadm backup -R with pop3c using dovecot 2.2.4 (50117c22151e) (auto build) and it works fine, and indeed very fast.
Funny, I ran into a segmentation fault by mistakenly passing pop3c_password instead of imapc_password, thought it would be worth posting.
Fixed: http://hg.dovecot.org/dovecot-2.2/rev/1a5d92b8d3d5
I'm using the cutting-edge dovecot over NFS storing to the production server running 2.1.7. I'll be seeing "Corrupted index cache file .. Invalid magic in hole header" when every account logs in for the first time post-migration. Is this OK? I'm using Maildir.
You should run at least v2.1.11, which adds support for v2.2 index changes. Now when those errors are happening, the entire cache file gets deleted and the performance gets worse.
participants (2)
-
Gedalya
-
Timo Sirainen