[Dovecot] pop3c migration?

Gedalya gedalya at gedalya.net
Wed Jul 17 13:36:12 EEST 2013 

Hi,

I'm running dsync migrations using imapc and the source IMAP server is 
just too slow. It has taken 8 hours to migrate a mailbox with 47,000 
messages. It seems most of the mailboxes are never accessed with IMAP, 
so it would be just as good to migrate them via POP3 which should work 
faster in my case.

Is this actually meant to work?

doveadm -D -o pop3c_user=xxxxxx at example.com -o 'pop3c_password=*******' 
-o pop3c_host=120.121.122.123 backup -u yyyyy at example.com -R pop3c:

gives:

doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded: 
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined 
symbol: acl_user_module (this is usually intentional, so just ignore 
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: 
undefined symbol: expire_set_init (this is usually intentional, so just 
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: 
undefined symbol: quota_user_module (this is usually intentional, so 
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined 
symbol: i_stream_create_deflate (this is usually intentional, so just 
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined 
symbol: fts_list_backend (this is usually intentional, so just ignore 
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting: 
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting: 
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001, 
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private, 
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
doveadm(yyyyy at example.com): Debug: Namespace inbox: type=private, 
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes 
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++: 
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, 
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt=
dsync(yyyyy at example.com): Debug: Effective uid=1001, gid=1001, 
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private, 
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=, 
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
Segmentation fault


remembering something about that... :

# mkdir /tmp/pop3c
# chown vmail:vmail /tmp/pop3c/
# doveadm -D -o pop3c_user=xxxxxx at example.com -o 
'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u 
yyyyy at example.com -R pop3c:/tmp/pop3c
doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded: 
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory: 
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined 
symbol: acl_user_module (this is usually intentional, so just ignore 
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: 
undefined symbol: expire_set_init (this is usually intentional, so just 
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: 
undefined symbol: quota_user_module (this is usually intentional, so 
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined 
symbol: i_stream_create_deflate (this is usually intentional, so just 
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined 
symbol: fts_list_backend (this is usually intentional, so just ignore 
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting: 
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting: 
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001, 
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private, 
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
doveadm(yyyyy at example.com): Debug: Namespace inbox: type=private, 
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes 
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++: 
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=, 
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt=
dsync(yyyyy at example.com): Debug: Effective uid=1001, gid=1001, 
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private, 
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=, 
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes 
location=pop3c:/tmp/pop3c
dsync(yyyyy at example.com): Debug: fs: root=/tmp/pop3c/., 
index=/tmp/pop3c, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from 
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from 
/stor/real/domains/example.com/yy/yyyyy/Maildir: mode=0700 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : /tmp/pop3c/./INBOX doesn't 
exist yet, using default permissions
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from 
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Looking up IP 
address
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Connecting to 
120.121.122.123:110
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Authenticating 
as xxxxxx at example.com
dsync(yyyyy at example.com): Panic: file mail-storage.c: line 1311 
(mailbox_get_metadata): assertion failed: ((items & 
MAILBOX_METADATA_GUID) == 0 || !guid_128_is_empty(metadata_r->guid))
dsync(yyyyy at example.com): Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(+0x45caa) [0xb761ccaa] -> 
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) 
[0xb761cdc7] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb761d647] 
-> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x120) 
[0xb76f3040] -> doveadm() [0x80740bc] -> 
doveadm(dsync_worker_mailbox_iter_next+0x21) [0x8071c11] -> doveadm() 
[0x806c1f7] -> doveadm(dsync_brain_sync+0x6c0) [0x806b890] -> 
doveadm(dsync_brain_sync_all+0x18) [0x806c4f8] -> doveadm() [0x806961b] 
-> doveadm() [0x8069bbf] -> doveadm() [0x80541ec] -> 
doveadm(doveadm_mail_try_run+0x257) [0x8054ee7] -> doveadm(main+0x391) 
[0x805d0c1] -> 
/lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) 
[0xb748ae46] -> doveadm() [0x8054001]
Aborted


Now I'm lost...
Am I in the right direction?

Thanks,

Gedalya


# dovecot -n
# 2.1.17: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-686-pae i686 Debian 7.1
auth_default_realm = example.com
auth_mechanisms = plain login cram-md5
auth_verbose = yes
auth_verbose_passwords = plain
default_client_limit = 8096
dict {
   expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
   quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
imapc_host = mail01.example.com
mail_gid = vmail
mail_plugins = listescape
mail_prefetch_count = 50
mail_uid = vmail
namespace {
   hidden = yes
   list = no
   location = pop3c:
   prefix = POP3-MIGRATION-NS/
}
namespace inbox {
   inbox = yes
   location =
   prefix =
   separator = /
   type = private
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
plugin {
   expire = Trash
   expire2 = Trash.*
   expire3 = Junk
   expire_dict = proxy::expire
   pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX
   sieve = ~/.dovecot.sieve
   sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve
   sieve_dir = ~/sieve
}
pop3c_host = 120.121.122.123
postmaster_address = postmaster at example.com
service auth-worker {
   user = $default_internal_user
}
service auth {
   client_limit = 20480
}
service dict {
   unix_listener dict {
     mode = 0600
     user = vmail
   }
}
service imap-login {
   process_min_avail = 4
   service_count = 0
   vsz_limit = 128 M
}
service imap {
   process_limit = 10240
}
service lmtp {
   inet_listener lmtp {
     address = 0.0.0.0
     port = 7025
   }
}
service pop3-login {
   process_min_avail = 4
   service_count = 0
   vsz_limit = 128 M
}
service pop3 {
   process_limit = 4096
}
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
   driver = prefetch
}
userdb {
   args = /etc/dovecot/dovecot-sql.conf.ext
   driver = sql
}
verbose_proctitle = yes
protocol doveadm {
   mail_plugins = listescape pop3_migration
}
protocol lmtp {
   mail_plugins = listescape sieve
}
protocol imap {
   mail_max_userip_connections = 25
   mail_plugins = listescape imap_quota
   ssl_cert = </etc/dovecot/imap.example.com.crt
   ssl_key = </etc/dovecot/imap.example.com.key
}
protocol pop3 {
   mail_max_userip_connections = 10
   mail_plugins = listescape
   pop3_fast_size_lookups = yes
   pop3_no_flag_updates = yes
   pop3_reuse_xuidl = no
   pop3_save_uidl = yes
   pop3_uidl_format = %08Xu%08Xv
   ssl_cert = </etc/dovecot/pop.example.com.crt
   ssl_key = </etc/dovecot/pop.example.com.key
}

/etc/dovecot/dovecot-sql.conf.ext :
driver = mysql
connect = .....
default_pass_scheme = PLAIN

password_query = SELECT password, \
   '/stor/real/domains/%d/%2n/%n' AS userdb_home, \
   'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as userdb_mail, \
   concat('*:storage=', quota_mb, 'M') as userdb_quota_rule, \
   'vmail' AS userdb_uid, 'vmail' AS userdb_gid \
   FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0

user_query = SELECT '/stor/real/domains/%d/%2n/%n' AS home, \
   'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as mail, \
   concat('*:storage=', quota_mb, 'M') as quota_rule, \
   'vmail' AS uid, 'vmail' AS gid \
   FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0

iterate_query = SELECT userid as user, domain FROM email where deleted=0

More information about the dovecot mailing list