[Dovecot] pop3c migration?
Gedalya
gedalya at gedalya.net
Wed Jul 17 13:36:12 EEST 2013
Hi,
I'm running dsync migrations using imapc and the source IMAP server is
just too slow. It has taken 8 hours to migrate a mailbox with 47,000
messages. It seems most of the mailboxes are never accessed with IMAP,
so it would be just as good to migrate them via POP3 which should work
faster in my case.
Is this actually meant to work?
doveadm -D -o pop3c_user=xxxxxx at example.com -o 'pop3c_password=*******'
-o pop3c_host=120.121.122.123 backup -u yyyyy at example.com -R pop3c:
gives:
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_init (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined
symbol: i_stream_create_deflate (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_list_backend (this is usually intentional, so just ignore
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting:
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting:
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
doveadm(yyyyy at example.com): Debug: Namespace inbox: type=private,
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++:
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=,
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt=
dsync(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=,
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
Segmentation fault
remembering something about that... :
# mkdir /tmp/pop3c
# chown vmail:vmail /tmp/pop3c/
# doveadm -D -o pop3c_user=xxxxxx at example.com -o
'pop3c_password=*******' -o pop3c_host=120.121.122.123 backup -u
yyyyy at example.com -R pop3c:/tmp/pop3c
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules
doveadm(root): Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_listescape_plugin.so
doveadm(root): Debug: Loading modules from directory:
/usr/lib/dovecot/modules/doveadm
doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
undefined symbol: expire_set_init (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
undefined symbol: quota_user_module (this is usually intentional, so
just ignore this message)
doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined
symbol: i_stream_create_deflate (this is usually intentional, so just
ignore this message)
doveadm(root): Debug: Skipping module doveadm_fts_plugin, because
dlopen() failed:
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined
symbol: fts_list_backend (this is usually intentional, so just ignore
this message)
doveadm(yyyyy at example.com): Debug: Added userdb setting:
mail=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: Added userdb setting:
plugin/quota_rule=*:storage=5120M
doveadm(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
doveadm(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
doveadm(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
doveadm(yyyyy at example.com): Debug: Namespace inbox: type=private,
prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/stor/real/domains/example.com/yy/yyyyy/Maildir
doveadm(yyyyy at example.com): Debug: maildir++:
root=/stor/real/domains/example.com/yy/yyyyy/Maildir, index=, control=,
inbox=/stor/real/domains/example.com/yy/yyyyy/Maildir, alt=
dsync(yyyyy at example.com): Debug: Effective uid=1001, gid=1001,
home=/stor/real/domains/example.com/yy/yyyyy
dsync(yyyyy at example.com): Debug: Namespace : type=private,
prefix=POP3-MIGRATION-NS/, sep=, inbox=no, hidden=yes, list=no,
subscriptions=yes location=pop3c:
dsync(yyyyy at example.com): Debug: fs: root=, index=, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace inbox: type=private, prefix=,
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=pop3c:/tmp/pop3c
dsync(yyyyy at example.com): Debug: fs: root=/tmp/pop3c/.,
index=/tmp/pop3c, control=, inbox=, alt=
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from
/stor/real/domains/example.com/yy/yyyyy/Maildir: mode=0700 gid=-1
dsync(yyyyy at example.com): Debug: Namespace : /tmp/pop3c/./INBOX doesn't
exist yet, using default permissions
dsync(yyyyy at example.com): Debug: Namespace : Using permissions from
/tmp/pop3c/.: mode=0755 gid=-1
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Looking up IP
address
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Connecting to
120.121.122.123:110
dsync(yyyyy at example.com): Debug: pop3c(120.121.122.123): Authenticating
as xxxxxx at example.com
dsync(yyyyy at example.com): Panic: file mail-storage.c: line 1311
(mailbox_get_metadata): assertion failed: ((items &
MAILBOX_METADATA_GUID) == 0 || !guid_128_is_empty(metadata_r->guid))
dsync(yyyyy at example.com): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x45caa) [0xb761ccaa] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47)
[0xb761cdc7] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb761d647]
-> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x120)
[0xb76f3040] -> doveadm() [0x80740bc] ->
doveadm(dsync_worker_mailbox_iter_next+0x21) [0x8071c11] -> doveadm()
[0x806c1f7] -> doveadm(dsync_brain_sync+0x6c0) [0x806b890] ->
doveadm(dsync_brain_sync_all+0x18) [0x806c4f8] -> doveadm() [0x806961b]
-> doveadm() [0x8069bbf] -> doveadm() [0x80541ec] ->
doveadm(doveadm_mail_try_run+0x257) [0x8054ee7] -> doveadm(main+0x391)
[0x805d0c1] ->
/lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6)
[0xb748ae46] -> doveadm() [0x8054001]
Aborted
Now I'm lost...
Am I in the right direction?
Thanks,
Gedalya
# dovecot -n
# 2.1.17: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-686-pae i686 Debian 7.1
auth_default_realm = example.com
auth_mechanisms = plain login cram-md5
auth_verbose = yes
auth_verbose_passwords = plain
default_client_limit = 8096
dict {
expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
imapc_host = mail01.example.com
mail_gid = vmail
mail_plugins = listescape
mail_prefetch_count = 50
mail_uid = vmail
namespace {
hidden = yes
list = no
location = pop3c:
prefix = POP3-MIGRATION-NS/
}
namespace inbox {
inbox = yes
location =
prefix =
separator = /
type = private
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
expire = Trash
expire2 = Trash.*
expire3 = Junk
expire_dict = proxy::expire
pop3_migration_mailbox = POP3-MIGRATION-NS/INBOX
sieve = ~/.dovecot.sieve
sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve
sieve_dir = ~/sieve
}
pop3c_host = 120.121.122.123
postmaster_address = postmaster at example.com
service auth-worker {
user = $default_internal_user
}
service auth {
client_limit = 20480
}
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
service imap-login {
process_min_avail = 4
service_count = 0
vsz_limit = 128 M
}
service imap {
process_limit = 10240
}
service lmtp {
inet_listener lmtp {
address = 0.0.0.0
port = 7025
}
}
service pop3-login {
process_min_avail = 4
service_count = 0
vsz_limit = 128 M
}
service pop3 {
process_limit = 4096
}
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol doveadm {
mail_plugins = listescape pop3_migration
}
protocol lmtp {
mail_plugins = listescape sieve
}
protocol imap {
mail_max_userip_connections = 25
mail_plugins = listescape imap_quota
ssl_cert = </etc/dovecot/imap.example.com.crt
ssl_key = </etc/dovecot/imap.example.com.key
}
protocol pop3 {
mail_max_userip_connections = 10
mail_plugins = listescape
pop3_fast_size_lookups = yes
pop3_no_flag_updates = yes
pop3_reuse_xuidl = no
pop3_save_uidl = yes
pop3_uidl_format = %08Xu%08Xv
ssl_cert = </etc/dovecot/pop.example.com.crt
ssl_key = </etc/dovecot/pop.example.com.key
}
/etc/dovecot/dovecot-sql.conf.ext :
driver = mysql
connect = .....
default_pass_scheme = PLAIN
password_query = SELECT password, \
'/stor/real/domains/%d/%2n/%n' AS userdb_home, \
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as userdb_mail, \
concat('*:storage=', quota_mb, 'M') as userdb_quota_rule, \
'vmail' AS userdb_uid, 'vmail' AS userdb_gid \
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
user_query = SELECT '/stor/real/domains/%d/%2n/%n' AS home, \
'maildir:/stor/real/domains/%d/%2n/%n/Maildir' as mail, \
concat('*:storage=', quota_mb, 'M') as quota_rule, \
'vmail' AS uid, 'vmail' AS gid \
FROM email WHERE userid = '%n' AND domain = '%d' and deleted=0
iterate_query = SELECT userid as user, domain FROM email where deleted=0
More information about the dovecot
mailing list