[Dovecot] Deleting metadata smashes file dovecot.dict
Hi,
I am using dovecot (see dovecot -n output below) with metadata plugin in my own plugin. I want to insert and delete some large metadata (>4KiB). It seems, that the delete in a second call of my plugin smashes the file dovecot.dict in users dir:
1st call: *dovecot.dict is empty -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 <1KiB *dovecot.dict contains two entries
2nd call: -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 <1KiB *dovecot.dict contains the two entries + some waste
If all eight steps are called together (in the same plugin context) the file dovecot.dict seems to be ok.
I have created a test plugin to reproduce this behaviour: http://temp-share.com/show/dPf3myu5W. It needs the notify and (patched) metadata plugin to run. To get it work i copy an email from one IMAP folder to another (to trigger the copy notifier).
It seems, that there is an issue on writing metadata/dict, can anybody reproduce it or knows what is wrong?
TIA
Andre
missing patch and info: metadata patch ============== --- a/src/metadata-backend.c +++ b/src/metadata-backend.c @@ -130,7 +130,8 @@ metadata_set_entry(struct metadata_entry *entry, struct mail_user *user) { if (!metadata_entry_is_valid(entry)) return -METADATA_ERROR_INVALID; - if (strlen(metadata_entry_get_value(entry)) > muser->set->maxsize) + if (metadata_entry_get_value(entry) != NULL + && strlen(metadata_entry_get_value(entry)) > muser->set->maxsize) return -METADATA_ERROR_TOOLARGE; if (count_entries(muser) > muser->set->maxentries) return -METADATA_ERROR_TOOMANY; --- a/Makefile.am +++ b/Makefile.am @@ -1,3 +1,6 @@ ACLOCAL_AMFLAGS = -I m4 SUBDIRS = src + +pkginc_libdir=$(dovecot_pkgincludedir)/metadata +nodist_pkginc_lib_HEADERS = metadata-config.h --- a/src/Makefile.am +++ b/src/Makefile.am @@ -42,7 +42,7 @@ lib90_imap_annotatemore_plugin_la_LIBADD = \ lib80_metadata_plugin.la endif -noinst_HEADERS = \ +headers = \ metadata-backend.h \ metadata-entry.h \ metadata-entry-private.h \ @@ -55,3 +55,6 @@ noinst_HEADERS = \ imap-arg-ext.h \ mailbox-ext.h \ str-ext.h + +pkginc_libdir=$(dovecot_pkgincludedir)/metadata +pkginc_lib_HEADERS = $(headers) dovecot -n ========== # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 2.6.35-1exp2-intern-k8-kvm x86_64 Debian 6.0.5 ext3 debug_log_path = /var/opt/dovecot/log/dovecot.log default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_uid = 0 info_log_path = /var/opt/dovecot/log/dovecot.log lmtp_check_rcpt = yes lmtp_rcpt_to_parameters { * = i: ignorequota = b:plugin/quota_ignore xtolain = s:plugin/lain xtomsgtype = s:plugin/mailtype_xtomsgtype xtospam = b:plugin/spamflag_xtospam xtovacation = B:plugin/sieve_xtovacation } log_path = /var/opt/dovecot/log/dovecot.log log_timestamp = "%y%m%H%M%S " mail_gid = mail mail_home = /var/spool/dovecot/%2.256Hu/%3.4096Hu/%u mail_location = mdbox:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u mail_plugin_dir = /opt/app/dovecot/lib/dovecot mail_plugins = notify metadata test mail_uid = mail namespace inbox { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = nocache=y /var/spool/dovecot/deny deny = yes driver = passwd-file } passdb { args = nopassword=y driver = static } plugin { acl = vfile autocreate = INBOX.Drafts autocreate2 = INBOX.Sent autocreate3 = INBOX.Trash metadata_dict = file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict metadata_maxsize = 10240 quota = dict:User quota::file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict quota_rule = *:bytes=999999 quota_rule2 = *:messages=999 quota_rule3 = *:mailboxes=99 quota_rule4 = *:levels=9 quota_rule5 = INBOX.Voicebox:ignore } service anvil { unix_listener anvil { group = dovecot mode = 0660 } } service auth-worker { user = $default_internal_user } service auth { unix_listener auth-userdb { group = dovecot mode = 0660 } } service config { unix_listener config { group = dovecot mode = 0660 } } service dict { unix_listener dict { mode = 0600 user = mail } } service lmtp { inet_listener lmtp { port = 2003 } } ssl = no userdb { args = /opt/app/dovecot/etc/dovecot/dovecot-tdbfcall.conf.ext driver = tdbfcall } protocol imap { mail_plugins = notify metadata test autocreate imap_metadata imap_annotatemore } protocol pop3 { pop3_uidl_format = %v.%u }
On reading 'big' metadata entries i got a similiar behaviour (metadata_maxsize = 10240). The command getmetadata returns no data. I have created a dovecot.dict with a single entry (key=priv/mailbox/fba9aa2cf41721501f340000d6254462/private/vendor/test/10458d10cf1f2a5023750000d6254462/2) to reproduce that.
I expected to get the value with command:
. getmetadata inbox /private/vendor/test/10458d10cf1f2a5023750000d6254462/2 . OK Completed.
It seems to me that there is a bug in metadata plugin. Anyone any ideas?
TIA Andre
Hi
it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes.
Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)?
Regards Andre
On 20.8.2012, at 16.35, Andre Gröbe wrote:
it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes.
Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)?
Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1.
Hi Timo,
thx for reply.
[tss@iki.fi; Mo 20 Aug 2012 20:22:58 CEST]
Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)?
Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1.
Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops.
Best regards Andre
On 21.8.2012, at 13.10, Andre Gröbe wrote:
[tss@iki.fi; Mo 20 Aug 2012 20:22:58 CEST]
Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)?
Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1.
Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops.
Does the attached patch fix it? (not a very good fix)
Hi Timo,
[tss@iki.fi; Di 21 Aug 2012 12:22:04 CEST]
Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops.
Does the attached patch fix it? (not a very good fix)
No, it doesn't. Still got no response on large metadata and writing destroys the dictionary.
Best Regards Andre
Hi Timo,
[Timo Sirainen; Di 21 Aug 2012 12:22:04 CEST]
Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops.
Does the attached patch fix it? (not a very good fix)
No, it doesn't. Still got no response on large metadata and writing destroys the dictionary.
Sorry for bumping - is there a chance to get it work with dovecot 2.1.7?
TIA Andre
On 08/08/12 14:23, Andre Gröbe wrote:
I am using dovecot (see dovecot -n output below) with metadata plugin in my own plugin. I want to insert and delete some large metadata (>4KiB). It seems, that the delete in a second call of my plugin smashes the file dovecot.dict in users dir:
That's an error in dict-file.c, file_dict_refresh():
while ((key = i_stream_read_next_line(input)) != NULL && (value = i_stream_read_next_line(input)) != NULL) {
If reading the value requires reading from the stream (because the value doesn't fit into the input buffer), the key is overwritten. The attached patch duplicates the key before reading the value.
On 28.8.2012, at 17.42, Ewald Dieterich wrote:
On 08/08/12 14:23, Andre Gröbe wrote:
I am using dovecot (see dovecot -n output below) with metadata plugin in my own plugin. I want to insert and delete some large metadata (>4KiB). It seems, that the delete in a second call of my plugin smashes the file dovecot.dict in users dir:
That's an error in dict-file.c, file_dict_refresh():
while ((key = i_stream_read_next_line(input)) != NULL && (value = i_stream_read_next_line(input)) != NULL) {
If reading the value requires reading from the stream (because the value doesn't fit into the input buffer), the key is overwritten. The attached patch duplicates the key before reading the value.
Committed: http://hg.dovecot.org/dovecot-2.1/rev/6e53209030f6
participants (3)
-
Andre Gröbe
-
Ewald Dieterich
-
Timo Sirainen