Timo Sirainen wrote:

Getting backtrace of the crash could also show where the problem is. http://dovecot.org/bugreport.html

Hi,

later I will try to backport the Dovecot 1.0 Debian package to Etch, but for now I can only provide you a backtrace:

(gdb) bt full #0 0x0808a778 in index_storage_get_status_locked () No symbol table info available. #1 0x0808bbf8 in index_mailbox_sync_deinit () No symbol table info available. #2 0xb7fd5c6d in sync_mailbox () from /usr/lib/dovecot/modules/imap/lib10_mbox_snarf_plugin.so No symbol table info available. #3 0xb7fd5c91 in mbox_snarf () from /usr/lib/dovecot/modules/imap/lib10_mbox_snarf_plugin.so No symbol table info available. #4 0xb7fd5eef in mbox_snarf_sync_init () from /usr/lib/dovecot/modules/imap/lib10_mbox_snarf_plugin.so No symbol table info available. #5 0x080614cc in imap_sync_nonselected () No symbol table info available. #6 0x080599d3 in _cmd_select_full () No symbol table info available. #7 0x08059b97 in cmd_select () No symbol table info available. #8 0x0805b4e2 in _client_input () No symbol table info available. #9 0x080b8826 in io_loop_handler_run () No symbol table info available. #10 0x080b7bd8 in io_loop_run () No symbol table info available. #11 0x080632de in main () No symbol table info available.

HTH, Bernd

Timo Sirainen wrote:

Anyway, updated http://dovecot.org/patches/mbox-snarf-plugin.c

Hi,

thanks, this code seems to work with the original Debian package.

It managed to copy the contents of a mbox from /var/mail/$user to /home/$user/.imap_mail/inbox, well done!

I have yet to conduct more testing as Dovecot+Exim will be handling 100+ users soon. Exims current setup delivers directly to directory /home/user/.imap_mail, where spam mails can optionally be delivered into different mbox files. /home is subject to user-based filesystem quotas, also a policy not to bounce any mail is in place.

My idea is to let Exim deliver everything to /var/mail/$user, where enough space is present, and let Dovecot snarf the mails to /home, if quota allows.

Is it possible to snarf more than one mbox from /var/mail to /home/$user, like this?

/var/mail/$user -> /home/.imap_mail/inbox /var/mail/$user.spam -> /home/.imap_mail/Spam /var/mail/$user.virus -> /home/.imap_mail/Virus

Snarfing only one mbox removes the possibility to let Exim move recognized spam mails not into the inbox.

Since using the snarf plugin I have this additional line in syslog:

Jun 4 15:28:27 bach dovecot: IMAP(kuhls): Timeout leak: 0x808ad10

Anything to worry about?

Greetings, Bernd Kuhls

Timo Sirainen wrote:

...

Jun 4 15:28:27 bach dovecot: IMAP(kuhls): Timeout leak: 0x808ad10

Fixed and uploaded new version.

Hi,

thanks, the log message disappeared.

In your original posting you said:

if all the messages couldn't be copied, for example because of out of quota, then none of the messages are copied. I suppose that could be a problem. It might be possible to fix that by changing:

  if (mailbox_copy(dest_trans, mail, 0, NULL, NULL) < 0) {
  	if (!mail->expunged) {
  		ret = -1;
  		break;
  	}
  }

to just:

  if (mailbox_copy(dest_trans, mail, 0, NULL, NULL) < 0) {
  	if (!mail->expunged)
  		break;
  }

But I'm not sure about that. Try and tell me. :)

In your new patch you changed exactly this code, did you have a reason to do so? I have yet to test your patch in an over-quota scenario, so I can´t tell you my experience with the patch.

Greetings, Bernd Kuhls

Hi,

Bernd Kuhls wrote:

It managed to copy the contents of a mbox from /var/mail/$user to /home/$user/.imap_mail/inbox, well done!

I have yet to conduct more testing

I did that now and I found another problem.

If /var/mail/$user does not exist, its created by Exim when delivering a mail, the content of the file looks like this:

From kuhls@localhost Sun Jun 10 22:00:16 2007 Return-path: <kuhls@localhost> Envelope-to: overquota@localhost Delivery-date: Sun, 10 Jun 2007 22:00:16 +0200 Received: from withoutHELO (noIP) by localhost (Exim 4.63) id 1HxTa9-0005dE-0Q; Sun, 10 Jun 2007 22:00:13 +0200 To: overquota@localhost Subject: test Date: Sun, 10 Jun 2007 22:00:16 +0200

test

This mailbox is snarfed by Dovecot each time the Inbox is opened, because the mail is not deleted from /var/mail$user, although Dovecot successfully snarfed the mail to /home/$user/.imap_mail/inbox. /home/$user is not overquota.

I found the reason for this, in contrast to the bevaviour described above my first test was done with a /var/mail/$user mbox which had an uw-imap style index, like this:

From MAILER_DAEMON Sat Apr 21 19:48:13 2007 Date: Sat, 21 Apr 2007 19:48:13 +0200 From: Mail System Internal Data <MAILER-DAEMON@localhost> Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA Message-ID: <1177177693@localhost> X-IMAP: 1177169873 0000000082 NonJunk Status: RO

This text is part of the internal format of your mail folder, and is not a real message. It is created automatically by the mail system software. If deleted, important folder data will be lost, and it will be re-created with the data reset to initial values.

After Exim delivered a mail to a file having this content, Dovecot is able to snarf the mail to /home/$user/.imap_mail/inbox _and_ delete it from /var/mail/$user.

How to solve this problem? /var/mail is normally empty and only used by Exim delivering mails when the user is overquota. I could use a cronjob-script-solution to create mbox files with uw-imap indexes for all users present on the system, but it would be an ugly hack.

Greetings, Bernd Kuhls

Timo Sirainen wrote:

On 10.6.2007, at 23.10, Bernd Kuhls wrote:

...

This mailbox is snarfed by Dovecot each time the Inbox is opened, because the mail is not deleted from /var/mail$user, although Dovecot successfully snarfed the mail to /home/$user/.imap_mail/inbox. /home/$user is not overquota.

Check your error log. I'm pretty sure there's something in there (http://wiki.dovecot.org/Logging).

Hi,

nothing interesting found. The only message I saw, and which I would see disabled, is this one:

Jun 10 22:33:33 bach dovecot: IMAP(ulfb): stat() failed with mbox file /var/mail/someuser: No such file or directory

someuser is a valid mail user, where Dovecot uses /home/someuser/.imap_mail/ to store email, but Exim has never created /var/mail/someuser because the user never was over-quota. Having this log message all the time when 100+ users are using their mail accounts is a bit nasty;-)

Here are the log entries of a user with /var/mail/$user present:

Jun 10 22:22:27 bach dovecot: imap-login: Login: user=<user>, method=plain, rip=x.x.x.x, lip=y.y.y.y.y, TLS Jun 10 22:27:24 bach dovecot: IMAP(user): Disconnected: Logged out

During this imap session the mails from /var/mail/$user were snarfed to $HOME/.imap_mail/inbox, but the snarfed mails from /var/mail/$user were not deleted.

Jun 10 22:28:47 bach dovecot: imap-login: Login: user=<user>, method=plain, rip=x.x.x.x, lip=y.y.y.y, TLS Jun 10 22:28:53 bach dovecot: IMAP(user): Disconnected: Logged out Jun 10 22:29:00 bach dovecot: imap-login: Login: user=<user>, method=plain, rip=x.x.x.x, lip=y.y.y.y, TLS Jun 10 22:29:03 bach dovecot: IMAP(user): Disconnected: Logged out

Then I restarted the mailclient (Thunderbird 2.0) twice, the mails from /var/mail/$user were snarfed again and I saw these mails three times in my imap inbox. No other log messages are present and /var/mail/$user still holds the mails already snarfed three times. Only from a file with mbox index the mails get deleted during snarf process by Dovecot.

Greetings, Bernd Kuhls

Bernd Kuhls wrote:

I am using Dovecot 1.0.0, might updating to 1.0.1 help fixing some of the problems mentioned above?

Hi,

updated to Dovecot 1.0.3, but all the mentioned problems still exist.

Greetings, Bernd

Hi,

Timo Sirainen wrote:

On Fri, 2007-07-06 at 11:54 +0200, Bernd Kuhls wrote:

...

• The index for $HOME/.imap_mail/inbox is created in $HOME/.imap_mail/.imap/inbox/, while the indexes for other mbox files are created in /var/mail/indexes/%u, like its supposed to be. How to disable creating index files in $HOME altogether without using links?

This is a bit problematic to fix because of how it works internally.

ok, not a big problem for me. I can live with the current situation.

...

• if /var/mail/$user does not exist, syslog is cluttered with these messages for each POP3/IMAP login:

  Jun 10 22:33:33 server dovecot: IMAP(someuser): stat() failed with mbox file /var/mail/someuser: No such file or directory

I can't reproduce this.

Since your patch below the snarf-bug is gone, Dovecot creates a zero-byte file for /var/mail/$user now.

Looks like this is broken only with mbox_lazy_writes=yes, which is why I didn't notice it first.

And looks like it's a bug in Dovecot itself. Fixed: http://hg.dovecot.org/dovecot-1.0/rev/1b6c41524c6a

Big thanks! Your patch fixed the bug, now I can throw away my ugly work-arounds for this. Weird enough I did not activate mbox_lazy_writes.

# dovecot -n # 1.0.3: /etc/dovecot/dovecot.conf log_path: /var/log/dovecot/dovecot.main log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_extra_groups: mail mail_location: mbox:~/.imap_mail:INBOX=/var/mail/%u:INDEX=/var/mail/indexes/%u mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mbox_snarf mail_plugins(imap): quota imap_quota mbox_snarf mail_plugins(pop3): mbox_snarf mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle netscape-eoh tb-extra-mailbox-sep delay-newmail imap_client_workarounds(imap): outlook-idle netscape-eoh tb-extra-mailbox-sep delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: fs mbox_snarf: ~/.imap_mail/inbox

Happy greetings, Bernd Kuhls