[Dovecot] mdbox and folders containing both messages and sub-folders
I've got Dovecot 2.2r5 (Ubuntu package version 2:2.2.0~rc5-0~auto+1 from the "testing-auto/dovecot-2.2" repository on xi.rename-it.nl) running on a brand-new Ubuntu 12.04.2 server.
I'm trying to configure Dovecot (using the "mdbox" mailbox format) to allow folders to contain both messages and sub-folders. However, it's not working consistently.
I need to have folders with both messages and sub-folders because I'm trying to migrate mail from an existing (but soon-to-be-retired) Cyrus server, and the existing body of mail has folders of this sort.
I'm currently using the following configuration line:
mail_location = mdbox:~/mdbox:DIRNAME=.indexes:INDEX=~/mdbox/indexes
and am using the Larch program to do an IMAP-to-IMAP copy of one of my family member's mail from the old Cyrus server to the new Dovecot server. When I examine the "mailboxes" directory tree, I see some directories that (correctly) contain both the renamed ".indexes" directory and also directories for IMAP subfolders. However, the Larch output is showing many errors of the form "mailbox cannot contain messages", and some of the messages could not be migrated because of this problem.
Am I doing something wrong? Or is the "folders containing both messages and sub-folders" feature simply broken for the "mdbox" format -- meaning presumably that I'll be forced either to use the Maildir format, or else abandon Dovecot entirely and install Cyrus on my new mail server instead?
Rich Wales richw@richw.org
On 06/04/2013 4:51 PM, Rich Wales wrote:
Am I doing something wrong? Or is the "folders containing both messages and sub-folders" feature simply broken for the "mdbox" format -- meaning presumably that I'll be forced either to use the Maildir format, or else abandon Dovecot entirely and install Cyrus on my new mail server instead?
mdbox folders with messages and subfolders work very well. I migrated from another imap server using imapsync without problems.
On 6.4.2013, at 23.51, Rich Wales richw@richw.org wrote:
and am using the Larch program to do an IMAP-to-IMAP copy of one of my family member's mail from the old Cyrus server to the new Dovecot server. When I examine the "mailboxes" directory tree, I see some directories that (correctly) contain both the renamed ".indexes" directory and also directories for IMAP subfolders. However, the Larch output is showing many errors of the form "mailbox cannot contain messages", and some of the messages could not be migrated because of this problem.
What is the error message exactly? I'm wondering if the error is coming from Dovecot or if Larch is somehow confused internally and giving bogus errors.
Am I doing something wrong? Or is the "folders containing both messages and sub-folders" feature simply broken for the "mdbox" format -- meaning presumably that I'll be forced either to use the Maildir format, or else abandon Dovecot entirely and install Cyrus on my new mail server instead?
It works fine in mdbox, you can't disable it even if you tried.
Repying to Timo Sirainen:
What is the error message exactly? I'm wondering if the error is coming from Dovecot or if Larch is somehow confused internally and giving bogus errors.
Certainly a reasonable question. I tried to enable full debugging output in both Dovecot and Larch, but I couldn't get anything helpful.
Is there a way to make Dovecot report all IMAP commands and responses, so I can see exactly what the client is asking for, and exactly why Dovecot is refusing to do it?
Alternatively, does anyone here know Ruby (the language Larch is written in) well enough to say how to make its low-level IMAP module give a full logging of all commands and responses?
For what it may or may not be worth, I'm trying my same mail migration test using Maildir instead of mdbox, and it appears to be working fine.
Rich Wales richw@richw.org
Am 07.04.2013 20:08, schrieb Rich Wales:
Repying to Timo Sirainen:
What is the error message exactly? I'm wondering if the error is coming from Dovecot or if Larch is somehow confused internally and giving bogus errors.
Certainly a reasonable question. I tried to enable full debugging output in both Dovecot and Larch, but I couldn't get anything helpful.
Is there a way to make Dovecot report all IMAP commands and responses, so I can see exactly what the client is asking for, and exactly why Dovecot is refusing to do it?
verbose logging and mail logger plugin
http://wiki2.dovecot.org/Logging http://wiki2.dovecot.org/Plugins/MailLog
Alternatively, does anyone here know Ruby (the language Larch is written in) well enough to say how to make its low-level IMAP module give a full logging of all commands and responses?
no , perhaps try imapsync, it ever worked for me
For what it may or may not be worth, I'm trying my same mail migration test using Maildir instead of mdbox, and it appears to be working fine.
show dovecot conf and log
Rich Wales richw@richw.org
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Replying to Timo Sirainen:
But didn't you say Larch was giving some errors already?
Yes, but the Larch error message said only "mailbox cannot contain messages", followed by the name of the folder into which it couldn't store a message.
I looked at the Larch source code (written in Ruby, a language I have never used and don't really know at all), and this particular error message appears to be generated when Larch is unable to select or create a given mailbox. Exactly what the IMAP rejection response from Dovecot is, I can't tell; it's not being reported anywhere as far as I can see.
Maybe Larch has some mbox-specific code and it thinks that dbox looks similar..
Larch isn't accessing Dovecot's internal mailbox structure at all -- it's just talking to the IMAP server. I'm not sure what sort of "mbox-specific code" could be involved in such a scenario. Is there anything about mbox vs. dbox that would be reflected in the way a Dovecot server behaves when using IMAP?
I think what I really need here is a way to look at the actual, literal IMAP commands and responses (not just abstract interpretations of the errors from Dovecot, Larch, or any other program). Without that information, I don't see how I can tell what the problem is, or whether it's in Dovecot or Larch.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Replying to Robert Schetterer:
Is there a way to make Dovecot report all IMAP commands and responses, so I can see exactly what the client is asking for, and exactly why Dovecot is refusing to do it?
verbose logging and mail logger plugin
http://wiki2.dovecot.org/Logging http://wiki2.dovecot.org/Plugins/MailLog
I enabled these -- at least, I think I did; see my "doveconf -n" output below -- but I'm still not getting a full description of the IMAP commands and responses. Is there any way to see the actual, literal IMAP dialogue?
Alternatively, does anyone here know Ruby (the language Larch is written in) well enough to say how to make its low-level IMAP module give a full logging of all commands and responses?
no, perhaps try imapsync, it [has always] worked for me
I'm confused by the licensing status of Imapsync. The "official" web site for Imapsync says it is a commercial product (50 EUR). I found another web site (fedorahosted.org) that claims to have the latest Imapsync as a free download, but I'm unsure as to whether this copy is legitimate or pirated. Can anyone clear this up for me? I had originally preferred to use Imapsync, but I'm not prepared to pay 50 EUR for it, and I don't want to use a pirated "free" copy.
show dovecot conf and log
OK, see below for my current output of "doveconf -n". Note that I'm trying out Maildir right now; when I was using mdbox, my "mail_location" configuration line read as follows:
mail_location = mdbox:~/mdbox:DIRNAME=.indexes:INDEX=~/mdbox/indexesor like this:
mail_location = mdbox:~/mdbox:DIRNAME=.indexesAs for the log output (in /var/log/mail.log), there were no error messages of any kind. There were a bunch of "Debug: Namespace" messages saying "<path> doesn't exist yet, using default permissions", but that's all. I'm including (see below) an excerpt from my log output, showing the results of one session.
Again, I'm assuming at this point that I'm going to need to see the literal, low-level IMAP commands and responses before I can tell if this is a bug in Dovecot, a bug in Larch, or something else.
Rich Wales richw@richw.org
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# 2.2.rc5 (c24783148d78): /etc/dovecot/dovecot.conf # OS: Linux 3.5.0-26-generic x86_64 Ubuntu 12.04.2 LTS auth_verbose = yes mail_debug = yes mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = . } passdb { driver = pam } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = " imap" ssl_cert =
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
imap-login: Login: user=<randerzo>, method=PLAIN, rip=127.0.0.1,
lip=127.0.0.1, mpid=29976, secured, session=
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Rich Wales richw@richw.org
On 2013-04-07 3:42 PM, Rich Wales richw@richw.org wrote:
I'm confused by the licensing status of Imapsync. The "official" web site for Imapsync says it is a commercial product (50 EUR). I found another web site (fedorahosted.org) that claims to have the latest Imapsync as a free download, but I'm unsure as to whether this copy is legitimate or pirated. Can anyone clear this up for me? I had originally preferred to use Imapsync, but I'm not prepared to pay 50 EUR for it, and I don't want to use a pirated "free" copy.
https://github.com/imapsync/imapsync
Note the contents of the license file:
NO LIMIT PUBLIC LICENSE
Version 0, June 2012Gilles LAMIRAL La Billais 35580 Baulon France NO LIMIT PUBLIC LICENSE Terms and conditions for copying, distribution, modification or anything else. 0 No limit to do anything with this work and this license. 1 GOTO 0
--
Best regards,
Charles
Replying to Charles Marcus:
https://github.com/imapsync/imapsync Note the contents of the license file:
Quite bizarre. So the author of the program is selling it for 50 EUR, but at the same time he has released the latest version of the source (the same as what he's selling) via a free licence. (!?!)
OK, thanks for setting my mind at ease.
Rich Wales richw@richw.org
Quoting Rich Wales richw@richw.org:
Replying to Charles Marcus:
https://github.com/imapsync/imapsync Note the contents of the license file:
Quite bizarre. So the author of the program is selling it for 50 EUR, but at the same time he has released the latest version of the source (the same as what he's selling) via a free licence. (!?!)
- There's no guarantee that what's on github is the most recent
version of the software. - There's no guarantee that any future improvements will be released
as open source (as opposed to guaranteed lifetime updates). - Looks like at least a windows compiled version of the script is provided.
These are all common tactics for releasing open source code while
maintaining value-added perks to aid in funding the software
development.
michael
Am 07.04.2013 22:49, schrieb Rich Wales:
Replying to Charles Marcus:
https://github.com/imapsync/imapsync Note the contents of the license file:
Quite bizarre. So the author of the program is selling it for 50 EUR, but at the same time he has released the latest version of the source (the same as what he's selling) via a free licence. (!?!)
OK, thanks for setting my mind at ease.
Rich Wales richw@richw.org
my Last info is ,that you have to pay for the windows version only
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
On 7.4.2013, at 21.08, Rich Wales richw@richw.org wrote:
Repying to Timo Sirainen:
What is the error message exactly? I'm wondering if the error is coming from Dovecot or if Larch is somehow confused internally and giving bogus errors.
Certainly a reasonable question. I tried to enable full debugging output in both Dovecot and Larch, but I couldn't get anything helpful.
But didn't you say Larch was giving some errors already?
Is there a way to make Dovecot report all IMAP commands and responses, so I can see exactly what the client is asking for, and exactly why Dovecot is refusing to do it?
http://wiki2.dovecot.org/Debugging/Rawlog
For what it may or may not be worth, I'm trying my same mail migration test using Maildir instead of mdbox, and it appears to be working fine.
Maybe Larch has some mbox-specific code and it thinks that dbox looks similar..
On 07.04.2013 20:08, wrote Rich Wales:
Repying to Timo Sirainen:
What is the error message exactly? I'm wondering if the error is coming from Dovecot or if Larch is somehow confused internally and giving bogus errors.
Certainly a reasonable question. I tried to enable full debugging output in both Dovecot and Larch, but I couldn't get anything helpful.
Is there a way to make Dovecot report all IMAP commands and responses, so I can see exactly what the client is asking for, and exactly why Dovecot is refusing to do it?
http://wiki2.dovecot.org/Debugging/Rawlog
Alternatively, does anyone here know Ruby (the language Larch is written in) well enough to say how to make its low-level IMAP module give a full logging of all commands and responses?
For what it may or may not be worth, I'm trying my same mail migration test using Maildir instead of mdbox, and it appears to be working fine.
Rich Wales richw@richw.org
Replying to e-frog:
Thanks. I'll try this and (hopefully) be able to tell what's going on.
Rich Wales richw@richw.org
I retried my test with raw IMAP logging in effect. Each test involved importing about 900 messages into an empty account on my Dovecot server, using the Larch program.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
When I use the mdbox format, there are a bunch of LIST and LSUB commands whose responses mention INBOX.Trash with the \Noselect and \HasChildren flags (the Trash folder for this particular account happens to contain subfolders). Before trying to save a message in the Trash folder, the client did a "SUBSCRIBE INBOX.Trash" (which succeeded) followed by a "STATUS INBOX.Trash" (which failed with "NO Mailbox doesn't exist").
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
When I use the Maildir format, the raw log never mentions the \Noselect flag in connection with the INBOX.Trash folder. Before trying to save a message in the Trash folder, the client did a "CREATE INBOX.Trash", "SUBSCRIBE INBOX.Trash", "STATUS INBOX.Trash", and "SELECT INBOX.Trash" -- all of which succeeded -- and then an "APPEND INBOX.Trash" command was given, which also succeeded.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
So, with the mdbox format, Larch did not CREATE the folder before trying to SELECT it. And Dovecot was flagging INBOX.Trash with a \Noselect flag (which was not the case with the Maildir format).
I'm not familiar enough with Dovecot or IMAP to know exactly what this difference means, but hopefully it will reveal something to someone here. Let me know if you need to see other stuff in these raw logs that I didn't mention here.
Rich Wales richw@richw.org
On 8.4.2013, at 1.51, Rich Wales richw@richw.org wrote:
When I use the mdbox format, there are a bunch of LIST and LSUB commands whose responses mention INBOX.Trash with the \Noselect and \HasChildren flags (the Trash folder for this particular account happens to contain subfolders). Before trying to save a message in the Trash folder, the client did a "SUBSCRIBE INBOX.Trash" (which succeeded) followed by a "STATUS INBOX.Trash" (which failed with "NO Mailbox doesn't exist").
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
When I use the Maildir format, the raw log never mentions the \Noselect flag in connection with the INBOX.Trash folder.
Yeah, this is the difference between Dovecot's LAYOUT=fs and LAYOUT=maildir++ backends, which is also exposed to IMAP clients. With maildir++ there are no \Noselect folders possible. Whether that's a good or a bad thing I'm not entirely sure. I've been wondering earlier also if I should change the behavior somewhat, or make it configurable.
So, with the mdbox format, Larch did not CREATE the folder before trying to SELECT it. And Dovecot was flagging INBOX.Trash with a \Noselect flag (which was not the case with the Maildir format).
I'd say this is a Larch bug. It's relying on some specific way that some IMAP servers work.
participants (7)
-
Charles Marcus
-
e-frog
-
Michael M Slusarz
-
Oscar del Rio
-
Rich Wales
-
Robert Schetterer
-
Timo Sirainen