It's not just that mail is dissappearing, it's that mail is being DELETED completely from random opened folders. We have verified that no other clients(pop3 or imap) are accessing the servers and deleting the mail.

When a user opens a folder, the mail is there and then it's not.

I've looked at the "cur" directory of the imap maildir folder and yup, the mail is no longer there.

Very frustrating and my users are about to fire us because of this issue.

Brent

Quoting Paul Tansom paul@aptanet.com:

Obantec Support wrote:

...

----- Original Message ----- From: bclements@io-networks.com To: dovecot@dovecot.org Sent: Tuesday, August 09, 2005 8:23 PM Subject: [Dovecot] Mail disappearing

...

My users are complaining that their mail is dissappearing out of folders or if they go into a folder, the mail will be there one minute, the next it isn't.

...

Anybody see this before...and if so..is there a fix?

I'm running dovecot 99.11

Hi

Yes a user reported this tonight. 30MB of emails gone! not happy and wants me to restore from backup. ( i had to explain we only backup website data).

running FC3 with 0.99 but a test of 1.0 stable is on the cards.

all i want is a stable pop3 server like i had on RH8.0

Mark

Please don't scare the wotsits out of me, I've just abandoned Courier IMAP for two reasons:

1. it seems incapable of handling subfolders anywhere but below the Inbox and the only help suggested is that I should ask the people writing my mail client to fix it - so that covers Microsoft (x2), Mozilla (x2), Eudora, Evolution and Squirrel mail for starters!

2. this is the relevant one here - I kept loosing mail within my IMAP folders. I don't know whether this one may be relevant here, but the issue was basically a visibility thing. If I accessed the folders directly from Mutt on the server all the mail was all there. If I used a mail client then the mail would not be there. Once in a while a folder would show up some mail, but others would not see it, and once I'd hit one of these the only folder that would update properly was the Inbox until I restarted. Admittedly this was not mail dissappearing, simply not appearing in the first place, but I thought I'd mention it in case the description prompted someone to have a bright idea about what would be causing the reference problems in Dovecot :) From what I could see the client and server just stopped talking to each other. I certainly got no error messages logged. I'm using Thunderbird at the moment, and I think the problem was primarily in the two Mozilla based mail clients, and Outlook (+ Express). I seem to recall Evolution being OK, but sadly Evolution is deceased on my system and I've not worked out whether this is my own problem or a Debian issue yet.

As an aside, I've been very pleased with Dovecot so far, especially having spent so long trying to get Courier working properly. So far I have Exim 4 working with it and Procmail. My next stage of mail upgrade is to rebuild my filtering with Mailscanner, Spamassassing and ClamAV as well as hooking up IMP for webmail. I also need to build in virtual user support with probably an LDAP backend. By then I hope there'll be a version in Debian Backports for shared folders ;) I may hold off a bit on this though.

-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/

This message was sent using IMP, the Internet Messaging Program.

Your answer doesn't make sense.

Here's an example. I don't necessarily upgrade RHEL3 to RHEL4 just because it's out.

Are my issues a known issue with dovecot 99.11? If so, then I'll upgrade.

Has anyone seen these types of issues before with this particular version of dovecot?

Brent

Quoting Bob Hall rjhjr@cox.net:

On Tue, Aug 09, 2005 at 04:55:33PM -0500, bclements@io-networks.com wrote:

...

It's not just that mail is dissappearing, it's that mail is being DELETED completely from random opened folders. We have verified that no other clients(pop3 or imap) are accessing the servers and deleting the mail.

When a user opens a folder, the mail is there and then it's not.

I've looked at the "cur" directory of the imap maildir folder and yup, the mail is no longer there.

Very frustrating and my users are about to fire us because of this issue.

Your first post said you were using 99.11. If you are having problems with an outdated version of dovecot, I don't understand why you don't update to at least 99.14. Particularly since your users are about to fire you. Doesn't that suggest that it's time for an upgrade?

Bob Hall

This message was sent using IMP, the Internet Messaging Program.

Freebsd 4.10 Filesystem is UFS2 Generic P4 with 256 megs of ram The problems were discovered this morning by the president of a larger client. Logfiles do not show a thing This is a existing install from one or two years ago and it was not a migration.

Brent

Quoting "albi@scii.nl" albi@scii.nl:

On Tue, 9 Aug 2005 17:28:40 -0500 bclements@io-networks.com wrote:

...

Are my issues a known issue with dovecot 99.11? If so, then I'll upgrade.

i don't know an answer to that, but imho it would make sense telling the list here :

• which OS+version you use
• which filesystem-type you use (yes, filesystem-probs can make files vanish is my experience)
• which kind of machine you use
• since when exactly did these problems occur
• do logfiles show any related messages
• did you migrate from e.g. courier and follow the migration-tips

This message was sent using IMP, the Internet Messaging Program.

bclements@io-networks.com wrote:

Freebsd 4.10 Filesystem is UFS2 Generic P4 with 256 megs of ram The problems were discovered this morning by the president of a larger client. Logfiles do not show a thing This is a existing install from one or two years ago and it was not a migration.

Brent

...

i don't know an answer to that, but imho it would make sense telling the list here :

• which OS+version you use
• which filesystem-type you use (yes, filesystem-probs can make files vanish is my experience)
• which kind of machine you use
• since when exactly did these problems occur
• do logfiles show any related messages
• did you migrate from e.g. courier and follow the migration-tips

Also - which storage format are you using? mbox or Maildir?

From memory, Timo spent AGES working subtle problems out of the mbox code (which only added to the already numerous reasons I chose Maildir :) I don't remember clearly, but if you are using mbox, I can easily believe that one of those bugs could possibly have caused mail loss.

Every time I think of mbox, I think about that "all your eggs in one basket" phrase....

-- Curtis Maloney

On Tue, Aug 09, 2005 at 04:55:33PM -0500, bclements@io-networks.com wrote:

It's not just that mail is dissappearing, it's that mail is being DELETED completely from random opened folders. We have verified that no other clients(pop3 or imap) are accessing the servers and deleting the mail.

What client is the user using when this 'deletion' occurs?  What
protocol, IMAP or POP?



						John

-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.

"My other computer is your windows box." Ralf Hildebrandt

bclements@io-networks.com wrote:

It's not just that mail is dissappearing, it's that mail is being DELETED completely from random opened folders. We have verified that no other clients(pop3 or imap) are accessing the servers and deleting the mail.

When a user opens a folder, the mail is there and then it's not.

This sounds suspiciously like someone has turned on the anti-spam feature of their mail client with "delete mails detected as SPAM" selected instead of safely moving it to a SPAM folder for review.

Any chance they're the sort of user to turn on good sounding features without checking how they work? (I know, I know... most of management, and half the rest of the users fit this category :)

-- Curtis Maloney

The "dot" notation for folder hierarchy doesn't matter. Its the implementation of the imap server itself as to how it handles the folder hierarchy.

When I migrated from uw-imap to courier, I patched courier so that it would show all folders at the root level rather than under INBOX, so that end-users wouldnt notice a thing during migration. There was a patch floating around on the net but its out-dated now. Involves doing your own patches now.

The hard-line stance of courier that all IMAP clients should support the IMAP NAMESPACE feature is not practical. So this is a big draw card to dovecot for me. Steve

----- Original Message ----- From: "Chris Green" chris@areti.co.uk To: dovecot@dovecot.org Sent: Wednesday, August 10, 2005 8:04 PM Subject: Re: [Dovecot] Mail disappearing

On Tue, Aug 09, 2005 at 10:46:57PM +0100, Paul Tansom wrote:

...

Please don't scare the wotsits out of me, I've just abandoned Courier IMAP for two reasons:

1. it seems incapable of handling subfolders anywhere but below the Inbox and the only help suggested is that I should ask the people writing my mail client to fix it - so that covers Microsoft (x2), Mozilla (x2), Eudora, Evolution and Squirrel mail for starters!

Of course it can't because it uses the brain-dead convention of folders all being files at the same level with '.' separators in the filenames indicating folders.

Thus a folder called 'holidays' is actually a directory called 'INBOX.holidays' and a folder called 'holidays/france' is a directory called 'INBOX.holidays.france'. Why any sort of mail server would want to construct a folder hierarchy like this I can never understand. It's not just Courier either, there are a few maildir based servers which do it.

(I'm not absolutely sure that the directories have the INBOX stuck on the front but however it's done they have to be all rooted in the same place because of the stupid idea of using '.' in the name rather than a really separate directory)

-- Chris Green (chris@areti.co.uk)

"Never ascribe to malice that which can be explained by incompetence."

Chris Green wrote:

On Tue, Aug 09, 2005 at 10:46:57PM +0100, Paul Tansom wrote:

...

Please don't scare the wotsits out of me, I've just abandoned Courier IMAP for two reasons:

1. it seems incapable of handling subfolders anywhere but below the Inbox and the only help suggested is that I should ask the people writing my mail client to fix it - so that covers Microsoft (x2), Mozilla (x2), Eudora, Evolution and Squirrel mail for starters!

Of course it can't because it uses the brain-dead convention of folders all being files at the same level with '.' separators in the filenames indicating folders.

Thus a folder called 'holidays' is actually a directory called 'INBOX.holidays' and a folder called 'holidays/france' is a directory called 'INBOX.holidays.france'. Why any sort of mail server would want to construct a folder hierarchy like this I can never understand. It's not just Courier either, there are a few maildir based servers which do it.

(I'm not absolutely sure that the directories have the INBOX stuck on the front but however it's done they have to be all rooted in the same place because of the stupid idea of using '.' in the name rather than a really separate directory)

Sorry, I'm not following this one. The file structure behind my new Dovecot installation is exactly the same as that used behing my Courier one - and I've rebuilt from scratch on a new box, so the structure has been created cleanly by Dovecot. This may be related to the way the Debian package has been configured, I'm not sure.

The key difference is in the display on the client. With Courier it works on the basis that the Inbox (in my Maildir directory, which contains the cur, new and tmp subdirectories for it) has all the other folders below it (stored in physical directories within the same Maildir directory, using . seperators for the folder structure as seen by the client). Dovecot on the other hand, takes all these physical directories in my Maildir folder and displays them off the root of the virtual folder tree as seen by the mail client. This to me makes sense, and although I can see that the NAMESPACE feature is defined to enable this to be flexible, I don't see an issue with setting the default action sensibly.

From the Googling I did, and playing with the settings in the various mail clients, it seems that they do support automatic setting of the Namespace - certainly Thunderbird changed this according to the server it connected to. Now the Namespace setting in the client may not be the IMAP NAMESPACE RFC setting, I've not actually managed to get my head around the spec document yet! That said, Courier sets the Personal Namespace to "INBOX." in the client and Shared Namespace to "shared.", which matches what it details in its FAQ. Dovecot sets the Personal Namespace to "" and the Shared Namespace is left blank (as I would expect not being supported yet). The physical directory structures are identical.

The key difference in physical directory structure is that with Dovecot folders that are created below the Inbox are actually defined with .INBOX. at the beginning. Folders off the root are just preceded with a .

Anyway, this may be getting OT a bit. The only issue I currently have is that I don't always get an update of the count of messages in folders that I auto file mail into with Procmail. This could be a client issue though, I've not yet investigated properly.

-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/

Chris Green wrote:

On Wed, Aug 10, 2005 at 02:31:40PM +0100, Paul Tansom wrote:

...

Chris Green wrote:

...

On Tue, Aug 09, 2005 at 10:46:57PM +0100, Paul Tansom wrote:

...

Please don't scare the wotsits out of me, I've just abandoned Courier IMAP for two reasons:

1. it seems incapable of handling subfolders anywhere but below the Inbox and the only help suggested is that I should ask the people writing my mail client to fix it - so that covers Microsoft (x2), Mozilla (x2), Eudora, Evolution and Squirrel mail for starters!

Of course it can't because it uses the brain-dead convention of folders all being files at the same level with '.' separators in the filenames indicating folders.

Thus a folder called 'holidays' is actually a directory called 'INBOX.holidays' and a folder called 'holidays/france' is a directory called 'INBOX.holidays.france'. Why any sort of mail server would want to construct a folder hierarchy like this I can never understand. It's not just Courier either, there are a few maildir based servers which do it.

(I'm not absolutely sure that the directories have the INBOX stuck on the front but however it's done they have to be all rooted in the same place because of the stupid idea of using '.' in the name rather than a really separate directory)

Sorry, I'm not following this one. The file structure behind my new Dovecot installation is exactly the same as that used behing my Courier one - and I've rebuilt from scratch on a new box, so the structure has been created cleanly by Dovecot. This may be related to the way the Debian package has been configured, I'm not sure.

The key difference is in the display on the client. With Courier it works on the basis that the Inbox (in my Maildir directory, which contains the cur, new and tmp subdirectories for it) has all the other folders below it (stored in physical directories within the same Maildir directory, using . seperators for the folder structure as seen by the client).

I thought (it is a while ago though) that Courier IMAP's maildirs had *directory* names like:-

fola.folb
fola.folb.folc
fola.fold.fole

and that all these directories were rooted in the same place. There was no actual hierachy at all. The folders just had long names with dots in that the Courier IMAP server fed to the client such that they looked like real directories.

For a remote IMAP client this makes little difference except that of having them all rooted in the same place. For a local MUA it can be a significant issue.

I really have never understood why any maildir server (IMAP or otherwise) would want to do this. It's not how qmail does it.

Yes, that's exactly the way it works, but then that is exactly the way Dovecot works too - not that I'm speaking from a position of authority, just from my two recent installations from the Debian Sarge packages.

I have to admit it seemed a bit strange to me too, but when using Mutt for local mail access I've found there is little difference between typing a / and typing a . when separating the directory locations.

I'd assumed (well, such as I'd really considered it much) that this was a standard way of doing things (either documented or undocumented), but that is clearly a rash assumption. I've not had much experience of qmail, I did have an installation to work with once and found the bizarre way it stored its configuration files as multiple hidden . files in the /var directory structure (iirc) to be such an odd and annoying way of working that I never really considered it as an option for my own installations. A bit like Cyrus IMAP where it has its own way of configuring something (I really can't remember what now) that meant it wasn't suited to any form of local server based mail access - as an aside I see that this too uses the . separator for the folder hierarchy.

-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/

Chris Green wrote:

Thus a folder called 'holidays' is actually a directory called 'INBOX.holidays' and a folder called 'holidays/france' is a directory called 'INBOX.holidays.france'. Why any sort of mail server would want to construct a folder hierarchy like this I can never understand.

IMAP pretty much requires that you can find all the folder names pretty quickly. Having folders stored like this means you can find all folders with a single directory scan, without having to walk a whole filesystem tree.

bclements@io-networks.com wrote:

...

Chris Green wrote: Quoting Gregory Bond gnb@itga.com.au:

...

Thus a folder called 'holidays' is actually a directory called 'INBOX.holidays' and a folder called 'holidays/france' is a directory called 'INBOX.holidays.france'. Why any sort of mail server would want to construct a folder hierarchy like this I can never understand.

IMAP pretty much requires that you can find all the folder names pretty quickly. Having folders stored like this means you can find all folders with a single directory scan, without having to walk a whole filesystem tree.

Can we get back on topic guys?

Ok, maybe someone can enlighten me.

Can IMAP servers, any IMAP server just delete mail without a client requesting that it doe so?

Well being software and computer based I wouldn't categorically rule out anything in terms of possible bugs. I would, however, suggest that the likelihood is there is another cause somewhere - i.e. something that is requesting the deletion of the mail.

As such, are all the clients using the same mail client? Are they all in the same office? Perhaps someone has, as has been suggested, found a new feature that they've all tried. Is this still happening, or has it happened on a couple of occasions and not since?

You mentioned in a previous post that you are running on version 99.11. Is there a more recent package available for your distribution? I'm running Debian and on 0.99.14. Although I've not read about any issues of this sort, I can't see that it would necessarily hurt to upgrade to a the latest (stable) version. Although in the true debug process it would be preferable to identify exactly what is happening, sometimes this isn't practical - namely when somebody is breathing down your neck to get something fixed. In these cases the 'have you rebooted it?' Windows style fix might give you breathing space!

As a fallback measure, and not knowing your exact setup, is it worth setting up something like Procmail to deliver all mail into another location *as well as* the users mail folder. This way you have access to restoring any mail that comes in, not just that available from the last backup (I run a nightly backup of critical folders and weekly of the rest - keeping week end and month end backups for longer periods on a rotating basis).

OK, the usual disclaimers apply here - backup, backup, backup; and not having been using Dovecot for long (weeks in fact) and not having delved into the code I'm by no means an expert. It has, however, been frequently recommended to me - and I finally gave in and tried it :)

-- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/

On Tue, Aug 09, 2005 at 08:42:07PM +0100, Obantec Support wrote:

Hi

Yes a user reported this tonight. 30MB of emails gone! not happy and wants me to restore from backup. ( i had to explain we only backup website data).

Are you sure that the user's POP client is not set to remove
mail from the server?  This sounds like a user related issue and
not necessarily a dovecot issue.



						John

-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.

"My other computer is your windows box." Ralf Hildebrandt