December 2006 - dovecot

[Dovecot] Huge IMAP folder takes forever
by Peter Lindgren 10 Dec '06

10 Dec '06

Hi! Just updated to 1.0 rc15, but the problem remains. I think it could be client-dependent to some extent. I have only IMAPS access and Maildir. I have one mailbox which contains some 10,200 mails. That mailbox takes forever (have waited 30 minutes and then aborted) to open with Thunderbird 1.5.0.7. Other mailboxes (like dovecot :-), only 6984 mails) opens quickly. When Thunderbird tries to open this folder, the imap process on the server uses roughly all cpu resources. There isn't much disk I/O, at least as far as I can tell by listening at the server. The dovecot is upgraded from 0.99.something. Should I remove index files first, and in that case which? Is there something cached in Thunderbird? Here's the directory listing for the troublesome folder: -rw------- 1 peter peter 28 Jan 7 2006 .customflags -rw------- 1 peter peter 49248 Feb 7 2006 .imap.index -rw------- 1 peter peter 420998 Feb 7 2006 .imap.index.data -rw------- 1 peter peter 10256 Feb 7 2006 .imap.index.log -rw------- 1 peter peter 49200 Feb 7 2006 .imap.index.tree drwx------ 2 peter peter 686592 Dec 10 15:06 cur -rw------- 1 peter peter 64 Jun 6 2006 dovecot-keywords -rw------- 1 peter peter 619832 Dec 10 15:04 dovecot-uidlist -rw------- 1 peter peter 162992 Dec 10 15:21 dovecot.index -rw------- 1 peter peter 8826880 Dec 9 15:36 dovecot.index.cache -rw------- 1 peter peter 33776 Dec 10 15:22 dovecot.index.log -rw------- 1 peter peter 193660 Dec 3 12:50 dovecot.index.log.2 drwx------ 2 peter peter 512 Dec 10 15:06 new drwx------ 2 peter peter 512 Dec 10 15:04 tmp Thanks in advance, Peter -- Peter Lindgren, dataingenjör E-post: peter "at" norrskenkonsult punkt com http://www.norrskenkonsult.com

1 2

[Dovecot] Configuring for virtual users
by Gerard Seibert 10 Dec '06

10 Dec '06

I haven't tried this yet; however, I would like to know if it would work. Using Postfix, I am setting up numerous virtual users. These users obviously do not have an account on the mail server. I have created a "domain.net" account. Assuming that Postfix is properly set up to deliver to the specified accounts, which will be held under the "domain.net" directory, will this work in the dovecot.conf file: # default_mail_env = maildir:/var/mail/%d/%u or do I need to modify it somehow? -- Gerard

2 1

[Dovecot] Full text search indexing
by Timo Sirainen 10 Dec '06

10 Dec '06

There are now two full text search backends in CVS HEAD: - squat: My own full text search index based on ideas from Cyrus Squat indexes. Supports substring searches. - lucene: Uses CLucene library. It should be pretty easy to add support for more backends. As I've explained a couple of times already, IMAP RFC says that searches are done by matching substrings, so the Lucene backend isn't fully RFC compliant. Currently it's anyway used to optimize standard SEARCH BODY/TEXT rules, but I'll probably make it optional and instead add new X-NONEXACT-BODY/TEXT rules. Then it will be possible to use both Lucene and Squat backends to optimize different searches (eg. Lucene for your modified webmail and Squat for standard IMAP clients). The Squat index finally worked better than I thought. It has two limitations: - Search strings must be at least 4 characters long - Those 4 characters can't contain spaces So eg. searching for "ip add" doesn't work, but "ip addr" does. By "doesn't work" I mean it fallbacks to the slow old way of going through all the mails. The space limitation exists only because I thought that such searches are going to be rare, and this reduces the index size by 20-50%. So the Squat indexing works by looking at 4 character blocks. So when searching for "address": - Get a list of UIDs for "ress" - Filter out UIDs which don't appear for "dres" - The same for "ddre" and "addr" - Open all the messages that are left in the UID list and drop out the messages where the whole "address" word doesn't exist. I think searching the words backwards usually finds fewer hits and makes the search faster. As for the index sizes, indexing my dovecot mbox with 7495 messages and size of 32479980 bytes it produced: dovecot.index.search : 2166526 bytes dovecot.index.search.uids : 6074155 bytes So that's about 25% of the mailbox size. Last I checked, Cyrus Squat took about 120% (with different messages). Anyway as you can see the uids file is larger than the search trie file, so the number of messages is more important in determining the index size than the number of bytes. The initial index build is pretty slow, but adding new messages to the existing index should be pretty fast. The new messages aren't indexed until client does a SEARCH BODY/TEXT. I'm not sure if the messages should be optionally indexed immediately as they're seen (or even added by Dovecot LDA). Oh and another difference to Cyrus Squat is that Dovecot's version indexes 16bit characters and not bytes. So non-English messages get indexed much more nicely. If you want to try this yourself, you'll need to add: protocol imap { .. mail_plugins = fts fts_squat } plugin { fts = squat }

1 0

Re: [Dovecot] Blocked
by Joseph Tam 09 Dec '06

09 Dec '06

On Fri, 8 Dec 2006, dovecot-request(a)dovecot.org wrote: > Date: Fri, 08 Dec 2006 12:33:04 +0100 > From: Jos Chrispijn <jos(a)webrz.net> > Subject: Re: [Dovecot] Blocked > > ... > I block pop3 to force my users to use webmail (that is in fact using IMAP). > What mechanism should trigger sending a message that pop3 is blocked? > Once this message is send, it is either send by postfix to alternative > email account (Postfix forward) or just read once thru Webmail. In the > mean time this user is trying to establish pop3 on a 24/7 basis $-) If you are not selective on which users get the message (i.e. you want this notice to go out to all POP users), then you can do what the previous poster said and make a read-only mailbox and redirect the user's INBOX to this file (via symlinks, or dovecot.conf or what-have-you). This is probably the easier route. Another easy solution might to use tcp-wrappers and reply to all connections to port 110 with -ERR Please contact ... Some (most?) mail readers will pop this message to the user. Another solution which I employed (for another reason) is to make your own mini-dameon which enters into a fake POP session. Just allow any authentication information and server up your fake notice. The POP3 protocol is fairly simple and you only have to implement a small subset of commands: user(noop), pass(noop), list, stat, retr, top, rset(noop), noop(noop), dele(noop), quit. I return a "OK" for most of these commands and most of them are noops. I wrote a patch for qpopper to do this but you could probably modify the source to make it a stand-alone server: http://www.math.ubc.ca/~tam/Software/Qpopper.M2W/m2w.patch For the curious, the reason why I had to make this bizarre patch: http://www.math.ubc.ca/~tam/Software/Qpopper.M2W/ Joseph Tam <tam(a)math.ubc.ca>

1 0

[Dovecot] Dovecot+outlook express timeout error
by Artur Słowik 09 Dec '06

09 Dec '06

Hi When my outlook express client first time connect to Dovecot pop3 server, I receive error timeout 60s. When I close client and run client again then client receive mail correctly. I have in config file: //pop3_client_workarounds = outlook-no-nuls, oe-ns-eoh// best regards Arti

1 0

Re: [Dovecot] Blocked
by Kenny Dail 08 Dec '06

08 Dec '06

> I block pop3 to force my users to use webmail (that is in fact using IMAP). > What mechanism should trigger sending a message that pop3 is blocked? > Once this message is send, it is either send by postfix to alternative > email account (Postfix forward) or just read once thru Webmail. In the > mean time this user is trying to establish pop3 on a 24/7 basis $-) What's wrong with just ignoring the login attempts? Why not stop listening for POP3 altogether? Or, you can block the particular user at the firewall. Or if you have a special someone who is allowed POP3 you can give them an alternate port to login with, and not listen at all on port 110 if it really bothers you. -- Kenny Dail <kend(a)amigo.net>

1 0

Re: [Dovecot] Blocked
by Jos Chrispijn 08 Dec '06

08 Dec '06

Dominic Marks: > An alternative which does this is to replace their mailbox with > a read-only mbox or maildir containing a single mail with the > 'you have been blocked' message you want. Obviously you would > need to allow them to log in again, but they would only receive > the message you choose. I block pop3 to force my users to use webmail (that is in fact using IMAP). What mechanism should trigger sending a message that pop3 is blocked? Once this message is send, it is either send by postfix to alternative email account (Postfix forward) or just read once thru Webmail. In the mean time this user is trying to establish pop3 on a 24/7 basis $-) Jos

2 1

[Dovecot] Blocked
by Jos Chrispijn 08 Dec '06

08 Dec '06

I have blocked some pop3 user sessions: Dec 8 00:03:38 xxxxxx pop3-login: Disconnected: user=<johndoe> Is there a way of sending a message to a user that constantly tries to establish a block pop3 session? Example: From: noreply(a)mysite.com Subject: Your POP3 session Message: Hello, Your POP3 session is blocked; please contact systemmanagement --

3 2

Re: [Dovecot] Blocked
by Jos Chrispijn 08 Dec '06

08 Dec '06

Alan Premselaar schreef (08-12-06 3:06): > if you're blocking their POP3 access, how are those users ever supposed > to receive that mail notification? > That's simple: i am blocking them on username. Example: I am blocking johndoe's pop3 session. In Postfix I forward all mail for johndoe to johndoe(a)another.email.adress.com So johndoe can still receive mail. > Also, what if it's a brute force bot or something trying to access a > random blocked account via POP3? (alright, not as likely, but still > possible) > There you've got a point... > a more suitable solution would be if there was a way to return an error > during the authentication process that was returned as a result of > trying to authenticate to a blocked user. > The text I am sending could be any text, from 'your pop3 box = blocked' to '<any errornumber> on your POP3 box' Jos

1 0

[Dovecot] specifying INBOX with maildir for mail_location
by Post Master 08 Dec '06

08 Dec '06

I'd like to store all of my mail in maildir, but I want to store the inbox and indices on fast storage and everything else on deep storage. Does the following configuration value for mail_location accomplish this? mail_location = maildir:/data/deep/%Ld/mail/%Ln:INBOX=/data/fast/%Ld/mail/%Ln:INDEX=/data/fast/%Ld/indices/%Ln ------------------------------------------------------------------------ If not, does the following configuration accomplish this with namespaces? mail_location = maildir:/data/fast/%Ld/mail/%Ln:INDEX=/data/fast/%Ld/indices/%Ln namespace private { separator = / inbox = yes hidden = yes prefix = location = } namespace private { separator = / hidden = yes prefix = ??? location = maildir:/data/deep/%Ld/mail/%Ln/??? } ------------------------------------------------------------------------ Actually, I can't quite finish the namespace configuration. I'd like to set something like: prefix = * location = maildir:/data/deep/%Ld/mail/%Ln But I suspect the '*' won't actually match everything, and even if it did, I suppose it would also match the inbox. I began with listing any and every typical folder in its own namespace like: namespace private { separator = / hidden = yes prefix = drafts location = maildir:/data/deep/%Ld/mail/%Ln/drafts } namespace private { separator = / hidden = yes prefix = templates location = maildir:/data/deep/%Ld/mail/%Ln/templates } namespace private { separator = / hidden = yes prefix = sent location = maildir:/data/deep/%Ld/mail/%Ln/sent } and so on, but that doesn't seem very elegant. Also, namespaces scare me for client compatibility. Am I missing something obvious?

2 2