- dovecot - dovecot.org

[Dovecot] I need IMAP stress testing tool with SEARCH command
by Ernesto Revilla 14 Mar '10

14 Mar '10

Hi. Does anyone know of a good imap stress testing tool? And one which supports SEARCH commands? I have to compare Courier and Dovecot. I already did some tests using a modified postal/rabid package. I would be thankfully for any hints. Best regards. Erny Yaco Sistemas Spain

2 2

[Dovecot] Bug in driver-mysql.c + fix
by Alain Williams 13 Mar '10

13 Mar '10

I tried to use MySQL stored procedures from dovecot: password_query = CALL user_pass_check('%n', '%d', '%w') user_query = CALL user_info('%n', '%d') This failed with the message: User query failed: PROCEDURE imap.user_info can't return a result set in the given context The root of this problem is that mysql_real_connect() needs to be called with option CLIENT_MULTI_RESULTS and mysql_next_result() called to retrieve extra results (that will not exist in the way that we use it). I attach a patch that fixes this problem -- the patch is against dovecot-1.2.10. This works for me ... but could probably do with testing by other people. BTW: I got the same problem with exim this morning, wrote a patch that has now been accepted. The dovecot patch is similar. I am looking to use mysql procedures, there are some interesting things that can be done. Two suggestions that I have will help with this: 1) There be variable (say) %o - this be the obscured password, ie what password_query returns. 2) that dovecot look for either ''password_query'' as above, or ''password_check''. password_check would NOT return a password, but would be given %o and determine itself if the password is correct. It would return the other values (user, userdb_home, ...) and auth_result that would encode success/retry/fail (0, 1, 2 - or maybe more structured [**]) and auth_reason some human readable reason. The ''nologin'' value encodes some of this. The motivation for this is that my stored procedure will record the number of successive login failures and lock the account after 3 of them. It would also be possible to do time based restrictions & the such. Also: by passing %o the password is not sent in plain to the database server - which will increase security. I will publish my stored procedures when done. Regards [**] eg taking ideas from the HTTP codes: 200 - OK 300 5 - try again in 5 minutes 301 2 9 - try again on tuesday at 9am 400 - Login forbidden, no reason given 401 - bad username and/or password 402 - account locked administratively 403 - too many failed login attempts 500 - authentication system error The above would allow a native language version of auth_reason to be produced -- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php Past chairman of UKUUG: http://www.ukuug.org/ #include <std_disclaimer.h>

2 5

[Dovecot] Plugin like zlib
by Alex Baule 13 Mar '10

13 Mar '10

Hi everyone. I rebuild a plugin, based on zlib plugin. The changes between this plugin and zlib, is the zlib stuffs is replaced by open, read, lseek, close. With "plain-text" files, its works ok. So, my file is encrypt, and the result from decrypt file is different from fread. Like zlib, i read from fread the size variable, but the value added in seek_offset and pos is the result from decrypt, and it can be dfferent from fread. Like zlib, my file size is different from email. In zlib the file is smaller than the email. In my file, the file is bigger than the email. OK, lets go to the problem now. when the email arrives in the new folder, by imap, the stat (read my file by "while (i_stream_emexis_read(stream) > 0)" like zlib) and the seek works fine....the email is readed and the final stat size is the decrypted size of my email (smaller than original file). When I receive the email, clicking in "Send/Receive", The read function is aborted because the size to read (like zlib) is negative. I put some debug in my plugin, and print some stuffs about the functions. Email decryt size = 49556 Email encrypt size = 49568 Attachements: mail-arrived, is when the email arrives, and it's ok the read. mail-receive, is when the email is received, and have the error. maillog is the /var/log/mail The bigger difference from zlib is the i_stream_zlib_read part.... do { ret = read(zstream->fd ,crypttext, size); fprintf(zstream->debug,"DENTRO DA READ RET = %d -- ERRNO = %d -- STR = |%s|\n", ret, errno, strerror(errno)); if(ret > 0){ EVP_DecryptUpdate(&ctx, plaintext, &plain_len, crypttext, ret); memcpy(stream->w_buffer + stream->pos , plaintext, plain_len); bzero (plaintext, DEFAULT_MAX_BUFFER_SIZE); EVP_DecryptFinal(&ctx, plaintext, &plain_len_final); memcpy(stream->w_buffer + stream->pos + plain_len , plaintext, plain_len_final); } } while (ret < 0 && errno == EINTR && stream->istream.blocking); *retc *= plain_len + plain_len_final; if (ret == 0 ) { /* EOF */ stream->istream.eof = TRUE; return -1; } if (ret < 0) { if (errno == EAGAIN) { i_assert(!stream->istream.blocking); ret = 0; // } else { // i_assert(errno != 0); // stream->istream.stream_errno = errno; // return -1; } } fprintf(zstream->debug,"READ RET = %d -- ERRNO = %d -- STR = |%s|\n", ret, errno, strerror(errno)); fprintf(zstream->debug,"stream->pos = %d -- zstream->seek_offset = %d\n", stream->pos, zstream->seek_offset); zstream->seek_offset += *retc*; stream->pos += *retc*; i_assert(ret > 0); fprintf(zstream->debug,"READ RET DECRYPT = %d -- stream->pos = %d -- zstream->seek_offset = %d\n", ret, stream->pos, zstream->seek_offset); fprintf(zstream->debug,"********************** FINISH READ *********************\n"); fflush(zstream->debug); return *retc;* The* retc* is the result from decrypt. In mail-receive file (line 38), there is the error, at some point along the read/send email, the stream->pos (its returned by i_stream_compress, with stream->pos -= stream->skip) is changed out from context and the size to read is negative, but i don't understand why. Someone can help me find out what happens ?

2 1

[Dovecot] Sendmail / Dovecot Config
by John Moorhouse 13 Mar '10

13 Mar '10

I'm trying to set up dovecot on a Fedora box for about 5 users, I have a raid 5 array that is used for home folders etc mounted on /home, and I've created a directory /home/mail, which I have used a sym link to map /var/mail to. I've created folders for each of the users with relevant permissions and edited the dovecote config with:- namespace private { separator = / prefix = "#dbox/" location = dbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no # for v1.1+ } namespace private { separator = / prefix = location = dbox:/var/mail/%u/mail } I was hoping that this would create a inbox in the root of each users folder in the mail directory and a sub folder to hold any user created folders. This all seems to work as I can create folders, copy emails to them r-arrange the folders etc as I want via a mail client using imap on 143. My only problem is that I have also configured send mail as best as I know to listen on port 25 as an SMPT client but any emails sent to it to one of the email addresses are saved by sendmail to /var/spool/mail/username and not picked up by dovecot Thanks John

2 2

[Dovecot] imap support for attachement deletion?
by Louis-David Mitterrand 13 Mar '10

13 Mar '10

Hi, Does dovecot support in-place attachement deletion? Mutt currently uploads a new message after deleting attachements. Is there a better, imap way to do it?

1 0

[Dovecot] how quotas works with postfix and dovecot
by Wilberth Pérez 13 Mar '10

13 Mar '10

Hi everybody any one knows, how i could edit dovecot to assign user quotas ? I have now configured my dovecot.conf on this way: protocol imap { listen = *:143 mail_plugins = quota imap_quota } protocol pop3 { listen = *:110 mail_plugins = quota } plugin { quota = fs:INBOX:mount=/ quota_warning = storage=95%% /usr/local/bin/quota-warning.sh 95 } however when a test my quota: quota -v user1 Disk quotas for user1 (uid 107): Filesystem *usage* *quota* limit timeleft files quota limit timeleft /var/mail 16911 10240 20480 NOT STARTED 49 1000 1500 dovecot or postfix (i don't know who really should sendme the messsage) never sendme an quota warning someone knows what happen ? -- -------------------------------------------------------- LCC Wilberth de Jesús Pérez Segura CCSA- Administración de Servicios y Seguridad de las TI Correo:wilberth.perez@uady.mx Universidad Autónoma de Yucatán Secretaría General Coordinación Administrativa de Tecnologías de Información RIUADY C-59 x Av. Itzáes (999)923-74-28 Ext. 1117 Mérida, Yucatán, México 2010 -------------------------------------------------------- -- -------------------------------------------------------- LCC Wilberth de Jesús Pérez Segura CCSA- Administración de Servicios y Seguridad de las TI Correo: wilberth.perez(a)uady.mx Universidad Autónoma de Yucatán Secretaría General Coordinación Administrativa de Tecnologías de Información RIUADY C-59 x Av. Itzáes (999)923-74-28 Ext. 1117 Mérida, Yucatán, México 2010 --------------------------------------------------------

3 5

[Dovecot] Emails Disappearing -- Mystery
by Mario Antonio 12 Mar '10

12 Mar '10

I got a case that I haven't been able to resolve. I hope somebody can give me some hints. One of our users is reporting emails disappearing in front of his eyes, (Right ... emails filters at the client side ....) Server does not have global sieve filter enabled (no file present), even though this configuration is present: sieve_global_path=/vmail/globalsieverc User does not have any sieve scripts enabled at the server side. Ok, 1) The user shuts down his apple mail software, his Iphone, and the Web interface: 2010-03-11 15:52:57 IMAP(my_user(a)mydomain.com): Info: Connection closed bytes=2188/6499 2010-03-11 15:52:57 IMAP(my_user(a)mydomain.com): Info: Connection closed bytes=3887/7336 2010-03-11 15:52:57 IMAP(my_user(a)mydomain.com): Info: Connection closed bytes=5873/68309 2010-03-11 15:52:57 IMAP(my_user(a)mydomain.com): Info: Disconnected in IDLE bytes=4514/547136 2) He generates the troublesome emails from a notification system in Craiglist .... We can see the email hitting the inbox (we also see in Postfix this email going to Dovecot) 2010-03-11 15:54:48 deliver(my_user(a)mydomain.com): Info: From=noreply(a)craigslist.org : msgid=<20100311205447.0C0FE118B04(a)web8p.int.craigslist.org> : saved mail to INBOX 3) Email is deleted !!! 2010-03-11 15:54:49 IMAP(my_user(a)mydomain.com): Info: delete: uid=10719, flags=(\Deleted \Recent $Junk), msgid=<20100311205447.0C0FE118B04(a)web8p.int.craigslist.org>, from=craigslist.org <noreply(a)craigslist.org>, subject=POST/EDIT/DELETE : (accounting/financejobs) Test Post 2010-03-11 15:54:49 IMAP(my_user(a)mydomain.com): Info: expunge: uid=10719, flags=(\Deleted \Recent $Junk), msgid=<20100311205447.0C0FE118B04(a)web8p.int.craigslist.org>, from=craigslist.org <noreply(a)craigslist.org>, subject=POST/EDIT/DELETE : (accounting/finance jobs) Test Post , size=2776 Who is deleting this email ? I can see the flag $Junk ... perhaps some filter somewhere is seeing that flag and proceeding to delete it. If I understand sieve filters at the server side, they will act in the deliver process. So once the email is saved into INBOX, sieve filters will not count. Am I right? Any Hints? # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686-bigmem i686 Debian 5.0.4 xfs Regards, Mario Antonio

2 2

Re: [Dovecot] nosql (it was mogilefs)
by fernando＠dfcom.com.br 12 Mar '10

12 Mar '10

Yes, I´ve read these doc. I was just considering if someone had any experience with Mogile in email environment. Well, actually I´m trying to figure out if one of those nosql systems can store and scale email systems (ie, mailboxes - instead of maildir). I know mysql is always quoted as an option and discussed sometimes in the list. I think it´s an unnecessary overhead and bottleneck. As brainstorm. How about Cassandra, Tokyo-Tirant, Redis, and so on. Have you ever worked or tested one of them at production or even testing environment ? What do you think ? Best Regards, Fernando >---- Original Message ---- >From: Robert Schetterer <robert(a)schetterer.org> >To: dovecot(a)dovecot.org >Sent: Qua, Mar 10, 2010, 4:46 PM >Subject: Re: [Dovecot] mogilefs > >Am 10.03.2010 20:23, schrieb fernando(a)dfcom.com.br: >> Hi list, >> >> Have you ever heard or tried mogilefs? (http://danga.com/mogilefs/) >> >> Do you think suitable for dovecot/mail environments ? >> >> Regards, >> Fernando >> > >sounds not ready for mail > >MogileFS is not: > > * POSIX Compliant -- you don't run regular Unix applications or >databases against MogileFS. It's meant for archiving write-once files >and doing only sequential reads. (though you can modify a file by way of >overwriting it with a new version) Notes: > o Yes, this means your application has to specifically use a >MogileFS client library to store and retrieve files. The steps in >general are 1) talk to a tracker about what you want to put or get, 2) >read/write to one of the places it told you you could (it'll pick >storage node(s) for you as part of its load balancing), using HTTP GET/PUT > o We've prototyped a FUSE binding, so you could use MogileFS >without application support, but it's not production-ready. > > >-- >Best Regards > >MfG Robert Schetterer > >Germany/Munich/Bavaria

2 1

[Dovecot] patch: allow proxy to lookup host by name
by Martin F. Foster 12 Mar '10

12 Mar '10

This patch allows the dovecot proxy processes to lookup the destination host by name instead of IP address. Tested agains 1.2.10, expected to work against 1.2.11. The patch is pretty straightforward, it's making it work within the restrictions of the login process that's more interesting. I have made some changes to the wiki (pending approval) to - enhance the discussion of proxy with LDAP http://wiki.dovecot.org/PasswordDatabase/ExtraFields/Proxy - discuss the implication of this patch w.r.t. the login_chroot configuration parameter: http://wiki.dovecot.org/PasswordDatabase/ExtraFields/Proxy/HostLookup Feedback is appreciated, particulary from a security standpoint. Cheers, -Martin Foster martin_foster(a)netlog.net dump of wiki info, for the lazy. It will look better if viewed from the URL: http://wiki.dovecot.org/PasswordDatabase/ExtraFields/Proxy/HostLookup For all released versions of Dovecot, the "host" referred to in a proxy or proxy_maybe lookup must be an IP address see: the main PasswordDatabase/ExtraFields/Proxy <http://wiki.dovecot.org/PasswordDatabase/ExtraFields/Proxy> page for more info A patch is available to enable lookups, but it has some imporant caveats due to the nature of the LoginProcess <http://wiki.dovecot.org/LoginProcess> that the proxy function is a part of. This page exists to discuss these. The Problem Proxying is done from the dovecot login processes, both pop3-login & imap-login call code in src/login-common/login-proxy.c to handle the proxying to the given host. If the host is not an IP address, name resolution must be done. Two options from dovecot's configuration: 1. Dovecot running with roots Dovecot is started as root and drops privileges later; this is the recommended way of running Dovecot. Set by the config option: login_chroot = yes. In this mode, the login process is chrooted to login_dir, from which the proposed patch adds name resolution. Resolution is done via dovecot's net_gethostbyname() function - a wrapper for getaddrinfo/gethostbyname depending on compile-time platform support. These functions need access to the name service switch configuration, then whatever dependencies are required to consult the host resolution databases stipulated. so, if the /etc/nsswitch.conf hosts entry is: hosts: dns files Then the system will need access to /etc/resolv.conf and the nss_dns libraries for "dns" lookup, and /etc/hosts for "files" lookup. The exact files will vary by platform/operating system, but these all need to be available in the chroot for the lookup to succeed. and, if dovecot.conf has: login_chroot = yes login_dir = /var/run/dovecot/login login_user = dovecotuser Then for the nsswitch.conf entry above, the following files need to be copied (not symbolically linked!) to the chroot. This example is for a 64-bit RHEL5 system, with 64-bit dovecot daemon. Ownership of the new directories must be set to whatever the login_user is set to. mkdir /var/run/dovecot/login/etc mkdir /var/run/dovecot/login/lib64 cp /etc/nsswitch.conf /etc/resolv.conf /etc/hosts /var/run/dovecot/login/etc cp /lib64/libnss_dns.so.2 /var/run/dovecot/login/lib64 chown -R dovecotuser:dovecotuser /var/run/dovecot/login/etc chown -R dovecotuser:dovecotuser /var/run/dovecot/login/lib64 Remember that the ownership of the login_dir itself *must not* be changed. So: # ls -ld /var/run/dovecot/login/ drwxr-x--- 4 root dovecotuser 4096 Mar 12 06:48 /var/run/dovecot/login/ Troubleshooting If the lookup fails because of a perceived lookup-in-chroot issue, an error message of this form will be printed to the logs dovecot: pop3-login: proxy(test(a)domain1.test): cannot resolve mailhost.domain1.test. If name resolution is working outside dovecot, it may be a chroot issue. See LoginProcess on wiki, and login_dir& login_chroot in config. Clients will receive a much less descript general error message, for example with POP3: -ERR [IN-USE] Account is temporarily unavailable. Things to check 1. does name resolution work for normal users? the dovecot user? 2. are all the files required by the Name Service Switch's configuration available? Use a process tracing tool such as strace or truss against the pop3-login or imap-login processes to check. 3. are other security measures interfering? eg: 1. SELinux <http://en.wikipedia.org/wiki/Security-Enhanced_Linux> (RedHat <http://www.redhat.com/rhel/> EL, CentOS, Oracle EL) 2. AppArmor <http://en.wikipedia.org/wiki/AppArmor> (SuSE, Ubuntu?) 3. RBAC (Solaris, OpenSolaris <http://en.wikipedia.org/wiki/Open_Solaris>) Caveats This procedure allows the chroot'ed login to do something, which forms a security risk if the libraries in the chroot are exploitable. It will be up to the operator to ensure that the copy of the files in the chroot get updated whenver the copies in the real filesystem do. On systems where the chroot is on the same filesystem/partition that the original file copies are, hard links (again: no soft links!) could be used to accomplish this. 2. Dovecot running without roots (RootLess) Dovecot is started as whatever user/group is used to start the master daemon. See the discussion on Rootless <http://wiki.dovecot.org/HowTo/Rootless> for reasons why this is not the recommended way of running Dovecot, which includes a discussion on how this is more likely to be less secure. In dovecot.conf, rootless is selected by setting: login_chroot = no In this mode, the login process will have access to the required Name Service Switch configuration files and their dependencies if the user that started dovecot did. No further configuration is required for the proxy process to lookup hostnames.

4 3

[Dovecot] Scalability plans: Abstract out filesystem and make it someone else's problem
by Timo Sirainen 12 Mar '10

12 Mar '10

This is something I figured out a few months ago, mainly because this one guy at work (hi, Stu) kept telling me my multi-master replication plan sucked and we should use some existing scalable database. (I guess it didn't go exactly like that, but that's the result anyway.) So, my current plan is based on a couple of observations: * Index files are really more like memory dumps. They're already in an optimal format for keeping them in memory, so they can be just mmap()ed and used. Doing some kind of translation to another format would just make it more complex and slower. * I can change all indexing and dbox code to not require any locks or overwriting files. I just need very few filesystem operations, primarily the ability to atomically append to a file. * Index and mail data is very different. Index data is accessed constantly and it must be very low latency or performance will be horrible. It practically should be in memory in local machine and there shouldn't normally be any network lookups when accessing it. * Mail data on the other hand is just written once and usually read maybe once or a couple of times. Caching mail data in memory probably doesn't help all that much. Latency isn't such a horrible issue as long as multiple mails can be fetched at once / in parallel, so there's only a single latency wait. So the high level plan is: 1. Change the index/cache/log file formats in a way that allows lockless writes. 2. Abstract out filesystem accessing in index and dbox code and implement a regular POSIX filesystem support. 3. Make lib-storage able to access mails in parallel and send multiple "get mail" requests in advance. (3.5. Implement async I/O filesystem backend.) 4. Implement a multi-master filesystem backend for index files. The idea would be that all servers accessing the same mailbox must be talking to each others via network and every time something is changed, push the change to other servers. This is actually very similar to my previous multi-master plan. One of the servers accessing the mailbox would still act as a master and handle conflict resolution and writing indexes to disk more or less often. 5. Implement filesystem backend for dbox and permanent index storage using some scalable distributed database, such as maybe Cassandra. This is the part I've thought the least about, but it's also the part I hope to (mostly) outsource to someone else. I'm not going to write a distributed database from scratch.. This actually should solve several issues: * Scalability, of course. It'll be as scalable as the distributed database being used to store mails. * NFS reliability! Even if you don't care about any of these alternative databases, this still solves NFS caching problems. You'd keep using the regular POSIX FS API (or async FS api) but with the in-memory index "cache", so only a single server is writing to mailbox indexes at the same time. * Shared mailboxes. The filesystem API is abstracted, so it should be possible to easily add another layer to handle accessing other users' mails from both local and remote servers. This should finally make it possible to easily support shared mailboxes with system users.

17 82