- dovecot - dovecot.org

[Dovecot] Director pop-login and imap-login processes exiting on signal 11
by Andy Dills 09 Apr '12

09 Apr '12

We recently upgraded our cluster to 2.1.3, to enable director proxying. Everything appears to be working fine for the most part; the only odd thing is that I'm seeing a lot of entries in the logs like this: Apr 7 02:18:05 mail-out06 dovecot: pop3-login: Fatal: master: service(pop3-login): child 75029 killed with signal 11 (core not dumped - set service pop3-login { drop_priv_before_exec=yes }) This is on the proxy side, not that backend side. When I try to get a dump out of it, and add drop_prive_before_exec and chroot= to the pop3-login statement on the proxy, I keep running into permissions errors with the various service sockets. Any suggestions? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---

2 2

[Dovecot] v2.0.20 released
by Timo Sirainen 09 Apr '12

09 Apr '12

http://dovecot.org/releases/2.0/dovecot-2.0.20.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.20.tar.gz.sig + doveadm user: Added -m parameter to show some of the mail settings. - doveadm import didn't import messages' flags - Make sure IMAP clients can't create directories when accessing nonexistent users' mailboxes via shared namespace. - Dovecot auth clients authenticating via TCP socket could have failed with bogus "PID already in use" errors.

1 0

[Dovecot] Director simplification?
by Timo Sirainen 09 Apr '12

09 Apr '12

An idea I just had: Director basically works by assigning the backend IP address by: ip = vhosts[ md5(username) mod vhosts_count ].ip The rest of director is about what happens when vhosts[] or vhosts_count changes. What about instead doing this on IP address level? ip = ip_pool[ md5(username) mod ip_pool_size ] When a backend dies, you'll reassign the backend's IPs to other backends. Each backend should have many IPs. The main restriction here is that the IP pool cannot change without stopping the entire Dovecot. But if you initially allocate enough IPs, that shouldn't be a problem. And the advantage of this over the current director? To guarantee that one director can't break others, because they don't need to communicate with each others. The disadvantage of course is that it's a little less flexible and requires more planning ahead. The IP address reassignment would also need some distro-specific scripts. This could be implemented as an alternative director-lite or something. The doveadm director status-related commands could still work with it.

1 0

[Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders
by Thomas von Eyben 09 Apr '12

09 Apr '12

Hi, I am seeing a >10x as slow performance when trying to complete a "send/receive" from an Outlook 2010 client to Dovecot via IMAP, but only when the LAN is fully loaded with other traffic, EG file copying. It seems the problem is when outlook is trying to identify folders that have changed since last "send/receive" thus traversing the hierachy. Observations: - Apple's Mail.app does not have problems when exposed to the same environment. - Eliminating Outlook is not an option. - The tests have been performed during the easter vacation, thus almost no other clients are using the network/mailserver. Under normal load there are 2-300 users using the network. Description of the environment: Server sw: Mac OS X Server 10.6.8 running dovecot: 1.1.20apple0.5 (OS = fully updated 10.6.x) Server hw: Xserve Quadcore intel Xeon 2.26 GHz 12 GB RAM - Never having a cpu load > 20%, mail is stored on a Promise vtrack RAID connected via Fibre Channel Client: Windows 7, Outlook 2010 The client have maybe 50 folders (and each folder about a handfull of subfolders) the size of the mailbox is around 3-5 GB. Test results: CLIENT-1 is having the problems when CLIENT-2 is using all the (100Mbps) bandwidth eg. copying files to MAIL-SRV. If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone. NB.: I have not (yet) tested if the problem also exists when CLIENT-2 generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am expecting the same problems). When dumping the traffic on CLIENT-1 (with Wireshark) one thing is catching my eyes for the 'bad' case: There is a very long delay after each "REQUEST IDLE" until the next "REQUEST DONE" ca. 0,3 seconds. Comparing to the 'good' setup the pause at the same place in the communication is less than 1 ms ! So why this delay? Where/how shall I continue my debugging? - Run dtruss/dtrace scripts on the server? - Get dovecot to output more debug info (I guess it's doing it's best so no problems will be seen there…)? Are there any dovecot configurations that can be altered to workaround [what I think is a limitation in Outlook's IMAP implementation]? The current dovecot configuration is Apple's defaults (+POP3 disabled)) I am seing one warning from dovecotd -n though: - But I see the same warning on a clean installed not-yet-configured OS X Server so I guess it's not 'that' bad: Warning: fd limit 256 is lower than what Dovecot can use under full load (more than 456). Either grow the limit or change login_max_processes_count and max_mail_processes settings Physical setup: +---------------+ +------------------------------+ +-----------------+ | CLIENT-1 | | CLIENT-2 | | CLIENT-3 | | Outlook "bad" | | Traffic generator, eg. Samba | | Outlook "good" | +---------------+ +------------------------------+ +-----------------+ | | | +-------------------------------------------------+ | | 100Mbps switch (clients) | | +-------------------------------------------------+ | | | +------------------------------------------------------------------------+ | 1Gbps switch (for servers) | +------------------------------------------------------------------------+ | | +-----------------------------+ +-----------+ | MAIL-SRV: Dovecot, AFP, SMB | | OTHER-SRV | +-----------------------------+ +-----------+ Wireshark dump - IMAP communication: |Time | 10.211.55.3 | | | | 10.0.0.10 | |9.851 | Response: gatq OK L |IMAP: Response: gatq OK Logged in. | |(49433) <------------------ (143) | |9.852 | Request: o47u SELEC |IMAP: Request: o47u SELECT "1_GROUPS" | |(49433) ------------------> (143) | |9.853 | Response: * FLAGS ( |IMAP: Response: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) | |(49433) <------------------ (143) | |9.854 | Request: 3y4b NOOP |IMAP: Request: 3y4b NOOP | |(49433) ------------------> (143) | |9.854 | Response: 3y4b OK N |IMAP: Response: 3y4b OK NOOP completed. | |(49433) <------------------ (143) | |9.855 | Request: 4vlj IDLE |IMAP: Request: 4vlj IDLE | |(49433) ------------------> (143) | |9.856 | Response: + idling |IMAP: Response: + idling | |(49433) <------------------ (143) | |10.108 | Request: DONE |IMAP: Request: DONE | |(49433) ------------------> (143) | |10.108 | Response: 4vlj OK I |IMAP: Response: 4vlj OK Idle completed. | |(49433) <------------------ (143) | |10.108 | Request: wh89 SELEC |IMAP: Request: wh89 SELECT "1_GROUPS.Adm" BR Thomas von Eyben

4 4

[Dovecot] Setting ACL for master user after login
by PL MB 09 Apr '12

09 Apr '12

Hello, I'd like to log in to normal user accounts as a master user but retain the normal users' ACLs. The Master Users page on the Dovecot 1.x wiki (1) says that I can set the master user's ACLs in a postlogin script. The documentation for master users on the 2.x wiki (2) no longer has any statements about master user ACLs. Has something important in this regard changed? Can I no longer override the ACLs in a postlogin script? Thanks in advance. (1) http://wiki.dovecot.org/Authentication/MasterUsers (2) http://wiki2.dovecot.org/Authentication/MasterUsers

2 1

[Dovecot] POP3 dele to Trash?
by Kelsey Cummings 09 Apr '12

09 Apr '12

To simplify recovery of accidentally deleted messages we'd like to send all messages that are deleted by pop clients to their Trash folder. (Which is auto expired already.) This allows a POP client who deletes an important email to restore it themselves by logging into our webmail client and move the lost message from Trash to the Inbox where it would be fetched by their client again. Has anyone already done this? Should this be possible via a plugin? I see the deleted-to-trash imap plugin. We are using Maildir if it makes a difference. -- Kelsey Cummings - kgc(a)corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407

2 2

[Dovecot] distributed mdbox
by James Devine 09 Apr '12

09 Apr '12

Anyone know how to setup dovecot with mdbox so that it can be used through shared storage from multiple hosts? I've setup a gluster volume and am sharing it between 2 test clients. I'm using postfix/dovecot LDA for delivery and I'm using postal to send mail between 40 users. In doing this, I'm seeing these errors in the logs Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count 272 -> 271 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=3768 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but next_uid = 517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=4220 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=5088 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index This is my dovecot config currently: jdevine@test-gluster-client2:~> dovecot -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 lock_method = dotlock mail_fsync = always mail_location = mdbox:~/mdbox mail_nfs_index = yes mail_nfs_storage = yes mmap_disable = yes passdb { driver = pam } protocols = " imap" ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd }

5 19

[Dovecot] Using a namespace for providing access to mail snapshots for user based on-demand restoration of email backups
by Charles Marcus 09 Apr '12

09 Apr '12

Hi all, I'm planning on implementing this in my new upcoming dovecot instance, and would like to hear thoughts on how best to accomplish this. We will be paying Timo's support company to do the work, but obviously, the less work in the form of coding he has to do to get this working (I'm hoping it won't be a lot), the more money it will save us... ;) First - I currently use rsnapshot to backup emails, so that is the use-case I'm most interested in getting working. It is rsync based, and like other rsync based backup programs it uses hardlinks to save storage space - so you can have a *lot* of backups (going back months, or even years), where each snapshot only adds a little more to the total disk space being used. The snapshots are stored with the following filesystem layout: /path/to/snapshotsdir/hourly.0 ... /path/to/snapshotsdir/hourly.4 /path/to/snapshotsdir/daily.0 ... /path/to/snapshotsdir/daily.7 /path/to/snapshotsdir/weekly.0 ... /path/to/snapshotsdir/weekly.4 /path/to/snapshotsdir/monthly.0 ... /path/to/snapshotsdir/monthly.12 /path/to/snapshotsdir/yearly.0 ... /path/to/snapshotsdir/yearly.5 The 'names' (hourly, daily, weekly, monthly, yearly) are arbitrary (this is a bit confusing to people new to rsnapshot), and would *not* be used for displaying the mail folders to the users - it is the Date/Time stamps of each of the snapshot dirs above that would be used to display the folder names under the 'Time Machine' namespace. This is, I imagine, the part that will need some actual coding by Timo to get working - maybe just some new config variables added to the namespace code for mapping the date/time stamps of the directories to user friendly folder names in the namespace. That said, I'd like to design this and have it coded such that it will work with almost any type of backup storage that stores snapshots as date/time stamped directories like this (there must be others, right?). Also, it goes without saying that this code will be (if Timo is ok with it) part of the core dovecot code going forward, so anyone else will be able to benefit from it. What I'm envisioning is something like this... 1. Define a namespace - for this example we'll call it 'Time Machine' 2. Under this namespace, each user will see their, and *only* their snapshots So, each user would see something like this: My Mail Account Inbox Drafts Templates Sent Time Machine (sorted above user created folders if possible) -4/3/12, 8:00am (first subfolder) Inbox Drafts etc... (all other folders and sub-folders shown here) +4/3/12, 12:00pm (first subfolder) etc... Other User Folders ... Or even better, I'm thinking some magical code that can group them by Date, like: -4/3/12 (first subfolder) -8:00am (next sub-folder) Inbox Drafts Etc... (all folders and sub-folders shown here) +12:00pm +4:00pm +8:00pm +4/4/12 etc... Comments? Suggestions? Flames? -- Best regards, Charles

3 4

Re: [Dovecot] Using a namespace for providing access to mail snapshots for user based on-demand restoration of email backups
by Joseph Tam 09 Apr '12

09 Apr '12

A timely topic as I was just mulling over ways to provide this to my users. Charles Marcus wrote: > The snapshots are stored with the following filesystem layout: > > /path/to/snapshotsdir/hourly.0 > ... This is familiar to NetApp users. > The 'names' (hourly, daily, weekly, monthly, yearly) are arbitrary (this > is a bit confusing to people new to rsnapshot), and would *not* be used > for displaying the mail folders to the users - it is the Date/Time > stamps of each of the snapshot dirs above that would be used to display > the folder names under the 'Time Machine' namespace. This is, I imagine, > the part that will need some actual coding by Timo to get working - > maybe just some new config variables added to the namespace code for > mapping the date/time stamps of the directories to user friendly folder > names in the namespace. > > That said, I'd like to design this and have it coded such that it will > work with almost any type of backup storage that stores snapshots as > date/time stamped directories like this (there must be others, right?). One idea is to take this complexity entirely out of dovecot and create a synthetic filesystem using hard or soft links (as rsnapshot has done) and create your own, with whatever weird and wonderful naming scheme you want. /path/to/TimeMachine/<friendlylabel>/<user> -> /path/to/snapshotsdir/<snaplabel>/<user> namespace { prefix = TimeMachine location = maildir:/path/to/TimeMachine:INDEX=MEMORY ... } This might not be very scalable depending on how big your userbase is. I would probably define memory indices for this namespace, and take the performance hit on the assumption that access will be a once-in-a- while thing. On-disk indices will probably get out of date with each snapshot rollover, if if you have a lot of snapshots/mailboxes, it could consume a non-trivial amount of space without a lot of benefits. Or you could run a cron script to rename or remove old indices, but that seems more trouble than it's worth. One other consideration (at least for me) is if the INBOX and personal mail folders are stored in two separate FS's. It would be nice to fuse the two sets of backups under the same namespace, but I don't know how the namespace prefix matching works and whether you can define hierarchical namespaces like namespace { prefix = backup/inbox location = mbox:/path/to/inbox-snapdir/%u ... } namespace { prefix = backup/mail location = mbox:/path/to/mail-snapdir/%u ... } The above can also be accomplished with a synthetic filesystem. Joseph Tam <jtam.home(a)gmail.com>

2 1

[Dovecot] 2.1.3: Overly lax FETCH parsing
by Michael M Slusarz 09 Apr '12

09 Apr '12

These commands both work in Dovecot (as of 2.1.3): 1 UID FETCH 1:* () (CHANGEDSINCE 1) * 1 FETCH (UID 30097 MODSEQ (19554)) [...] 1 OK Fetch completed. 2 FETCH 1:* () (CHANGEDSINCE 1) * 2 FETCH (MODSEQ (19554)) [...] 2 OK Fetch completed. However, RFC 3501 indicates that an empty FETCH attribute list is incorrect: fetch = "FETCH" SP sequence-set SP ("ALL" / "FULL" / "FAST" / fetch-att / "(" fetch-att *(SP fetch-att) ")") fetch-att = "ENVELOPE" / "FLAGS" / "INTERNALDATE" / "RFC822" [".HEADER" / ".SIZE" / ".TEXT"] / "BODY" ["STRUCTURE"] / "UID" / "BODY" section ["<" number "." nz-number ">"] / "BODY.PEEK" section ["<" number "." nz-number ">"] In other words, fetch-att is not allowed to be empty. (RFC 4466 does not alter this requirement when using FETCH modifiers). While useful that Dovecot is more liberal about what it receives, 3501 seems pretty clear that incorrect FETCH parameters must return a BAD. I can verify that the above commands fail on Cyrus. I don't disagree that the way Dovecot handles this is correct - or at least is the way that I wish the RFCs were written. Since this seems a bit redundant, but is necessary under the ABNF: 1 UID FETCH 1:* UID (CHANGEDSINCE 1) -or- 1 UID FETCH 1:* (UID) (CHANGEDSINCE 1) michael

2 1