From mcbdovecot at robuust.nl Thu Mar 1 00:37:24 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Wed, 29 Feb 2012 23:37:24 +0100 (CET) Subject: [Dovecot] remove messages once downloaded In-Reply-To: References: <4F4C8DB2.9060302@mobilia.it> <6477DAB9-6840-410D-875D-5885A7E9962D@leuxner.net> Message-ID: On Tue, 28 Feb 2012, Thomas Leuxner wrote: > Am 28.02.2012 um 09:46 schrieb Thomas Leuxner: > >> Assuming this is the task you could also use: >> >> $ doveadm expunge -u jane.doe at example.org before 2012-02-01 unseen >> >> Play around with the scope a bit before you actually run it: >> >> $ doveadm search -u jane.doe at example.org before 2012-02-01 unseen | wc -l > > Of course this was supposed to read 'seen' :/ Maybe so, maybe not. I've seen people with over 2GB of unread mail in their inbox alone. Subscribed to dozens of mailing lists, and just not reading 95% of all incoming email. Don't know how pop handles 'leave on server' with regard to mail storage. Will messages loose their 'unseen' state when doing something like a TOP msgnumber maxint or something similar? Will messages loose their 'unseen' state when an RSET is issued after downloading? If you don't want people to keep a lot of mail in popboxes, quota and sending out warning messages would be the better way to handle this, I think. Applying 'magic' to customer's email might not be seen as a very polite action. Just my 2 cents.. -- Maarten From dbenfell at gmail.com Thu Mar 1 02:54:40 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 16:54:40 -0800 Subject: [Dovecot] need simpler directions on user authentication Message-ID: <20120301005440.GA5290@-e> Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From dbenfell at gmail.com Thu Mar 1 03:40:56 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 17:40:56 -0800 Subject: [Dovecot] need simpler instructions for user authentication Message-ID: Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com From stan at hardwarefreak.com Thu Mar 1 04:15:27 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 20:15:27 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4BB559.6050405@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> Message-ID: <4F4EDBBF.40004@hardwarefreak.com> On 2/27/2012 10:54 AM, Charles Marcus wrote: > These two locations will be connected via a private Gb ethernet > connection, and each location will have its own internet connection (I > think - still waiting on some numbers to present to the owner to see > what he wants to do in that regard, but that will be my recommendation), > so bandwidth for replication won't be an issue. Say you're a boutique mail services provider or some such. In your own datacenter you have a Dovecot server w/64 processors, 512GB RAM, and 4 dual port 8Gb fiber channel cards. It's connected via 8 redundant fiber channel links to 4 SAN array units, each housing 120 x15k SAS drives, 480 drives total, ~140,000 random IOPs. This gear eats 36U of a 40U rack, and about $400,000 USD out of your wallet. In the remaining 4U at the top of the rack you have a router, with two GbE links connected to the server, and an OC-12 SONET fiber link (~$15k-20k USD/month) to a national ISP backbone. Not many years ago OC-12s comprised the backbone links of the net. OC-48s handle that today. Today OC-12s are most often used to link midsized ISPs to national ISPs, act as the internal backbone of midsized ISPs, and link large ISPs' remote facilities to the backbone. Q: How many concurrent IMAP clients could you serve with this setup before hitting a bottleneck at any point in the architecture? What is the first bottleneck you'd run into? The correct answer to this question, and the subsequent discussion that will surely take place, may open your eyes a bit, and prompt you to rethink some of your assumptions that went into the architectural decisions you've presented here. -- Stan From david at davidrenstrom.com Thu Mar 1 04:53:31 2012 From: david at davidrenstrom.com (=?us-ascii?Q?David_Renstrom?=) Date: Thu, 1 Mar 2012 03:53:31 +0100 Subject: [Dovecot] Permission problems Message-ID: <34D750D1EC41408BA8AAA495077048CD@daviddator> Hi, I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and Postfixadmin on Fedora. Almost everything works correctly now. The only thing bugging me right now is that I always have to change the access privileges of the directories in /var/run/dovecot/ manually to 777 to make it work. As you understand, I'm not particularly happy about this as it doesn't feel secure at all. And I also have to do this every time I reboot since Dovecot resets the old access priviliges (600). As you see I've got a vmail user who belongs to the mail group. Almost all directories under /var/run/dovecot belong to user root and group root. They all have the permissions rw--------. Below is some output from the log file dovecot.log: Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Mar 01 03:31:17 master: Error: service(auth): command startup failed, throttling Mar 01 03:31:46 imap-login: Info: Disconnected (auth failed, 1 attempts): method=PLAIN, rip=94.254.99.6, lip=91.123.204.174 What is wrong here and how can I correct it? Thanks in advance! /David R. From stan at hardwarefreak.com Thu Mar 1 06:16:47 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 22:16:47 -0600 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: <4F4EF82F.6090306@hardwarefreak.com> On 2/29/2012 8:53 PM, David Renstrom wrote: > Hi, > > I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and > Postfixadmin on Fedora. Almost everything works correctly now. The only > thing bugging me right now is that I always have to change the access > privileges of the directories in /var/run/dovecot/ manually to 777 to make > it work. As you understand, I'm not particularly happy about this as it > doesn't feel secure at all. And I also have to do this every time I reboot > since Dovecot resets the old access priviliges (600). > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. Is this a Fedora Dovecot RPM, or did you use source? Do you have selinux/apparmor enabled? Do you like shellfish? -- Stan From tss at iki.fi Thu Mar 1 06:30:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:30:19 +0200 Subject: [Dovecot] need simpler instructions for user authentication In-Reply-To: References: Message-ID: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> On 1.3.2012, at 3.40, David Benfell wrote: > My situation is this: I have postfix running on a Linode under Arch Linux. > A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." You're missing userdb. > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: userdb { driver = passwd } From tss at iki.fi Thu Mar 1 06:34:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:34:19 +0200 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: On 1.3.2012, at 4.53, David Renstrom wrote: > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. That's how they should be mostly. > Below is some output from the log file dovecot.log: > > Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory > /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) > missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Why is your auth process running as vmail? Show your dovecot -n output From dbenfell at gmail.com Thu Mar 1 07:53:37 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 21:53:37 -0800 Subject: [Dovecot] SOLVED - Re: need simpler instructions for user authentication In-Reply-To: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> References: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> Message-ID: <20120301055337.GC25866@-e> On Thu, Mar 01, 2012 at 06:30:19AM +0200, Timo Sirainen wrote: > > You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: Sorry. The Arch Linux package is dovecot-2.1.1-3. > > userdb { > driver = passwd > } It *looks* like this worked. Thank you *very* much! David Benfell benfell at parts-unknown.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From joshua at hybrid.pl Thu Mar 1 10:21:50 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Thu, 1 Mar 2012 09:21:50 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? Message-ID: Hi all, While reading the "Multiple locations, 2 servers - planning questions..." thread, it just hit me that I was planning to do something that seems to be a bit hazardous... I am setting up an high-availability server set, which consists of two (or more) servers with common disk space, all set behind redundant hardware load balancers. At first, there will be two servers and disk space will be kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what creates kind of networked RAID1 storage space). This will be done mostly to keep WWW service available in case when one of servers fails. However, if we have everything redundant, why not have the same with SMTP and POP3/IMAP? But - won't anything fail if two (or more) dovecots are accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If it is possible, that would be a good solution for the mentioned thread. If not - where are possible dangers, and are there any plans to make it possible in the future? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From jtam.home at gmail.com Thu Mar 1 10:44:58 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Thu, 1 Mar 2012 00:44:58 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid I would like to run various doveadm commands that involves all (mail) users like doveadm expunge -A mailbox Trash savedbefore 30d but any doveadm command that uses "-A" to iterate through all users will stop processing at the first account with UIDname mapping for utilities like ls, chown, etc. There are various workaround like iterating manually, or setting first_valid_uid to zero, or even reordering users to put all system accounts at the end, but is there a better way to do this? Joseph Tam From CMarcus at Media-Brokers.com Thu Mar 1 13:43:47 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 01 Mar 2012 06:43:47 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4EDBBF.40004@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> Message-ID: <4F4F60F3.5050508@Media-Brokers.com> On 2012-02-29 9:15 PM, Stan Hoeppner wrote: > Q: How many concurrent IMAP clients could you serve with this setup > before hitting a bottleneck at any point in the architecture? No idea how to calculate it... > What is the first bottleneck you'd run into? Unless this is a trick question, the OC-12 link (since it is only 644Mb), and the next bottleneck would be the 2 GbE server connections to the router (are these bonded? if so, what mode?... > The correct answer to this question, and the subsequent discussion that > will surely take place, may open your eyes a bit, and prompt you to > rethink some of your assumptions that went into the architectural > decisions you've presented here. Since the vast majority of our connections will be *local*, I'm unconcerned about the internet connect speeds (one office has a 100/10Mb Cable (Comcast Business Class) connection, the other will have a 100/100Mb fiber/ethernet connection). My main priority is that the user experience at each physical location be optimal, which is why I'm more focused on making sure each offices users are connected to only the local server for all services (file/print/mail). I also neglected to mention how each server would be physically connected to the network, which I guess I should have done, since I'm fairly sure that will be the bottleneck I should mostly be concerned about... My choices are, as I see it, single GbE connections, or add some multiport GbE cards (these Dells support up to 3 PCIe cards) and bond some ports together for each VM. 10GbE is simply not in our price range (and I don't think we need it anyway), although I did stumble on these while googling and am waiting on pricing, since they claim to be 'much cheaper': http://www.mellanox.com/ethernet/ Since neither the multi-port GbE cards or decent switches that have reliable support for bonding/teaming are really not that expensive (especially when comparing to 10GbE solutions), I don't really see any reason *not* to do this (at a minimum I'd get redundancy if one of the ports on the server failed), but I'm also not sure which mode would be best - round-robin or IEEE 802.3ad dynamic link aggregation? Obviously, I don't have the experience or expertise to answer these questions myself (never analyzed IMAP traffic to have an idea of the bandwidth each user uses, and probably wouldn't trust my efforts if I made the attempt). Hopefully, there are some people here who have a rough idea, which is why I brought this question up here. Oh - and I am/will be working with a local I.T. services company to help with the design and implementation (since obviously I don't have the experience to do this myself), and will be asking them these same questions, I just like to usually know the general answers to questions like this ahead of time, so that I know if the guys I'm hiring know what they are doing and are giving me the best options for my budget. Thanks for your thoughts... -- Best regards, Charles From mark at ecs.vuw.ac.nz Thu Mar 1 13:49:11 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:49:11 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> Message-ID: <4F4F6237.1060100@ecs.vuw.ac.nz> On 02/29/12 23:41, Timo Sirainen wrote: >> Oh, yes, this is a bug in Dovecot.. > > Hmm. Or it looked like a bug, since it replied only with "+", so I thought all auth mechanisms would have such a bug, but no.. So I'm not really sure why it's not sending more data. I don't have a Kerberos setup to test this with. v2.1's GSSAPI code is anyway identical to v2.0's. With auth debugging on a successful connection gives: Mar 2 00:33:34 bats dovecot: auth: Debug: auth client connected (pid=1584) Mar 2 00:33:34 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49116 Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 YIGaBgkqhkiG9xIBAgICAG+BijCBh6ADAgEFoQMCAQ+iezB5oAMCARKicgRwXldfEmBHqH3DiVbw7aXtx54iBNjo1Rv/KxBSK5G3TmYFm3YskYN/23EiaOQ0Tdyi4bc4jhv5cFWMpH/xM89wAFJVW8Ue27/fmCasfDWXE+i4TKA3UCm78Wy8YyiNVae8X341LspBk86R1Zl5MNRMvA== Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): Negotiated security layer Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 BQQF/wAMAAwAAAAA47846FHFUOykdXinGYvMKwH///8= Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: client out: OK 1 user=mark and the failing kmail gives Mar 2 00:38:08 bats dovecot: auth: Debug: auth client connected (pid=2720) Mar 2 00:38:08 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49118 resp= Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:38:08 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:38:08 bats dovecot: auth: Debug: client in: CONT Mar 2 00:38:08 bats dovecot: auth: GSSAPI(mark at ECS.VUW.AC.NZ,130.195.5.88): Invalid base64 data in continued response Mar 2 00:38:08 bats dovecot: auth: Debug: client out: FAIL 1 reason=Invalid base64 data in continued response so what bit of the code should I be looking at to see what happens between the "security context state completed" and the "client out"? cheers mark From tss at iki.fi Thu Mar 1 13:52:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 13:52:34 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6237.1060100@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> Message-ID: <1330602754.2081.34.camel@innu> On Fri, 2012-03-02 at 00:49 +1300, Mark Davies wrote: > so what bit of the code should I be looking at to see what happens > between the "security context state completed" and the "client out"? All of the code is in mech-gssapi.c Are these working vs. non-working Dovecots in same or different servers? From joseba.torre at ehu.es Thu Mar 1 13:52:55 2012 From: joseba.torre at ehu.es (Joseba Torre) Date: Thu, 01 Mar 2012 12:52:55 +0100 Subject: [Dovecot] need simpler directions on user authentication In-Reply-To: <20120301005440.GA5290@-e> References: <20120301005440.GA5290@-e> Message-ID: <4F4F6317.2090605@ehu.es> El 01/03/12 01:54, David Benfell escribi?: > Hi all, > > My situation is this: I have postfix running on a Linode under Arch > Linux. A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." > > I do not want virtual users. I do not particularly want SQL or LDAP. I > do have multiple domains. I do use Maildir. I already have SSL > certificates for the server which I did manage to get dovecot to quit > complaining about. > > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? > > Thank you! > David Benfell > dbenfell at gmail.com If you give us some information maybe we could help you. At least, we need the output of doveadm config -n and the relevant log lines. From mark at ecs.vuw.ac.nz Thu Mar 1 13:57:56 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:57:56 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330602754.2081.34.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> Message-ID: <4F4F6444.1050107@ecs.vuw.ac.nz> On 03/02/12 00:52, Timo Sirainen wrote: >> so what bit of the code should I be looking at to see what happens >> between the "security context state completed" and the "client out"? > > All of the code is in mech-gssapi.c Yes, I'm just trying to work out the flow of the calls in and out of there. > Are these working vs. non-working Dovecots in same or different servers? All the working and non working connections are against a single dovecot instance, just using different clients. cheers mark From tss at iki.fi Thu Mar 1 14:04:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 14:04:30 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6444.1050107@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> Message-ID: <1330603470.2081.37.camel@innu> On Fri, 2012-03-02 at 00:57 +1300, Mark Davies wrote: > > On 03/02/12 00:52, Timo Sirainen wrote: > >> so what bit of the code should I be looking at to see what happens > >> between the "security context state completed" and the "client out"? > > > > All of the code is in mech-gssapi.c > > Yes, I'm just trying to work out the flow of the calls in and out of there. The problem is that that mech_gssapi_sec_context() calls gss_accept_sec_context(), which is supposed to return some output in output_token, but it doesn't. So I don't think there's anything in Dovecot code that is helpful in debugging this. You'd have to look into the GSSAPI/Kerbereros libraries. > > Are these working vs. non-working Dovecots in same or different servers? > > All the working and non working connections are against a single dovecot > instance, just using different clients. Oh. So GSSAPI in general is working, just not with kmail. I think if you downgraded to Dovecot v2.0 in your current system it would fail as well. The difference between your previously working system and currently working system is the GSSAPI/Kerberos libraries. From campbell at cnpapers.com Thu Mar 1 16:16:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 01 Mar 2012 09:16:39 -0500 Subject: [Dovecot] Multiple namespaces seems to be used at the same time Message-ID: <4F4F84C7.1060502@cnpapers.com> I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Thanks for any help. The learning curve is getting less steep, but it's still a ways off to the top of the hill. steve campbell From xchris89x at googlemail.com Thu Mar 1 17:08:28 2012 From: xchris89x at googlemail.com (Chris) Date: Thu, 1 Mar 2012 16:08:28 +0100 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: 2012/2/28 Ralf Hildebrandt : > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. Is that fixed? -- Chris From tss at iki.fi Thu Mar 1 18:15:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 18:15:05 +0200 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: On 28.2.2012, at 20.46, Ralf Hildebrandt wrote: > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. I can't reproduce this. From slusarz at curecanti.org Thu Mar 1 20:03:46 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 11:03:46 -0700 Subject: [Dovecot] LIST-STATUS issue Message-ID: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: >> Timestamp: Thu, 01 Mar 2012 10:55:47 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. 3 seconds later... >> Timestamp: Thu, 01 Mar 2012 10:55:50 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: * OK [XPROXYREUSE] IMAP connection reused by squirrelmail-imap_proxy S: 1 OK User logged in C: 2 EXAMINE IN.dovecot (QRESYNC (1254166849 25800 18484,20260,20393,20705,20860:20868)) S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft impflag1 impflag2 impflag0 Junk seen NonJunk) S: * OK [PERMANENTFLAGS ()] Read-only mailbox. S: * 13 EXISTS S: * 0 RECENT S: * OK [UNSEEN 5] First unseen. S: * OK [UIDVALIDITY 1254166849] UIDs valid S: * OK [UIDNEXT 20869] Predicted next UID S: * OK [HIGHESTMODSEQ 25800] Highest S: 2 OK [READ-ONLY] Select completed. [...] C: 7 SEARCH RETURN (COUNT) UNSEEN S: * ESEARCH (TAG "7") COUNT 8 S: 7 OK Search completed (0.000 secs). I can confirm that I did not receive 8 dovecot list messages in 3 seconds. :) Once the mailbox is accessed, I am seeing correct counts in LIST-STATUS return again: >> Timestamp: Thu, 01 Mar 2012 10:57:28 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 8) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. I am using imapproxy for these connections, so all access are happening in the same Dovecot access. michael From tss at iki.fi Thu Mar 1 20:22:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 20:22:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: On 1.3.2012, at 20.03, Michael M Slusarz wrote: > Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... > > UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: Only with LIST-STATUS, or also STATUS itself? And with what kind of config (doveconf -n)? Can you reproduce this? LIST-STATUS should use the exact same code as STATUS. STATUS should use pretty much the same code as SELECT. mailbox_list_index=yes makes a difference, but other than that I can't really think of a reason. From slusarz at curecanti.org Thu Mar 1 21:17:41 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 12:17:41 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 1.3.2012, at 20.03, Michael M Slusarz wrote: > >> Looked at the hg commits since 2.1.1, but didn't see anything that >> was exactly on-point to this... >> >> UNSEEN counts seem to be broken when done in a LIST-STATUS call. I >> see this, for example: > > Only with LIST-STATUS, or also STATUS itself? And with what kind of > config (doveconf -n)? Can you reproduce this? I've been seeing this intermittently the last few days. When I notice it, I try to reproduce and never can. Then it eventually comes back, as in the first time I access the next morning. Maybe its some kind of cache/timeout issue. (e.g. - I can't reproduce right now) > LIST-STATUS should use the exact same code as STATUS. STATUS should > use pretty much the same code as SELECT. mailbox_list_index=yes > makes a difference, but other than that I can't really think of a > reason. # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-1-ARCH x86_64 auth_mechanisms = plain login namespace { inbox = yes location = maildir:~/Maildir prefix = separator = . type = private } namespace { location = maildir:~/mailtest/mboxtesting2 prefix = "#shared." separator = . type = shared } namespace { location = mbox:~/mailtest/mail prefix = "#public2." separator = . type = public } passdb { args = dovecot driver = pam } plugin { acl = vfile } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_cert = I'm running imapproxy as shown at http://wiki2.dovecot.org/HowTo/ImapcProxy. In fact, that's my config in the wiki. 8-) It's been working, but has performance issues when the Exchange server that's hard-coded as imapc_host=xxx.xxx.xxx.xxx doesn't happen to be the user's home exchange server. I'd like to point dovecot at the correct Exchange server based on an LDAP query, and in fact, have an LDAP search that works: DC=example,DC=com (&(objectCategory=person)(objectClass=user)(!(userAccountcontrol:1.2.840.113556.1.4.803:=2))(sAMAccountName=username)) With the exchange server being returned in the msExchHomeServerName property as: /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Can anybody toss me a clue? Once I get it working, I'll update the wiki. Thanks! Terry From dbenfell at gmail.com Fri Mar 2 03:23:41 2012 From: dbenfell at gmail.com (David Benfell) Date: Thu, 1 Mar 2012 17:23:41 -0800 Subject: [Dovecot] Thanks, was Re: need simpler instructions for user authentication Message-ID: Thanks to everyone who responded to my request. And I apologize that this message isn't properly threaded. I'm in trouble whenever I have to send mail from my gmail account because I automatically pull it down into the same place from which I send my regular mail (from benfell at parts-unknown.org ) and I haven't configured that to deal with the gmail account. That makes it difficult to reply properly to mail on this list which I have subscribed to with the gmail account so that I'd actually receive it even when I'm having problems with my main mail system. Timo Sirainen responded that I needed a userdb and offered that I should probably add the following to dovecot.conf : userdb { driver = passwd } As near as I can tell--and it's been over a day now--that solved the problem. Thanks! From stan at hardwarefreak.com Fri Mar 2 03:38:13 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 01 Mar 2012 19:38:13 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F502485.9070503@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > On 2012-02-29 9:15 PM, Stan Hoeppner wrote: >> Q: How many concurrent IMAP clients could you serve with this setup >> before hitting a bottleneck at any point in the architecture? > > No idea how to calculate it... The correct answer is approximately 120,000 concurrent users, based on an assumed average of ~3MB-5MB of ram consumed in all processes for each user. >> What is the first bottleneck you'd run into? > > Unless this is a trick question, the OC-12 link (since it is only > 644Mb), and the next bottleneck would be the 2 GbE server connections to > the router (are these bonded? if so, what mode?... It was a bit of a trick question, with a somewhat elaborate setup, designed to shift your focus/thinking. Apparently I failed in my effort here. The correct answer is that RAM will be the first bottleneck. Then disk IOPS, finally followed by the OC-12 assuming we beef up the others. > Since the vast majority of our connections will be *local*, I'm > unconcerned about the internet connect speeds (one office has a 100/10Mb > Cable (Comcast Business Class) connection, the other will have a > 100/100Mb fiber/ethernet connection). You didn't grasp why I used the OC-12 in my example. It had nothing to do with LAN/WAN, local or remote, but the total users/traffic a 600Mb/s link can carry. > My main priority is that the user experience at each physical location > be optimal, which is why I'm more focused on making sure each offices > users are connected to only the local server for all services > (file/print/mail). A single MAN (Metropolitan Area Network) 1000BASE-LX link, good for 5km, likely what you will have, is more than sufficient to carry the 2nd office site traffic while keeping all of your servers/etc where they are now. > My choices are, as I see it, single GbE connections, or add some > multiport GbE cards (these Dells support up to 3 PCIe cards) and bond > some ports together for each VM. 10GbE is simply not in our price range > (and I don't think we need it anyway), although I did stumble on these > while googling and am waiting on pricing, since they claim to be 'much > cheaper': With specs like that you must be supporting 100,000 users. ;) > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Your company/employer has less than 250 users IIRC. Is this right? You're a media company that works with files much larger than the average company. Is that correct? Let's cut to the chase shall we? Your 1000BASE-LX MAN link has an after link overhead bandwidth of just over 100MB/s full duplex. To put this into real world perspective, you can copy a single 4.7GB DVD in 47 seconds, or 1 in each direction in the same time, 2 total, 9.4GB total. You can copy 20 full DVDs over this link, 10 in each direction, in less than 8 minutes. Add heavy IMAP traffic for 500 concurrent users and it's still less than 10 minutes and the IMAP users won't have a clue if the switch VLAN QOS is setup correctly. You see GbE as mundane, slow, because it has been ubiquitous for some time, being a freebie on both servers and desktops. This is why I used the OC-12 example at $15K/month, hoping you'd start to grasp that cost has little direct relationship to performance. GbE is "free" now because the cost of the silicon to drive a 1000MHz signal over 300 meters of copper wire is no longer higher than for 100BASE-T. Here's another comparison. All internet backbone links are OC-48 at 2.5Gb/s. It takes only 2.5 GbE links to equal a backbone link. Backbone links carry the traffic of *millions* of users, all applications, all data stream types. And that's *only* 250MB/s. So, the point is, a single 1000BASE-LX MAN link is far more than plenty to carry all of the traffic you'll throw at it, and quite a bit more, with some minor QOS configuration. Consider how much money, time, and duplication of services and servers you are going to save now that you realize you need nothing other than the 1000BASE-LX MAN link, and closet switches at the second office site? Get yourself a qualified network architect. Pay for a full network traffic analysis. He'll attach sniffers at multiple points in your network to gather traffic/error/etc data. Then you'll discuss the new office, which employees/types with move there, and you'll be able to know almost precisely the average and peak bandwidth needs over the MAN link. He'll very likely tell you the same thing I have, that a single gigabit MAN link is plenty. If you hire him to do the work, he'll program the proper QOS setup to match the traffic patterns gleaned from the sniffers. -- Stan From sdavies at sdc.com.au Fri Mar 2 04:14:04 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Fri, 2 Mar 2012 12:44:04 +1030 Subject: [Dovecot] Log sybnch error Message-ID: <201203021244.05034.sdavies@sdc.com.au> My mail log has many entries like: Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: Extension header update points outside header size Version 2.1.1 dovecot -n gives: # 2.1.1: /usr/etc/dovecot/dovecot.conf # OS: Linux 2.6.33.5-desktop-2mnb i686 Mandriva Linux 2010.2 mail_access_groups = mail mail_privileged_group = mail passdb { driver = pam } protocols = pop3 imap service anvil { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = mode = 0600 user = } unix_listener anvil { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = root vsz_limit = 18446744073709551615 B } service config { chroot = client_limit = 0 drop_priv_before_exec = no executable = config extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = config unix_listener config { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service dict { chroot = client_limit = 1 drop_priv_before_exec = no executable = dict extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dict { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service director { chroot = . client_limit = 0 drop_priv_before_exec = no executable = director extra_groups = fifo_listener login/proxy-notify { group = mode = 00 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener director-admin { group = mode = 0600 user = } unix_listener login/director { group = mode = 00 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service dns_client { chroot = client_limit = 1 drop_priv_before_exec = no executable = dns-client extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dns-client { group = mode = 0666 user = } unix_listener login/dns-client { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = port = 143 ssl = no } inet_listener imaps { address = port = 993 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service indexer-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = indexer-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 10 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service indexer { chroot = client_limit = 0 drop_priv_before_exec = no executable = indexer extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service ipc { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = ipc extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener ipc { group = mode = 0600 user = } unix_listener login/ipc-proxy { group = mode = 0600 user = $default_login_user } user = $default_internal_user vsz_limit = 18446744073709551615 B } service lmtp { chroot = client_limit = 1 drop_priv_before_exec = no executable = lmtp extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = lmtp service_count = 0 type = unix_listener lmtp { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service pop3-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = pop3-login extra_groups = group = idle_kill = 0 inet_listener pop3 { address = port = 110 ssl = no } inet_listener pop3s { address = port = 995 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = pop3 service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service pop3 { chroot = client_limit = 1 drop_priv_before_exec = no executable = pop3 extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = pop3 service_count = 1 type = unix_listener login/pop3 { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service ssl-params { chroot = client_limit = 0 drop_priv_before_exec = no executable = ssl-params extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = startup unix_listener login/ssl-params { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service stats { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = stats extra_groups = fifo_listener stats-mail { group = mode = 0600 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener stats { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } ssl_cert = References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> Message-ID: <4F503FFD.5040504@r.paypc.com> > My initial tests for CLucene were that it would take 30% of mailbox size > (compared to 50% for Xapian). But this was before I actually implemented > it to Dovecot.. I haven't really looked at how large the indexes > actually are. Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. =R= From tss at iki.fi Fri Mar 2 09:31:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:31:01 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F503FFD.5040504@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: On 2.3.2012, at 5.35, Robin wrote: > >> My initial tests for CLucene were that it would take 30% of mailbox size >> (compared to 50% for Xapian). But this was before I actually implemented >> it to Dovecot.. I haven't really looked at how large the indexes >> actually are. > > Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. From dchenusa at yahoo.com Fri Mar 2 09:33:14 2012 From: dchenusa at yahoo.com (D Chen) Date: Thu, 1 Mar 2012 23:33:14 -0800 (PST) Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! Message-ID: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! Thx. From tss at iki.fi Fri Mar 2 09:36:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:36:40 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203021244.05034.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: On 2.3.2012, at 4.14, Stephen Davies wrote: > My mail log has many entries like: > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error > at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: > Extension header update points outside header size What filesystem is this? Are you using NFS or some other remote/shared filesystem? Do these messages repeat more than once for the same mailbox? Have you recently upgraded from Dovecot v1.x? From slusarz at curecanti.org Fri Mar 2 09:53:27 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 00:53:27 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Message-ID: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Quoting Michael M Slusarz : > Quoting Timo Sirainen : > >> On 1.3.2012, at 20.03, Michael M Slusarz wrote: >> >>> Looked at the hg commits since 2.1.1, but didn't see anything that >>> was exactly on-point to this... >>> >>> UNSEEN counts seem to be broken when done in a LIST-STATUS call. >>> I see this, for example: >> >> Only with LIST-STATUS, or also STATUS itself? And with what kind of >> config (doveconf -n)? Can you reproduce this? > > I've been seeing this intermittently the last few days. When I > notice it, I try to reproduce and never can. Then it eventually > comes back, as in the first time I access the next morning. Maybe > its some kind of cache/timeout issue. (e.g. - I can't reproduce > right now) It happened again. Once it got around midnight local time I stopped seeing correct STATUS updates with the UNSEEN count. Running two MUAs: I opened a mailbox with new messages in one, and the next time the other MUA (webmail) polled it magically started seeing unseen messages again. And now if I change the seen flag in one MUA, it will continue to be updated (as normal) in the other MUA. michael From tss at iki.fi Fri Mar 2 10:27:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:27:16 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Message-ID: <1330676836.2081.46.camel@innu> On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > It happened again. Once it got around midnight local time I stopped > seeing correct STATUS updates with the UNSEEN count. Running two > MUAs: I opened a mailbox with new messages in one, and the next time > the other MUA (webmail) polled it magically started seeing unseen > messages again. And now if I change the seen flag in one MUA, it will > continue to be updated (as normal) in the other MUA. Next time try talking IMAP protocol directly? First giving the same LIST STATUS command, and if it doesn't show the updated counts, try STATUS directly and finally EXAMINE. I don't see anything in the code that could explain why this could be happening. What filesystem are you using? From tss at iki.fi Fri Mar 2 10:45:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:45:51 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: Message-ID: <1330677951.2081.49.camel@innu> On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: > I am setting up an high-availability server set, which consists of two (or > more) servers with common disk space, all set behind redundant hardware > load balancers. At first, there will be two servers and disk space will be > kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what > creates kind of networked RAID1 storage space). This will be done mostly > to keep WWW service available in case when one of servers fails. > > However, if we have everything redundant, why not have the same with SMTP > and POP3/IMAP? But - won't anything fail if two (or more) dovecots are > accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If both servers randomly access users' mails, with NFS you'll have some trouble, with OCFS2 probably less trouble. But in both cases you'll have better performance and no problems if you use Dovecot director in both servers (install both director and backend to both servers). http://wiki2.dovecot.org/Director From stan at hardwarefreak.com Fri Mar 2 11:17:21 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 02 Mar 2012 03:17:21 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F509021.2050202@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Expanding on my previous statements, and hopefully answering some questions here, or at least getting in the ballpark, lets see what a single GbE link is capable of. Let's assume an average transfer size of SMTP/IMAP email including headers is roughly 4096 bytes, or 32768 bits. TCP over GbE after all framing and protocol overhead = 992,697,000 bits/sec maximum bandwidth with jumbo frames = 941,482,000 bits/sec max without jumbo frames We'll go without jumbo frames in our example. Every GbE interface on one router segment must support jumbo or you can't enable it. If you do, interfaces that don't do jumbo will have bad to horrible performance, or maybe not work at all. Many workstation NICs don't do jumbo frames as well as many commercial routers. Typical IMAP command payload is absolutely tiny, so we'll concentrate on response traffic. Theoretical steady state IMAP server to client 4KB message transfer rates: = 28,731 msgs/sec = 1,723,905 msgs/minute = 103,434,301 msgs/hour = 2,482,423,242 msgs/day General file transfer bandwidth, 5MB JPG: = 22 files/sec = 1,346 files/minute = 80,808 files/hour = 1,939,393 files/day General file transfer bandwidth, 100MB TIFF: = 1 files/sec = 67 files/minute = 4,040 files/hour = 96,969 files/day General file transfer bandwidth, 500MB video file: = 1 files 4.5 seconds = 10 files 44.6 seconds = 100 files 7.4 minutes As you can see, a single GbE interface has serious capacity and will probably easily carry your inter-site traffic without needing duplicate servers at the second site. You mentioned putting multiple GbE interfaces on your servers. Very, very few servers *need* 900+ Mb/s of bandwidth, however having two links is good for redundancy. So I'd not worry about the aggregation performance, only the proper and seamless failover functionality. I obviously haven't seen your workflows Charles, but I recall you do a lot of media work. By 'you' I mean Media Brokers. So obviously your users will be hitting the network harder than average office workers. I'm taking that into account. My gut instinct, based on experience and the match, is that a single GbE inter site MAN link will be plenty, without the need to duplicate server infrastructure. Again, have a qualified network architect sniff your current network traffic patterns, and discuss with you the anticipated user traffic at the 2nd site to determine your average and peak inter-site b/w needs. The average will absolutely be much less than 1Gb/s, but the peak may be well above 1Gb/s. You can still avoid the myriad problems/costs of server duplication without incurring significant additional link costs. There are a couple of options that should be available to you: 1. A second fiber pair and GbE link You might negotiate a burst contract. You pay a flat monthly rate for a base bit rate of X and pay extra for bursts. Burst contract availability will depend on the provider's network topology. If at any point they're aggregating multiple customer's traffic on a single trunk fiber pair a burst contract should be available. Burst contract allow them to oversubscribe their trunks, just as ISPs and broadband providers do. Your network architect should be able to assist you in figuring out what you'd want for your base and peak bit rates for such a contract. Why pay for 1000Mb/s from 8pm to 6am if you're only using 20Kb/s? 2. Add a second GbE link on a different transceiver wavelength using a prism on each end to transmit both links on one fiber pair. This is typically cheaper when the provider has limited fiber runs in a given area or to a given building. You may or may not be able to save money with a burst contract in this scenario. Talk to your provider and find out what your options are. Wait until your architect has finished your network analysis before speaking to the provider. Treat this link as a traditional WAN link. Do NOT treat it as simply another switch segment. Put an IP router on each side of the GbE MAN link and create a separate IP subnet for hosts and devices in the new office. By doing this you keep broadcast traffic from traversing the link. This includes things like ARP discovery, DHCP, NTP broadcast, and most importantly: broadcast traffic from disk imaging software. If you don't make this an IP routed link, network disk imaging traffic will traverse the MAN link just as it traverses your entire switched LAN. This could be anywhere from 25-80MB/s (200-640Mb/s) of broadcast traffic. You obviously don't want this clogging the link. You *might* be able to eliminate broadcast traffic using special VLAN configurations on sufficiently advanced layer2-7 "switch routers", but it's cheaper and fool proof when done with standard IP routers. Again, chat with your architect. With this being a routed connection, and broadcast traffic being eliminated, any services that rely on broadcast traffic will need to be duplicated or tweaked accordingly. You will need a DHCP server in the new office. The router should be able to serve DHCP, unless you're currently serving some custom scope it can't handle. If you rely on broadcast for WINS, or have any other Microsoft services that rely on broadcast, you will need to address those. If you currently use NTP broadcast for time updates you'll need another NTP server in the new office. Again, the router should be able to broadcast NTP updates. The solutions to these things have been around forever, so I'm not going to go into all of them, but you need to be aware. You'll need to discuss these things with your network architect or a qualified Microsoft consultant. If you run no MS servers and don't use broadcast, then no need to worry about. And hooray for you, no MS! :) This may be of interest given the topic. At a previous $dayjob a few years back, we ran the traffic of about 580 desktops/wireless laptops through a single GbE uplink into an 11 blade server farm backed by a small fiber channel SAN. Blade-blade IP traffic was through a dedicated 14x6 port GbE switch module, so things like vmotion, backups, etc worked at full boogie. But the uplink from the switch module in the BladeCenter to the Cisco 5000 core switch was a single copper GbE uplink. All user traffic flowed over this link. We never had performance issues. We'd configured QOS to keep the IP phones happy but that's about it for traffic shaping. Before I left I jacked in a 2nd GbE uplink for redundancy and configured Cisco's link aggregation protocol. We didn't notice a performance difference. I could have aggregated 6 GbE uplinks. One did the job, two gave resiliency, more would have just wasted ports on the core switch. Hope you find this educational/informational/useful Charles, and maybe others. -- Stan From dovecot at r.paypc.com Fri Mar 2 11:51:53 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 01:51:53 -0800 Subject: [Dovecot] fts size In-Reply-To: References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: <4F509839.8010801@r.paypc.com> > No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. That sounds like a great deal to me! I'm glad you're still interested enough in it. =R= From Ralf.Hildebrandt at charite.de Fri Mar 2 12:25:01 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:25:01 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from Message-ID: <20120302102501.GZ11180@charite.de> Hi! Starting with 2.1.1 we suddely encounter quite a lot of these messages: Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) While this has (assumedly) been working with 2.0.18. Another issue with this: This fixes ONE file, and throws an error. Repeatedly accessing this folder fixes more files, until at some point all files were fixed. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From mailing at securitylabs.it Fri Mar 2 12:30:04 2012 From: mailing at securitylabs.it (mailing at securitylabs.it) Date: Fri, 02 Mar 2012 11:30:04 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <4F50A12C.4080304@securitylabs.it> Il 02/03/2012 11:25, Ralf Hildebrandt ha scritto: > Hi! > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) > > While this has (assumedly) been working with 2.0.18. Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. > > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, apparently it happens only on servers with qmail, not on servers with exim or dovecot as lda: Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: Broken physical size for mail UID 40669 Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) From tss at iki.fi Fri Mar 2 12:32:08 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:32:08 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F509839.8010801@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> <4F509839.8010801@r.paypc.com> Message-ID: On 2.3.2012, at 11.51, Robin wrote: >> No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. > > That sounds like a great deal to me! I'm glad you're still interested enough in it. Having more choices is always good. :) From Ralf.Hildebrandt at charite.de Fri Mar 2 12:34:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:34:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <4F50A12C.4080304@securitylabs.it> References: <20120302102501.GZ11180@charite.de> <4F50A12C.4080304@securitylabs.it> Message-ID: <20120302103445.GC11180@charite.de> > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, > apparently it happens only on servers with qmail, not on servers with > exim or dovecot as lda: I'm using the dovecot LDA, but then it's not clear if the messages affected are REALLY old and thus might predate the use of the dovecot LDA... > > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename > has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, > to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index > cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: > Broken physical size for mail UID 40669 > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Fri Mar 2 12:37:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:37:06 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S .. > While this has (assumedly) been working with 2.0.18. Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. > Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. From Ralf.Hildebrandt at charite.de Fri Mar 2 12:43:33 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:43:33 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> Message-ID: <20120302104333.GD11180@charite.de> * Timo Sirainen : > On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > .. > > While this has (assumedly) been working with 2.0.18. > > Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. COuld be! > > Another issue > > with this: This fixes ONE file, and throws an error. Repeatedly > > accessing this folder fixes more files, until at some point all files > > were fixed. > > Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Ah yes, good idea Mar 2 11:39:39 postamt dovecot: imap-login: Login: user=, method=PLAIN, rip=141.42.206.38, lip=141.42.206.36, mpid=28959, secured Mar 2 11:39:41 postamt dovecot: imap(user): Error: Cached message size smaller than expected (168202 < 170440) Mar 2 11:39:41 postamt dovecot: imap(user): Error: Maildir filename has wrong S value, renamed the file from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ Mar 2 11:39:41 postamt dovecot: imap(user): Error: Corrupted index cache file /home/g/z/user/Maildir/.Partys/dovecot.index.cache: Broken physical size for mail UID 81 Mar 2 11:39:41 postamt dovecot: imap(user): Error: read(/home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ) failed: Input/output error (uid=81) Mar 2 11:39:41 postamt dovecot: imap(user): Disconnected: Internal error occurred. Refer to server log for more information. [2012-03-02 11:39:41] in=735 out=5258 Look at that renaming operation: It simply reused the same name: from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ > Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. In the case above that mail was gzipped twice :( -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From CMarcus at Media-Brokers.com Fri Mar 2 14:40:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 02 Mar 2012 07:40:54 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F4D009F.7000107@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> Message-ID: <4F50BFD6.5010808@Media-Brokers.com> On 2012-02-28 11:28 AM, Charles Marcus wrote: > On 2012-02-28 11:05 AM, kfx wrote: >> Ok I feel ashame... it was a third party init scrip who was the >> problem :( >> >> Sorry for the noise and thank you for dovecot > > So... you're saying that Thunderbird now correctly uses server side search? Please respond... I need to know whether or not I need to pursue this, since we use Thunderbird in house and will be switching soon to dovecot... Thanks, -- Best regards, Charles From slusarz at curecanti.org Fri Mar 2 20:27:42 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 11:27:42 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <1330676836.2081.46.camel@innu> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> Message-ID: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Quoting Timo Sirainen : > On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > >> It happened again. Once it got around midnight local time I stopped >> seeing correct STATUS updates with the UNSEEN count. Running two >> MUAs: I opened a mailbox with new messages in one, and the next time >> the other MUA (webmail) polled it magically started seeing unseen >> messages again. And now if I change the seen flag in one MUA, it will >> continue to be updated (as normal) in the other MUA. > > Next time try talking IMAP protocol directly? First giving the same LIST > STATUS command, and if it doesn't show the updated counts, try STATUS > directly and finally EXAMINE. Weird. In the MUA I was seeing this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. Then I did this: slusarz at bigworm % dovecot * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.dovecot" * STATUS "IN.dovecot" (UNSEEN 7) * LIST () "." "INBOX" * STATUS "INBOX" (UNSEEN 1) 1 OK List completed. Sure enough, I went back to the MUA and now see this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 7) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. The only difference... in the MUA I am enabling QRESYNC. This is what the previous commands look like (before reaching the LIST-STATUS command): >> Timestamp: Fri, 02 Mar 2012 11:17:56 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: 1 OK User logged in C: 2 ENABLE QRESYNC S: * ENABLED QRESYNC S: 2 OK Enabled. C: 3 STATUS IN.horde.cvs (MESSAGES UIDNEXT UIDVALIDITY HIGHESTMODSEQ) S: * STATUS "IN.horde.cvs" (MESSAGES 11 UIDNEXT 16767 UIDVALIDITY 1239077891 HIGHESTMODSEQ 31409) S: 3 OK Status completed. Next time I see this issue, I will try to go back and issue ENABLE QRESYNC from the command line. > I don't see anything in the code that could explain why this could be > happening. What filesystem are you using? Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. michael From tss at iki.fi Fri Mar 2 20:36:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 20:36:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: On 2.3.2012, at 20.27, Michael M Slusarz wrote: >> I don't see anything in the code that could explain why this could be >> happening. What filesystem are you using? > > Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. jfs used to have a problem with not updating directory's mtime when link()ing files to it, which caused Dovecot not to notice new mails. This got fixed a few years ago though. But it is a bit suspicious that this is happening with jfs.. From dovecot at r.paypc.com Sat Mar 3 00:13:18 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 14:13:18 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F50BFD6.5010808@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> Message-ID: <4F5145FE.3070301@r.paypc.com> On 3/2/2012 4:40 AM, Charles Marcus wrote: > Please respond... I need to know whether or not I need to pursue this, > since we use Thunderbird in house and will be switching soon to dovecot... This mailing list is for dovecot, not Thunderbird support. The lack of replies to Thunderbird usage questions no doubt reflects this. I would look at the GUI interface and/or "manual" for Thunderbird to find the answer to that question. I suspect there is a check-box or configuration item that's been right in front of you all along that you've not thought twice about. =R= From mcguire at neurotica.com Sat Mar 3 00:18:12 2012 From: mcguire at neurotica.com (Dave McGuire) Date: Fri, 02 Mar 2012 17:18:12 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F514724.7070001@neurotica.com> On 03/02/2012 05:13 PM, Robin wrote: > On 3/2/2012 4:40 AM, Charles Marcus wrote: >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. Please forgive me for jumping in, but I believe this is very much on-topic. It isn't a matter of "Thunderbird support", it's a matter of Dovecot interoperability. Please DO keep stuff like this on-list. -Dave -- Dave McGuire, AK4HZ New Kensington, PA From sdavies at sdc.com.au Sat Mar 3 01:45:02 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Sat, 3 Mar 2012 10:15:02 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: <201203031015.02716.sdavies@sdc.com.au> No NFS. The file system is local. Yes. There are multiple copies of the message for multiple mailboxes for each of at least two users. Yes. Did recently upgrade from 1.2.15. Cheers and thanks, Stephen On Fri, 2 Mar 2012 06:06:40 PM Timo Sirainen wrote: > On 2.3.2012, at 4.14, Stephen Davies wrote: > > My mail log has many entries like: > > > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization > > error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS > > Emails/dovecot.index: Extension header update points outside header size > > What filesystem is this? Are you using NFS or some other remote/shared > filesystem? > > Do these messages repeat more than once for the same mailbox? Have you > recently upgraded from Dovecot v1.x? -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From slusarz at curecanti.org Sat Mar 3 02:48:09 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 17:48:09 -0700 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data Message-ID: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> I'm seeing this: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) It should be: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) since the RFC 2822 quoting characters must be removed. michael From slusarz at curecanti.org Sat Mar 3 03:48:23 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 18:48:23 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 2.3.2012, at 20.27, Michael M Slusarz wrote: > >>> I don't see anything in the code that could explain why this could be >>> happening. What filesystem are you using? >> >> Nothing fancy: jfs on Archlinux. I've been using it for years - >> it's not something I have recently changed. > > jfs used to have a problem with not updating directory's mtime when > link()ing files to it, which caused Dovecot not to notice new mails. > This got fixed a few years ago though. But it is a bit suspicious > that this is happening with jfs.. I can now verify that QRESYNC is triggering this behavior. * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 ENABLE QRESYNC * ENABLED QRESYNC 1 OK Enabled. 2 LIST "" (IN.horde.dev) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.horde.dev" * STATUS "IN.horde.dev" (UNSEEN 0) 2 OK List completed. 3 EXAMINE IN.horde.dev * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded NonJunk impflag0 impflag1) * OK [PERMANENTFLAGS ()] Read-only mailbox. * 3 EXISTS * 1 RECENT * OK [UNSEEN 3] First unseen. * OK [UIDVALIDITY 1255685339] UIDs valid * OK [UIDNEXT 2805] Predicted next UID * OK [HIGHESTMODSEQ 8266] Highest 3 OK [READ-ONLY] Select completed. (The unseen message arrived about 7 minutes before I issued these commands, so it didn't sneak in between command #2 and #3). michael From busseniu at in.tum.de Sat Mar 3 14:33:27 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Sat, 03 Mar 2012 13:33:27 +0100 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F4DF07A.7020408@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> Message-ID: <4F520F97.5030002@in.tum.de> On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: > when the private namespace has "prefix = INBOX." and you use doveadm > fetch to search for "mailbox INBOX", then it prints every message twice: Apparently the bug has been introduced with this changeset: changeset: 14112:f5353573d3a0 user: Timo Sirainen date: Sun Feb 12 02:50:49 2012 +0200 summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. http://hg.dovecot.org/dovecot-2.1/rev/f5353573d3a0 Cheers, Christoph From netwiz at crc.id.au Sat Mar 3 14:07:44 2012 From: netwiz at crc.id.au (Steven Haigh) Date: Sat, 03 Mar 2012 23:07:44 +1100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <20110311215739.GD13492@state-of-mind.de> References: <20110311215739.GD13492@state-of-mind.de> Message-ID: <4F520990.2000903@crc.id.au> Hi all, I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... -- Steven Haigh Email: netwiz at crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 >Timo, > >would you consider adding support for "IMAP LIST Extension for >Special-Use >Mailboxes" any time near >in the >future? > >I would really love to get rid of all those folders created by all >those >different mail clients just because they can't agree to use the same >folder >for special purpose. > >Obviously clients need to support it too. Having Dovecot support it >certainly >would make them adopt the standard sooner. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4952 bytes Desc: S/MIME Cryptographic Signature URL: From CMarcus at Media-Brokers.com Sat Mar 3 14:52:10 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 07:52:10 -0500 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5213FA.30700@Media-Brokers.com> On 2012-03-02 2:33 AM, D Chen wrote: > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start > successfully with lots of errors i.e. "dovecot: doveconf: Warning: > ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on > Ubuntu 11.10, by following the Postfix installation and configuration > instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section > on page 190, after run "sudo apt-get install dovecot-common", it > requires to edit the section of "auth default" and the "socket > listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my > /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth > default" "socket listen" ! Distro specific questions are usually much better asked on the distro support lists... -- Best regards, Charles From eliezer at ec.hadorhabaac.com Sat Mar 3 15:03:40 2012 From: eliezer at ec.hadorhabaac.com (Eliezer Croitoru) Date: Sat, 03 Mar 2012 15:03:40 +0200 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5216AC.3030602@ec.hadorhabaac.com> On 02/03/2012 09:33, D Chen wrote: use the command dovecot -n to get dovecot settings output and we can try to help you a bit. Regards, Eliezer > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. > "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! > > I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a > dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? > > BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! > > Thx. From arnaud.abelard at univ-nantes.fr Sat Mar 3 18:06:17 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Sat, 03 Mar 2012 17:06:17 +0100 Subject: [Dovecot] keywords/flags questions Message-ID: <4F524179.2040407@univ-nantes.fr> Hello, I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? For example: . STORE 1:1 flags &AOA-_refaire * 1 FETCH (FLAGS (&aoa-_refaire)) This makes retrieving the keyword properly impossible. What did I miss? My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? Thanks in advance, Arnaud -- Arnaud Ab?lard jabber: arnaud.abelard at univ-nantes.fr / twitter: ArnY Administrateur Syst?me DSI Universit? de Nantes - From public-mail at alekciy.ru Sat Mar 3 18:51:28 2012 From: public-mail at alekciy.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KHRg9C90LTRg9C60L7Qsg==?=) Date: Sat, 3 Mar 2012 20:51:28 +0400 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: 3 ????? 2012??. 20:06 ???????????? Arnaud Ab?lard ???????: > But docevot save keywords in lowercase Yes. http://www.dovecot.org/list/dovecot/2011-April/058493.html From tss at iki.fi Sat Mar 3 19:05:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 3 Mar 2012 19:05:13 +0200 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: <5CA4B56F-D26E-492B-9B4D-9BD8E6EAD018@iki.fi> On 3.3.2012, at 18.06, Arnaud Ab?lard wrote: > I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. > > First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? > > For example: > . STORE 1:1 flags &AOA-_refaire > * 1 FETCH (FLAGS (&aoa-_refaire)) > > This makes retrieving the keyword properly impossible. What did I miss? Sorry, doesn't work like that. This was recently discussed in imap-protocol mailing list though, but nothing has come of it. > My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? For now yes. I've been thinking about doing something about this for last 8 years, but it has never become a real problem so I haven't bothered. From trashcan at odo.in-berlin.de Sat Mar 3 20:03:42 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:03:42 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <1330346709.11500.324.camel@innu> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: Hi -- On 27.02.2012, at 13:45, Timo Sirainen wrote: > On Thu, 2012-02-23 at 20:55 +0100, Michael Grimm wrote: >> My working 2.0.18 syntax threw the following error: >> >> vmail> dsync -v -f -u test ssh vmail at remote-host.tld dsync -v -f -u test >> doveadm(vmail): Fatal: Unknown print formatter: -u >> dsync-local(test): Error: read() from worker server failed: EOF > > You left out "mirror" from that command, but after adding it the latest > hg version works. I did use 'mirror', I just forgot to paste it. JFTR: vmail> dovecot --version 20120303 (1002733ca266+) vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test dsync-local(test): Error: remote: dsync: illegal option -- f doveadm dsync-server [-u |-A] [-S ] dsync-local(test): Error: read() from worker server failed: EOF If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. >> Now I switched to the recommended new syntax as stated in http://wiki2.dovecot.org/Upgrading/2.1: >> >> vmail> doveadm sync -v -f -u test ssh vmail at remote-host.tld doveadm sync -v -f -u test >> doveadm: illegal option -- v >> doveadm sync [-u |-A] [-S ] [-fR] [-m ] > > The -v parameter is in wrong place now, needs to be "doveadm -v sync". Ah, yes. That was my mistake, sorry. >> After some trial by error I finally found a working syntax: >> >> vmail> doveadm sync -u test -f ssh vmail at remote-host.tld doveadm dsync-server -u test > > Oh, hmm. I hadn't thought about this problem, it shouldn't have been > necessary to give the dsync-server parameter. But I guess there's not a > whole lot of other possibilities to do this. Hmm. > > BTW. I think you can do this simply: > > doveadm sync -f -u test test at remote-host.tld vmail> doveadm sync -f -u test ssh vmail at remote-host.tld dsync-local(test): Error: remote: dsync-server: Command not found. dsync-local(test): Error: read() from worker server failed: EOF Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that I'm the only one reporting that. How could I help to debug this issue? Regards, Michael From tlx at leuxner.net Sat Mar 3 20:10:35 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Sat, 3 Mar 2012 19:10:35 +0100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <4F520990.2000903@crc.id.au> References: <20110311215739.GD13492@state-of-mind.de> <4F520990.2000903@crc.id.au> Message-ID: <83D77B81-EC49-4755-A866-E30B41E8B246@leuxner.net> Am 03.03.2012 um 13:07 schrieb Steven Haigh: > I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... Yes it was. It has been discussed extensively: http://www.dovecot.org/list/dovecot-news/2012-February/000213.html http://www.dovecot.org/list/dovecot/2011-December/062327.html Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From trashcan at odo.in-berlin.de Sat Mar 3 20:12:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:12:21 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <8D5B4989-571D-4F5D-927E-65198CECFADD@odo.in-berlin.de> Hi -- On 03.03.2012, at 19:03, Michael Grimm wrote: > Thus I can't test user suggestion. s/user/your/ Sorry, Michael From anyaddress at gmx.net Sat Mar 3 22:08:48 2012 From: anyaddress at gmx.net (Tom Fernandes) Date: Sat, 3 Mar 2012 21:08:48 +0100 Subject: [Dovecot] directly addressable public folders issues Message-ID: <201203032108.49489.anyaddress@gmx.net> Hi, I would like to have an address info at example.com whose mails are stored in a public folder. I also want certain users to be able to create sieve-filter-rules and subfolders. From what I understand it's a good idea to have a separate location for home and for mail_location. For my normal accounts I have: home = /var/vmail/ mail_location = ~/Maildir Is there a way to have the same for public folders? This are my current settings: namespace public { separator = / prefix = public/ location = maildir:/var/vmail/public subscriptions = no } user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The LDAP-homeDirectory-attribute for info at example.com is "public/.info" Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which is good. The MUA seems to read to read from /var/vmail/public/.info though. How can I make the MUA read from /var/vmail/public/.info/Maildir instead? And in case this works - is it possible to have only certain users modify the sieve-rule for this public folder or do I definitely need a passdb-entry for info at example.com? Modifying scripts via commandline is not an option... I'm using dovecot 1.2. regards, Tom Fernandes From CMarcus at Media-Brokers.com Sat Mar 3 22:14:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:14:55 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F527BBF.3060607@Media-Brokers.com> On 2012-03-02 5:13 PM, Robin wrote: > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? -- Best regards, Charles From CMarcus at Media-Brokers.com Sat Mar 3 22:20:24 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:20:24 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F509021.2050202@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> Message-ID: <4F527D08.6070508@Media-Brokers.com> Thanks very much for taking the time for your detailed reply, Stan, but I'll need more time to study it... On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > My gut instinct, based on experience and the match, is that a single GbE > inter site MAN link will be plenty, without the need to duplicate server > infrastructure. I just wanted to point out one thing - I have two primary goals - yes, one is to maximize performance, but the other is accomplish a level of *redundancy*... Also - I already have the servers (I have 3 Poweredge 2970's available to me, only one of which is currently being used)... So, the only extra expenses involved will be relatively minor hardware expenses (multi-port Gb NICs), and some consulting services for making sure I implement the VM environment (including the routing) correctly. So, honestly, we'd be incurring most of these expenses anyway, even if we didn't set up redundant servers, so I figure why not get redundancy too (now is the time to get the boss to pay for it)... -- Best regards, Charles From trashcan at odo.in-berlin.de Sat Mar 3 22:27:11 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 21:27:11 +0100 Subject: [Dovecot] Dovecot clustering with dsync-based replication In-Reply-To: <1330437834.2081.2.camel@innu> References: <1330437834.2081.2.camel@innu> Message-ID: Hi -- On 28.02.2012, at 15:03, Timo Sirainen wrote: > This document describes a design for a dsync-replicated Dovecot cluster. Whow! That's more than interesting, that's a real bummer ;-) At least for my setup of redundant mail servers. Looking forward to test it, Michael From piotr-l at netexpert.pl Sat Mar 3 22:41:24 2012 From: piotr-l at netexpert.pl (Piotr NetExpert) Date: Sat, 03 Mar 2012 21:41:24 +0100 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F5281F4.1070503@netexpert.pl> > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Take into account costs of administering a more complex environment too. -- pozdrawiam Piotr Szafarczyk http://www.netexpert.pl From bradley.giesbrecht at gmail.com Sat Mar 3 23:16:31 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sat, 3 Mar 2012 13:16:31 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F527BBF.3060607@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> Message-ID: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > On 2012-03-02 5:13 PM, Robin wrote: >> This mailing list is for dovecot, not Thunderbird support. The lack of >> replies to Thunderbird usage questions no doubt reflects this. > > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? Show dovecot misbehaving. On Feb 28, 2012, at 6:57 AM, Timo Sirainen wrote: > On Tue, 2012-02-28 at 15:47 +0100, kfx wrote: >>> Did you enable the 'Run search on server' option in the Advanced Search >>> window? Doing this *should* result in Thunderbird using dovecots indexes >>> server side. >>> >> >> Yes I did. >> >> Some more info: >> >> by telnet'ing directly and issuing: >> c search text pattern >> * SEARCH 1208 >> c OK Search completed (0.003 secs). > > So, Solr in Dovecot works perfectly. > >> But the same search in thunderbird return "No matches found" :( > > Thunderbird problem, nothing you can do about it from Dovecot's side. Regards, Bradley Giesbrecht From stan at hardwarefreak.com Sun Mar 4 02:51:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 03 Mar 2012 18:51:39 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F52BC9B.9000005@hardwarefreak.com> On 3/3/2012 2:20 PM, Charles Marcus wrote: > Thanks very much for taking the time for your detailed reply, Stan, but > I'll need more time to study it... > > On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > >> My gut instinct, based on experience and the match, is that a single GbE >> inter site MAN link will be plenty, without the need to duplicate server >> infrastructure. > > I just wanted to point out one thing - I have two primary goals - yes, > one is to maximize performance, but the other is accomplish a level of > *redundancy*... What type of redundancy are you looking for? I.e. is one reason for duplicating servers at site #2 to avoid disruption in the event the MAN link fails? Do you currently have redundant GbE links to each closet switch stack in site #1, and also redundant switches in the datacenter? I.e. do you skip a beat if a core or closet switch fails? If you do not currently have, nor plan to create such network redundancy internally at site #1, then why build application redundancy with the single goal of mitigating failure of a single network link? Do you have reason to believe there is a higher probability of failure of the MAN link than any other single link in the current network? > Also - I already have the servers (I have 3 Poweredge 2970's available > to me, only one of which is currently being used)... > > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Again, you don't need multi-port GbE NICs or bonding for performance--a single GbE link is all each server needs. Your switches should be able to demonstrate that, without even needing a sniffer, assuming they're decent managed units. If you're after link redundancy, use two single port NICs per server, or one mobo mounted port and once single port NIC. Most dual port NICs duplicate the PHYs but not the ethernet chip nor power circuits, etc. Thus, when a dual port NIC fails you usually loose both ports. > So, honestly, we'd be incurring most of these expenses anyway, even if > we didn't set up redundant servers, so I figure why not get redundancy > too (now is the time to get the boss to pay for it)... Don't forget power backup at site #2. Probably not a huge cost in the overall scheme of things, but it's still another $5000 or so. In summary, my advice is: 1. One 1000Mb MAN link is plenty of bandwidth for all users at site #2 including running internet traffic through site #1, saving the cost of an internet pipe at site #2 2. If truly concerned about link failure, get a backup 100Mb/s link, or get two GbE links with a burst contract, depending on price 3. Keep your servers in one place. If you actually desire application level redundancy (IMAP, SMB/CIFS, etc) unrelated to a network link failure, then do your clustering etc "within the rack". It will be much easier to manage and troubleshoot this than two datacenters w/ all kinds of replication etc between them 4. If site #1 is not already link redundant, it makes little sense to make a big redundancy push to cover a possible single network link failure, regardless of which link 5. Building a 2nd datacenter and using the MAN link for data replication gives you no performance advantage, and may actually increase overall utilization, vs using the link as a regular trunk 6. *Setup QOS appropriately to maintain low latency of IMAP and other priority data, giving a back seat to SMB/CIFS/FTP/HTTP and other bulk transfer protocols* With proper QOS the single GbE MAN link will simply scream for everyone, regardless of saturation level -- Stan From dchenusa at yahoo.com Sun Mar 4 08:25:12 2012 From: dchenusa at yahoo.com (D Chen) Date: Sat, 3 Mar 2012 22:25:12 -0800 (PST) Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... Message-ID: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! ?? admin at server:/etc/dovecot$ doveconf -n ? ? # 2.0.13: /etc/dovecot/dovecot.conf ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? # OS: Linux 3.0.0-16-server x86_64 Ubuntu 11.10? ? ? log_timestamp = "%Y-%m-%d %H:%M:%S " ? ? mail_location = maildir:~/Maildir ? ? mail_privileged_group = mail ? ? managesieve_notify_capability = mailto ? ? managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ? ? passdb { ? ? ? driver = pam - Ignored: ? ? } ? ? passdb { ? ? ? driver = pam ? ? } ? ? plugin { ? ? ? sieve = ~/.dovecot.sieve ? ? ? sieve_dir = ~/sieve ? ? } ? ? protocols = imap pop3 sieve ? ? service auth { ? ? ? unix_listener /var/spool/postfix/private/auth-client { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? unix_listener /var/spool/postfix/private/dovecot-auth { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? user = root ? ? } ? ? ssl_cert = was automatically rejected:%n%r ? ? } From tss at iki.fi Sun Mar 4 12:44:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 04 Mar 2012 12:44:46 +0200 Subject: [Dovecot] dsync replication available for testing Message-ID: <4F53479E.40703@iki.fi> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Below is a configuration for virtual user setup. System user configuration works pretty much the same, except doveadm/ssh is run as root. Try first that dsync works successfully with ssh in host1: doveadm sync -u user at domain remote:vmail at host2.example.com and also in host2: doveadm sync -u user at domain remote:vmail at host1.example.com ------ mail_plugins = $mail_plugins notify replication service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } From tss at iki.fi Sun Mar 4 13:05:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:05:26 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: On 3.3.2012, at 20.03, Michael Grimm wrote: > vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test > dsync-local(test): Error: remote: dsync: illegal option -- f > doveadm dsync-server [-u |-A] [-S ] > dsync-local(test): Error: read() from worker server failed: EOF > > If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 >> doveadm sync -f -u test test at remote-host.tld > > vmail> doveadm sync -f -u test ssh vmail at remote-host.tld > dsync-local(test): Error: remote: dsync-server: Command not found. > dsync-local(test): Error: read() from worker server failed: EOF Remove the "ssh" parameter from the middle and change vmail@ to test@ > Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I > can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's a good idea. In the latest hg version the preferred way is: doveadm sync -f -u test remote:vmail at host It automatically adds the remote -u test. > Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). > The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that > I'm the only one reporting that. How could I help to debug this issue? Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or other scripts that "rm" mails? From tss at iki.fi Sun Mar 4 13:13:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:13:18 +0200 Subject: [Dovecot] directly addressable public folders issues In-Reply-To: <201203032108.49489.anyaddress@gmx.net> References: <201203032108.49489.anyaddress@gmx.net> Message-ID: <7E27E7D3-DCE8-4FF9-9689-24815D2895CB@iki.fi> On 3.3.2012, at 22.08, Tom Fernandes wrote: > I would like to have an address info at example.com whose mails are stored in a > public folder. .. > From what I understand it's a good idea to have a separate location for home and > for mail_location. > > For my normal accounts I have: > home = /var/vmail/ > mail_location = ~/Maildir mail_location = maildir:~/Maildir to unnecessary avoid autodetection. > Is there a way to have the same for public folders? This are my current > settings: > > namespace public { > separator = / > prefix = public/ > location = maildir:/var/vmail/public > subscriptions = no > } That's ok. > user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The "mail" isn't necessary here, since it's already globally set. > The LDAP-homeDirectory-attribute for info at example.com is "public/.info" That's not going to work too well. > Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which > is good. The MUA seems to read to read from /var/vmail/public/.info though. > > How can I make the MUA read from /var/vmail/public/.info/Maildir instead? You can't. > And in case this works - is it possible to have only certain users modify the > sieve-rule for this public folder or do I definitely need a passdb-entry for > info at example.com? > Modifying scripts via commandline is not an option... The way I did it was to make info@, sales@ and others aliases to "company" user, which is a rather regular user (except can't actually log in). For this "company" user I've a Sieve script that puts the mails into the proper mailbox, e.g. fileinto "public/info". From tss at iki.fi Sun Mar 4 13:21:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:21:07 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Message-ID: <75EBCBDD-B30A-401F-A6D9-517C03B1873B@iki.fi> On 3.3.2012, at 3.48, Michael M Slusarz wrote: > I can now verify that QRESYNC is triggering this behavior. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/8cbc130c2b72 http://hg.dovecot.org/dovecot-2.1/rev/31ae11fe18b2 From trashcan at odo.in-berlin.de Sun Mar 4 13:31:45 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:31:45 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:05, Timo Sirainen wrote: > On 3.3.2012, at 20.03, Michael Grimm wrote: >> vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test >> dsync-local(test): Error: remote: dsync: illegal option -- f >> doveadm dsync-server [-u |-A] [-S ] >> dsync-local(test): Error: read() from worker server failed: EOF >> >> If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. > > Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: > http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 Ok, that means it has always been ignored in 2.0.x ;-) >>> doveadm sync -f -u test test at remote-host.tld >> >> vmail> doveadm sync -f -u test ssh vmail at remote-host.tld >> dsync-local(test): Error: remote: dsync-server: Command not found. >> dsync-local(test): Error: read() from worker server failed: EOF > > Remove the "ssh" parameter from the middle and change vmail@ to test@ That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: ssh -p 1234 vmail at remote-host.tld Sorry, I should have mentioned that before. In your other mail about 'dsync replication' you refer to a config option: #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} Would that allow for ssh options to be set? >> Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. >> Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. > > In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's > a good idea. In the latest hg version the preferred way is: > > doveadm sync -f -u test remote:vmail at host See above regarding ssh options. >> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >> I'm the only one reporting that. How could I help to debug this issue? > > Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or > other scripts that "rm" mails? No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. But no scripts or such are used to remove mail. I'm running mdbox only. Thanks and regards, Michael From trashcan at odo.in-berlin.de Sun Mar 4 13:34:03 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:34:03 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Hi -- On 04.03.2012, at 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Great news. I would love to test it, if I will be able to run this on a test account, only. All other users should become synced the "old way" for the time being. Would that be possible with the current implementation? Regards, Michael From tss at iki.fi Sun Mar 4 13:35:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:35:04 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Message-ID: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> On 4.3.2012, at 13.31, Michael Grimm wrote: > That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: > > ssh -p 1234 vmail at remote-host.tld > > Sorry, I should have mentioned that before. > > In your other mail about 'dsync replication' you refer to a config option: > #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > Would that allow for ssh options to be set? Yes. >> doveadm sync -f -u test remote:vmail at host > > See above regarding ssh options. So this works by changing the dsync_remote_cmd. >>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>> I'm the only one reporting that. How could I help to debug this issue? >> >> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >> other scripts that "rm" mails? > > No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. > But no scripts or such are used to remove mail. I'm running mdbox only. By "undeletable" do you mean you have mails that always come back after expunging them? I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they don't contain any sensitive information. From tss at iki.fi Sun Mar 4 13:38:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:38:14 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Message-ID: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> On 4.3.2012, at 13.34, Michael Grimm wrote: > On 04.03.2012, at 11:44, Timo Sirainen wrote: > >> In dovecot-2.1 hg you can now test dsync-based replication. > > Great news. I would love to test it, if I will be able to run this on a test > account, only. All other users should become synced the "old way" for the time > being. > > Would that be possible with the current implementation? 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) 2) You can enable replication plugin only for one user by changing mail_plugins setting via userdb extra fields. Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So you could even ignore 1) and just let it sync everyone at startup. From trashcan at odo.in-berlin.de Sun Mar 4 13:41:47 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:41:47 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> Message-ID: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:35, Timo Sirainen wrote: > On 4.3.2012, at 13.31, Michael Grimm wrote: >> That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: >> >> ssh -p 1234 vmail at remote-host.tld >> >> Sorry, I should have mentioned that before. >> >> In your other mail about 'dsync replication' you refer to a config option: >> #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} >> >> Would that allow for ssh options to be set? > > Yes. Good news. >>>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>>> I'm the only one reporting that. How could I help to debug this issue? >>> >>> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >>> other scripts that "rm" mails? >> >> No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. >> But no scripts or such are used to remove mail. I'm running mdbox only. > > By "undeletable" do you mean you have mails that always come back after expunging them? Yes. Deleting by the client will return them after the next dsync run. > I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they > don't contain any sensitive information. From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Thanks and regards, Michael From tss at iki.fi Sun Mar 4 13:54:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:54:34 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Message-ID: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> On 4.3.2012, at 13.41, Michael Grimm wrote: >> By "undeletable" do you mean you have mails that always come back after expunging them? > > Yes. Deleting by the client will return them after the next dsync run. > >> I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they >> don't contain any sensitive information. > > From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Just one mailbox where that consistently happens is enough: 1. Expunge the mail 2. Get a copy of the dbox-Mails/dovecot.index, dbox-Mails/dovecot.index.log and dbox-Mails/dovecot.index.log.2 from both servers 3. Run dsync, and verify that the message is undeleted 4. Get another copy of the indexes from both servers From tss at iki.fi Sun Mar 4 14:14:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:14:23 +0200 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data In-Reply-To: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> References: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> Message-ID: <90B35FA4-651C-40CA-8149-8FE7E3E09E50@iki.fi> On 3.3.2012, at 2.48, Michael M Slusarz wrote: > I'm seeing this: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > It should be: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > since the RFC 2822 quoting characters must be removed. Oops. This has been buggy forever. Added the fix to all Dovecot hg trees. From tss at iki.fi Sun Mar 4 14:33:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:33:35 +0200 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F520F97.5030002@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> <4F520F97.5030002@in.tum.de> Message-ID: <8AAD13E5-CE14-44BF-9CD9-DDB984B0BF31@iki.fi> On 3.3.2012, at 14.33, Christoph Bu?enius wrote: > On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: >> when the private namespace has "prefix = INBOX." and you use doveadm >> fetch to search for "mailbox INBOX", then it prints every message twice: > > Apparently the bug has been introduced with this changeset: > > changeset: 14112:f5353573d3a0 > user: Timo Sirainen > date: Sun Feb 12 02:50:49 2012 +0200 > summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/bbe6b6c2ee99 From tss at iki.fi Sun Mar 4 14:35:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:35:18 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203031015.02716.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> Message-ID: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> On 3.3.2012, at 1.45, Stephen Davies wrote: > No NFS. The file system is local. > > Yes. There are multiple copies of the message for multiple mailboxes for each > of at least two users. But does the error keep repeating for the same mailbox? It's supposed to fix itself automatically after logging the error once. > Yes. Did recently upgrade from 1.2.15. I think in earlier versions mbox used somewhat different index file structures and now Dovecot logs some errors about them. Anyway, one sure way to fix this is to just delete all the .imap/ directories. From tss at iki.fi Sun Mar 4 14:36:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:36:23 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Message-ID: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> On 4.3.2012, at 8.25, D Chen wrote: > Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf So, doveconf -n > dovecot-new.conf mv dovecot-new.conf /etc/dovecot/dovecot.conf That should do it. From tss at iki.fi Sun Mar 4 14:41:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:41:20 +0200 Subject: [Dovecot] Multiple namespaces seems to be used at the same time In-Reply-To: <4F4F84C7.1060502@cnpapers.com> References: <4F4F84C7.1060502@cnpapers.com> Message-ID: <42CE6C31-D246-4AE1-9E79-3DD457E20E39@iki.fi> On 1.3.2012, at 16.16, Steve Campbell wrote: > I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. > > On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. > > The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Difficult to say without knowing 1) doveconf -n output and 2) .subscriptions file contents. > Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Yes, assuming you have: mail_location = mbox:~/mail (with maybe the :INBOX=/var/mail/%u) Also the .subscriptions needs to be in ~/mail/ then. From tss at iki.fi Sun Mar 4 14:45:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:45:48 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> Message-ID: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> On 2.3.2012, at 0.35, Terry Carmen wrote: > With the exchange server being returned in the msExchHomeServerName property as: > > /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername > > I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. .. > The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Do all of the values have the same prefix? Then I guess you can do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%49.100$.example.com If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%-10$.example.com There's no otherwise nice way to parse this string. From tss at iki.fi Sun Mar 4 14:47:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:47:34 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <28E8C0DA-B388-42F0-B39E-B08CA7960D09@iki.fi> On 1.3.2012, at 10.44, Joseph Tam wrote: > I would like to run various doveadm commands that involves all (mail) users like > > doveadm expunge -A mailbox Trash savedbefore 30d > > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users > > However, these accounts are system accounts (locked password, no shell) > and are in userdb to provide UID<->name mapping for utilities like ls, > chown, etc. What userdb are you using? userdb passwd should already skip users that aren't in the valid range. And what Dovecot version are you using? From tss at iki.fi Sun Mar 4 14:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:48:53 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f From tss at iki.fi Sun Mar 4 14:51:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:51:03 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users And one more thing: Does it really even stop there? Looking at the code it's supposed to log an error and continue to next user. Note that it says "Failed to iterate through SOME users". From tss at iki.fi Sun Mar 4 15:32:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:32:15 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: <20120224012247.GA6512@krell.zikzak.de> <1330342560.11500.308.camel@innu.invalid> Message-ID: On 29.2.2012, at 3.03, Andreas M. Kirchwitz wrote: > Timo Sirainen wrote: > >>> There seems to be a new dependency in some modules (eg, lib-storage, >>> libdovecot-lda, libdovecot-ssl) on OpenSSL. In Dovecot 2.0, those >>> modules didn't require OpenSSL, but 2.1 does. >>> >>> For the linking process the path to the OpenSSL library isn't >>> specified properly (SSL_LIBS). Dovecot fails to build if OpenSSL >>> is in a non-standard path. (Haven't checked if SSL_CFLAGS isn't >>> properly used as well.) >> >> Maybe http://hg.dovecot.org/dovecot-2.1/rev/c07415305d9e fixes >> everything? > > That's the way to go. Makes things better, but I've found three more > dependencies. This patch is against the daily snapshot 20120228. > (Sorry for the changes to Makefile.in which you won't need. But this way > I don't have to rebuild Makefile.in from Makefile.am when compiling. ;-) The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 From tss at iki.fi Sun Mar 4 15:36:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:36:59 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> On 28.2.2012, at 19.45, Steve Platt wrote: > Most of this is working but I'm stuck on how to convert users' mail folders > from the existing setup to the new one. I'm using the convert plugin but of > course the problem is that the plugin executes as the "vmail" user and cannot > access the existing mail folders that belong to the users: and I'd be worried > if it could, of course! Convert plugin also has some other problems. > I have the idea that I should be able to run some command (as a privileged > user) on the mail server and have it do the conversion for me, changing the > ownership/permissions on the way. > > Can convert-tool do this? Possibly, but I remember it had some problems. The best solution would be to use Dovecot v2.0's dsync. Also you can use one of the scripts in http://wiki2.dovecot.org/Migration/MailFormat such as mb2md. > I'd prefer to go with the automatic (plugin) conversion if I can bodge the > ownership issues somehow. Failing that, some tool or script may be the next > best answer. You could set mail_drop_priv_before_exec=yes, mail_access_groups=vmail and chgrp vmail, chmod g+rw the old mailboxes. From tss at iki.fi Sun Mar 4 16:10:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:10:28 +0200 Subject: [Dovecot] [PATCH] Pop3 order in courier migration script In-Reply-To: <4F4B2F62.1020204@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> Message-ID: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> On 27.2.2012, at 9.23, Christoph Bu?enius wrote: > I found a problem in the courier conversion script (courier-dovecot-migrate.pl). In some cases, it does not correctly preserve the order of POP3 UIDLs. Thanks, updated. BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. From tss at iki.fi Sun Mar 4 16:23:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:23:39 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302104333.GD11180@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: >> Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. > > In the case above that mail was gzipped twice :( Yes, looks like Dovecot can't correctly fix the wrong S size for gzipped mails. I don't know if I should bother fixing it, especially since in your case the doubly-gzipped mails will look corrupted to user.. From terry at cnysupport.com Sun Mar 4 16:48:17 2012 From: terry at cnysupport.com (Terry Carmen) Date: Sun, 04 Mar 2012 09:48:17 -0500 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> Message-ID: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> ----- Message from Timo Sirainen ---------    Date: Sun, 4 Mar 2012 14:45:48 +0200    From: Timo Sirainen Subject: Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location      To: Terry Carmen      Cc: dovecot at dovecot.org > On 2.3.2012, at 0.35, Terry Carmen wrote: >> With the exchange server being returned in the msExchHomeServerName >> property as: >> >> /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername >> >> I believe this should somehow end up in the userdb section, >> which currently contains "driver = prefetch", but can't seem to >> figure out specifically what should be there. > .. > The only important part is "cn=exchangeservername", which is > the machine name and would need to be prepended to example.com to > get the fqdn. > Do all of the values have the same prefix? Then I guess you can do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%49.100$.example.com > > If the prefix differs, but all of the exchange server names have > the same length, for example 10, you can also do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%-10$.example.com > There's no otherwise nice way to parse this string. If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). homeservers.txt: user1:exch1.example.com user2:exch1.example.com user3:exch1.example.com user4:exch2.example.com Is it possible to do a lookup in a text file to get this? Terry From tss at iki.fi Sun Mar 4 16:58:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:58:59 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> Message-ID: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> On 4.3.2012, at 16.48, Terry Carmen wrote: >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >> >> If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: >> >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >> There's no otherwise nice way to parse this string. > > > If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. OK, so if the prefix or suffix isn't always the same length you can't do the above. > I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). > > homeservers.txt: > user1:exch1.example.com > user2:exch1.example.com > user3:exch1.example.com > user4:exch2.example.com > > Is it possible to do a lookup in a text file to get this? If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile Example line: user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com Note that you can't then return any userdb fields from passdb ldap lookup. From CMarcus at Media-Brokers.com Sun Mar 4 17:27:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:27:40 -0500 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <4F5389EC.1040503@Media-Brokers.com> On 2012-03-04 7:36 AM, Timo Sirainen wrote: > So, doveconf -n> dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. One suggestion... since 'doveconf -n' is sort of a clone of 'postconf -n', maybe it would be a good idea to clone the postfix way for upgrading the configuration file as well? Postfix does it as: postfix upgrade-configuration Man page details for options are here: http://www.postfix.org/postfix.1.html I hate to keep suggesting that you 'copy' anyone or anything, but if you *are*, I think postfix is one of the ones you'd want to emulate... ;) Just a thought... it would be more intuitive for those of us who use postfix with dovecot (and I think there are a lot)... -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 4 17:57:45 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:57:45 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> Message-ID: <4F5390F9.4000301@Media-Brokers.com> On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: > On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >> On 2012-03-02 5:13 PM, Robin wrote: >>> This mailing list is for dovecot, not Thunderbird support. The lack of >>> replies to Thunderbird usage questions no doubt reflects this. >> What precisely about a possible bug with *any* IMAP client when >> using dovecot+fts makes you think that this is not on topic for the >> dovecot list? > Show dovecot misbehaving. The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:21:13 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:21:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? > > The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? well as I wrote in the mini-tutorial, if you use roundcube search, you will see that it uses solr. So from my point of view it would be a Thunderbird thing. -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From CMarcus at Media-Brokers.com Sun Mar 4 18:42:04 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 11:42:04 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> Message-ID: <4F539B5C.3030003@Media-Brokers.com> On 2012-03-04 11:21 AM, Christian Roessner wrote: >> The OP showed where *something* was misbehaving - maybe you should >> read an entire thread before jumping in? >> >> The bottom line, though, until it can be determined that it *is* a >> Thunderbird bug, we won't know if it is a dovecot bug or not, will >> we? > well as I wrote in the mini-tutorial, if you use roundcube search, ? First post from you in this thread, much less a reference to some mini-tutorial you wrote > you will see that it uses solr. So from my point of view it would be > a Thunderbird thing. As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... Looks like he isn't interested in replying (or has unsubbed from the list), so looks like there is no point in pursuing this at this point. I'll just have to wait until we get switched over to dovecot, and see if we have any problems with fts... -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:59:42 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:59:42 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F539B5C.3030003@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> Message-ID: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> >>> > >> well as I wrote in the mini-tutorial, if you use roundcube search, > > ? First post from you in this thread, much less a reference to some mini-tutorial you wrote This: http://www.roessner-network-solutions.com/2012/02/19/full-text-search-with-solr-and-dovecot-on-ubuntu-10-04/ is from my blog ;) > >> you will see that it uses solr. So from my point of view it would be >> a Thunderbird thing. > > As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... > ok -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From bradley.giesbrecht at gmail.com Sun Mar 4 19:39:37 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sun, 4 Mar 2012 09:39:37 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: On Mar 4, 2012, at 7:57 AM, Charles Marcus wrote: > On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: >> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >>> On 2012-03-02 5:13 PM, Robin wrote: >>>> This mailing list is for dovecot, not Thunderbird support. The lack of >>>> replies to Thunderbird usage questions no doubt reflects this. > >>> What precisely about a possible bug with *any* IMAP client when >>> using dovecot+fts makes you think that this is not on topic for the >>> dovecot list? > >> Show dovecot misbehaving. > > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? I have been reading this thread from the beginning. You asked the question: On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. At what point should this issue be taken to a Thunderbird support venue? I will butt out now. From CMarcus at Media-Brokers.com Sun Mar 4 20:03:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 13:03:14 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> Message-ID: <4F53AE62.3000005@Media-Brokers.com> On 2012-03-04 11:59 AM, Christian Roessner wrote: >>> you will see that it uses solr. So from my point of view it would be >>> a Thunderbird thing. >> As I said, I would like confirmation *from the OP* about his last >> comment that his problem with Thunderbird was actually fixed by fixing >> whatever 'third party init script who was the problem'... > ok One other thing I neglected to mention - I'll probably use fts+lucene, since it appears to be simpler (is a plugin and only requires clucene as a dependency)... Next is to push for full support in Thunderbird for a per account config option to simply run all searches on the server Should only be enabled for an account the server for which support fts search indexes): https://bugzilla.mozilla.org/show_bug.cgi?id=564168 -- Best regards, Charles From trashcan at odo.in-berlin.de Sun Mar 4 23:39:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 22:39:22 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:38, Timo Sirainen wrote: > On 4.3.2012, at 13.34, Michael Grimm wrote: >> On 04.03.2012, at 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. >> >> Great news. I would love to test it, if I will be able to run this on a test >> account, only. All other users should become synced the "old way" for the time >> being. >> >> Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration > to return only one test user for replicator that avoids it. (You may be able to > do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > 2) You can enable replication plugin only for one user by changing mail_plugins > setting via userdb extra fields. > > Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So > you could even ignore 1) and just let it sync everyone at startup. Does that mean that the new functionality (queue) does only run dsync replication the usual way whenever new mail arrives? That's at least what I read in your code committed today (but I'm not that good in reading code I do have to confess). If you could approve my assumption, I'm willing to give it a try to all users. Regards, Michael From stan at hardwarefreak.com Mon Mar 5 00:29:32 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 16:29:32 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <4F53ECCC.7060302@hardwarefreak.com> On 3/4/2012 11:39 AM, Bradley Giesbrecht wrote: > It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. Most software contains workarounds to bugs/misfeatures in other vendors' programs. Dovecot already has many: NFS: mmap_disable = no mail_nfs_index = no Workarounds for various client bugs: delay-newmail: netscape-eoh: tb-extra-mailbox-sep: To state that a problem in other software that interacts with Dovecot is not worth discussing seems a bit naive, or arrogant, or both. Given how long it takes, never in some cases, for Mozilla to fix IMAP related problems in TBird, you can't blame the OP for looking in other directions for a solution. Note the bug I filed 2+ years on broken IMAP custom header search: https://bugzilla.mozilla.org/show_bug.cgi?id=546925 2 years later and it's not even been assigned to a dev... -- Stan From p at state-of-mind.de Mon Mar 5 00:47:13 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Sun, 04 Mar 2012 23:47:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53ECCC.7060302@hardwarefreak.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> Message-ID: <4F53F0F1.9010002@state-of-mind.de> On 04.03.2012 23:29, Stan Hoeppner wrote: > not worth discussing seems a bit naive, or arrogant, or both. Given how > long it takes, never in some cases, for Mozilla to fix IMAP related > problems in TBird, you can't blame the OP for looking in other > directions for a solution. Note the bug I filed 2+ years on broken IMAP > custom header search: > > https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > > 2 years later and it's not even been assigned to a dev... We started buying features/fixes. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5372 bytes Desc: S/MIME Cryptographic Signature URL: From stan at hardwarefreak.com Mon Mar 5 01:07:36 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 17:07:36 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F0F1.9010002@state-of-mind.de> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> Message-ID: <4F53F5B8.8070105@hardwarefreak.com> On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how >> long it takes, never in some cases, for Mozilla to fix IMAP related >> problems in TBird, you can't blame the OP for looking in other >> directions for a solution. Note the bug I filed 2+ years on broken IMAP >> custom header search: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 >> >> 2 years later and it's not even been assigned to a dev... > > We started buying features/fixes. Does Mozilla have a page listing such services and prices, err, required/expected donation amounts? -- Stan From sdavies at sdc.com.au Mon Mar 5 01:18:40 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Mon, 5 Mar 2012 09:48:40 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> Message-ID: <201203050948.40819.sdavies@sdc.com.au> Sorry. I wasn't clear. The message did repeat for the same mailbox. eg Mar 5 09:41:40 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42304 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:41:44 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42392 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42480 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Drafts/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Templates/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=12964 for /home/scldad/Mail/Mail/.imap/Sent/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42568 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Junk/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Outbox/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=60240 for /home/scldad/Mail/Mail/.imap/storage1/dovecot.index: Extension header update points outside header size I have deleted the .imap directories and the message seems to have disappeared. Cheers and thanks, Stephen On Sun, 4 Mar 2012 11:05:18 PM Timo Sirainen wrote: > On 3.3.2012, at 1.45, Stephen Davies wrote: > > No NFS. The file system is local. > > > > Yes. There are multiple copies of the message for multiple mailboxes for > > each of at least two users. > > But does the error keep repeating for the same mailbox? It's supposed to > fix itself automatically after logging the error once. > > > Yes. Did recently upgrade from 1.2.15. > > I think in earlier versions mbox used somewhat different index file > structures and now Dovecot logs some errors about them. > > Anyway, one sure way to fix this is to just delete all the .imap/ > directories. -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From amk at spamfence.net Mon Mar 5 02:32:17 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Mon, 5 Mar 2012 00:32:17 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: Message-ID: Hello Timo! Timo Sirainen wrote: > The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: > http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 Thanks for this patch. I've applied it to the dovecot-20120303 nightly snapshot. The good news is, compilation works fine. The bad news is, the libraries and binaries don't work because they don't find the custom SSL libraries. Greetings, Andreas =============================================================================== $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install =============================================================================== $ ldd src/*/.libs/*.so src/auth/.libs/libauthdb_imap.so: linux-gate.so.1 => (0x0013a000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0091d000) librt.so.1 => /lib/librt.so.1 (0x003c7000) libc.so.6 => /lib/libc.so.6 (0x00c6a000) libdl.so.2 => /lib/libdl.so.2 (0x009a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00491000) /lib/ld-linux.so.2 (0x007a4000) src/lib-dovecot/.libs/libdovecot.so: linux-gate.so.1 => (0x0053c000) libdl.so.2 => /lib/libdl.so.2 (0x0056d000) librt.so.1 => /lib/librt.so.1 (0x00925000) libc.so.6 => /lib/libc.so.6 (0x00626000) /lib/ld-linux.so.2 (0x00a61000) libpthread.so.0 => /lib/libpthread.so.0 (0x003ec000) src/lib-lda/.libs/libdovecot-lda.so: linux-gate.so.1 => (0x00b75000) libdovecot-storage.so.0 => /usr/local/src/dovecot-20120303/src/lib-storage/.libs/libdovecot-storage.so.0 (0x00c9a000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0062d000) librt.so.1 => /lib/librt.so.1 (0x00b3d000) libc.so.6 => /lib/libc.so.6 (0x00110000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x002f1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ab1000) /lib/ld-linux.so.2 (0x00f23000) src/lib-sql/.libs/libdovecot-sql.so: linux-gate.so.1 => (0x006d3000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0096c000) libdl.so.2 => /lib/libdl.so.2 (0x0078c000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x00119000) /lib/ld-linux.so.2 (0x00731000) libpthread.so.0 => /lib/libpthread.so.0 (0x00569000) src/lib-ssl-iostream/.libs/libdovecot-ssl.so: linux-gate.so.1 => (0x00ea1000) libdl.so.2 => /lib/libdl.so.2 (0x00b31000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00110000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00dcf000) librt.so.1 => /lib/librt.so.1 (0x00fa5000) libc.so.6 => /lib/libc.so.6 (0x002d3000) /lib/ld-linux.so.2 (0x002b4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d3c000) src/lib-storage/.libs/libdovecot-storage.so: linux-gate.so.1 => (0x002ee000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00395000) libdl.so.2 => /lib/libdl.so.2 (0x00958000) librt.so.1 => /lib/librt.so.1 (0x00333000) libc.so.6 => /lib/libc.so.6 (0x00d45000) /lib/ld-linux.so.2 (0x008e9000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f06000) src/login-common/.libs/libdovecot-login.so: linux-gate.so.1 => (0x00d66000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00c82000) librt.so.1 => /lib/librt.so.1 (0x00f64000) libc.so.6 => /lib/libc.so.6 (0x00110000) libdl.so.2 => /lib/libdl.so.2 (0x00b26000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /lib/ld-linux.so.2 (0x00520000) =============================================================================== $ ldd /usr/local/Dovecot-20120303/lib/dovecot/*.so /usr/local/Dovecot-20120303/*bin/* /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so: linux-gate.so.1 => (0x00230000) librt.so.1 => /lib/librt.so.1 (0x00b69000) libc.so.6 => /lib/libc.so.6 (0x00231000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a82000) /lib/ld-linux.so.2 (0x007eb000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_imap_acl_plugin.so: linux-gate.so.1 => (0x004fb000) lib01_acl_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so (0x00c6f000) librt.so.1 => /lib/librt.so.1 (0x0061d000) libc.so.6 => /lib/libc.so.6 (0x001dd000) libpthread.so.0 => /lib/libpthread.so.0 (0x00fae000) /lib/ld-linux.so.2 (0x00b89000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_lazy_expunge_plugin.so: linux-gate.so.1 => (0x00e5b000) librt.so.1 => /lib/librt.so.1 (0x00847000) libc.so.6 => /lib/libc.so.6 (0x00110000) libpthread.so.0 => /lib/libpthread.so.0 (0x0039d000) /lib/ld-linux.so.2 (0x00a4e000) /usr/local/Dovecot-20120303/lib/dovecot/lib05_snarf_plugin.so: linux-gate.so.1 => (0x001e2000) librt.so.1 => /lib/librt.so.1 (0x00441000) libc.so.6 => /lib/libc.so.6 (0x00220000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cfa000) /lib/ld-linux.so.2 (0x00acd000) /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so: linux-gate.so.1 => (0x00fa9000) librt.so.1 => /lib/librt.so.1 (0x00bb4000) libc.so.6 => /lib/libc.so.6 (0x00d5e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00be7000) /lib/ld-linux.so.2 (0x0055c000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_imap_quota_plugin.so: linux-gate.so.1 => (0x00426000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x004c9000) librt.so.1 => /lib/librt.so.1 (0x00e35000) libc.so.6 => /lib/libc.so.6 (0x009b5000) libpthread.so.0 => /lib/libpthread.so.0 (0x00146000) /lib/ld-linux.so.2 (0x00507000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_trash_plugin.so: linux-gate.so.1 => (0x00baf000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x00d8b000) librt.so.1 => /lib/librt.so.1 (0x0041e000) libc.so.6 => /lib/libc.so.6 (0x00bb6000) libpthread.so.0 => /lib/libpthread.so.0 (0x0088a000) /lib/ld-linux.so.2 (0x00a6d000) /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00f77000) libc.so.6 => /lib/libc.so.6 (0x00146000) libpthread.so.0 => /lib/libpthread.so.0 (0x00df9000) /lib/ld-linux.so.2 (0x004dd000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_autocreate_plugin.so: linux-gate.so.1 => (0x005d2000) librt.so.1 => /lib/librt.so.1 (0x007ed000) libc.so.6 => /lib/libc.so.6 (0x00262000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b48000) /lib/ld-linux.so.2 (0x00243000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_expire_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x0099e000) libc.so.6 => /lib/libc.so.6 (0x0013c000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a93000) /lib/ld-linux.so.2 (0x00460000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so: linux-gate.so.1 => (0x002c1000) librt.so.1 => /lib/librt.so.1 (0x00ef2000) libc.so.6 => /lib/libc.so.6 (0x00c3a000) libpthread.so.0 => /lib/libpthread.so.0 (0x0028d000) /lib/ld-linux.so.2 (0x003f5000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_listescape_plugin.so: linux-gate.so.1 => (0x0027c000) librt.so.1 => /lib/librt.so.1 (0x007cf000) libc.so.6 => /lib/libc.so.6 (0x00427000) libpthread.so.0 => /lib/libpthread.so.0 (0x00110000) /lib/ld-linux.so.2 (0x00d25000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_mail_log_plugin.so: linux-gate.so.1 => (0x00cff000) lib15_notify_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so (0x00a2b000) librt.so.1 => /lib/librt.so.1 (0x00152000) libc.so.6 => /lib/libc.so.6 (0x00355000) libpthread.so.0 => /lib/libpthread.so.0 (0x00680000) /lib/ld-linux.so.2 (0x00c16000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_virtual_plugin.so: linux-gate.so.1 => (0x0074d000) librt.so.1 => /lib/librt.so.1 (0x008d2000) libc.so.6 => /lib/libc.so.6 (0x004c2000) libpthread.so.0 => /lib/libpthread.so.0 (0x009c5000) /lib/ld-linux.so.2 (0x0038e000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so: linux-gate.so.1 => (0x004ab000) libz.so.1 => /lib/libz.so.1 (0x0091c000) libbz2.so.1 => /lib/libbz2.so.1 (0x00c4d000) librt.so.1 => /lib/librt.so.1 (0x00b30000) libc.so.6 => /lib/libc.so.6 (0x0075b000) libpthread.so.0 => /lib/libpthread.so.0 (0x00611000) /lib/ld-linux.so.2 (0x00bfd000) /usr/local/Dovecot-20120303/lib/dovecot/lib21_fts_squat_plugin.so: linux-gate.so.1 => (0x00df4000) lib20_fts_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so (0x00ea8000) librt.so.1 => /lib/librt.so.1 (0x0021d000) libc.so.6 => /lib/libc.so.6 (0x00bd1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d63000) /lib/ld-linux.so.2 (0x00b83000) /usr/local/Dovecot-20120303/lib/dovecot/lib30_imap_zlib_plugin.so: linux-gate.so.1 => (0x0059f000) lib20_zlib_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so (0x00652000) librt.so.1 => /lib/librt.so.1 (0x00f2d000) libc.so.6 => /lib/libc.so.6 (0x00735000) libz.so.1 => /lib/libz.so.1 (0x00110000) libbz2.so.1 => /lib/libbz2.so.1 (0x00125000) libpthread.so.0 => /lib/libpthread.so.0 (0x00136000) /lib/ld-linux.so.2 (0x001f9000) /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so: linux-gate.so.1 => (0x004c0000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x006b2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00992000) /lib/ld-linux.so.2 (0x00d92000) /usr/local/Dovecot-20120303/lib/dovecot/lib95_imap_stats_plugin.so: linux-gate.so.1 => (0x0076b000) lib90_stats_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so (0x00b90000) librt.so.1 => /lib/librt.so.1 (0x00215000) libc.so.6 => /lib/libc.so.6 (0x0021e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /lib/ld-linux.so.2 (0x00e48000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-lda.so: linux-gate.so.1 => (0x00d82000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x009bb000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00542000) libc.so.6 => /lib/libc.so.6 (0x002f6000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x006a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00189000) /lib/ld-linux.so.2 (0x002d7000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-login.so: linux-gate.so.1 => (0x00f96000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00e56000) librt.so.1 => /lib/librt.so.1 (0x00371000) libc.so.6 => /lib/libc.so.6 (0x00168000) libdl.so.2 => /lib/libdl.so.2 (0x00623000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ccd000) /lib/ld-linux.so.2 (0x0074f000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so: linux-gate.so.1 => (0x00bf3000) libdl.so.2 => /lib/libdl.so.2 (0x00a54000) librt.so.1 => /lib/librt.so.1 (0x00ad8000) libc.so.6 => /lib/libc.so.6 (0x00e63000) /lib/ld-linux.so.2 (0x00b1c000) libpthread.so.0 => /lib/libpthread.so.0 (0x005ac000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-sql.so: linux-gate.so.1 => (0x008b7000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00909000) libdl.so.2 => /lib/libdl.so.2 (0x005f5000) librt.so.1 => /lib/librt.so.1 (0x008bc000) libc.so.6 => /lib/libc.so.6 (0x00675000) /lib/ld-linux.so.2 (0x004bc000) libpthread.so.0 => /lib/libpthread.so.0 (0x00184000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-ssl.so: linux-gate.so.1 => (0x00ef2000) libdl.so.2 => /lib/libdl.so.2 (0x0033f000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00a3d000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x0034e000) librt.so.1 => /lib/librt.so.1 (0x002ea000) libc.so.6 => /lib/libc.so.6 (0x00110000) /lib/ld-linux.so.2 (0x007d4000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so: linux-gate.so.1 => (0x0089e000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00be9000) libdl.so.2 => /lib/libdl.so.2 (0x00852000) librt.so.1 => /lib/librt.so.1 (0x001aa000) libc.so.6 => /lib/libc.so.6 (0x00442000) /lib/ld-linux.so.2 (0x00b8a000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b0e000) /usr/local/Dovecot-20120303/bin/doveadm: linux-gate.so.1 => (0x00c6a000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00110000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00acf000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x0066e000) libc.so.6 => /lib/libc.so.6 (0x00247000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x001f4000) librt.so.1 => /lib/librt.so.1 (0x00ab5000) libfreebl3.so => /lib/libfreebl3.so (0x003d1000) /lib/ld-linux.so.2 (0x00228000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cc3000) /usr/local/Dovecot-20120303/bin/doveconf: linux-gate.so.1 => (0x00830000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x0032a000) libc.so.6 => /lib/libc.so.6 (0x00876000) libdl.so.2 => /lib/libdl.so.2 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x002b6000) /lib/ld-linux.so.2 (0x007d2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /usr/local/Dovecot-20120303/bin/dsync: linux-gate.so.1 => (0x00c8b000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00257000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00662000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00d15000) libc.so.6 => /lib/libc.so.6 (0x00dd1000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x009b3000) librt.so.1 => /lib/librt.so.1 (0x0037f000) libfreebl3.so => /lib/libfreebl3.so (0x00110000) /lib/ld-linux.so.2 (0x00a8e000) libpthread.so.0 => /lib/libpthread.so.0 (0x001e9000) /usr/local/Dovecot-20120303/sbin/dovecot: linux-gate.so.1 => (0x00f00000) libcap.so.2 => /lib/libcap.so.2 (0x0037c000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00555000) libc.so.6 => /lib/libc.so.6 (0x0080b000) libattr.so.1 => /lib/libattr.so.1 (0x00ece000) libdl.so.2 => /lib/libdl.so.2 (0x00ea2000) librt.so.1 => /lib/librt.so.1 (0x00dce000) /lib/ld-linux.so.2 (0x00d49000) libpthread.so.0 => /lib/libpthread.so.0 (0x00bb2000) =============================================================================== From isolecki at gmail.com Mon Mar 5 03:42:12 2012 From: isolecki at gmail.com (Ian Solecki) Date: Sun, 4 Mar 2012 20:42:12 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine Message-ID: Hello, hoping someone here might be able to help me or at least point me in the right direction. My company recently (last week) moved to a new dedicated server for website and email hosting. It is a fairly run-of-the-mill Linux machine running cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail server. When we first set up the server, I was able to create a mailbox, access it via any desktop mail client, webmail, AND by setting it up as a basic POP3 account on my BlackBerry. Mail was running fine to and from the BlackBerry, no problems. I deleted that account from my BlackBerry as it was a test account, and went to add my actual account but was unable to do so. Received the "Cannot log in. Verify your email address, user name and password. If the error persists, contact (my domain name)" message. I tried the test account that had been running successfully and sending/receiving emails not minutes earlier, and it would not set up either. I have since tried multiple accounts on multiple BlackBerry devices (different models) on multiple carriers on several of my different domains (all of which point to the same server, of course), to no avail. Yet, any of these accounts still work flawlessly in any desktop mail client (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these BlackBerrys work flawlessly with any other mail server. The username to log in to these mailboxes is not standard, it's mailboxname+ example.com if the email address is mailboxname at example.com and the mail server is mail.example.com. So, I know I have to access the "Advanced Settings" in BlackBerry email setup in order to put this username in. Still, no effect. So, there is something wrong with how mail SETUP works (not sending/receiving, though that may also not work, I have no way of knowing now) between RIM and my server, and it's something that has changed since the server was set up a week ago. My carrier(s) are clueless, my dedicated server provider (Lunarpages) is clueless. Can anyone help? From gedalya at gedalya.net Mon Mar 5 04:52:06 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 04 Mar 2012 21:52:06 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F542A56.6070104@gedalya.net> Do you have the dovecot logs? What do they say about connections coming from RIM? On 03/04/2012 08:42 PM, Ian Solecki wrote: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > From dchenusa at yahoo.com Mon Mar 5 05:57:16 2012 From: dchenusa at yahoo.com (Dennis Chen) Date: Sun, 4 Mar 2012 19:57:16 -0800 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Thanks Timo, I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? I also posted the similar question to ubuntuforums, but not much response. Hope you understand my point. It's very frustrated! Sent from my iPhone On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > On 4.3.2012, at 8.25, D Chen wrote: > >> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! > > v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > >> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf > > So, doveconf -n > dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. > From tss at iki.fi Mon Mar 5 08:41:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:41:36 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Message-ID: Dovecot v2.0 changed the settings a lot compared to v1.x. There are no longer auth default or socket listen sections. The doveconf -n generated dovecot.conf should contain all of the settings that you had in v1.x, converted for v2.0. So you should be able to use it directly without problems. If you want, you could look for Dovecot's example-config that probably comes with Ubuntu (in /usr/share/doc/dovecot*/ maybe?), copy those to /etc/dovecot/ and change the settings in there based on the generated dovecot.conf. In any case you shouldn't try to add those v1.x-specific things back there anymore, since they'll add back the "obsolete settings" warnings. On 5.3.2012, at 5.57, Dennis Chen wrote: > Thanks Timo, > > I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) > > Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? > > I also posted the similar question to ubuntuforums, but not much response. > > Hope you understand my point. > > It's very frustrated! > > > Sent from my iPhone > > On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > >> On 4.3.2012, at 8.25, D Chen wrote: >> >>> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! >> >> v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: >> >>> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf >> >> So, doveconf -n > dovecot-new.conf >> mv dovecot-new.conf /etc/dovecot/dovecot.conf >> >> That should do it. >> > From tss at iki.fi Mon Mar 5 08:43:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:43:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Message-ID: <86F8DB00-F1B0-4666-B3EC-B3EA25F87C0B@iki.fi> On 4.3.2012, at 23.39, Michael Grimm wrote: >> Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So >> you could even ignore 1) and just let it sync everyone at startup. > > Does that mean that the new functionality (queue) does only run dsync replication > the usual way whenever new mail arrives? That's at least what I read in your code > committed today (but I'm not that good in reading code I do have to confess). > > If you could approve my assumption, I'm willing to give it a try to all users. Yes, the replicator simply runs "doveadm sync -u user at domain -d" (and sometimes with -f). The -d gets the default location from mail_replica setting. From robert at schetterer.org Mon Mar 5 08:48:01 2012 From: robert at schetterer.org (Robert Schetterer) Date: Mon, 05 Mar 2012 07:48:01 +0100 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F5461A1.7000100@schetterer.org> Am 05.03.2012 02:42, schrieb Ian Solecki: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > this is the dovecot mail list, not support for blackberry, we cant help unless you have dovecot configs and dediacted logs to your problem -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From p at state-of-mind.de Mon Mar 5 08:53:39 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Mon, 5 Mar 2012 07:53:39 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F5B8.8070105@hardwarefreak.com> References: <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> <4F53F5B8.8070105@hardwarefreak.com> Message-ID: <20120305065339.GC5094@state-of-mind.de> Stan, * Stan Hoeppner : > On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > > > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > > > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how > >> long it takes, never in some cases, for Mozilla to fix IMAP related > >> problems in TBird, you can't blame the OP for looking in other > >> directions for a solution. Note the bug I filed 2+ years on broken IMAP > >> custom header search: > >> > >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > >> > >> 2 years later and it's not even been assigned to a dev... > > > > We started buying features/fixes. > > Does Mozilla have a page listing such services and prices, err, > required/expected donation amounts? to my knowledge they don't have a page listing services and prices. Recently they discussed pros and cons of crowd sourcing, but without much progress. I can get you in contact with one of the TB programmers, who implemented features for us, if you want to. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3603 bytes Desc: not available URL: From bra at fsn.hu Mon Mar 5 09:25:34 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 08:25:34 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <4F546A6E.6020400@fsn.hu> Hi, On 03/04/12 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Everything > isn't finished yet, but it appears to work and I've enabled it for my > @dovecot.fi mails. Some issues: > > - public namespace isn't replicated at all > - shared namespace is replicated, but not private mail flags > - I've only tested SSH replication setup now, not director > replication setup (and director setup is still missing many things) > - SSH replication setup uses aggregator process, which isn't really > necessary and can probably be avoided in future Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. (if I understand things correctly) Thanks for working on this. From janfrode at tanso.net Mon Mar 5 10:35:25 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 09:35:25 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <20120305083525.GA20889@dibs.tanso.net> On Sun, Mar 04, 2012 at 01:38:14PM +0200, Timo Sirainen wrote: > > > > Great news. I would love to test it, if I will be able to run this on a test > > account, only. All other users should become synced the "old way" for the time > > being. > > > > Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) IMHO it would be great if it didn't sync all users. We probably av have hundreds of thousands of inactive users that we would like to sync at a later point. Also when we provision users that's just an entry in a LDAP-directory without any files or directories. So dovecot shouldn't create any directories for these before they've received mail or logged in. So, ideally (for us), dovecot should keep a log over which accounts are active (has received or checked mail), and only sync users that has been active for the last $timeperiode on startup. -jf From ccourvoisier70 at yahoo.com Mon Mar 5 11:51:19 2012 From: ccourvoisier70 at yahoo.com (Charles C) Date: Mon, 5 Mar 2012 09:51:19 +0000 (GMT) Subject: [Dovecot] 1.0beta to latest and greatest? Message-ID: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> dear dovecot users, This is my first post and with a rather embarassing question. To soften the question up I just want to say I've been very pleased with Dovecot thus far and its low maintenance requirements - perhaps to pleased! I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Thanks for your input. Charles From tss at iki.fi Mon Mar 5 11:56:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 11:56:30 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> Message-ID: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> On 4.3.2012, at 13.54, Timo Sirainen wrote: > On 4.3.2012, at 13.41, Michael Grimm wrote: > >>> By "undeletable" do you mean you have mails that always come back after expunging them? >> >> Yes. Deleting by the client will return them after the next dsync run. Luckily this just started happening to me as well. After some debugging I found and fixed the problem: http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 From tss at iki.fi Mon Mar 5 12:08:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:08:35 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F546A6E.6020400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: On 5.3.2012, at 9.25, Attila Nagy wrote: > On 03/04/12 11:44, Timo Sirainen wrote: >> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >> > Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. Sure the idea is to improve the performance :) There are two ways: 1) Use longer running SSH sessions which dsync more than one user at a time. 2) Use TCP connections instead of SSH. > It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. From tss at iki.fi Mon Mar 5 12:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:41:30 +0200 Subject: [Dovecot] 1.0beta to latest and greatest? In-Reply-To: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> References: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> Message-ID: On 5.3.2012, at 11.51, Charles C wrote: > I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Just delete the indexes and you don't have to worry about problems related to them. The config file is different though and you'll probably have to spend some time converting it. (I'm assuming you're using mbox/maildir, not the broken dbox implementation.) From tss at iki.fi Mon Mar 5 12:45:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:45:26 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305083525.GA20889@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> Message-ID: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> On 5.3.2012, at 10.35, Jan-Frode Myklebust wrote: >> 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > IMHO it would be great if it didn't sync all users. We probably av have > hundreds of thousands of inactive users that we would like to sync at a > later point. Also when we provision users that's just an entry in a > LDAP-directory without any files or directories. So dovecot shouldn't > create any directories for these before they've received mail or logged in. > > So, ideally (for us), dovecot should keep a log over which accounts are > active (has received or checked mail), and only sync users that has been > active for the last $timeperiode on startup. Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. From CMarcus at Media-Brokers.com Mon Mar 5 13:13:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:13:07 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F547CAB.2030005@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F547CAB.2030005@gmail.com> Message-ID: <4F549FC3.5030602@Media-Brokers.com> On 2012-03-05 3:43 AM, kadafax at gmail.com wrote: > Le 02/03/12 13:40, Charles Marcus a ?crit : >> On 2012-02-28 11:28 AM, Charles Marcus wrote: >>> On 2012-02-28 11:05 AM, kfx wrote: >>>> Ok I feel ashame... it was a third party init scrip who was the >>>> problem :( >>> So... you're saying that Thunderbird now correctly uses server side >>> search? >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > Yes, for me thunderbird correctly use server side search. Just got confirmation from the OP that his problem with Thunderbird indeed is now gone, so it was a 3rd party init script causing his problems... One less thing to worry about... -- Best regards, Charles From CMarcus at Media-Brokers.com Mon Mar 5 13:24:28 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:24:28 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <1330441042.2081.24.camel@innu> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> Message-ID: <4F54A26C.4040005@Media-Brokers.com> On 2012-02-28 9:57 AM, Timo Sirainen wrote: > So, Solr in Dovecot works perfectly. Timo, a follow-up on this... Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? I don't have a dovecot test server set up yet, but even if I did, I wouldn't really know what to look for or how to test this myself... Thanks, -- Best regards, Charles From mark at ecs.vuw.ac.nz Mon Mar 5 13:26:22 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Tue, 06 Mar 2012 00:26:22 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330603470.2081.37.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> <1330603470.2081.37.camel@innu> Message-ID: <4F54A2DE.5000500@ecs.vuw.ac.nz> On 03/02/12 01:04, Timo Sirainen wrote: > The difference between your previously working system and currently > working system is the GSSAPI/Kerberos libraries. Just to close this thread off, seems that the bug was in the cyrus-sasl libraries that kmail uses. Reverting from the 2.1.25 version that the latest was trying to use to 2.1.23 that was on the older systems got it working again. When I get a moment I'll try and work out what specifically changed. cheers mark From bra at fsn.hu Mon Mar 5 14:15:39 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 13:15:39 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: <4F54AE6B.4060400@fsn.hu> On 03/05/12 11:08, Timo Sirainen wrote: > On 5.3.2012, at 9.25, Attila Nagy wrote: > >> On 03/04/12 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>> >> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. > Sure the idea is to improve the performance :) There are two ways: > > 1) Use longer running SSH sessions which dsync more than one user at a time. > > 2) Use TCP connections instead of SSH. Don't forget about connection pooling to get concurrency. :) BTW, despite being somewhat harder to implement, I personally like native connections better. > >> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. > The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. > It depends. For a moderately loaded server I get this: # time ssh root at be02 "echo 1" 1 0.000u 0.009s 0:00.30 0.0% 0+0k 0+0io 0pf+0w ICMP echo RTT is 0.878 ms. So the ssh connection adds ~29 ms overhead to each sync request. Yes, dsync seems to need some optimizations too. :) I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. The method was similar to yours: - an external library wrote modified user ids to a file - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) The runs were longer and longer... BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... From janfrode at tanso.net Mon Mar 5 14:35:08 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 13:35:08 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> Message-ID: <20120305123508.GA22845@dibs.tanso.net> On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: > > > > So, ideally (for us), dovecot should keep a log over which accounts are > > active (has received or checked mail), and only sync users that has been > > active for the last $timeperiode on startup. > > Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. > .. or we could keep touching /activemailaccounts/$address in post-login scripts, and run "doveadm sync" for any user updated the last $timeperiode and avoid the need for SQL-userdatabase. But we still don't have a last-login update on lmtp delivery... or has this changed? -jf From tss at iki.fi Mon Mar 5 14:48:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 14:48:40 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54AE6B.4060400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: On 5.3.2012, at 14.15, Attila Nagy wrote: >>> On 03/04/12 11:44, Timo Sirainen wrote: >>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>> >>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >> Sure the idea is to improve the performance :) There are two ways: >> >> 1) Use longer running SSH sessions which dsync more than one user at a time. >> >> 2) Use TCP connections instead of SSH. > Don't forget about connection pooling to get concurrency. :) There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. > BTW, despite being somewhat harder to implement, I personally like native connections better. Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. >>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >> > It depends. For a moderately loaded server I get this: > # time ssh root at be02 "echo 1" I meant doveadm/dsync costs, ssh startup is rather slow. > Yes, dsync seems to need some optimizations too. :) > I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. > The method was similar to yours: > - an external library wrote modified user ids to a file > - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) > > The runs were longer and longer... dsync doesn't currently take enough advantage of modseqs and send only the changed data. > BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... Definitely doesn't help. From apm at one.com Mon Mar 5 15:01:54 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 14:01:54 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST Message-ID: <4F54B942.9070005@one.com> Hi, I noticed a difference between courier and dovecot, and I'm not sure which of them is wrong wrt. RFC3501 - if any. I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: INBOX INBOX.Trash INBOX.INBOX.folder INBOX.INBOX.folder.a INBOX.INBOX.folder.b The INBOX.INBOX folder does not exist on disk and is not subscribed. Courier responds to: . list "" "*" with * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" But dovecot does not list that folder using "*". However, if you issue: . list "" "INBOX.%" Dovecot answers: * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". I know the recommended client way is to use "%", but I'm still curious about which is the correct behaviour. /Peter From bra at fsn.hu Mon Mar 5 15:11:06 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 14:11:06 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: <4F54BB6A.20702@fsn.hu> On 03/05/12 13:48, Timo Sirainen wrote: > On 5.3.2012, at 14.15, Attila Nagy wrote: > >>>> On 03/04/12 11:44, Timo Sirainen wrote: >>>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>>> >>>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >>> Sure the idea is to improve the performance :) There are two ways: >>> >>> 1) Use longer running SSH sessions which dsync more than one user at a time. >>> >>> 2) Use TCP connections instead of SSH. >> Don't forget about connection pooling to get concurrency. :) > There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. Good to hear. > >> BTW, despite being somewhat harder to implement, I personally like native connections better. > Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. Yes. For large installations there may be some backend channel already (SSL tunnels, IPSec etc), so it seems to be OK. > >>>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >>> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >>> >> It depends. For a moderately loaded server I get this: >> # time ssh root at be02 "echo 1" > I meant doveadm/dsync costs, ssh startup is rather slow. I see. Running from network makes this worse slightly. Long running processes with long running connections rule. :) > >> Yes, dsync seems to need some optimizations too. :) >> I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. >> The method was similar to yours: >> - an external library wrote modified user ids to a file >> - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) >> >> The runs were longer and longer... > dsync doesn't currently take enough advantage of modseqs and send only the changed data. Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. > >> BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... > Definitely doesn't help. I know, we are working on this. :) From tss at iki.fi Mon Mar 5 16:37:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:37:44 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54BB6A.20702@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> <4F54BB6A.20702@fsn.hu> Message-ID: <8E6FD156-E7F5-49BD-9C3A-1F012E600DD8@iki.fi> On 5.3.2012, at 15.11, Attila Nagy wrote: >> dsync doesn't currently take enough advantage of modseqs and send only the changed data. > Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? > I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? The plan is to get it working with at least a few thousand users to several tens of thousands. > BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. Yes, that's also in my future plans, but it's a larger change. Also I don't think Cassandra (or any nosql?) still supports application-level merging of data after split brain. From tss at iki.fi Mon Mar 5 16:45:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:45:55 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54B942.9070005@one.com> References: <4F54B942.9070005@one.com> Message-ID: On 5.3.2012, at 15.01, Peter Mogensen wrote: > I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: > > INBOX > INBOX.Trash > INBOX.INBOX.folder > INBOX.INBOX.folder.a > INBOX.INBOX.folder.b > > The INBOX.INBOX folder does not exist on disk and is not subscribed. > > Courier responds to: > . list "" "*" > with > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" I'm surprised Courier would return this. > But dovecot does not list that folder using "*". But it returns all of the mailboxes under INBOX.INBOX, right? > However, if you issue: > . list "" "INBOX.%" > > Dovecot answers: > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. > This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Also if you used LAYOUT=fs in Dovecot, it would always show the \Noselect mailboxes because they happen to exist physically. From tss at iki.fi Mon Mar 5 16:48:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:48:11 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54A26C.4040005@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> Message-ID: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> On 5.3.2012, at 13.24, Charles Marcus wrote: > On 2012-02-28 9:57 AM, Timo Sirainen wrote: >> So, Solr in Dovecot works perfectly. > > Timo, a follow-up on this... > > Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... > > Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? The regular IMAP protocol supports searching only from the selected folder. But I guess this quickfilter search also searches from only the selected folder. So I don't see a problem. From tss at iki.fi Mon Mar 5 16:56:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:56:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305123508.GA22845@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> <20120305123508.GA22845@dibs.tanso.net> Message-ID: On 5.3.2012, at 14.35, Jan-Frode Myklebust wrote: > On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: >>> >>> So, ideally (for us), dovecot should keep a log over which accounts are >>> active (has received or checked mail), and only sync users that has been >>> active for the last $timeperiode on startup. >> >> Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. >> > > .. or we could keep touching /activemailaccounts/$address in post-login > scripts, and run "doveadm sync" for any user updated the last $timeperiode > and avoid the need for SQL-userdatabase. But we still don't have a > last-login update on lmtp delivery... or has this changed? It would be pretty simple to write such a plugin that globally does it for all imap/pop3/lmtp. Here, works for v2.0 and v2.1: http://dovecot.org/patches/2.1/lastaccess-plugin.c From apm at one.com Mon Mar 5 16:56:52 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 15:56:52 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> Message-ID: <4F54D434.6090300@one.com> On 2012-03-05 15:45, Timo Sirainen wrote: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > I'm surprised Courier would return this. > >> But dovecot does not list that folder using "*". > > But it returns all of the mailboxes under INBOX.INBOX, right? Yes. And they exists on disk and are subscribed to. >> However, if you issue: >> . list "" "INBOX.%" >> >> Dovecot answers: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. Seems reasonable. >> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". > > What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. /Peter From tss at iki.fi Mon Mar 5 17:08:02 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 17:08:02 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54D434.6090300@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> Message-ID: <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> On 5.3.2012, at 16.56, Peter Mogensen wrote: >>> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". >> >> What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. > > Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. Couldn't the scripts be simply fixed to figure out that if foo.bar.baz is returned without foo or foo.bar, then just internally assume them being there as \noselect? From CMarcus at Media-Brokers.com Mon Mar 5 18:12:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 11:12:19 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> Message-ID: <4F54E5E3.2010509@Media-Brokers.com> On 2012-03-05 9:48 AM, Timo Sirainen wrote: > On 5.3.2012, at 13.24, Charles Marcus wrote: >> Thunderbird has a 'Quickfilter Toolbar' with a little searchbox >> that applies a filter of the current folder message view pane (to >> show you only messages in the pane that meet the criteria >> specified). The default criteria that are selected are just >> 'Sender', 'Recipients' and 'Subject', but you can also select >> 'Body' (and a few others like 'Unread', 'Contain attachments', >> etc), but the 'Body' criteria is the one that would want/need to >> use the fts indexes, so... >> >> Can dovecots fts indexes be used in a case like this? In other >> words, will it 'just work'? Or will it *not* work? Or, would the >> client need to specify the folder in the commands it sends to >> limit the search and/or results to just the currently selected >> folder? Or is this even possible for such a simple/limited use >> filtering mechanism? > The regular IMAP protocol supports searching only from the selected > folder. Interesting, thanks... so, just guessing, most likely Thunderbird simply iterates over each folder in an account when searching an entire account and 'Run search on server' is checked (only available in the Advanced Search window)... One last question then (couldn't find an answer on the wiki)... I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: https://bugzilla.mozilla.org/show_bug.cgi?id=541400 This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? This would pretty much neutralize/work around the above bug. > But I guess this quickfilter search also searches from only the > selected folder. So I don't see a problem. Correct, and thanks for the confirmation... -- Best regards, Charles From tss at iki.fi Mon Mar 5 19:01:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:01:45 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54E5E3.2010509@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> Message-ID: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> On 5.3.2012, at 18.12, Charles Marcus wrote: > One last question then (couldn't find an answer on the wiki)... > > I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... > > What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? Lucene doesn't really work that way. It only searches full words. But some words are "stop words" that are ignored, such as "a" or "the" in the English language. So fts-lucene won't find anything when you search for "a", but other character searches will return all messages that contain it as a full word. (Or: That's how I think it works, too lazy to test it now.) > The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: > > https://bugzilla.mozilla.org/show_bug.cgi?id=541400 > > This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... I doubt this is going to be a problem with Dovecot+fts. The search results will be pretty much instantaneous, even if the search matches all of the messages. > Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? Nope. From CMarcus at Media-Brokers.com Mon Mar 5 19:11:50 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 12:11:50 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> Message-ID: <4F54F3D6.9040508@Media-Brokers.com> On 2012-03-05 12:01 PM, Timo Sirainen wrote: > On 5.3.2012, at 18.12, Charles Marcus wrote: >> Thunderbird has an annoying behavior where it sends a new/separate >> query each time a character is typed, beginning with the very first >> character: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=541400 >> >> This causes massive performance degradation on my clients system >> that currently uses Courier-IMAP (this is the one we'll soon be >> converting to dovecot) when the users (usually accidentally >> (select the 'Body' criteria in the Quickfilter searchbox... > I doubt this is going to be a problem with Dovecot+fts. The search > results will be pretty much instantaneous, even if the search matches > all of the messages. Cool... I guess I'll just wait and see then, and revisit this if we run into problems... Thanks Timo... -- Best regards, Charles From joshua at hybrid.pl Mon Mar 5 19:25:43 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Mon, 5 Mar 2012 18:25:43 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: <1330677951.2081.49.camel@innu> References: <1330677951.2081.49.camel@innu> Message-ID: On Fri, 2 Mar 2012, Timo Sirainen wrote: > On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: >> However, if we have everything redundant, why not have the same with SMTP >> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > If both servers randomly access users' mails, with NFS you'll have some > trouble, with OCFS2 probably less trouble. But in both cases you'll have > better performance and no problems if you use Dovecot director in both > servers (install both director and backend to both servers). > http://wiki2.dovecot.org/Director Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From tss at iki.fi Mon Mar 5 19:53:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:53:12 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: <1330677951.2081.49.camel@innu> Message-ID: <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> On 5.3.2012, at 19.25, Jacek Osiecki wrote: >>> However, if we have everything redundant, why not have the same with SMTP >>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > >> If both servers randomly access users' mails, with NFS you'll have some >> trouble, with OCFS2 probably less trouble. But in both cases you'll have >> better performance and no problems if you use Dovecot director in both >> servers (install both director and backend to both servers). >> http://wiki2.dovecot.org/Director > > Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... You can safely do that with director. Also the problem with NFS isn't locks, but caching. > Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). From sam at robots.org.uk Mon Mar 5 20:45:36 2012 From: sam at robots.org.uk (Sam Morris) Date: Mon, 05 Mar 2012 18:45:36 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users Message-ID: <1330973136.70967.33.camel@leela.office.red-redemption.com> The attached patch makes it possible for Kerberos principals to be associated with a password database entry by adding a new "k5principals" passdb setting. A client that successfully authenticates using GSSAPI will be able to log in as any user who has been associated with the client's Kerberos principal. This means that users can now use their Kerberos identities to access virtual mail accounts. The patch definitely needs review by someone familiar with Dovecot. It works for me on a small test installation using the passwd-file backend. Things that should probably be improved: 1. The list of authorized principals is stored in struct auth_request. I would prefer to store it in struct gssapi_auth_request, but auth-request.c does not know about structs that are specific to the different authentication plugins. This could be fixed in a more general way by adding a new function to struct mech_module to allow authplugins to read fields during passdb lookups. 2. The gssapi authplugin now does a credential lookup in order to trigger parsing of the k5principals setting. In order for this to work, auth_request_set_username is now called before mech_gssapi_userok. AFAICT the only impact of this is that messages logged by this function (and the functions it calls) will now use the name of the virtual account. 3. The credentials lookup triggers an info log message saying that credentials for GSSAPI were requested, "but we have only (e.g.) MD5-CRYPT". The authplugin doesn't actually want the credential, but I think that the only way the authplugin can trigger a passdb lookup is by requesting it. 4. The final part of the code in mech_gssapi_unwrap was moved to the callback that's triggered when the credentials lookup is complete. The code still needs access to the GSSAPI data, so the buffer pointer & length are now stored in struct gssapi_auth_request, making the inbuf parameter to the mech_gssapi_{sec_context,wrap,unwrap} functions superfluous. The parameters should be removed. 5. The k5principals list won't be processed on Solaris. The code added to the end of mech_gssapi_krb5_userok would have to be moved to a separate function and then be called from the Solaris code. 6. GCC tells me about assignment to incompatible pointer types in the code that iterates through gssapi_k5principals. I must be missing something. The patch is licensed under the MIT license. Please let me know what you think. -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_1.patch Type: text/x-patch Size: 6953 bytes Desc: not available URL: From tss at iki.fi Mon Mar 5 20:52:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 20:52:19 +0200 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: <1330973136.70967.33.camel@leela.office.red-redemption.com> References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: On 5.3.2012, at 20.45, Sam Morris wrote: > 3. The credentials lookup triggers an info log message saying that > credentials for GSSAPI were requested, "but we have only (e.g.) > MD5-CRYPT". The authplugin doesn't actually want the credential, > but I think that the only way the authplugin can trigger a > passdb lookup is by requesting it. I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". From campbell at cnpapers.com Mon Mar 5 21:30:30 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Mon, 05 Mar 2012 14:30:30 -0500 Subject: [Dovecot] Shared mboxes Message-ID: <4F551456.102@cnpapers.com> I've been looking at some documentation on shared mail accounts. But I'm getting mixed thoughts on how this can or should be done. I use mbox for all my pop and imap folders since I've converted from a uw-imap server. The first thing that makes me wonder about setup is that I've been told to not use maildir and mbox on the same machine, although I'm not really sure why since it seems this would work OK, but anyway, I'm guessing I should stick with mbox for the shared accounts. Secondly, I'm sure I'd need a namespace to use which ever format, so there's private, public, and shared types. Most of the stuff I'm reading seems to suggest "public" as a type instead of "shared". So what's shared for anyway? I want to use this shared account so that email can be sent to this account, and be shared by only a few people, but I'm reading where locks and such don't work with mbox, so in my mind, how do you avoid corruption and why not just make a normal account and let people hack away at the data? I've not even got to the questions in my mind about how to set up the account, but figured if I could get the above straight, I might be able to fuddle my way through it. Help would be truly appreciated. steve campbell From tom at talpey.com Tue Mar 6 00:06:20 2012 From: tom at talpey.com (Tom Talpey) Date: Mon, 05 Mar 2012 17:06:20 -0500 Subject: [Dovecot] POP3C storage backend Message-ID: <4F5538DC.4060802@talpey.com> I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I don't see anything in the 2.1 doc directory or in the wiki. Can this be used to synchronize dovecot with external pop servers? Doing away with my current fetchmail and lmtp solution for this would be quite interesting. Thanks for any pointers to configuring and using this, if so... From stan at hardwarefreak.com Tue Mar 6 01:16:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Mon, 05 Mar 2012 17:16:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F551456.102@cnpapers.com> References: <4F551456.102@cnpapers.com> Message-ID: <4F55495B.10609@hardwarefreak.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. But I'm > getting mixed thoughts on how this can or should be done. > > I use mbox for all my pop and imap folders since I've converted from a > uw-imap server. The first thing that makes me wonder about setup is that > I've been told to not use maildir and mbox on the same machine, although > I'm not really sure why since it seems this would work OK, but anyway, > I'm guessing I should stick with mbox for the shared accounts. > > Secondly, I'm sure I'd need a namespace to use which ever format, so > there's private, public, and shared types. Most of the stuff I'm reading > seems to suggest "public" as a type instead of "shared". So what's > shared for anyway? > > I want to use this shared account so that email can be sent to this > account, and be shared by only a few people, but I'm reading where locks > and such don't work with mbox, so in my mind, how do you avoid > corruption and why not just make a normal account and let people hack > away at the data? > > I've not even got to the questions in my mind about how to set up the > account, but figured if I could get the above straight, I might be able > to fuddle my way through it. > > Help would be truly appreciated. Start here: http://wiki.dovecot.org/SharedMailboxes -- Stan From kgc at corp.sonic.net Tue Mar 6 03:33:32 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Mon, 5 Mar 2012 17:33:32 -0800 Subject: [Dovecot] Master Users Message-ID: <20120306013332.GE16881@corp.sonic.net> I have a setup where I need to use a Master User account to login on behalf of users normally authed via PAM. Is there any existing mechanism that will allow master users to be wired down to specific ip address rather than having these very magic user/pass combos be valid from any random host? It would be totally acceptable to be able to say that master logins were only valid from a specific list of hosts rather than wiring specific master users to specific hosts. -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtam.home at gmail.com Tue Mar 6 04:01:43 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 5 Mar 2012 18:01:43 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On Sun, 4 Mar 2012, Timo Sirainen writes: > > I would like to run various doveadm commands that involves all (mail) users like > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > but any doveadm command that uses "-A" to iterate through all users will > > stop processing at the first account with UID > What userdb are you using? userdb passwd should already skip users that > aren't in the valid range. And what Dovecot version are you using? passwd-file under dovecot 2.0.16. > And one more thing: Does it really even stop there? Looking at the code > it's supposed to log an error and continue to next user. Note that it says > "Failed to iterate through SOME users". The wording did not escape my notice, which is why I suspect it's not doing what it was designed to do. This is my test: # Command doveadm mailbox list -A # Start of password file sysdaemon:*:500:500:System daemon:/:/dev/null ... and the rest ... # dovecot.conf ... first_valid_uid = 10000 first_valid_gid = 10000 ... In this situation, doveadm will exit immediately with an UID error message. If I change the UID>10000, it will produce the analogous GID error message. If I satify both UID and GID constraints, it will fail on the next daemon entry. If I move the sysdaemon entry all the way to the bottom of the pasword file, I get user1 saved-messages user1 sent-mail user1 postponed-msgs user1 temp user1 temp/temp user1 INBOX user2 sent-mail user2 101 user2 345 user2 ckf ... all user's mailbox with UID>10000, then ... doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop privileges: Mail access for users with UID 500 not permitted (see first_valid_uid in config file, uid from userdb lookup). doveadm(sysdaemon): Error: User init failed doveadm: Error: Failed to iterate through some users > Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? Sorry for this error mismatch -- I cut&pasted the wrong test output; however, the problem I witnessed applies to both UID and GID (if either constraint is not met, user iteration terminates). > I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f It looks like I'll be upgrading. Oh, I just spotted this in the ChangeLog -- maybe you are undoing this? (2010-10-21) * src/auth/auth-settings.c, src/auth/auth-settings.h, src/auth/userdb- passwd.c: auth: userdb passwd iteration now lists only users within first_valid_uid..last_valid_uid range. [745ef289b0ea] Joseph Tam From apm at one.com Tue Mar 6 09:17:32 2012 From: apm at one.com (Peter Mogensen) Date: Tue, 06 Mar 2012 08:17:32 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> Message-ID: <4F55BA0C.5090606@one.com> On 2012-03-05 16:36, Timo Sirainen wrote: >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > Neither behavior is wrong, just different. :) Ok... I were in doubt if I had missed something from the RFC. However... for testing, I tried to create "INBOX.INBOX" on dovecot. But then dovecot answers NO and complains that the folder already exists. Though it's still not on disk and dovecot still doesn't list it with "*". /Peter From frank at moltke28.B.Shuttle.DE Tue Mar 6 09:37:45 2012 From: frank at moltke28.B.Shuttle.DE (Frank Elsner) Date: Tue, 6 Mar 2012 08:37:45 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve Message-ID: (auto-added) Hello all, I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 installed and working. But I've problems to get the avelsieve plugin for squirrelmail working with dovecot. The "Message Filters" show up in "Options" of squirrelmail, but "Could not log on to timsieved daemon on your IMAP server ........." dovecot log shows: Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured Where to look for configuration error(s)? | root at seymour: /usr/local/dovecot/src<156> dovecot -n | # 2.1.1: /usr/local/dovecot/etc/dovecot/dovecot.conf | # OS: Linux 2.6.35.14-106.fc14.i686.PAE i686 Fedora release 14 (Laughlin) ext3 | default_vsz_limit = 512 M | disable_plaintext_auth = no | first_valid_uid = 200 | last_valid_uid = 65534 | listen = * | lmtp_save_to_detail_mailbox = yes | login_greeting = c64.shuttle.de - IMAPs Service (dovecot) ready. | login_log_format_elements = %u %r %c | mail_location = maildir:/var/spool/mail/%u | mail_log_prefix = "%Us(%u,%r): " | mail_plugin_dir = /usr/dovecot/lib/dovecot/ | mail_plugins = " notify quota" | managesieve_notify_capability = mailto | managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave | passdb { | args = dovecot | driver = pam | } | plugin { | mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append | mail_log_fields = uid box from subject msgid size flags | mail_log_group_events = yes | quota = maildir:User quota | quota_rule = *:storage=2G | quota_rule2 = Trash:storage=+100M | sieve = ~/.sieve | sieve_dir = ~/sieve | } | postmaster_address = postmaster at moltke28.b.shuttle.de | protocols = imap sieve | service anvil { | client_limit = 1027 | } | service auth { | unix_listener auth-client { | group = exim | mode = 0660 | user = exim | } | } | service imap-login { | inet_listener imap { | port = 143 | } | inet_listener imaps { | port = 993 | ssl = yes | } | process_limit = 512 | process_min_avail = 10 | } | service imap-postlogin { | executable = script-login /usr/dovecot/bin/imap-post-login | } | service imap { | executable = imap imap-postlogin | } | service lmtp { | inet_listener lmtp { | address = 0.0.0.0 | port = 24 | } | } | service managesieve-login { | inet_listener sieve { | port = 4190 | } | } | service pop3-login { | inet_listener pop3 { | port = 110 | } | inet_listener pop3s { | port = 995 | ssl = yes | } | } | service pop3 { | process_limit = 1024 | } | ssl_cert = References: <1330677951.2081.49.camel@innu> <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> Message-ID: <4F55C9DB.2070809@mobilia.it> Il 05/03/2012 18.53, Timo Sirainen ha scritto: > On 5.3.2012, at 19.25, Jacek Osiecki wrote: > >>>> However, if we have everything redundant, why not have the same with SMTP >>>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? >>> If both servers randomly access users' mails, with NFS you'll have some >>> trouble, with OCFS2 probably less trouble. But in both cases you'll have >>> better performance and no problems if you use Dovecot director in both >>> servers (install both director and backend to both servers). >>> http://wiki2.dovecot.org/Director >> Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... > You can safely do that with director. > > Also the problem with NFS isn't locks, but caching. After reading a little bit, it seems that Director does the job of a decent load balancer, but in the middle instead of in front of your servers, I've limited problems with NFS by using "sticky" connections with long timeouts in my load balancer, unless they're disconnected for days, they'll always end up going through the same server for POP3/IMAP conections. Doesn't work great for the SMTP/LDA part though. > >> Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? > Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). > In the rare case it does happen, NFS locking and concurrent_connections set to one has seemed to reduce my problems to a minimum.. I like the Director idea though, since it's content aware it isn't organizing connections based on port/IP, but on the the actual users, especially if it does so with the LDA, it seems like an excellent solution to collisions (I guess they're called this) .. I wish it had been a reality when I was building my servers. From stephan at rename-it.nl Tue Mar 6 10:40:44 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 06 Mar 2012 09:40:44 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve In-Reply-To: (auto-added) References: (auto-added) Message-ID: <4F55CD8C.20108@rename-it.nl> On 3/6/2012 8:37 AM, Frank Elsner wrote: > Hello all, > > I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 > installed and working. But I've problems to get the avelsieve plugin for > squirrelmail working with dovecot. > > The "Message Filters" show up in "Options" of squirrelmail, but > "Could not log on to timsieved daemon on your IMAP server ........." > > dovecot log shows: > > Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured You should try to capture traffic between client and server with ngrep, e.g. sudo ngrep -d lo port 4190 However, I've noticed that avelsieve uses STARTTLS even on localhost, so if you want to see anything intelligible, you will have to turn that off temporarily. As far as I know, there is also a means to instruct managesieve-login to write its traffic somewhere (a login 'rawlog'), but I can't find where it is documented right now. > | protocol lmtp { > | mail_plugins = " notify quota quota" > | } > | protocol lda { > | mail_plugins = " notify quota quota" > | } > | protocol imap { > | imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags > | imap_logout_format = [%i/%o] > | mail_max_userip_connections = 0 > | mail_plugins = " notify quota mail_log quota imap_quota listescape" > | } Why do you have duplicate "quota" entries here? Also, "sieve" plugin is missing from lmtp and lda. Still, ManageSieve should accept connections with this config. Regards, Stephan. From dovecot at arvoreen.net Tue Mar 6 13:29:13 2012 From: dovecot at arvoreen.net (Pol Bettinger) Date: Tue, 06 Mar 2012 12:29:13 +0100 Subject: [Dovecot] LDAP auth_bind fails Message-ID: <4F55F509.4000507@arvoreen.net> Hello, I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. Dovecot version 2.1.1 (I started with 2.1.0 and hoped 2.1.1 would fix it) I tried to play around with the base, pass_attrs,pass_filter to no avail but didn't succeed. Looking at a wireshark trace i only saw 7 packets and it seemed to me dovecot did only an anonymous bind. any help would appreciated Sincerely Pol Bettinger output of mail.log: Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5#011service=imap#011secured#011lip=192.168.16.27#011rip=192.168.16.20#011lport=993#011rport=51838 Mar 6 12:16:34 Dell dovecot: auth: Debug: client out: CONT#0112#011PDQ1NjgyMjE3NjYyMDk3NjkuMTMzMTAzMjU5NEBEZWxsPg== Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: CONT Mar 6 12:16:34 Dell dovecot: auth: Debug: password(arvi at arvoreen.net,192.168.16.20): passdb doesn't support credential lookups Mar 6 12:16:36 Dell dovecot: auth: Debug: client out: FAIL#0112#011user=arvi at arvoreen.net output of dovecot -n: # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic i686 Ubuntu 11.10 ext4 auth_debug = yes auth_default_realm = arvoreen.net auth_mechanisms = plain digest-md5 cram-md5 auth_verbose = yes base_dir = /var/run/dovecot/ mail_location = maildir:/var/mail/%d/%1n/%n:INDEX=/var/indexes/%d/%1n/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Drafts { auto = create special_use = \Drafts } mailbox Junk { auto = create special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = create special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap_pass.conf.ext driver = ldap } plugin { sieve = /var/sieve/%d/%1n/%n sieve_dir = /var/sieve/%d/%1n/%n } protocols = imap lmtp sieve service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = From tss at iki.fi Tue Mar 6 13:43:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 6 Mar 2012 13:43:26 +0200 Subject: [Dovecot] LDAP auth_bind fails In-Reply-To: <4F55F509.4000507@arvoreen.net> References: <4F55F509.4000507@arvoreen.net> Message-ID: On 6.3.2012, at 13.29, Pol Bettinger wrote: > I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. .. > Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5 CRAM-MD5 can't work with auth_bind. http://wiki2.dovecot.org/Authentication/Mechanisms#Non-plaintext_authentication From jernej.porenta at arnes.si Tue Mar 6 15:28:50 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 6 Mar 2012 14:28:50 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) Message-ID: Heya, We are expiriencing issues with dovecot 2.1.1 on Linux with weird filenames in home directory of username. We are using mbox IMAP folders, with no special changes (mail_location = mbox:~/:INBOX=%h/.mailbox). Mar 6 13:37:17 machine dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 6 13:37:17 machine dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb79450] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb794a6] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb78963] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c87ebd5] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c88c12c] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_next+0x1b4) [0x2ba41c88c494] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c885342] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_next+0x234) [0x2ba41c885604] -> dovecot/imap [0x40b2d1] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410427] -> dovecot/imap [0x40f4cd] -> dovecot/imap [0x40f582] -> dovecot/imap(client_handle_input+0x3f) [0x40f6cf] -> dovecot/imap(client_input+0x62) [0x410052] -> /opt/dovecot Mar 6 13:37:17 machine dovecot: imap(username): Fatal: master: service(imap): child 20873 killed with signal 6 (core dumps disabled) The bug is reproducible by using home folder structure available from: http://bit.ly/x8pTXS AFAIK, the problem lies in processing the file list of home folder, which can contain filenames that do not have proper UTF-8 encoding of filenames, which causes dovecot to crash. On the other hand, UTF-8 filenames created on the system by hand (using touch), are not displayed in IMAP LIST command (sample is included in the folder structure; single letter file). Cheers, Jernej From campbell at cnpapers.com Tue Mar 6 16:28:55 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 09:28:55 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F55495B.10609@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> Message-ID: <4F561F27.5000102@cnpapers.com> On 3/5/2012 6:16 PM, Stan Hoeppner wrote: > On 3/5/2012 1:30 PM, Steve Campbell wrote: >> I've been looking at some documentation on shared mail accounts. But I'm >> getting mixed thoughts on how this can or should be done. >> >> I use mbox for all my pop and imap folders since I've converted from a >> uw-imap server. The first thing that makes me wonder about setup is that >> I've been told to not use maildir and mbox on the same machine, although >> I'm not really sure why since it seems this would work OK, but anyway, >> I'm guessing I should stick with mbox for the shared accounts. >> >> Secondly, I'm sure I'd need a namespace to use which ever format, so >> there's private, public, and shared types. Most of the stuff I'm reading >> seems to suggest "public" as a type instead of "shared". So what's >> shared for anyway? >> >> I want to use this shared account so that email can be sent to this >> account, and be shared by only a few people, but I'm reading where locks >> and such don't work with mbox, so in my mind, how do you avoid >> corruption and why not just make a normal account and let people hack >> away at the data? >> >> I've not even got to the questions in my mind about how to set up the >> account, but figured if I could get the above straight, I might be able >> to fuddle my way through it. >> >> Help would be truly appreciated. > Start here: > http://wiki.dovecot.org/SharedMailboxes That's where most of my questions originated, but thanks for the reply. (Sorry for the first response - I sent it to the poster, not the list). Maybe I'm misunderstanding concepts here and I'm trying to use something I don't need to use. I'm really new to dovecot, and as I learn all the ins and outs, I'm finding a lot of this doesn't seem to be "turning on any light bulbs" until after I've played with it a while. What I've done in the past with the old imap server is to create an account (unix account), so the smtp server puts the mbox (what is referred to as the INbox) in /var/spool/mail. Users who needed to "share" this mailbox would be give the account user name and the password for this account and would add an Imap account to their mail client. This would sometimes cause locking problems or client corruption due to email removals mostly. This is basically a normal, non-shared account. Now that I've moved to dovecot on a new, updated server, I'd like to use the facilities of dovecot for the truly shared accounts. I'm not sure if I need to create the account like before, but seems like I'd have to in order to get the smtp server to deliver new email to /var/spool/mail/%u. As I see it, I've got to create a namespace for shared accounts and configure this on the multiple-user's clients so that when they access the Inbox and imap files under /home/%u/mail, they don't butt heads, so they're some locking involved. I could use acls for this, but don't have to according to the documentation. I can grant permissions to each user that is included in the acl, and I can create dovecot "groups" to use as a basis for this permission. I'm hoping this is pretty much the way it's done, and I want to keep with mbox format for all files and folders. I'm also hoping that this is the way it's supposed to be used, but I get conflicting ideas about what the documentation is really telling me. Anyway, I'll play with this and see where I get. I've still not found out where to create these dovecot "groups" other than it seems to use a userdb file somewhere. Thanks for the help so far steve > From khoroshyy at gmail.com Tue Mar 6 18:58:55 2012 From: khoroshyy at gmail.com (Khoroshyy Petro) Date: Tue, 6 Mar 2012 17:58:55 +0100 Subject: [Dovecot] Dovecot saves mails in "wrong" folder. Message-ID: Hi all I have installed dovecot 1.2.15 and try to use it together with offlineimap and gnus. my problem is that it saves emails into /var/mail/petro instead of ~/Maildir Thanks. Petro. This is my .dovecot.conf default_mail_env = maildir:%h/Maildir And this is my .offlineimaprc [general] accounts = Gmail maxsyncaccounts = 1 [Account Gmail] localrepository = Local remoterepository = Remote [Repository Local] type = IMAP remotehost = localhost port = 143 remoteuser = petro [Repository Remote] type = IMAP remotehost = imap.gmail.com remoteuser = myname at gmail.com ssl = yes maxconnections = 1 realdelete = no folderfilter = lambda foldername: foldername in ['INBOX'] -- From kgc at corp.sonic.net Tue Mar 6 19:33:08 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Tue, 06 Mar 2012 09:33:08 -0800 Subject: [Dovecot] Master Users In-Reply-To: <20120306013332.GE16881@corp.sonic.net> References: <20120306013332.GE16881@corp.sonic.net> Message-ID: <4F564A54.9050400@corp.sonic.net> On 03/05/12 17:33, Kelsey Cummings wrote: > I have a setup where I need to use a Master User account to login on > behalf of users normally authed via PAM. Is there any existing mechanism > that will allow master users to be wired down to specific ip address rather Ah, found it. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/AllowNets -K From sam at robots.org.uk Tue Mar 6 20:12:01 2012 From: sam at robots.org.uk (Sam Morris) Date: Tue, 06 Mar 2012 18:12:01 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: <1331057521.84875.2.camel@leela.office.red-redemption.com> On Mon, 2012-03-05 at 20:52 +0200, Timo Sirainen wrote: > On 5.3.2012, at 20.45, Sam Morris wrote: > > > 3. The credentials lookup triggers an info log message saying that > > credentials for GSSAPI were requested, "but we have only (e.g.) > > MD5-CRYPT". The authplugin doesn't actually want the credential, > > but I think that the only way the authplugin can trigger a > > passdb lookup is by requesting it. > > I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". Thanks for pointing that out. Here's a newer version of the patch with that change. I also realised that the gss_buffer is not required in the code that runs once the passdb lookup is complete, so I removed the code that stashes it in struct gssapi_auth_request. Regards, -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_2.patch Type: text/x-patch Size: 6020 bytes Desc: not available URL: From stan at hardwarefreak.com Tue Mar 6 22:17:22 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 06 Mar 2012 14:17:22 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F561F27.5000102@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> Message-ID: <4F5670D2.9090004@hardwarefreak.com> On 3/6/2012 8:28 AM, Steve Campbell wrote: >> http://wiki.dovecot.org/SharedMailboxes > That's where most of my questions originated, but thanks for the reply. Steve, all the information you need is behind that link. > Maybe I'm misunderstanding concepts here Very possibly. > What I've done in the past with the old imap server is to create an > account (unix account), so the smtp server puts the mbox (what is > referred to as the INbox) in /var/spool/mail. Users who needed to > "share" this mailbox would be give the account user name and the > password for this account and would add an Imap account to their mail > client. This would sometimes cause locking problems or client corruption > due to email removals mostly. This is basically a normal, non-shared > account. Locking problems with multiple users hitting mbox files is unavoidable. The same is true when a single user hits an mbox from multiple client devices simultaneously--PC, smart phone, tablet, etc. Which is why you do not want to use mbox file format for shared mailboxes, but maildir instead, because each email is a separate file. Please note, from the link I provided: ********************************************************************** Maildir: Per-user \Seen flag With Maildir a dovecot-shared file controls if the \Seen flags are shared or private. The file must be created separately inside each Maildir, although if the file already exists in the Maildir root it's automatically copied for newly created mailboxes. If dovecot-shared file doesn't exist in Maildir, the \Seen flags are shared. If it exists, the \Seen flag state is stored only in the user's index files. By making each user have their own private index files, you can make the \Seen flag private for the users. ********************************************************************** Simple concept above: each user of the shared mailbox sees "new" mail. One user accessing new mail and marking it as read doesn't mark that message as read for other shared users. You can not do this with mbox file format, only maildir. ********************************************************************** Maildir: Keyword sharing Make sure you don't try to use per-user CONTROL directory. Otherwise dovecot-keywords file doesn't get shared and keyword mapping breaks. Other mailbox formats Currently you can't have any per-user flags with other mailbox formats than Maildir. ********************************************************************** -- Stan From campbell at cnpapers.com Tue Mar 6 23:01:08 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 16:01:08 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F5670D2.9090004@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> Message-ID: <4F567B14.3030908@cnpapers.com> On 3/6/2012 3:17 PM, Stan Hoeppner wrote: > On 3/6/2012 8:28 AM, Steve Campbell wrote: > >>> http://wiki.dovecot.org/SharedMailboxes >> That's where most of my questions originated, but thanks for the reply. > Steve, all the information you need is behind that link. I've gone over that set of links on that page a dozen times. Perhaps I'm trying to put a square peg in a round hole by using mbox, but they keep providing information on it, so I guess I was just pounding away. But then there's that "don't use maildir and mbox together". All of the accounts on this server are carry-overs from the UW-IMAP server, so perhaps I should have converted those to maildir. Seems as though it's OK when they don't apply to the same type namespace. > >> Maybe I'm misunderstanding concepts here > Very possibly. > >> What I've done in the past with the old imap server is to create an >> account (unix account), so the smtp server puts the mbox (what is >> referred to as the INbox) in /var/spool/mail. Users who needed to >> "share" this mailbox would be give the account user name and the >> password for this account and would add an Imap account to their mail >> client. This would sometimes cause locking problems or client corruption >> due to email removals mostly. This is basically a normal, non-shared >> account. > Locking problems with multiple users hitting mbox files is unavoidable. > The same is true when a single user hits an mbox from multiple client > devices simultaneously--PC, smart phone, tablet, etc. Which is why you > do not want to use mbox file format for shared mailboxes, but maildir > instead, because each email is a separate file. Please note, from the > link I provided: I've experienced that type of locked mailbox before on the old server. Users insist on accessing their email account as a pop account on their desktop with the "check for new mail every so many minutes" turned on and still keep their smartphones on while accessing it as an imap account so they can still download the files to their desktop when they return. > > ********************************************************************** > Maildir: Per-user \Seen flag > > With Maildir a dovecot-shared file controls if the \Seen flags are > shared or private. The file must be created separately inside each > Maildir, although if the file already exists in the Maildir root it's > automatically copied for newly created mailboxes. If dovecot-shared file > doesn't exist in Maildir, the \Seen flags are shared. If it exists, the > \Seen flag state is stored only in the user's index files. By making > each user have their own private index files, you can make the \Seen > flag private for the users. > ********************************************************************** > > > Simple concept above: each user of the shared mailbox sees "new" mail. > One user accessing new mail and marking it as read doesn't mark that > message as read for other shared users. You can not do this with mbox > file format, only maildir. > > > ********************************************************************** > Maildir: Keyword sharing > > Make sure you don't try to use per-user CONTROL directory. Otherwise > dovecot-keywords file doesn't get shared and keyword mapping breaks. > > Other mailbox formats > > Currently you can't have any per-user flags with other mailbox formats > than Maildir. > ********************************************************************** So just to clarify, is it OK to have a maildir account setup on this server for these shared/imap access only accounts along with the mbox accounts already on there? Thanks for the patience and help steve From sdavies at sdc.com.au Wed Mar 7 01:00:50 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:30:50 +1030 Subject: [Dovecot] Log sync errors (again) Message-ID: <201203070930.50847.sdavies@sdc.com.au> As suggested earlier, I deleted all .imap directories and the log sync errors stopped - for a while. They have now returned. It seems to happen for every mailbox that gets accessed. Dovecot version 2.1.1 with pidgeonhole 3.0.0 on Mandriva Linux. Could this interfere with sieve filters? Several users have filters but none of them seem to do anything. Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=38708 for /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index: Extension header update points outside header size Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=41576 for /home/john/Mail/INBOX/.imap/Zerna/dovecot.index: Extension header update points outside header size Cheers and thanks, Stephen -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From sdavies at sdc.com.au Wed Mar 7 01:07:06 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:37:06 +1030 Subject: [Dovecot] Fscking warnings Message-ID: <201203070937.06545.sdavies@sdc.com.au> Google tells me that these "should go away" but they don't. Seems to happen continuously while a user is viewing email. Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Archive/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Davies/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/FieldNET/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Invoices Out/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Lawrence and Hanson/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Call/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Reset/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/River Murray/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Soil Moisture Alert/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Management Alarm/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Usage/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Zerna/dovecot.index -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From jk at jkart.de Wed Mar 7 01:19:10 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:19:10 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? Message-ID: <4F569B6E.1080905@jkart.de> Hello, you knows, that http://xi.rename-it.nl is down? -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Die Oper ist eine h?bsche Unterhaltung, die noch besser w?re, wenn nicht dabei gesungen w?rde. (Claude Debussy) From stephan at rename-it.nl Wed Mar 7 01:33:09 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 07 Mar 2012 00:33:09 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569B6E.1080905@jkart.de> References: <4F569B6E.1080905@jkart.de> Message-ID: <4F569EB5.7030204@rename-it.nl> On 3/7/2012 12:19 AM, Jim Knuth wrote: > Hello, > > you knows, that http://xi.rename-it.nl is down? > Yep, and back. Regards, Stephan. From jk at jkart.de Wed Mar 7 01:36:42 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:36:42 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569EB5.7030204@rename-it.nl> References: <4F569B6E.1080905@jkart.de> <4F569EB5.7030204@rename-it.nl> Message-ID: <4F569F8A.1090200@jkart.de> am 07.03.12 00:33 schrieb Stephan Bosch : > On 3/7/2012 12:19 AM, Jim Knuth wrote: >> Hello, >> >> you knows, that http://xi.rename-it.nl is down? >> > > Yep, and back. > > Regards, > > Stephan. wow. Thank you -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Dilettanten erkennt man an der Plumpheit ihrer Komplimente. Der routinierte Verf?hrer riskiert Kritik. (Cath?rine Deneuve) From 24x7server at 24x7server.net Wed Mar 7 04:22:23 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Wed, 7 Mar 2012 07:52:23 +0530 (Asi) Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file Message-ID: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> hi i am using qmailtoaster with dovecot version 2 mailbox format is maildir i have a domain with around 5000 users which are distributed over 2 servers webmail (squirrelmail) runs using dovecot v2 is being used from server number one server number 2 had all the data stored in it and also has pop and smtp running from it. i am not using dovecot for pop as yet on the server with dovecot i get such errors in the log file access to data on server number 2 is via nfs on server number 1 i get errors as such Error: fcntl(read-lock) locking failed for file Input/output error squirrelmail gives error imap connection closed and i am not able to login so i set the parameters as such in the dovecot conf file and the error stopped mmap_disable=yes dotlock_use_excl = yes lock_method = dotlock can somebody please advise me if the above is correct ? or is it preferred to use fcntl with lockd (note that my mailbox is maildir format) thanks very much for your help rajesh From jd.beaubien at gmail.com Wed Mar 7 05:19:05 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Tue, 6 Mar 2012 22:19:05 -0500 Subject: [Dovecot] mdbox + gzip and rsync Message-ID: Hi, After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip): "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation." How will the mailbox files (m.X) files be modified when I move or delete emails using mdbox+gzip. Will the resulting gzipped mdbox files be rsync-able or will they need a full re-upload? If I plan on using rsync for backups, am I better off not using the gzip feature (if i can spare the extra storage)??? Thanks, -JD From CMarcus at Media-Brokers.com Wed Mar 7 15:32:32 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 08:32:32 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? Message-ID: <4F576370.8040706@Media-Brokers.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. > But I'm getting mixed thoughts on how this can or should be done. This brings up a question I have been meaning to ask. One thing I want to do on my new converted system is to implement shared mail. There will be two different scenarios - users sharing 'folders', which looks to be fairly simple using virtual ACL files - but for the other scenario, I'm not sure about a specific requirement we will have... I want to give multiple people shared access to some actual accounts with all of the special use folders, with the following requirements: 1. They can all read/reply to new messages as they come in, 2. They use shared \seen, \replied and \forwarded flags, so once someone else has read/dealt with a message, the others see that, 3. When they reply to/forward a message, the Sent message gets saved to that accounts 'Sent' folder, 4. They can *move* messages to other folders in that account (ie, 'file' them), and last (this is the tricky part) 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. These emails deal with financial transactions (AP and AR issues) and Faxes, thus the requirement to not be able to delete them. Can this be accomplished with the current state of things? Or would this require some coding? If the latter, could it be done as a plug-in, or would it require changes to the core code? Thanks, -- Best regards, Charles From lists at wildgooses.com Wed Mar 7 18:39:13 2012 From: lists at wildgooses.com (Ed W) Date: Wed, 07 Mar 2012 16:39:13 +0000 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F578F31.3000303@wildgooses.com> > I want to give multiple people shared access to some actual accounts > with all of the special use folders, with the following requirements: I have done this (unsatisfactorarily) by making it a normal mail account with normal login credentials. Add it like any other mail account. It then satisfies all your requirements, although: behind a nat, on thunderbird and with condstore, I sometimes see read/unread get out of sync... Believed to be a thunderbird bug, but unsure. Easy to resync > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) Interested to hear proper answers... Ed W From wgillespie at es2eng.com Wed Mar 7 20:04:44 2012 From: wgillespie at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 11:04:44 -0700 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F57A33C.3050808@es2eng.com> On 3/7/2012 6:32 AM, Charles Marcus wrote: > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. If you are using ACLs, just don't give them the delete permission? But I guess now that I am thinking about it as I write, you did want them to be able to move the messages (which is really a copy + delete). So... maybe not. From e-frog at gmx.de Wed Mar 7 20:17:36 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 07 Mar 2012 19:17:36 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command Message-ID: <4F57A640.1030202@gmx.de> Hello Timo, There seems to be something broken in v2.1 latest hg version: # 2.1.1 (94de7605f50f) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL * OK Namespace completed. Please note that the "OK Namespace completed." is send untagged. It worked on below version: # 2.1.1 (315f0d8cc2b2) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL 1 OK Namespace completed. Thanks, e-frog From CMarcus at Media-Brokers.com Wed Mar 7 21:03:30 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 14:03:30 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F57A33C.3050808@es2eng.com> References: <4F576370.8040706@Media-Brokers.com> <4F57A33C.3050808@es2eng.com> Message-ID: <4F57B102.80400@Media-Brokers.com> On 2012-03-07 1:04 PM, Willie Gillespie wrote: > On 3/7/2012 6:32 AM, Charles Marcus wrote: >> 5. No one other than a designated user or users (Master User(s)? Users >> in a specified Group?) can delete any messages in this account, in any >> of the folders. > If you are using ACLs, just don't give them the delete permission? But I > guess now that I am thinking about it as I write, you did want them to > be able to move the messages (which is really a copy + delete). > > So... maybe not. Right... although my understanding is that dovecot does indeed use mv (at least on linux) to do moves when using maildir, so maybe there is a way... I'll wait and see what Timo says about this... no hurry, as I'm still in the design stage, this is just how I'd *like* it to work, but if it won't/can't, I'll figure something else out. Thanks for the replies so far... -- Best regards, Charles From micah at riseup.net Wed Mar 7 21:43:49 2012 From: micah at riseup.net (Micah Anderson) Date: Wed, 07 Mar 2012 14:43:49 -0500 Subject: [Dovecot] dot named folders Message-ID: <87aa3s2o3u.fsf@algae.riseup.net> When a user makes a folder called 'x.y' it actually creates a folder called 'x' with a folder called 'y' inside, rather than a folder called 'x.y'. I'm guessing this has to do with an internal folder separator namespace configuration, but I'm a bit confused by how this works. I'm using 2.0.15 with mdbox and this is what I have configured for my namespaces: namespace { separator = . prefix = inbox = yes } namespace { separator = . prefix = INBOX. inbox = no hidden = yes list = no } I migrated from courier maildirs, so perhaps I no longer need some of these now that the conversion is finished? thanks for any suggestions, I've got my head mixed up on this issue, micah -- From wgillespie+dovecot at es2eng.com Wed Mar 7 22:41:25 2012 From: wgillespie+dovecot at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 13:41:25 -0700 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <4F57C7F5.4030803@es2eng.com> On 03/07/2012 12:43 PM, Micah Anderson wrote: > > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. Correct. Similar to how in Linux, I could create a folder mkdir test1/test2 It will create test2 inside of test1. The difference being that IMAP doesn't necessarily need the parent mailbox to exist, where Linux would throw an error if test1/ didn't exist first. So basically, as far as I know, you can't have a folder with a "." in the name with the namespaces you have set up. From stan at hardwarefreak.com Wed Mar 7 22:47:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 14:47:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F567B14.3030908@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> Message-ID: <4F57C96F.7090602@hardwarefreak.com> On 3/6/2012 3:01 PM, Steve Campbell wrote: > I've experienced that type of locked mailbox before on the old server. > Users insist on accessing their email account as a pop account on their > desktop with the "check for new mail every so many minutes" turned on > and still keep their smartphones on while accessing it as an imap > account so they can still download the files to their desktop when they > return. Using IMAP on the phone and POP on the PC doesn't make any sense. Is there a (valid) reason why these people insist on this phone/IMAP and PC/POP setup? This seems seriously counter intuitive/productive. > So just to clarify, is it OK to have a maildir account setup on this > server for these shared/imap access only accounts along with the mbox > accounts already on there? Yes. With Dovecot it is possible to specify mail_location on a per user basis: http://wiki.dovecot.org/MailLocation You can even do a split mailbox type setup per user using multiple namespaces, for example specifying that INBOX use mbox with all other mail being stored in maildir format: http://wiki.dovecot.org/Namespaces > Thanks for the patience and help Sure thing. -- Stan From stan at hardwarefreak.com Wed Mar 7 23:03:35 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 15:03:35 -0600 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: <4F57CD27.3000207@hardwarefreak.com> On 3/6/2012 5:07 PM, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. Is this thread what "Google tells you"? http://dovecot.org/list/dovecot/2010-October/053909.html Timo is the creator of Dovecot, if you didn't know. So you can take his words for gospel. Also note his last statement in that thread: "The next time you could do it with dsync to avoid these kind of problems." It would seem you omitted a very important detail from your problem report, which is that you recently performed a migration. Please don't omit such critical details in future requests for help. Provide as much relevant detail as possible. This speeds the process up for everyone, and avoids guesswork on our part. -- Stan From M.Roos at roosit.eu Thu Mar 8 01:26:55 2012 From: M.Roos at roosit.eu (Marc) Date: Thu, 8 Mar 2012 00:26:55 +0100 Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap Message-ID: Anybody also experiencing that imap processes are kept running/open by mac osx mail.app, so eventually users are getting to the mail_max_userip_connections limit? Outlook / other clients seem to run fine. Thanks, Marc From jd.beaubien at gmail.com Thu Mar 8 03:30:26 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Wed, 7 Mar 2012 20:30:26 -0500 Subject: [Dovecot] Single instance storage Message-ID: I have read most of the doc on the dovecot website, and couldn't find any info on the single instance storage feature, so I'm posting my questions here. - Are these 3 parameters the only one necessary for single instance storage? I cannot find any doc on this feature on the website; is there anything specific I need to know about them? (the last one isn't exactly self-explanatory). - mail_attachment_dir = /srv/vmail/attachments - mail_attachment_hash = %{sha256} - mail_cache_min_mail_count = 2 - Is this feature ready for production? Thanks, -JD From schut at sarvision.nl Thu Mar 8 11:56:35 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 10:56:35 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. Message-ID: Hi, I'm currently migrating our old (colocated) mail server (running a [terribly outdated, I know] dovecot 1.1.11) to a new VPS (virtual private server). The old server was running gentoo linux (which is mainly the culprit of the old dovecot version: gentoo was too much trouble to keep updating); the new server will run debian (stable: 6). Debian currently has dovecot 1.2.15 in its repositories; not that much newer... I read in the docs about the auto-generated-from-hg debian dovecot packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what version to use... OK, 2.2 is development, which leaves the choice to: 1.2.15; 2.0.x, or 2.1.x. I would appreciate any consideration or thoughts on what version to choose. On a related note, there is the possibility to switch from maildir to dbox. I did not really find much pros or cons, except from performance and standards-compliance (ability to use e.g. mutt on the server itself). Any thoughts? About the server: we're just a small company. Think about 15 accounts, normal mail traffic, sometimes relatively large attachments (20mb+). Some accounts have many folders; some accounts are very large (5Gb+). Storage is on ext3, raid10. Performance has never been an issue; reliability and ease of maintenance is more important. Thanks, Vincent Schut. From trashcan at odo.in-berlin.de Thu Mar 8 11:59:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 10:59:37 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> Message-ID: HI -- On 05.03.2012 10:56, Timo Sirainen wrote: > On 4.3.2012, at 13.54, Timo Sirainen wrote: >> On 4.3.2012, at 13.41, Michael Grimm wrote: >>>> By "undeletable" do you mean you have mails that always come back >>>> after expunging them? >>> >>> Yes. Deleting by the client will return them after the next dsync >>> run. > > Luckily this just started happening to me as well. After some > debugging I found and fixed the problem: > > http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 I can confirm, that you fixed that issue successfully. Thanks and regards, Michael From trashcan at odo.in-berlin.de Thu Mar 8 12:26:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 11:26:56 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Hi -- On 04.03.2012 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. > Everything isn't finished yet, but it appears to work and I've > enabled > it for my @dovecot.fi mails. I did give it a try starting some days ago, and I can confirm that you are right, dsync replication can be used, but there are some issues, see below. Let me start with replicator's configuration ... > Below is a configuration for virtual user setup. [...] > service doveadm { > # if you're using a single virtual user, set this to > # start ssh as vmail (not root) > user = vmail > } ... that led to the following complaints at start-up: | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up | dovecot: doveadm: Error: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied | [...] | (repeatedly, presumably for the number of users in userdb?) Therefore, I modified dsync_remote_cmd ... > dsync_remote_cmd = ssh -p 1234 -l vmail %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} ... and used an empty 'service doveadm { }' instead. That worked, but I would love to run doveadm as vmail user (security), though. How should I do that without running into the error messages above? Now some observations regarding replicator: 1) I see a lot of error messages whenever replicator is in action like (although everything is being synced correctly): | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: save: box=INBOX, uid=27, msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld (admin), flags=() | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: flag_change: box=TEST, uid=27568, msgid=<20120307144810.6360A74F013 at example.tld>, size=435, from=test at example.tld, flags=(\Seen) JFTR: I do have mail_log plugin activated. Some testing results: 1) I ran a test by sending locally produced mails every other minute on both servers simultaneously. That test ran for ~5 hours. All mails became synced correctly, and no losses were observable, but some duplicates. 2) I did send 100 small test mails from a distant server to my mailservers (mx1 and mx2): a) replicator and dsync deactivated: received 100 distinct mails (57 at mx1, 43 at mx2). b) now, replicator active: 172 mails (100 distinct, a lot of duplicates (up to 8 incarnations of the very same mail). Ok, 2b) is a rather 'mailbomb-like' scenario, but it worries me a bit: One of my users is receiving mails from a mailing list that sends individual mails batch-wise ... 3) replicator active: 1000 mails sent ended in 4523 mails at every server. Well, that was a mailbomb :-) 4) replicator active: 100 (and even 1000) locally produced mails at one server only: all 100 (and 1000 mails) became synced, prefectly well, without duplicates. 5) replicator active: 100 locally produced mails at both servers simultaneously: 341 mails, thus a lot of multiple incarnations. (This test differed from 1) because all mails were sent in one batch.) Final note to these tests: It doesn't matter whether sieve with redirecting, or sieve with redirecting and copying, or no sieve at all has been involved. It seems to me, that whenever a larger number of mails arrive on both servers simultaneously, the replicator gets into trouble [1]. I am unsure if one can expect that a replicator should deal with such stress, though. Or? R?sum?: The overall performance of replicator is very good from my point of view for my conditions (handful users, average workload of roughly 1000 mails a day). Thank you for replicator and regards, Michael [1] JFTR: I did similar tests in the past with dsync running from cron every other minute with similar results. From tss at iki.fi Thu Mar 8 13:35:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 08 Mar 2012 13:35:34 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> References: <4F53479E.40703@iki.fi> <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Message-ID: <1331206534.2081.101.camel@innu> On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > Let me start with replicator's configuration ... > > > Below is a configuration for virtual user setup. > [...] > > service doveadm { > > # if you're using a single virtual user, set this to > > # start ssh as vmail (not root) > > user = vmail > > } > > ... that led to the following complaints at start-up: > > | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up > | dovecot: doveadm: Error: Error reading configuration: > net_connect_unix(/var/run/dovecot/config) failed: Permission denied > | [...] > | (repeatedly, presumably for the number of users in userdb?) You can do for example: service config { unix_listener config { user = vmail } } > Now some observations regarding replicator: > > 1) I see a lot of error messages whenever replicator is in action > like (although everything is being synced correctly): > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: save: box=INBOX, uid=27, > msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld > (admin), flags=() > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: flag_change: box=TEST, uid=27568, > msgid=<20120307144810.6360A74F013 at example.tld>, size=435, > from=test at example.tld, flags=(\Seen) > > JFTR: I do have mail_log plugin activated. Hmm. Right. I guess all the logging should go to the log files instead of via the ssh pipe. Of course that would also require that dsync has write access to your log files. > It seems to me, that whenever a larger number of mails arrive on both > servers simultaneously, > the replicator gets into trouble [1]. I am unsure if one can expect > that a replicator should > deal with such stress, though. Or? Were these mails delivered via LMTP or dovecot-lda? The locks should prevent duplicates I think, so there's something still going wrong. From Leo.Baltus at omroep.nl Thu Mar 8 13:56:41 2012 From: Leo.Baltus at omroep.nl (Leo Baltus) Date: Thu, 8 Mar 2012 12:56:41 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <4F459344.5020407@rename-it.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> Message-ID: <20120308115641.GB5700@omroep.nl> Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: > On 2/22/2012 12:15 AM, Adam Szpakowski wrote: > >On 22.02.2012 00:09, Timo Sirainen wrote: > >>Well, it would be possible to build a doveadm script that > >>deletes the duplicates after delivery, but currently there's no > >>implementation to avoid delivering duplicate Message-IDs in the > >>first place. > >> > >>I don't really like such a Message-ID-based deduplication > >>feature enabled by default, but something like this could be > >>nice: > >> > >>fileinto :copy :x-deduplicate "boss"; > >> > >>Anyway, probably not going to be implemented anytime soon. > >Maybe there is a way to use a procmail with something like this: > > > >:0 Wh: msgid.lock > >| formail -D 8192 .msgid.cache > > > >But is there a safe way to use it together with sieve? Using > >Pigeonhole Sieve Pipe Plugin? > > > > There are a few options: > > * You can use Procmail as primary delivery agent and invoke > dovecot-lda/sieve from within Procmail once Procmail has determined > that it is not a duplicate. > > * Invoke procmail from Sieve using the pipe extension (i.e. the > other way around). This has the disadvantage that Procmail will > have to take care of final delivery, meaning the Dovecot indexes are > not updated. > > * For Pigeonhole v0.3 there is the possibility to "filter" the > message through Procmail using the sieve_extprograms plugin, but I > haven't actually tested something like that. > > * I've just created an alternative that implements something similar > to the Procmail code you posted above, but from within Sieve itself. > It is a custom language extension called vnd.dovecot.duplicate and > it adds the "duplicate" test. This test keeps track of which > Message-IDs it has seen before in earlier deliveries and yields a > true result if the message was seen before, e.g.: > > require "vnd.dovecot.duplicate"; > > if duplicate { > discard; > } > > Read the specification for details ("name" argument is not yet implemented): > > http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate/raw-file/4b1dbda4d3fc/doc/rfc/spec-bosch-sieve-duplicate.txt > > The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate > > This plugin is only a few hours old, experimental, and largely > untested, so test it thoroughly before considering to use this. Read > the INSTALL file for compile and installation instructions. > > Comments are welcome. > I did some very basic testing and it seems to work fine. The example in spec-bosch-sieve-duplicate.txt however says: if duplicate { fileinto :create "Trash/Duplicate"; } This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' So this leads to: failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name I am not sure if this a bug or not, I suppose you know the rfc's better than I do, is the sieve language supposed to be agnostic of the internals of the storage-engine (dovecot)? -- Leo Baltus, internetbeheerder /\ NPO ICT Internet Services /NPO/\ Sumatralaan 45, 1217 GP Hilversum, Filmcentrum, west \ /\/ beheer at omroep.nl, 035-6773555 \/ From CMarcus at Media-Brokers.com Thu Mar 8 14:03:05 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 07:03:05 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F589FF9.7080608@Media-Brokers.com> On 2012-03-08 4:56 AM, Vincent Schut wrote: > The old server was running gentoo linux (which is mainly the culprit of > the old dovecot version: gentoo was too much trouble to keep updating); Please stop with the FUD... I've been running gentoo for 8+ years, and it is a *breeze* to keep updated, *especially* long term (since it is a 'rolling release' type of distro)... Yes, it actually does require some minimum amount of attention from the admin, like, say, once per week or once per month updates - buy so should *any* system... and yes, it does require a little more willingness to learn and 'get your hands dirty' (especially for the installation), but it is well worth it. Oh - and Portage rocks... :) -- Best regards, Charles From stephan at rename-it.nl Thu Mar 8 14:05:37 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 08 Mar 2012 13:05:37 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <20120308115641.GB5700@omroep.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> <20120308115641.GB5700@omroep.nl> Message-ID: <4F58A091.7090704@rename-it.nl> On 3/8/2012 12:56 PM, Leo Baltus wrote: > Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: >> The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate >> >> This plugin is only a few hours old, experimental, and largely >> untested, so test it thoroughly before considering to use this. Read >> the INSTALL file for compile and installation instructions. >> >> Comments are welcome. > I did some very basic testing and it seems to work fine. > > The example in spec-bosch-sieve-duplicate.txt however says: > > if duplicate { > fileinto :create "Trash/Duplicate"; > } > > This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' > > So this leads to: > failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name > > I am not sure if this a bug or not, I suppose you know the rfc's better > than I do, is the sieve language supposed to be agnostic of the > internals of the storage-engine (dovecot)? For Sieve, the mailbox name is pretty much opaque. Usually, it matches what is used through IMAP. http://tools.ietf.org/html/rfc5228#section-4.1 So, in your case, just use "Trash.Duplicate" instead. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 8 14:19:28 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 13:19:28 +0100 Subject: [Dovecot] dsync replication available for testing Message-ID: Hi -- On 08.03.2012 12:35, Timo Sirainen wrote: > On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > You can do for example: > > service config { > unix_listener config { > user = vmail > } > } I will try that later. >> It seems to me, that whenever a larger number of mails arrive on >> both >> servers simultaneously, the replicator gets into trouble [1]. I am >> unsure if one can expect that a replicator should deal with such >> stress, >> though. Or? > > Were these mails delivered via LMTP or dovecot-lda? LMTP > The locks should prevent duplicates I think, so there's something > still > going wrong. Just to be sure that I didn't misunderstand your proposed configuration: @mx1: plugin { mail_replica = remote:vmail at mx2.example.tld } @mx2: plugin { mail_replica = remote:vmail at mx1.example.tld } I do need to define one mail_replica plugin at each server pointing to the other one, correct? Regards, Michael From as at 3a.pl Thu Mar 8 14:24:13 2012 From: as at 3a.pl (Adam Szpakowski) Date: Thu, 08 Mar 2012 13:24:13 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F58A4ED.8070704@3a.pl> On 08.03.2012 10:56, Vincent Schut wrote: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... > I read in the docs about the auto-generated-from-hg debian dovecot > packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what > version to use... OK, 2.2 is development, which leaves the choice to: > 1.2.15; 2.0.x, or 2.1.x. > > I would appreciate any consideration or thoughts on what version to > choose. On several production machines we are using dovecot from debian testing repos, so 2.0.x. It's working stable for us and is quite easy to maintain. Please be careful and very selectively install packages from testing. If possible, the package dependences should be installed from stable/security. -- Adam Szpakowski From schut at sarvision.nl Thu Mar 8 15:53:54 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 14:53:54 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F589FF9.7080608@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: On 03/08/2012 01:03 PM, Charles Marcus wrote: > On 2012-03-08 4:56 AM, Vincent Schut wrote: >> The old server was running gentoo linux (which is mainly the culprit of >> the old dovecot version: gentoo was too much trouble to keep updating); > > Please stop with the FUD... > > I've been running gentoo for 8+ years, and it is a *breeze* to keep > updated, *especially* long term (since it is a 'rolling release' type of > distro)... Right. I should've known I shouln't mention anyone's favourite distro... :-) Hey, listen, sorry I offended you... its really nothing I have against gentoo, I'm sorry it might have sounded like that. It's just that I appeared not to have the time and energy to do regular updates, and when I tried to update something some months later, I had problems which I had no time and energy to start solving. Thus I decided a rolling distro was no good combination for my server and me. Which is why I will switch to a less rolling distro. That's really all there is to say about. I do still have a rolling distro which-will-not-be-named on my desktop, which I can and do update often and easy. > > Yes, it actually does require some minimum amount of attention from the > admin, like, say, once per week or once per month updates - buy so > should *any* system... and yes, it does require a little more > willingness to learn and 'get your hands dirty' (especially for the > installation), but it is well worth it. Yes, I have learned lots from some years with gentoo. No bad feelings. Just bad combo this time. > > Oh - and Portage rocks... :) > Well, yes, so does granite. Or iron maiden. Or whatever. As long as you like it :-) But maybe you also have something useful to say on the questions I *did* ask? About dovecot versions, and/or maildir vs. dbox for example? As the subject said, I was seeking advice, not rant nor war... Best, Vincent. From campbell at cnpapers.com Thu Mar 8 16:38:36 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 08 Mar 2012 09:38:36 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F57C96F.7090602@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> <4F57C96F.7090602@hardwarefreak.com> Message-ID: <4F58C46C.8000202@cnpapers.com> On 3/7/2012 3:47 PM, Stan Hoeppner wrote: > On 3/6/2012 3:01 PM, Steve Campbell wrote: > >> I've experienced that type of locked mailbox before on the old server. >> Users insist on accessing their email account as a pop account on their >> desktop with the "check for new mail every so many minutes" turned on >> and still keep their smartphones on while accessing it as an imap >> account so they can still download the files to their desktop when they >> return. > Using IMAP on the phone and POP on the PC doesn't make any sense. Is > there a (valid) reason why these people insist on this phone/IMAP and > PC/POP setup? This seems seriously counter intuitive/productive. The bulk of these type users are sales staff. They use their desktop when their in the office. For years, the only type of email account we used was pop just because that was the way it was. We used horde for webmail, which read these type of accounts just fine. Once they needed email in the field, it was necessary to either set up their phones to use pop and keep email on the server so that they could download the email to their desktop, or use imap on the phones. They typically don't use any folders they've created on the imap account when accessing mail on the desktop. It would be a nightmare going to each desktop, finding a time when each and every user would have the time to allow us to change things, and switching all of the accounts. It may not seem to be a good way of doing things, but it's just the way our system here has evolved. Now that we're down to skeleton-type staffing, it's not easy to find the time and manpower to accomplish change when it "ain't broke". The occasional locked mailbox was easier to resolve that the massive change to all user's accounts. This all came about because I installed a new server to replace the old, and dovecot became the pop/imap server. > >> So just to clarify, is it OK to have a maildir account setup on this >> server for these shared/imap access only accounts along with the mbox >> accounts already on there? > Yes. With Dovecot it is possible to specify mail_location on a per user > basis: > > http://wiki.dovecot.org/MailLocation > > You can even do a split mailbox type setup per user using multiple > namespaces, for example specifying that INBOX use mbox with all other > mail being stored in maildir format: > > http://wiki.dovecot.org/Namespaces > >> Thanks for the patience and help > Sure thing. Again, thanks for the help. From CMarcus at Media-Brokers.com Thu Mar 8 18:04:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 11:04:14 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: <4F58D87E.3040704@Media-Brokers.com> On 2012-03-08 8:53 AM, Vincent Schut wrote: > But maybe you also have something useful to say on the questions I *did* > ask? About dovecot versions, and/or maildir vs. dbox for example? As the > subject said, I was seeking advice, not rant nor war... Yeah, sorry, and I wasn't offended, I just dislike it when someone says something like that without clarification... As for version, it is generally recommended for obvious reasons to stay within the confines of your distros package manager unless you are comfortable installing from source. I've never used Debian, so can't speak to which repos you can safely use or the implications if you do... As for what mailbox format, there is no more 'dbox', it is either sdbox (like mbox one file per folder) or mdbox (multiple files per folder) - that said, mdbox seems to be the best general purpose, but my understanding is it can complicate things if something goes wrong, but it seems to be very solid. -- Best regards, Charles From micah at riseup.net Thu Mar 8 18:27:43 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:27:43 -0500 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> Message-ID: <87ty1zys5c.fsf@algae.riseup.net> Willie Gillespie writes: > On 03/07/2012 12:43 PM, Micah Anderson wrote: >> >> When a user makes a folder called 'x.y' it actually creates a folder >> called 'x' with a folder called 'y' inside, rather than a folder called >> 'x.y'. I'm guessing this has to do with an internal folder separator >> namespace configuration, but I'm a bit confused by how this works. > > Correct. > Similar to how in Linux, I could create a folder > mkdir test1/test2 > It will create test2 inside of test1. > > The difference being that IMAP doesn't necessarily need the parent mailbox to > exist, where Linux would throw an error if test1/ didn't exist first. > > So basically, as far as I know, you can't have a folder with a "." in the name > with the namespaces you have set up. That makes sense, however I'm not sure that I need these namespaces any longer if I no longer am using the maildir format (mdbox). In either case, it seems like the internal folder separator should not be exposed to the user like this. What is happening now is the user gets something other than they expect (a folder within a folder, instead of a folder with a dot in the name) because of some unknown internal configuration. If moving to mdbox is not enough to remove these namespace configurations that cause this, then it would be good if the user was unable to create such a folder, because it was prohibited, rather than creating something other than they expect. micah From micah at riseup.net Thu Mar 8 18:29:46 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:29:46 -0500 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. References: Message-ID: <87pqcnys1x.fsf@algae.riseup.net> Vincent Schut writes: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... No, Debian has 1.2.15 in its /stable (squeeze)/ repositories, there are newer versions available in other Debian repositories. micah From robert at schetterer.org Thu Mar 8 18:32:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Thu, 08 Mar 2012 17:32:49 +0100 Subject: [Dovecot] dot named folders In-Reply-To: <87ty1zys5c.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> Message-ID: <4F58DF31.3040203@schetterer.org> Am 08.03.2012 17:27, schrieb Micah Anderson: > Willie Gillespie writes: > >> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>> >>> When a user makes a folder called 'x.y' it actually creates a folder >>> called 'x' with a folder called 'y' inside, rather than a folder called >>> 'x.y'. I'm guessing this has to do with an internal folder separator >>> namespace configuration, but I'm a bit confused by how this works. >> >> Correct. >> Similar to how in Linux, I could create a folder >> mkdir test1/test2 >> It will create test2 inside of test1. >> >> The difference being that IMAP doesn't necessarily need the parent mailbox to >> exist, where Linux would throw an error if test1/ didn't exist first. >> >> So basically, as far as I know, you can't have a folder with a "." in the name >> with the namespaces you have set up. > > That makes sense, however I'm not sure that I need these namespaces any > longer if I no longer am using the maildir format (mdbox). > > In either case, it seems like the internal folder separator should not > be exposed to the user like this. What is happening now is the user gets > something other than they expect (a folder within a folder, instead of a > folder with a dot in the name) because of some unknown internal > configuration. > > If moving to mdbox is not enough to remove these namespace > configurations that cause this, then it would be good if the user was > unable to create such a folder, because it was prohibited, rather than > creating something other than they expect. > > micah > http://wiki.dovecot.org/Plugins/Listescape may help -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From e-frog at gmx.de Thu Mar 8 19:43:25 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 08 Mar 2012 18:43:25 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command In-Reply-To: <4F57A640.1030202@gmx.de> References: <4F57A640.1030202@gmx.de> Message-ID: <4F58EFBD.9080205@gmx.de> On 07.03.2012 19:17, wrote e-frog: > # 2.1.1 (94de7605f50f) > 1 namespace > * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL > * OK Namespace completed. > > Please note that the "OK Namespace completed." is send untagged. Ok, it's working again today with 2.1.1 (7a26c427fc78). From busseniu at in.tum.de Thu Mar 8 19:56:03 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Thu, 08 Mar 2012 18:56:03 +0100 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> Message-ID: <4F58F2B3.9070407@in.tum.de> On 03/04/2012 03:10 PM, Timo Sirainen wrote: > BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. Oh, I was not aware that this feature exists. I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Our migration process involves: 1) Converting the maildir from Courier using the Perl script 2) Converting to mdbox using dsync -R backup The POP3 ordering seems to get lost during the second step. I.e., if Dovecot is set up to server POP3 mails from a maildir having "O" flags, the POP3 ordering is as intended. After changing the configuration to mdbox format and converting the mails using dsync, the POP3 ordering is different. Is this known or am I missing something? (I tried Dovecot 2.1.1.) Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 20:46:50 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 18:46:50 +0000 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: Your message of "Sun, 04 Mar 2012 15:36:59 +0200." <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> Message-ID: Thank you for your help, Timo. > use Dovecot v2.0's dsync I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) on a dovecot (v1) installation; presumably with its own configuration file(s). > You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... Yes, I think we could do that; I should have thought of it myself, thanks again. I think there was one other problem with the automatic conversion which I've now remembered: I note that the first time a user connects to th eimap service dovecot creates their (virtual) home directory for them with all the right permissions. That's great and I use the existence of that directory as an indication to our MTA that the user wants delivery into the dovecot store rather than their old system mailbox. However once I tried using the convert plugin the process fails because (it seems) the conversion tries to take place before the home directory has been created. Is there any configuration change that might change this order? Can I configure the convert plugin on LDA delivery, for example, instead of as part of the "protocol imap" section? Many thanks, Steve Platt From tss at iki.fi Thu Mar 8 20:51:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 20:51:26 +0200 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <4F58F2B3.9070407@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> <4F58F2B3.9070407@in.tum.de> Message-ID: <0FEB6932-0FE5-42C6-B72F-3FD914B3A7BB@iki.fi> On 8.3.2012, at 19.56, Christoph Bu?enius wrote: > On 03/04/2012 03:10 PM, Timo Sirainen wrote: >> BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. > > Oh, I was not aware that this feature exists. > > I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Yeah, for now it's only for Maildir. Probably wouldn't be difficult to implement for dbox by adding it as dbox metadata (although how to add it there? dsync can't copy that). From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 21:04:47 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 19:04:47 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 Message-ID: I've set up a list of ciphers that excludes SSLv2 ciphers (and other weak ones) in the hope of preventing SSLv2 connections: ssl_cipher_list = TLSv1+HIGH : !SSLv2 : RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH However, this doesn't prevent the SSLv2 connection being allowed as our Nessus scans show and I'm tasked with trying to plug that "hole". I see Dovecot2 had the following change a year or so ago, in file src/login-common/ssl-proxy-openssl.c: - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); I tried making the same change to dovecot1's src tree on our test system and it seems to have the desired effect; however I am very hesitant about putting this into our production system without seeking advice here first :-) Have I missed anything that's obviously bad about doing this please? Thanks again, Steve Platt From markus at mpetri.org Thu Mar 8 21:18:12 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 8 Mar 2012 20:18:12 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 Message-ID: <20120308201812.2932e90c@legolas.home.ceotex.de> Hi, after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use shared folders with mutt anymore. 2.1 lists the shared namespace prefix once per user sharing an folder in LIST "" "%". I also noticed, that with 2.1 the user folder (Shared/) is no longer tagged as \NoSelect. Is this the intended behaviour and mutt simply cannot cope with it or is it a dovecot problem? Here an example with three users sharing a folder to the logged in user with Dovecot 2.1.1: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\HasChildren) "/" "Shared/test" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasChildren) "/" "Shared/test2" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. The same three users and config with Dovecot 2.0.18: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared/test" * LIST (\Noselect \HasChildren) "/" "Shared/test2" * LIST (\Noselect \HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. Markus # 2.1.1: /opt/dovecot-2.1/etc/dovecot/dovecot.conf # OS: Linux 3.2.0-1-amd64 x86_64 Debian wheezy/sid auth_mechanisms = plain login disable_plaintext_auth = no listen = 192.168.56.11 mail_location = maildir:~/Maildir mail_plugins = acl managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { inbox = yes location = prefix = separator = / subscriptions = yes type = private } namespace { inbox = no list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/index/shared/%%u prefix = Shared/%%u/ separator = / subscriptions = no type = shared } passdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } plugin { acl = vfile acl_anyone = allow acl_shared_dict = file:/var/lib/vdovecot/shared-mailboxes.db } protocols = imap service auth { unix_listener auth-userdb { mode = 0600 user = vdovecot } } ssl = no userdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } verbose_proctitle = yes protocol imap { mail_plugins = acl imap_acl } From tss at iki.fi Thu Mar 8 21:36:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 21:36:09 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <20120308201812.2932e90c@legolas.home.ceotex.de> References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: On 8.3.2012, at 21.18, Markus Petri wrote: > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > once per user sharing an folder in LIST "" "%". > > I also noticed, that with 2.1 the user folder (Shared/) is no > longer tagged as \NoSelect. > > Is this the intended behaviour and mutt simply cannot cope with it or > is it a dovecot problem? Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even if it did. Also Dovecot probably should add \Noselect, especially if the mailbox isn't really selectable (there's some weirdness between shared/user being equal to shared/user/INBOX, but I'm not sure what to do about it). From Bennett.Tony at con-way.com Fri Mar 9 01:23:00 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Thu, 8 Mar 2012 15:23:00 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? Message-ID: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. (The output of "dovecot -n" is at the bottom of this email.) I'm trying "baby steps" to get it up, before I give it the final configuration. (My apologies: I was pointed to RFC3501 and told to get an IMAP server, build it, configure it, and bring it up) What is currently occurring when I start dovecot is: Error: service(pop3-login): listen(::, 110) failed: Address already in use Error: service(pop3-login): listen(::, 995) failed: Address already in use Error: service(imap-login): listen(::, 143) failed: Address already in use Error: service(imap-login): listen(::, 993) failed: Address already in use Fatal: Failed to start listeners Using TRUSS and recompiling with log messages I've determined that dovecot is successfully creating and binding to AF_INET sockets... but is failing when trying to do the "bind" the same port to an AF_INET6 socket. The failure is "EADDRINUSE". The logic in the dovecot sources seems driven off of the define of HAVE_IPV6 (defined in config.h by configure) So, the questions I have are: - Is this the correct behavior - If this is the correct behavior, has this been tested against AIX 6.1, and if so, does anyone have an idea of what I did wrong...??? If it has not been tested against AIX 6.1 and is NOT the correct behavior, should I just change "config.h", and undefined HAVE_IPV6 ... or is there a better way to move beyond this issue... (like a change to "configure")??? Thanks, -tony Here is the output of "dovecot -n": # 2.1.1: /attic/usr/local/etc/dovecot/dovecot.conf # OS: AIX 1 00C30F654C00 default_login_user = dovecot disable_plaintext_auth = no namespace { inbox = yes location = mailbox { special_use = \Drafts name = Drafts } mailbox { special_use = \Junk name = Junk } mailbox { special_use = \Sent name = Sent } mailbox { special_use = \Sent name = Sent Messages } mailbox { special_use = \Trash name = Trash } prefix = name = inbox } passdb { args = scheme=CRYPT username_format=%u /attic/usr/local/etc/dovecot/users driver = passwd-file } service anvil-auth-penalty { name = anvil } service auth-worker { name = auth-worker } service auth-client { name = auth } service config { name = config } service dict { name = dict } service login/proxy-notify { name = director } service dns-client { name = dns_client } service doveadm-server { name = doveadm } service imap { name = imap-login } service login/imap { name = imap } service indexer-worker { name = indexer-worker } service indexer { name = indexer } service ipc { name = ipc } service lmtp { name = lmtp } service log-errors { name = log } service pop3 { name = pop3-login } service login/pop3 { name = pop3 } service login/ssl-params { name = ssl-params } service stats-mail { name = stats } ssl_cert = References: Message-ID: <201203091030.20828.sdavies@sdc.com.au> Yes that is the google thread that I saw. I don't see the relevance of your reference to dsync. As I read the man pages for dsync it is used to sync separate servers, to make backups or to convert mailbox formats. When I upgraded from 1.2.15 to 2.1.1 I saw nothing in the doco to suggest that dsync was relevant to my scenario. In a previous thread here (Log sync errors), Timo suggested that the migration fix was to delete all .imap directories. My understanding was that this should fix any differences between 1.2.15 files and 2.1.1. If that were the case, mentioning the migration again would seem irrelevant. However, it seems that deleting the .imap files did not fix the log sync errors or the fscking warnings. Both are still happening continuously. Cheers, Stephen On Thu, 8 Mar 2012 08:26:55 PM dovecot-request at dovecot.org wrote: > Date: Wed, 07 Mar 2012 15:03:35 -0600 > From: Stan Hoeppner > To: dovecot at dovecot.org > Subject: Re: [Dovecot] Fscking warnings > Message-ID: <4F57CD27.3000207 at hardwarefreak.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On 3/6/2012 5:07 PM, Stephen Davies wrote: > > Google tells me that these "should go away" but they don't. > > > > > > > > Seems to happen continuously while a user is viewing email. > > Is this thread what "Google tells you"? > > http://dovecot.org/list/dovecot/2010-October/053909.html > > Timo is the creator of Dovecot, if you didn't know. So you can take his > words for gospel. Also note his last statement in that thread: > > "The next time you could do it with dsync to avoid these kind of > problems." > > It would seem you omitted a very important detail from your problem > report, which is that you recently performed a migration. Please don't > omit such critical details in future requests for help. Provide as much > relevant detail as possible. This speeds the process up for everyone, > and avoids guesswork on our part. > > -- > Stan > > > ------------------------------ > > Message: 10 > Date: Thu, 8 Mar 2012 00:26:55 +0100 > From: "Marc" > To: > Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From mmielke at sapphire.gi Fri Mar 9 02:50:47 2012 From: mmielke at sapphire.gi (Martin Mielke) Date: Fri, 9 Mar 2012 00:50:47 +0000 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. Message-ID: Hi all, I have inherited an old Dovecot installation which is causing headaches almost every day. I know that one of the rules says "Don't bother asking questions about v0.99.x versions. They're no longer supported."...but please bear with me, this will be quick as I only need some advise from experienced Dovecot gurus out there. I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... Thanks a lot in advance! Regards, Martin From ybhu at hk1.ibm.com Fri Mar 9 10:09:37 2012 From: ybhu at hk1.ibm.com (Andy YB Hu) Date: Fri, 9 Mar 2012 16:09:37 +0800 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? Message-ID: In the real world, the mails which belongs to one thread could be dispersed in different mailbox, at least Inbox and SENT, so whether the command can search different mailboxes and grap them in one talk with dovecot? If Not, any other approach to do that? BTW, what the THREAD=REFS stands for? In the RFC, http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. From janfrode at tanso.net Fri Mar 9 10:31:34 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 9 Mar 2012 09:31:34 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <20120309083134.GA8248@dibs.tanso.net> On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: > > As for what mailbox format, there is no more 'dbox', it is either > sdbox (like mbox one file per folder) or mdbox (multiple files per > folder) - Sdbox is like maildir, one message per file, while mdbox is more like mbox: http://wiki2.dovecot.org/MailboxFormat/dbox > that said, mdbox seems to be the best general purpose, but > my understanding is it can complicate things if something goes > wrong, but it seems to be very solid. It's a leap of faith to go with dovecot's own format, and no longer be able to use grep and mutt to poke in mail folders directly, but as a serverside storage format it seems like the right way to go. -jf From varia at e-healthexpert.org Fri Mar 9 11:35:17 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Fri, 9 Mar 2012 09:35:17 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: References: Message-ID: <20120309093517.30979c04@e-healthexpert.org> On Thu, 08 Mar 2012 19:04:47 +0000, Steve Platt wrote: > I've set up a list of ciphers that excludes SSLv2 ciphers (and other > weak ones) in the hope of preventing SSLv2 connections: > > ssl_cipher_list = TLSv1+HIGH : !SSLv2 : > RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH > > I tried making the same change to dovecot1's src tree on our test > system and it seems to have the desired effect; No need to change sources. Try this and see if it serves your purpose: ssl = required ssl_cipher_list = HIGH:!SSLv2:!aNULL:!MD5!DES:!3DES M. From CMarcus at Media-Brokers.com Fri Mar 9 16:45:35 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 09 Mar 2012 09:45:35 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <4F5A178F.1060404@Media-Brokers.com> On Thu, Mar 09, 2012 at 12:30AM -0500, Jan-Frode Myklebust wrote: > On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: >> As for what mailbox format, there is no more 'dbox', it is either >> sdbox (like mbox one file per folder) or mdbox (multiple files per >> folder) - > > Sdbox is like maildir, one message per file, while mdbox is more > like mbox: > > http://wiki2.dovecot.org/MailboxFormat/dbox Wow, I've no idea how that bit of incorrect data got lodged inside my head. Thanks Jan-Frode for the correction! -- Best regards, Charles From steve.platt at mrc-bsu.cam.ac.uk Fri Mar 9 17:05:26 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Fri, 09 Mar 2012 15:05:26 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: Message from Mark Alan of "Fri, 09 Mar 2012 09:35:17 GMT." <20120309093517.30979c04@e-healthexpert.org> Message-ID: Hi Mark, I think I may not have been clear enough in my query, sorry! What I'm trying to do is to prevent SSLv2 connections being made to our IMAP server while allowing SSLv3 and TLSv1 connections. I think I've prevented the use of SSLv2 ciphers but this does not prevent SSLv2 protocol connections (as far as I can tell). (Once connected, the SSLv2 client finds it has no ciphers so the session fails at that point but this is not enough to satisfy our security audit. I want to disable the use of the SSLv2 protocol itself, not just the SSLv2 ciphers) steve.platt at mrc-bsu.cam.ac.uk said: > I see Dovecot2 had the following change a year or so ago, in file src/ > login-common/ssl-proxy-openssl.c: > > - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); > + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > > I tried making the same change to dovecot1's src tree on our test system and > it seems to have the desired effect ... I'm testing this by using: openssl s_client -ssl2 -connect mailhost:993 This should fail immediately with "ssl handshake failure" (for a happy audit!). Thanks again, Steve From user+dovecot at localhost.localdomain.org Fri Mar 9 22:19:15 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Fri, 09 Mar 2012 21:19:15 +0100 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> Message-ID: <4F5A65C3.4080404@localhost.localdomain.org> On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From Bennett.Tony at con-way.com Fri Mar 9 22:34:56 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Fri, 9 Mar 2012 12:34:56 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <4F5A65C3.4080404@localhost.localdomain.org> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> <4F5A65C3.4080404@localhost.localdomain.org> Message-ID: <9E085D377965634187A85638358AE61101A291DA92@DCXPRCL017.cnf.prod.cnf.com> Bingo... It worked... Thanks, Pascal -tony -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Pascal Volk Sent: Friday, March 09, 2012 12:19 PM To: Dovecot Mailing List Subject: Re: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From sca at andreasschulze.de Fri Mar 9 23:40:16 2012 From: sca at andreasschulze.de (Andreas Schulze) Date: Fri, 9 Mar 2012 22:40:16 +0100 Subject: [Dovecot] sieve and utf-7 foldernames Message-ID: <20120309214016.GA5584@doran.andreasschulze.de> Hi all, since many dovecot/pigeonhole versions I have an error: Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) setup: dovecot-2.1.1 / pigeonhole-0.3.0 postfix deliver to dovecot-lda .dovecot.sieve contains this: require ["fileinto","reject","vacation","relational","comparator-i;ascii-numeric","regex"]; if header :contains "To" "green at example.org" { fileinto "INBOX.gr&APw-n"; stop; } a mail to the mentioned address produces this logging: Mar 9 22:23:10 test dovecot: lda(foo): save: box=INBOX.gr&APw-n, uid=2, msgid=<4711 at example.org>, size=4642 also a mail_debug=yes log no more lines related to the foldername but at the end I found the mail not in the expected folder. It places in "INBOX.gr&-APw-n" notice the extra minus after & I think it's pigeonholes fault but I may be wrong ... # doveconf -n # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686 i686 Debian 5.0.9 ext3 ... lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = 127.0.0.1, ::1 protocols = " imap sieve" protocol lda { mail_plugins = quota notify mail_log sieve } ... Andreas From user+dovecot at localhost.localdomain.org Sat Mar 10 02:13:43 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Sat, 10 Mar 2012 01:13:43 +0100 Subject: [Dovecot] sieve and utf-7 foldernames In-Reply-To: <20120309214016.GA5584@doran.andreasschulze.de> References: <20120309214016.GA5584@doran.andreasschulze.de> Message-ID: <4F5A9CB7.1040804@localhost.localdomain.org> On 03/09/2012 10:40 PM Andreas Schulze wrote: > Hi all, > > since many dovecot/pigeonhole versions I have an error: > Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) > ? Behaves as documented - since Dovecot 1.2.0: Wiki > Upgrading > v1.1 to v1.2 > Sieve: * You should consider migrating from CMU Sieve to Dovecot Sieve (see the link for instructions) http://wiki.dovecot.org/LDA/Sieve/Dovecot#Migration_from_CMUSieve: * Be sure to use UTF8 for the mailbox argument of the fileinto command. Older CMUSieve installations used modified UTF7 (as IMAP does) for the mailbox parameter. If not adjusted, the new Sieve plugin will use the wrong folder name for storing the message. Regards, Pascal -- The trapper recommends today: cafefeed.1207001 at localdomain.org From tss at iki.fi Sat Mar 10 18:01:27 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:01:27 +0200 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. In-Reply-To: References: Message-ID: On 9.3.2012, at 2.50, Martin Mielke wrote: > I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... http://wiki2.dovecot.org/Upgrading/1.0 points out a few things: - rename .subscriptions -> subscriptions - rename .customflags -> dovecot-keywords - default pop3 UIDL format changed From tss at iki.fi Sat Mar 10 18:02:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:02:40 +0200 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? In-Reply-To: References: Message-ID: <4956B88C-3B55-460C-AAD2-E8253DA8627B@iki.fi> On 9.3.2012, at 10.09, Andy YB Hu wrote: > > In the real world, the mails which belongs to one thread could be dispersed > in different mailbox, at least Inbox and SENT, so whether the command can > search different mailboxes and grap them in one talk with dovecot? > > If Not, any other approach to do that? Create a virtual mailbox containing all mails. Then you see all the mails within a thread. http://wiki2.dovecot.org/Plugins/Virtual > BTW, what the THREAD=REFS stands for? In the RFC, > http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. It's from http://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05 which never made it to an actual RFC. From tss at iki.fi Sat Mar 10 18:04:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:04:44 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <79C53550-348C-4812-AAA1-2C3D6D9F59B4@iki.fi> On 8.3.2012, at 20.46, Steve Platt wrote: >> use Dovecot v2.0's dsync > > I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) > on a dovecot (v1) installation; presumably with its own configuration file(s). Yes, although in some situations it might write stuff to index files that v1.x complains about. But deleting index files afterwards fixes that. >> You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... > > Yes, I think we could do that; I should have thought of it myself, thanks > again. > > I think there was one other problem with the automatic conversion which I've > now remembered: I note that the first time a user connects to th eimap service > dovecot creates their (virtual) home directory for them with all the right > permissions. That's great and I use the existence of that directory as an > indication to our MTA that the user wants delivery into the dovecot store > rather than their old system mailbox. However once I tried using the convert > plugin the process fails because (it seems) the conversion tries to take place > before the home directory has been created. > > Is there any configuration change that might change this order? No. > Can I configure the convert plugin on LDA delivery, for example, instead of as > part of the "protocol imap" section? Yes. From tss at iki.fi Sat Mar 10 18:06:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:06:20 +0200 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <729DCBA4-E353-41EA-903D-0DDF897E5208@iki.fi> On 7.3.2012, at 21.43, Micah Anderson wrote: > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. > > I'm using 2.0.15 with mdbox and this is what I have configured for my > namespaces: > > namespace { > separator = . > prefix = > inbox = yes > } Keep this. > namespace { > separator = . > prefix = INBOX. > inbox = no > hidden = yes > list = no > } > > I migrated from courier maildirs, so perhaps I no longer need some of > these now that the conversion is finished? It depends on if you have any users whose clients are using INBOX. namespace. If there are, and you remove it, the users won't see anything except INBOX anymore. From tss at iki.fi Sat Mar 10 18:11:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:11:55 +0200 Subject: [Dovecot] Single instance storage In-Reply-To: References: Message-ID: <508532C3-B8D0-47E8-9566-6A570A3233F3@iki.fi> On 8.3.2012, at 3.30, Jean-Daniel Beaubien wrote: > I have read most of the doc on the dovecot website, and couldn't find any > info on the single instance storage feature, so I'm posting my questions > here. > > - Are these 3 parameters the only one necessary for single instance > storage? I cannot find any doc on this feature on the website; is there > anything specific I need to know about them? (the last one isn't exactly > self-explanatory). > - mail_attachment_dir = /srv/vmail/attachments > - mail_attachment_hash = %{sha256} > - mail_cache_min_mail_count = 2 > > - Is this feature ready for production? mail_cache_min_mail_count isn't related to single instance storage at all. I didn't really even remember that such a setting existed. I'm not sure if it's actually useful in any setups.. Maybe you were thinking about mail_attachment_min_size? Other than that, yeah, the mail_attachment_dir is really the only thing you need to set to enable SIS. From tss at iki.fi Sat Mar 10 18:13:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:13:39 +0200 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: On 7.3.2012, at 1.07, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. > > Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file > /home/john/Mail/INBOX/.imap/Archive/dovecot.index What Dovecot version? Anyway, something wrong in the mbox I guess. Just do rm -rf /home/john/Mail/INBOX/.imap/ From tss at iki.fi Sat Mar 10 18:20:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:20:33 +0200 Subject: [Dovecot] POP3C storage backend In-Reply-To: <4F5538DC.4060802@talpey.com> References: <4F5538DC.4060802@talpey.com> Message-ID: <7B049F23-AE14-4BCE-857C-91D70E02A7E8@iki.fi> On 6.3.2012, at 0.06, Tom Talpey wrote: > I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I > don't see anything in the 2.1 doc directory or in the wiki. Can this > be used to synchronize dovecot with external pop servers? Doing away > with my current fetchmail and lmtp solution for this would be quite > interesting. > > Thanks for any pointers to configuring and using this, if so... It could possibly be used to do that with http://wiki2.dovecot.org/Plugins/Snarf Although that would probably connect to POP3 server quite often. And when IDLEing it wouldn't see new POP3 mails (that would need a small change to snarf plugin). From tss at iki.fi Sat Mar 10 18:44:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:44:21 +0200 Subject: [Dovecot] dovecot Digest, Vol 107, Issue 20 Fscking warnings In-Reply-To: <201203091030.20828.sdavies@sdc.com.au> References: <201203091030.20828.sdavies@sdc.com.au> Message-ID: <7F823A5F-36DC-4444-A13E-3182FA243EE1@iki.fi> On 9.3.2012, at 2.00, Stephen Davies wrote: > However, it seems that deleting the .imap files did not fix the log sync errors > or the fscking warnings. > > Both are still happening continuously. If you're talking about errors like these: Mar 10 18:21:38 imap(tss): Error: Log synchronization error at seq=1,offset=26896 for /home/tss/mail/.imap/INBOX/dovecot.index: Extension header update points outside header size and the following fsck error, then deleting all of the .imap directories should get rid of them (maybe you didn't delete all of them? note that each subdirectory has its own, so there's more than just ~/mail/.imap/). Anyway, this is now also fixed: http://hg.dovecot.org/dovecot-2.0/rev/dc88712581c9 http://hg.dovecot.org/dovecot-2.1/rev/1289b79241bb From tss at iki.fi Sat Mar 10 18:50:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:50:15 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <7F351B29-16BF-4098-8C3A-3FC84D85ADD4@iki.fi> On 7.3.2012, at 15.32, Charles Marcus wrote: > 4. They can *move* messages to other folders in that account (ie, 'file' them), and last > > (this is the tricky part) > > 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. There is unfortunately no "default ACL" feature currently. Although you could somewhat easily add an ugly hack to the code for that. And I guess it wouldn't be difficult to implement it, maybe by reading it from $mail_root/dovecot-acl-default file or something.. So without code changes you could: - create all of the necessary folders - set such ACLs that user can't create any more folders - disallow expunging in all folders From tss at iki.fi Sat Mar 10 18:51:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:51:16 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F578F31.3000303@wildgooses.com> References: <4F576370.8040706@Media-Brokers.com> <4F578F31.3000303@wildgooses.com> Message-ID: <75B1D406-190B-4824-B575-E1AF1F76B207@iki.fi> On 7.3.2012, at 18.39, Ed W wrote: >> 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. > > Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) The maildir file's read permission doesn't matter, the parent cur/ or new/ directory's write permission matters. And removing those prevents moving mails from new/ to cur/ and from keeping the flag states in the filename.. Not very good. From tss at iki.fi Sat Mar 10 18:53:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:53:38 +0200 Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file In-Reply-To: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> References: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> Message-ID: <35158279-4A11-4806-A481-548314E27282@iki.fi> On 7.3.2012, at 4.22, Rajesh M wrote: > on server number 1 i get errors as such > Error: fcntl(read-lock) locking failed for file Input/output error > squirrelmail gives error imap connection closed and i am not able to login > > > so i set the parameters as such in the dovecot conf file and the error > stopped > > mmap_disable=yes > dotlock_use_excl = yes > lock_method = dotlock > > can somebody please advise me if the above is correct ? That should work. > or is it preferred to use fcntl with lockd That would probably be more efficient. > (note that my mailbox is maildir format) The fcntl locking is used for Dovecot index files, not for maildir files. From tss at iki.fi Sat Mar 10 18:56:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:56:50 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: References: Message-ID: On 7.3.2012, at 5.19, Jean-Daniel Beaubien wrote: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." > > How will the mailbox files (m.X) files be modified when I move or delete > emails using mdbox+gzip. Will the resulting gzipped mdbox files be > rsync-able or will they need a full re-upload? > > If I plan on using rsync for backups, am I better off not using the gzip > feature (if i can spare the extra storage)??? gzipping is irrelevant, the behavior is the same with and without gzip. The purging step recreates new mail files, so the new files will need to be fully uploaded with rsync. You might want to consider using dsync instead. From btb at bitrate.net Mon Mar 12 06:00:11 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Mon, 12 Mar 2012 00:00:11 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections Message-ID: hi- i have a configuration in which i'm using different passdb/userdb settings for each of imap, lmtp and smtp [without getting too far off on a tangent, this is so ldap group membership can be used to independently control authorization for receiving [lmtp], retrieving [imap], and sending [smtp/postfix] of email. each passdb/userdb uses a different search filter.] when i use this as my config: # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic-pae i686 Ubuntu 11.10 auth_debug = yes first_valid_gid = 2000 first_valid_uid = 2000 log_timestamp = "%d.%m.%Y %H.%M.%S " login_greeting = dovecot ready mail_debug = yes protocols = " imap lmtp" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service lmtp { inet_listener lmtp { address = 127.0.0.1 ::1 port = 10026 } } ssl_cert = Hello We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib - Does anybody know of any other project to create an easy-restore for Dovecot? Regards Maria From wouter at vdschagt.com Mon Mar 12 14:09:51 2012 From: wouter at vdschagt.com (Wouter van der Schagt) Date: Mon, 12 Mar 2012 13:09:51 +0100 Subject: [Dovecot] Dovecot LDA breaking .qmail forwarding? Message-ID: <00d201cd0049$07274880$1575d980$@vdschagt.com> Good morning all, I've a problem, I'm using the Dovecot LDA in my .qmail file: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER However when I add a forward, such as: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER &wouter at vdschagt.com The mail isn't forwarded and stays in the queue on the mail server, the same is the case when I reverse the lines. The error in the log file is: @400000004f5de5f7033bc434 delivery 824026: deferral: qmail-inject:_fatal:_qq_trouble_creating_files_in_queue_(#4.3.0)/system_erro r_calling_qmail-inject/ When using another LDA, for example when specifying the Maildir, it works, but then i cannot use Sieve scripts. Any ideas? Am I doing anything wrong? Sincerely, - Wouter van der Schagt From bind at enas.net Mon Mar 12 15:02:33 2012 From: bind at enas.net (Urban Loesch) Date: Mon, 12 Mar 2012 14:02:33 +0100 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <4F5DF3E9.9030703@enas.net> Hi, perhaps the mail_log plugin is what you need. Regards Urban On 12.03.2012 12:56, Maria Arrea wrote: > Hello > > We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: > > - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib > - Does anybody know of any other project to create an easy-restore for Dovecot? > > Regards > > Maria > From tss at iki.fi Mon Mar 12 15:04:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:04:10 +0200 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <1331557450.2081.112.camel@innu> On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From giles at coochey.net Mon Mar 12 15:09:52 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:09:52 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) Message-ID: <4F5DF5A0.9000609@coochey.net> Hi, I'm looking for a quick tool that can connect to my IMAP account grab all the messages in a particular folder and dump them to a mbox format file? Anyone know a quick easy tool to do that? This is a spam folder that I'd like to do some Bayes spam learning on, but since I've migrated to mdbox I don't think I can do this directly on the mailbox. Any thoughts appreciated. -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From maria_arrea at gmx.com Mon Mar 12 15:18:01 2012 From: maria_arrea at gmx.com (Maria Arrea) Date: Mon, 12 Mar 2012 14:18:01 +0100 Subject: [Dovecot] Question about folder creation/delete and logging Message-ID: <20120312131801.134760@gmx.com> mail_log plugin is just what we need, thank you for your support. Yes, we are using bacula+doveadm import for this project. If you are interested, we have in production an home-brew message-tracking system for our end-users. We parse qmail / postfix / clamav / spamassassin/ dovecot / sieve logs and insert them in a mysql database, and a logged user in our webapp can see what happened with her mails (sent mails or messages waiting for arrival). Affero GPL software, of course. Some screenshots (Spanish only, sorry) here: https://gestionproyectos.us.es/attachments/download/321/Seguimiento_de_mensajes___Vista_detalla_de_mensaje_1331558163319.png https://gestionproyectos.us.es/attachments/download/145/Captura_de_pantalla_2011-06-05_a_las_14.03.42.png https://gestionproyectos.us.es/attachments/download/156/Seguimiento_de_mensajes___Administraci%C3%B3n_1308042340487.png URL of the project (Seguimiento, spanish word for "tracking"): https://gestionproyectos.us.es/projects/seguimiento Regards Maria ----- Original Message ----- From: Timo Sirainen Sent: 03/12/12 02:04 PM To: Maria Arrea Subject: Re: [Dovecot] Question about folder creation/delete and logging On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From tss at iki.fi Mon Mar 12 15:19:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:19:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DF5A0.9000609@coochey.net> References: <4F5DF5A0.9000609@coochey.net> Message-ID: <1331558383.2081.114.camel@innu> On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: > Hi, > > I'm looking for a quick tool that can connect to my IMAP account grab > all the messages in a particular folder and dump them to a mbox format file? > > Anyone know a quick easy tool to do that? > > This is a spam folder that I'd like to do some Bayes spam learning on, > but since I've migrated to mdbox I don't think I can do this directly on > the mailbox. > > Any thoughts appreciated. With v2.0: dsync -m spam backup mbox:~/mbox-mails/ There are also doveadm move and doveadm import commands that can do this. From giles at coochey.net Mon Mar 12 15:37:43 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:37:43 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331558383.2081.114.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> Message-ID: <4F5DFC27.4020004@coochey.net> On 12/03/2012 13:19, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: >> Hi, >> >> I'm looking for a quick tool that can connect to my IMAP account grab >> all the messages in a particular folder and dump them to a mbox format file? >> >> Anyone know a quick easy tool to do that? >> >> This is a spam folder that I'd like to do some Bayes spam learning on, >> but since I've migrated to mdbox I don't think I can do this directly on >> the mailbox. >> >> Any thoughts appreciated. > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > There are also doveadm move and doveadm import commands that can do > this. > > OK, That would do a mailbox called spam no? I have a normal user who has an IMAP folder called 'Junk E-mail' Is there a similar syntax? -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From rob0 at gmx.co.uk Mon Mar 12 15:38:49 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 12 Mar 2012 08:38:49 -0500 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: References: Message-ID: <20120312133849.GS24983@harrier.slackbuilds.org> On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: > the problem with this is that while each of the passdb/userdb > configs for the various protocols does indeed work, if a result > is not found in one of them, the global passdb appears to then > function as a catch-all. > > how can i tell dovecot it doesn't need a global passdb? each > of the protocols' passdb/userdb configs is functioning as > desired, but having dovecot look elsewhere upon failure ends > up defeating the purpose. A simple workaround: use an empty passwd-file passdb as global. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From tss at iki.fi Mon Mar 12 15:39:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:39:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DFC27.4020004@coochey.net> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> Message-ID: <1331559583.2081.115.camel@innu> On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: > >> Any thoughts appreciated. > > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > > > There are also doveadm move and doveadm import commands that can do > > this. > > > > > OK, That would do a mailbox called spam no? > I have a normal user who has an IMAP folder called 'Junk E-mail' > > Is there a similar syntax? Yeah, the -m parameter just specifies the mailbox name. So: dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ If different accounts have different names, then I guess you'll just have to run dsync twice with each name. From giles at coochey.net Mon Mar 12 15:42:10 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:42:10 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331559583.2081.115.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> <1331559583.2081.115.camel@innu> Message-ID: <4F5DFD32.3010609@coochey.net> On 12/03/2012 13:39, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: >>>> Any thoughts appreciated. >>> With v2.0: dsync -m spam backup mbox:~/mbox-mails/ >>> >>> There are also doveadm move and doveadm import commands that can do >>> this. >>> >>> >> OK, That would do a mailbox called spam no? >> I have a normal user who has an IMAP folder called 'Junk E-mail' >> >> Is there a similar syntax? > Yeah, the -m parameter just specifies the mailbox name. So: > > dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ > > If different accounts have different names, then I guess you'll just > have to run dsync twice with each name. > > Thanks - was experimenting and got it. Works a treat! -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From micah at riseup.net Mon Mar 12 17:05:06 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:05:06 -0400 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> <4F58DF31.3040203@schetterer.org> Message-ID: <87k42pyi59.fsf@algae.riseup.net> Robert Schetterer writes: > Am 08.03.2012 17:27, schrieb Micah Anderson: >> Willie Gillespie writes: >> >>> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>>> >>>> When a user makes a folder called 'x.y' it actually creates a folder >>>> called 'x' with a folder called 'y' inside, rather than a folder called >>>> 'x.y'. I'm guessing this has to do with an internal folder separator >>>> namespace configuration, but I'm a bit confused by how this works. >>> >>> Correct. >>> Similar to how in Linux, I could create a folder >>> mkdir test1/test2 >>> It will create test2 inside of test1. >>> >>> The difference being that IMAP doesn't necessarily need the parent mailbox to >>> exist, where Linux would throw an error if test1/ didn't exist first. >>> >>> So basically, as far as I know, you can't have a folder with a "." in the name >>> with the namespaces you have set up. >> >> That makes sense, however I'm not sure that I need these namespaces any >> longer if I no longer am using the maildir format (mdbox). >> >> In either case, it seems like the internal folder separator should not >> be exposed to the user like this. What is happening now is the user gets >> something other than they expect (a folder within a folder, instead of a >> folder with a dot in the name) because of some unknown internal >> configuration. >> >> If moving to mdbox is not enough to remove these namespace >> configurations that cause this, then it would be good if the user was >> unable to create such a folder, because it was prohibited, rather than >> creating something other than they expect. >> >> micah >> > > http://wiki.dovecot.org/Plugins/Listescape > may help Interesting, thanks for the pointer, although I think I prefer if users are just prohibited from making a 'folder.withadot' and told that it is prohibited right away, rather than giving them a way to do it. micah -- From micah at riseup.net Mon Mar 12 17:10:46 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:10:46 -0400 Subject: [Dovecot] mdbox + gzip and rsync References: Message-ID: <87fwddyhvt.fsf@algae.riseup.net> Jean-Daniel Beaubien writes: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely. It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely. micah -- From tss at iki.fi Mon Mar 12 17:46:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 17:46:51 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: <87fwddyhvt.fsf@algae.riseup.net> References: <87fwddyhvt.fsf@algae.riseup.net> Message-ID: <1A1E352C-2A32-4C90-9357-A35C92D98875@iki.fi> On 12.3.2012, at 17.10, Micah Anderson wrote: > Jean-Daniel Beaubien writes: > >> After reading the following paragraph from the dovecot doc, I've been >> wondering how it would affect rsync (when combined with gzip): >> >> "Expunging a message only decreases the message's refcount. The space is >> later freed in "purge" step. This is typically done in a nightly cronjob >> when there's less disk I/O activity. The purging first finds all files that >> have refcount=0 mails. Then it goes through each file and copies the >> refcount>0 mails to other mdbox files (to the same files as where newly >> saved messages would also go), updates the map index and finally deletes >> the original file. So there is never any overwriting or file truncation." > > Interesting, so it would be recommended to those using mdbox format to > run a 'dovadm purge -A' every night to clean up these unused files? It > seems like without this, mail storage usage will just grow infinitely. Yes. > It does appear that using an rsync backup process for mdbox would not be > able to detect this and backups would also grow infinitely. rsync --delete would delete the old files, right? Anyway, I'd avoid using rsync for mdbox unless you're doing it on a filesystem snapshot. dsync backup should work better. From rtroy at ScienceTools.com Mon Mar 12 19:14:09 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) Subject: [Dovecot] Trouble adding sasl support via dovecot Message-ID: Hello Folks, I've been the admin of a site that uses Postfix with Dovecot on RedHat since, oh, gosh, maybe 1996? It's been a long time. I've never built it from source, though, just used the rpms (and I wonder if maybe that's my problem now). It just works, is reliable, and lets me be a very-part-time administrator. Repeatedly over the last few years I've been asked to have our mail system "join the modern age" and provide mail sending capabilities for clients that aren't on our internal network - via their smart-phones, from home, etc. OK... Well, way back when the site was set up, smtp servers didn't do any kind of "auth", but along the way to solving this problem (trying to configure pop-before-smtp, someone mentioned that Postfix now has an auth mechanism that uses Dovecot and I should use that instead! Great! ... Except that I've spent between 16 and 20 hours on this with no joy, and while I hate having to ask for help, it's time to ask what things that are obvious to the less ignorant that I must be doing wrong... Certainly, given the solid history of Postfix and Dovecot, I must be the problem! My problem statement is simply, "it should be working", but doesn't, and I don't get any announcement of "auth" when testing connections to Postfix as per directions here: http://www.postfix.org/SASL_README.html#server_test At least I haven't broken the normal functionality! I'm building a new server on the latest Fedora Core (16), but it's lacking in some hardware and won't be ready for a while, so I'm working with FC 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth scheme (which works fine and I'd hate to change it if I don't have to). The system has been up and functional on these versions for a couple of years, and quite stable, we just can't send if we're not local. When I do "postconf-a" it indicates cyrus and dovecot, so I take it that means Postfix has been built with sasl support. (I presume this means I don't have to compile it from source.) First Dovecot. Its set up to provide all protocols, but only imaps and pop3s have ports forwarded through the firewall. Plain-text auth is disabled, ssl is set to yes, ssl_listen is not specified, and the cert and key files are in the default locations - and work. No cipher list is used. Dovecot's chrooted. The protocol sections imap and pop3 take ALL the defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as I don't think it matters. "auth default {" has mechanisms set to cram-md5, digest-md5, plain, and login, with passdb passwd-file pointing to a file in /etc where the cram data goes. It's not using pam, and there's an OLD comment in the config: # Experience says we need an empty passdb - passwd group: which is followed by passdb passwd{}. Later, there's "userdb passwd {}. All of that was configured long ago and has been functional. The changes I've made to add sasl support primarily pertain to the "socket listen section of "auth default". There, the master section remains commented out while the client section has been uncommented, the path set to /var/spool/postfix/private/auth, mode set to 0660, and the user and group have been set to postfix. ...This is all as described here: http://www.postfix.org/SASL_README.html and http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL That's it for Dovecot. Now, to Postfix itself. >From the working environ, only listening on port 25, I simply added the following (as per directions already cited above): smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_security_options = noanonymous, noplaintext smtp_sasl_security_options = noanonymous, noplaintext smtpd_sasl_tls_security_options = noanonymous smtp_sasl_tls_security_options = noanonymous And, of course, permit_sasl_authenticated was added to smtpd_recipient_restrictions. I got the impression from the baove sources that Postfix will then use Dovecot's authentication mechanism via a socket it finds in its private/auth subdirectory. NOT documented in any of those places, someone suggested I must turn on TLS. OK... The documentation found here: http://www.postfix.org/TLS_README.html claims (intimates) that it's not possible to run a site on a self-signed certificate, however, there's ZERO budget for a signed certificate, so unless I can get one for ten bucks somewhere, that could be a deal-breaker here. However, we've been using self-signed certificates for a while now and wonder why an "exception" mechanism wouldn't exist. As that web page talks about "Netscape" I suspect it's very old and may no longer apply. In any event, I tried this, too (after trying without). On the good side, an available Android phone, previously reading fine, but unable to send, no longer complained when the setup was changed to the imap username and password, same server address, TLS security type, and the server port of 25. HOWEVER, no mail has passed through it successfully, it just gives no error whatsoever, so far, while the server's log reports "Relay access denied." Notably, when setting up TLS, Postfix complained when the smtpd_tls_key_file was incorrect, but did not complain when it was provided properly, suggesting it's reading and accepting my self-signed certificate and private key. Ideas, please?! And, by the way, what's port 465 all about? Some clients propose that's what should be used to send... Thanks in advance for your help, Richard From rtroy at ScienceTools.com Mon Mar 12 20:59:01 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 11:59:01 -0700 (PDT) Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: Message-ID: Hi All, it turned out to be the order of entries in stmpd_recipient_restrictions. Regards, Richard On Mon, 12 Mar 2012, Richard Troy wrote: > Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) > From: Richard Troy > To: postfix-users at cloud9.net, dovecot at dovecot.org > Subject: [Dovecot] Trouble adding sasl support via dovecot > > > Hello Folks, > > I've been the admin of a site that uses Postfix with Dovecot on RedHat > since, oh, gosh, maybe 1996? It's been a long time. I've never built it > from source, though, just used the rpms (and I wonder if maybe that's my > problem now). It just works, is reliable, and lets me be a very-part-time > administrator. > > Repeatedly over the last few years I've been asked to have our mail system > "join the modern age" and provide mail sending capabilities for clients > that aren't on our internal network - via their smart-phones, from home, > etc. OK... Well, way back when the site was set up, smtp servers didn't do > any kind of "auth", but along the way to solving this problem (trying to > configure pop-before-smtp, someone mentioned that Postfix now has an auth > mechanism that uses Dovecot and I should use that instead! Great! ... > Except that I've spent between 16 and 20 hours on this with no joy, and > while I hate having to ask for help, it's time to ask what things that are > obvious to the less ignorant that I must be doing wrong... Certainly, > given the solid history of Postfix and Dovecot, I must be the problem! > > My problem statement is simply, "it should be working", but doesn't, and I > don't get any announcement of "auth" when testing connections to Postfix > as per directions here: > > http://www.postfix.org/SASL_README.html#server_test > > At least I haven't broken the normal functionality! > > I'm building a new server on the latest Fedora Core (16), but it's lacking > in some hardware and won't be ready for a while, so I'm working with FC > 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth > scheme (which works fine and I'd hate to change it if I don't have to). > The system has been up and functional on these versions for a couple of > years, and quite stable, we just can't send if we're not local. > > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) > > First Dovecot. Its set up to provide all protocols, but only imaps and > pop3s have ports forwarded through the firewall. Plain-text auth is > disabled, ssl is set to yes, ssl_listen is not specified, and the cert and > key files are in the default locations - and work. No cipher list is used. > Dovecot's chrooted. The protocol sections imap and pop3 take ALL the > defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as > I don't think it matters. "auth default {" has mechanisms set to cram-md5, > digest-md5, plain, and login, with passdb passwd-file pointing to a file > in /etc where the cram data goes. It's not using pam, and there's an OLD > comment in the config: > > # Experience says we need an empty passdb - passwd group: > > which is followed by passdb passwd{}. Later, there's "userdb passwd {}. > > All of that was configured long ago and has been functional. > > The changes I've made to add sasl support primarily pertain to the "socket > listen section of "auth default". There, the master section remains > commented out while the client section has been uncommented, the path set > to /var/spool/postfix/private/auth, mode set to 0660, and the user and > group have been set to postfix. ...This is all as described here: > > http://www.postfix.org/SASL_README.html > and > http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL > > That's it for Dovecot. Now, to Postfix itself. > > >From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): > > smtpd_sasl_type = dovecot > smtpd_sasl_path = private/auth > smtpd_sasl_auth_enable = yes > broken_sasl_auth_clients = yes > smtpd_sasl_security_options = noanonymous, noplaintext > smtp_sasl_security_options = noanonymous, noplaintext > smtpd_sasl_tls_security_options = noanonymous > smtp_sasl_tls_security_options = noanonymous > > And, of course, permit_sasl_authenticated was added to > smtpd_recipient_restrictions. > > > I got the impression from the baove sources that Postfix will then use > Dovecot's authentication mechanism via a socket it finds in its > private/auth subdirectory. > > NOT documented in any of those places, someone suggested I must turn on > TLS. OK... > > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, however, there's ZERO budget for a signed certificate, so > unless I can get one for ten bucks somewhere, that could be a > deal-breaker here. However, we've been using self-signed certificates for > a while now and wonder why an "exception" mechanism wouldn't exist. As > that web page talks about "Netscape" I suspect it's very old and may no > longer apply. > > In any event, I tried this, too (after trying without). On the good side, > an available Android phone, previously reading fine, but unable to send, > no longer complained when the setup was changed to the imap username and > password, same server address, TLS security type, and the server port of > 25. HOWEVER, no mail has passed through it successfully, it just gives no > error whatsoever, so far, while the server's log reports "Relay access > denied." > > Notably, when setting up TLS, Postfix complained when the > smtpd_tls_key_file was incorrect, but did not complain when it was > provided properly, suggesting it's reading and accepting my self-signed > certificate and private key. > > Ideas, please?! > > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... > > > Thanks in advance for your help, > Richard > > -- Richard Troy, Chief Scientist Science Tools Corporation 510-717-6942 rtroy at ScienceTools.com, http://ScienceTools.com/ From terry at cnysupport.com Mon Mar 12 20:57:24 2012 From: terry at cnysupport.com (Terry Carmen) Date: Mon, 12 Mar 2012 14:57:24 -0400 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> Message-ID: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> On 03/04/2012 09:58 AM, Timo Sirainen wrote: > On 4.3.2012, at 16.48, Terry Carmen wrote: > >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >>> >>> If the prefix differs, but all of the exchange server names have >>> the same length, for example 10, you can also do: >>> >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >>> There's no otherwise nice way to parse this string. >> >> >> If by prefix, you mean the >> "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, >> they're different. > > OK, so if the prefix or suffix isn't always the same length you > can't do the above. > >> I could export the data to a text file as >> username:homeexchangeserver (or whatever other format is needed). >> >> homeservers.txt: >> user1:exch1.example.com >> user2:exch1.example.com >> user3:exch1.example.com >> user4:exch2.example.com >> >> Is it possible to do a lookup in a text file to get this? > > > If you can use userdb passwd-file and export the data to that file, > it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile > > Example line: > > user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com > > Note that you can't then return any userdb fields from passdb ldap lookup. That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. What would be perfect is if I could do something like this: //////////////////////////// http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying Proxy only server . . . In this document I assume that Dovecot is installed under /opt/dovecot, by default it is installed under /usr/local when compiling from source. Examples in this document are for MySQL but configs do not differ much with PostgreSQL. SQL table structure Create SQL table like CREATE TABLE proxy ( user varchar(255) NOT NULL, host varchar(16) default NULL, destuser varchar(255) default NULL, PRIMARY KEY (user) ); ////////////////////////////////////// All I really need is a way to lookup the user's home IMAP server when given the username, as above. Does imapproxy still support this 1.x feature? Thanks! Terry From CMarcus at Media-Brokers.com Mon Mar 12 21:02:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 12 Mar 2012 15:02:55 -0400 Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: References: Message-ID: <4F5E485F.40207@Media-Brokers.com> Since you got it working, I'll just comment on a couple of things... On Mon, 12 Mar 2012, Richard Troy wrote: > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) Correct... > From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): You really should separate AUTH to the port that is designed for it: port 587 (aka the 'submission' port/service)... just uncomment it (and its attendant lines) in master.cf > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, Where does it state any such thing? I've been using self-signed certs for 8+years with postfix... You do have to 'accept' the certs in the clients though, and that cn scare some users. I've had zero problems with this in Android, and none in recent versions of iOS, although earlier versions required you to install the cert manually (could be done using Safari on the iPhone)... Also, Outlook provides no simple way to Accept a Cert and store it permanently (Thunderbird does), so unless/until Outlook users import the Cert, they'll have to accept it each time they fire up Outlook and check mail. > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... It is the *deprecated* SMTPS (smtp over SSL). All modern clients can use the submission service, but some older versions of Outlook/Outlook Express can only use 465. It doesn't hurt anything to have it enabled, but you shoiuld absolutely tell all other clients to use the normal submissions service (STARTTLS on port 587). -- Best regards, Charles From andrei.michescu at miau.ca Tue Mar 13 07:41:53 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 01:41:53 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 Message-ID: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Hello, I'm using dovecot 2.1.1 with vpopmail 5.4.30 with multiples domains and I have problems setting up synchronization in between multiple computers. All act like master (my clients can connect to any of the them and read their emails either via POP3 either via IMAP, inbound email gets on any of the machines). Each machine is on a different continent, there is no shared drive in between and the synchronization is supposed to be asynchronous via cron scripts. To simplify the case, let's consider 2 machines (mx1.a and mx2.a) with 3 virtual domains (a and b and c). On both machine domain a is the default domain (needs only username to connect to imap). Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. I tried any of the following commands to synchronize the 2 machines: mx1.a$ doveadm -Dv sync -u user1 at a -f ssh mx2.a doveadm dsync-server -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a ssh mx2.a doveadm -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a user1 at a The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22). Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u and attempts to sync the first email in the default domain. Please advise. # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From nick.z.edwards at gmail.com Tue Mar 13 09:27:28 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 17:27:28 +1000 Subject: [Dovecot] upgrade convert omissions Message-ID: It did not convert over some things: protocol imap { listen = *:143 mail_plugins = quota imap_quota imap_client_workarounds = outlook-idle ssl_listen = *:993 } protocol pop3 { pop3_uidl_format = %f mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { mail_plugins = quota cmusieve quota_full_tempfail = no log_path = /var/log/dovecot/deliver.log deliver_log_format = msgid=%m: from=%f: %$ auth_socket_path = /var/run/dovecot/auth-master } I ended up with service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } It looks like service-foobar is replacing stuff inside protocol {} but protocol still exists in examples, kind of confusing. Oh what about service-pop3, different than examples and service-imap, where's pop3s ? I guess I'll just copy the service pop3 section from an example file. Does all this mean the protocol section is not needed? or everything inside of them I had is no longer valid? No LDA specific logging converted, it did not include the quota stuff in where it is needed (mail_plugins went MIA), but did inside converted plugin section. I did note the conversion warned that workarounds = outlook-idle is no longer needed, but nothing about the other stuff. Just concerns me if it ignored some needed stuff, what else did it ignore. Nik (who is rather reluctant to use series 2 and break everything and stay with 1.2) Below is the new conf file: auth_cache_negative_ttl = 0 auth_cache_ttl = 5 mins auth_mechanisms = plain login auth_verbose = yes disable_plaintext_auth = no first_valid_uid = 95 last_valid_uid = 95 listen = *,:: log_path = /var/log/dovecot/pop3.log login_log_format_elements = user=<%u> method=%m rip=%r %c mail_location = maildir:/vmail/%d/%n/Maildir mail_nfs_index = yes mail_nfs_storage = yes maildir_very_dirty_syncs = yes mmap_disable = yes passdb { args = /etc/dovecot-sql.conf driver = sql } plugin { quota = maildir quota_rule = *:storage=1000M quota_rule2 = Trash:storage=100M quota_rule3 = Junk:ignore quota_rule4 = Spam:ignore quota_warning = storage=90%% /usr/local/bin/quotawarn-90.sh quota_warning2 = storage=75%% /usr/local/bin/quotawarn-75.sh } pop3_lock_session = yes protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } user = vmail } service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } shutdown_clients = no ssl_cert = References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Message-ID: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> On 13.3.2012, at 7.41, Michescu Andrei wrote: > Initially I synchronize mx1.a with mx2.a using rsync. I check that I can > login using dovecot. .. > The only thing that happens is that the on each machine the folders get > doubled with some random extension (eg. Inbox becomes > Inbox_3e3ff3g3gb3bb3b22). This is kind of a feature. Currently if two mailboxes have a same name, but different GUID, dsync doesn't even try to merge them but instead renames one of them. So don't do initial sync with rsync, but with dsync. Alternatively you need to first get each mailbox assigned a GUID, for example: doveadm -A mailbox status guid '*' > Also, another bug, if there is a domain setup as default > (auth_default_realm) dsync simply ignores the specified -u and > attempts to sync the first email in the default domain. That can't be possible, something else is happening. What does dsync and auth log with debugs enabled when this happens? From tss at iki.fi Tue Mar 13 09:56:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 09:56:12 +0200 Subject: [Dovecot] upgrade convert omissions In-Reply-To: References: Message-ID: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> On 13.3.2012, at 9.27, Nick Edwards wrote: > It did not convert over some things: .. What Dovecot version did you use? In my test it converted everything (v2.0.18). Copy&pasting your config to a new file, adding ssl_cert + ssl_key and it produces output that converted everything (although cmusieve should be replaced with sieve): service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } } ssl_cert = It looks like service-foobar is replacing stuff inside protocol {} but > protocol still exists in examples, kind of confusing. Services have replaced some settings, not protocol itself. http://wiki2.dovecot.org/Services > Oh what about service-pop3, different than examples and service-imap, You had explicitly set listen/ssl_listen only for imap, not for pop3, so that's what the conversion did. pop3 uses the defaults. > where's pop3s ? It's enabled by default. But pop3s isn't a really a "protocol", so it's no longer treated specially. > I guess I'll just copy the service pop3 section from > an example file. No need to, the defaults are fine. Same with service imap actually, you could just remove it. From tss at iki.fi Tue Mar 13 10:06:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 10:06:23 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: On 12.3.2012, at 20.57, Terry Carmen wrote: >> If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile >> >> Example line: >> >> user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com >> >> Note that you can't then return any userdb fields from passdb ldap lookup. > > That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. Well, you could allow users to log in with any password and then let it just fail later at imapc login, but that's a bit ugly. You could also use passdb imap {} + userdb passwd-file {} with some extra work. The authentication would be done against the remote imap server, while the userdb_imapc_host would be looked up from the passwd-file. > What would be perfect is if I could do something like this: > > //////////////////////////// > > http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying > Proxy only server .. > All I really need is a way to lookup the user's home IMAP server when given the username, as above. > > Does imapproxy still support this 1.x feature? This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy From nick.z.edwards at gmail.com Tue Mar 13 10:15:09 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 18:15:09 +1000 Subject: [Dovecot] upgrade convert omissions In-Reply-To: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> References: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> Message-ID: On 3/13/12, Timo Sirainen wrote: > On 13.3.2012, at 9.27, Nick Edwards wrote: > >> It did not convert over some things: > .. > > What Dovecot version did you use? In my test it converted everything 1.2.17 -> 2.1.1 > > protocol pop3 { > mail_plugins = quota > pop3_client_workarounds = outlook-no-nuls oe-ns-eoh > pop3_uidl_format = %f > } > protocol lda { > auth_socket_path = /var/run/dovecot/auth-master > deliver_log_format = msgid=%m: from=%f: %$ > log_path = /var/log/dovecot/deliver.log > mail_plugins = quota cmusieve > quota_full_tempfail = no > } > >> It l any idea why it never copied over the protocol stuff? > Services have replaced some settings, not protocol itself. > http://wiki2.dovecot.org/Services thanks > >> I guess I'll just copy the service pop3 section from >> an example file. > > No need to, the defaults are fine. Same with service imap actually, you > could just remove it. OK Thanks From schut at sarvision.nl Tue Mar 13 10:46:04 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 09:46:04 +0100 Subject: [Dovecot] invalid mailbox name Message-ID: Hi, while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Migration is from dovecot 1.1.11 to 1.2.15. Thanks, Vincent. From tss at iki.fi Tue Mar 13 11:00:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:00:19 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 13.3.2012, at 10.46, Vincent Schut wrote: > Hi, > > while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: > > ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) > > I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... > > Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as "&-". From schut at sarvision.nl Tue Mar 13 11:10:48 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:10:48 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 03/13/2012 10:00 AM, Timo Sirainen wrote: > On 13.3.2012, at 10.46, Vincent Schut wrote: > >> Hi, >> >> while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: >> >> ERROR: Folder 'Organisations.RS Env& IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env& IJRS']) >> >> I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... >> >> Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? > > Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. > > Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". > Timo, thanks for the quick follow-up. However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: - when I want to create a folder with a "&" using a imap client, I have to type "&-"? or: - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) or: - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? or did you mean something else altogether ("don't use folders with & in their names, they're evil")? Vincent. From tss at iki.fi Tue Mar 13 11:15:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:15:45 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <66BBAF62-FBC2-452C-8A97-0A505515EC7E@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: >> Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. >> >> Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". >> > > Timo, > > thanks for the quick follow-up. > However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: > > - when I want to create a folder with a "&" using a imap client, I have to type "&-"? no. > or: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) no. > or: > - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? yes. > or did you mean something else altogether ("don't use folders with & in their names, they're evil")? When user types "&", the IMAP client should translate it to "&-". Also in filesystem it should be shown as "&-" (although this will be configurable in future). On input IMAP client will of course also see it as "&-" and should translate it back to "&" before making it visible to user. offlineimap works correctly in that it doesn't really need to do any kind of translation or validation, since it was originally IMAP client's fault for creating it and secondarily Dovecot's fault for allowing its creation. Although I guess offlineimap could have detected that this is an invalid mailbox name and translated it to something valid (this is how Dovecot v2.x's dsync works). From tss at iki.fi Tue Mar 13 11:20:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:20:47 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the & character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change & in there to &-, it would end up showing broken to user. Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. From schut at sarvision.nl Tue Mar 13 11:33:59 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:33:59 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> References: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> Message-ID: On 03/13/2012 10:20 AM, Timo Sirainen wrote: > On 13.3.2012, at 11.10, Vincent Schut wrote: > >> - the imap client (offlineimap in this case) should translate the "&" into"&-" on the fly (I can do that, you can give folder translation functions in offlineimap) > > Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the& character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change& in there to&-, it would end up showing broken to user. > > Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. > > Thanks Timo. Everything's clear now. There are only a few of these, I'll just rename them (and their line in the subscriptions file). Vincent. From marcio.merlone at a1.ind.br Tue Mar 13 16:36:32 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 11:36:32 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage Message-ID: <4F5F5B70.1020304@a1.ind.br> Hi people, When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is find -type d -name .Archives\* -print0 | \ while read -d $'\0' archive; do \ mv "$archive" /dead/Emails/jhon.doe/; \ ln -s "/dead/Emails/jhon.doe/$archive" .; done It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? Thanks in advance, best regards. -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 18:49:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 18:49:54 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F5B70.1020304@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> Message-ID: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> On 13.3.2012, at 16.36, Marcio Merlone wrote: > Hi people, > > When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is > > find -type d -name .Archives\* -print0 | \ > while read -d $'\0' archive; do \ > mv "$archive" /dead/Emails/jhon.doe/; \ > ln -s "/dead/Emails/jhon.doe/$archive" .; done > > It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I'm not sure what you mean by locks. dovecot-uidlist.lock? Anyway, the above is safe only if the IMAP client doesn't try to access the mailboxes during the move. Otherwise it can become confused. > I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? A perfectly working solution would be to (upgrade to v2.x and) switch to sdbox or mdbox format with alt storage enabled, then you could simply do: doveadm altmove -A mailbox 'Archives*' all From btb at bitrate.net Tue Mar 13 19:56:08 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Tue, 13 Mar 2012 13:56:08 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: <20120312133849.GS24983@harrier.slackbuilds.org> References: <20120312133849.GS24983@harrier.slackbuilds.org> Message-ID: <467D7FF4-02AB-4AB2-B1BA-26D50CC5A145@bitrate.net> On Mar 12, 2012, at 09.38, /dev/rob0 wrote: > On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: >> the problem with this is that while each of the passdb/userdb >> configs for the various protocols does indeed work, if a result >> is not found in one of them, the global passdb appears to then >> function as a catch-all. >> >> how can i tell dovecot it doesn't need a global passdb? each >> of the protocols' passdb/userdb configs is functioning as >> desired, but having dovecot look elsewhere upon failure ends >> up defeating the purpose. > > A simple workaround: use an empty passwd-file passdb as global. thanks, yeah. i've got what effectively accomplishes the same thing - a global ldap passdb within which the search filter always returns nothing. it sure seems like there would be a more logical method than this though. i looked briefly at the static password database, but at the moment it's intended application appears to be the opposite. if there were some argument such as deny=y or similar, it could be used. but ultimately, it would seem to make more sense to be able to simply tell dovecot that it doesn't need a global passdb, since obviously it doesn't. -b From andrei.michescu at miau.ca Tue Mar 13 20:22:16 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 14:22:16 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: Hello, Thank you for your reply. I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1 at b but they belong to first_user at a (which is NOT involved in this sync), BUT a is the default domain. Also for the first suggestion: 1) how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids 2) if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine Thank you, Andrei > On 13.3.2012, at 7.41, Michescu Andrei wrote: > >> Initially I synchronize mx1.a with mx2.a using rsync. I check that I can >> login using dovecot. > .. >> The only thing that happens is that the on each machine the folders get >> doubled with some random extension (eg. Inbox becomes >> Inbox_3e3ff3g3gb3bb3b22). > > This is kind of a feature. Currently if two mailboxes have a same name, > but different GUID, dsync doesn't even try to merge them but instead > renames one of them. > > So don't do initial sync with rsync, but with dsync. Alternatively you > need to first get each mailbox assigned a GUID, for example: doveadm -A > mailbox status guid '*' > >> Also, another bug, if there is a domain setup as default >> (auth_default_realm) dsync simply ignores the specified -u and >> attempts to sync the first email in the default domain. > > That can't be possible, something else is happening. What does dsync and > auth log with debugs enabled when this happens? > > > !DSPAM:4f5efb4c315461389012818! > > -------------- next part -------------- A non-text attachment was scrubbed... Name: output_doveadm.odt Type: application/vnd.oasis.opendocument.text Size: 16377 bytes Desc: not available URL: From e-frog at gmx.de Tue Mar 13 20:42:41 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 19:42:41 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors Message-ID: <4F5F9521.2060206@gmx.de> Hello Timo, I'm experimenting with 'doveadm backup' on 2.1.1 (latest hg, full dovecot -n output attached) and haven't managed to get it working. This is what I have done: 1. Create the directory /tmp/backup which is empty 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ Then I see the following errors: doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in dest (guid=9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=bd05451f2fbb574d40600000ec8d17cd) dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox 9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox bd05451f2fbb574d40600000ec8d17cd dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) It somehow finds INBOX in destination however the backup directory is newly created and empty. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: doveconf-n.txt URL: From tss at iki.fi Tue Mar 13 20:51:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 20:51:13 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> On 13.3.2012, at 20.22, Michescu Andrei wrote: > I'm attaching you the output for the 2nd bug. All the folders that you see > in there does not exists in user1 at b but they belong to first_user at a (which > is NOT involved in this sync), BUT a is the default domain. The output showed debug output from the local dsync, but not from the remote. I think you'll see that if you do: sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv dsync-server -u user1 at b Also it's possible that in v2.1.1 there was some bug related to this.. You could try the latest nightly snapshot that has several fixes related to dsync: http://www.dovecot.org/nightly/ > Also for the first suggestion: > > 1) how do you sync initially the 2 machines? Because if you create the > account on both machines, already the Inbox has 2 different guids doveadm sync should be run before the destination Maildir exists at all. If vpopmail creates that, I guess it would just have to be deleted manually.. > 2) if you know the guid, how do you change them? Because then I can do the > rsync and after I can correct the guid on the other machine The GUID is generated the first time it's used, which normally means when you run dsync for the first time. Alternatively you can also use doveadm to ask for the mailbox's GUID and it gets generated: doveadm mailbox status -u user at domain guid '*' Running rsync after this is done also copies the GUID (it's stored in dovecot-uidlist). From marcio.merlone at a1.ind.br Tue Mar 13 20:58:59 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 15:58:59 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> Message-ID: <4F5F98F3.1090601@a1.ind.br> Em 13-03-2012 13:49, Timo Sirainen escreveu: > On 13.3.2012, at 16.36, Marcio Merlone wrote: >> It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. > I'm not sure what you mean by locks. Operating system lock of open files (lsof). >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Sounds really nice. There are no 2.0 packages for Lucid tough, so will take a look at the latest stable Ubuntu, or perhaps give the beta a try (until it comes stable). Thanks for your prompt reply. :) -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 21:16:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 21:16:54 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5F9521.2060206@gmx.de> References: <4F5F9521.2060206@gmx.de> Message-ID: On 13.3.2012, at 20.42, e-frog wrote: > This is what I have done: > 1. Create the directory /tmp/backup which is empty > 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ .. > Then I see the following errors: > > doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. Try without mailbox_list_index=yes From e-frog at gmx.de Tue Mar 13 22:19:53 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 21:19:53 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: References: <4F5F9521.2060206@gmx.de> Message-ID: <4F5FABE9.3080200@gmx.de> On 13.03.2012 20:16, wrote Timo Sirainen: > On 13.3.2012, at 20.42, e-frog wrote: > >> This is what I have done: >> 1. Create the directory /tmp/backup which is empty >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > .. >> Then I see the following errors: >> >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > Try without mailbox_list_index=yes Yes, after disabling list indexes it works. From terry at cnysupport.com Tue Mar 13 23:44:40 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 17:44:40 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: <4F5FBFC8.3060306@cnysupport.com> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a regular dummy proxying setup. Sure you could still do > that, but it's not imapc proxying. > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. Thanks for the help! Terry From tss at iki.fi Wed Mar 14 00:05:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 00:05:14 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F5FBFC8.3060306@cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> Message-ID: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> On 13.3.2012, at 23.44, Terry Carmen wrote: > On 03/13/2012 04:06 AM, Timo Sirainen wrote: >> This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. If you find out that IMAP clients still don't work nicely with Exchange (apparently they have random problems, especially with shared mailboxes/accounts), you can still put imapc proxy in front of your currently working Dovecot proxy. :) From terry at cnysupport.com Wed Mar 14 00:29:09 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 18:29:09 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> Message-ID: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> ----- Message from Timo Sirainen --------- ? ? Date: Wed, 14 Mar 2012 00:05:14 +0200 ? ? From: Timo Sirainen Reply-To: Dovecot Mailing List Subject: Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location ? ? ? To: Terry Carmen ? ? ? Cc: dovecot at dovecot.org > On 13.3.2012, at 23.44, Terry Carmen wrote: >> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a >> regular dummy proxying setup. Sure you could still do that, but >> it's not imapc proxying. >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy >> The above URL worked beautifully and Dovecot is now running as a >> proxy for a dozen older Exchange servers on a private network. > If you find out that IMAP clients still don't work nicely with > Exchange (apparently they have random problems, especially with > shared mailboxes/accounts), you can still put imapc proxy in front > of your currently working Dovecot proxy. :) I'm going to hope everything is OK for a while, since my goal is to retire all the old Exchange servers and move all the users to dovecot/maildir within the next couple of months. However it's always nice to know there are options. 8-) Terry From andrei.michescu at miau.ca Wed Mar 14 07:25:09 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 01:25:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: Hello, Thank you very much... Using the nightly build and a combination of mailbox status + rsync + dsync made it happen. So the *full* procedure was: for every domain dom for every user u in dom doveadm mailbox status -u user at domain guid '*' rsync /home/vpopmail/domains/$dom/$u mx2.a:/home/vpopmail/domains/$dom/$u doveadm -Dv sync -u $u@$dom -f ssh mx2.a doveadm dsync-server -u $u@$dom loop $u loop $dom Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes). The problem comes when I start using the master-master model: emails starts getting duplicate with different ids. Here is the example: mx1.a receives an email for user1 (next line is the ls on the Maildir/new on mx1.a): -rw------- 1 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 mx2.a receives another email for user1: -rw------- 1 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx2,S\=273 cron job starts on mx1.a: sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 dsync-local(user1 at b): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at b): Info: INBOX: Couldn't keep all uids dsync-local(user1 at b): Warning: Mailbox changes caused a desync. You may want to run dsync again. --due to desync it runs again (via the cron script): sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 The expected result would be that in user1's Mailbox on both mx1.a and mx2.a there would be 2 files... which is not happening ls on mx1.a -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 -rw------- 1 vpopmail vchkpw 273 Mar 14 01:05 1331701504.32564.mx2,S\=273 -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331702193.M868989P24524.mx1,S\=278 ls on mx2.a -rw------- 1 vpopmail vchkpw 278 Mar 14 07:04 1331701451.24233.mx1,S\=278 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx1,S\=273 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331702193.M798223P32571.mx2,S\=273 As you can see on every machine the original email gets duplicated. Please advise on how I can fix this issue. Thank you and Best regards, Andrei > On 13.3.2012, at 20.22, Michescu Andrei wrote: > >> I'm attaching you the output for the 2nd bug. All the folders that you >> see >> in there does not exists in user1 at b but they belong to first_user at a >> (which >> is NOT involved in this sync), BUT a is the default domain. > > The output showed debug output from the local dsync, but not from the > remote. I think you'll see that if you do: > > sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv > dsync-server -u user1 at b > > Also it's possible that in v2.1.1 there was some bug related to this.. You > could try the latest nightly snapshot that has several fixes related to > dsync: http://www.dovecot.org/nightly/ > >> Also for the first suggestion: >> >> 1) how do you sync initially the 2 machines? Because if you create the >> account on both machines, already the Inbox has 2 different guids > > doveadm sync should be run before the destination Maildir exists at all. > If vpopmail creates that, I guess it would just have to be deleted > manually.. > >> 2) if you know the guid, how do you change them? Because then I can do >> the >> rsync and after I can correct the guid on the other machine > > The GUID is generated the first time it's used, which normally means when > you run dsync for the first time. Alternatively you can also use doveadm > to ask for the mailbox's GUID and it gets generated: > > doveadm mailbox status -u user at domain guid '*' > > Running rsync after this is done also copies the GUID (it's stored in > dovecot-uidlist). > !DSPAM:4f5f972f80146209382307! > > From CMarcus at Media-Brokers.com Wed Mar 14 12:58:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 06:58:54 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> Message-ID: <4F6079EE.4000201@Media-Brokers.com> On 2012-03-13 6:29 PM, Terry Carmen wrote: > I'm going to hope everything is OK for a while, since my goal is to retire > all the old Exchange servers and move all the users to dovecot/maildir > within the next couple of months. > > However it's always nice to know there are options. 8-) I'm currently looking at rolling out SOGo as part of a major reworking of their current infrastructure (will also include converting their old Courier-IMAP to dovecot 2.1.x among other things)... SOGo, as far as I can tell, is the best truly free and open source 'exchange clone' available that works extremely well with Thunderbird+Lightning (which is what my Client uses currently, but they are very dissatisfied with using Google Calendar for Shared calendars), Outlook and Apple Apps, as well as Android, Blackberry and Apple mobile devices - and their upcoming v2 (in beta now) will not only provide native Outlook support (no plugin needed), it will also (optionally) provide a Samba4 Active Directory server in my main Client's office - all with absolutely no licenses required. Commercial support is available from Inverse, the company created by the developers to provide said support services. I also learned something very interesting yesterday concerning SOGo and dovecot during a sales call with a SOGo rep, but I'll wait and see if Timo cares to chime in on this one... ;) -- Best regards, Charles From marcio.merlone at a1.ind.br Wed Mar 14 13:51:30 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 08:51:30 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F98F3.1090601@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> Message-ID: <4F608642.5060707@a1.ind.br> Em 13-03-2012 15:58, Marcio Merlone escreveu: > Em 13-03-2012 13:49, Timo Sirainen escreveu: >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Should this command be run every time a new folder (.Archives.2012 for example) is created or can this be automated, something like instructing deliver to do this to any Archives* is created? In other words, to cron or not to cron? Best regards. -- *Marcio Merlone* From tss at iki.fi Wed Mar 14 14:19:31 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 14:19:31 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F608642.5060707@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> Message-ID: <1331727571.2081.126.camel@innu> On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: > Em 13-03-2012 15:58, Marcio Merlone escreveu: > > Em 13-03-2012 13:49, Timo Sirainen escreveu: > >> A perfectly working solution would be to (upgrade to v2.x and) switch > >> to sdbox or mdbox format with alt storage enabled, then you could > >> simply do: doveadm altmove -A mailbox 'Archives*' all > Should this command be run every time a new folder (.Archives.2012 for > example) is created or can this be automated, something like instructing > deliver to do this to any Archives* is created? In other words, to cron > or not to cron? It's not deliver that creates it, it's the IMAP client. And it would have to be done when IMAP client copies mails there. But there's currently no easy way to automate that, so you'd need to do it in cron. Another possibility could be to add a new feature (plugin) that always immediately saves mails in Archives* mailboxes to alt storage. From tss at iki.fi Wed Mar 14 15:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 15:41:30 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5FABE9.3080200@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> Message-ID: <1331732490.2081.127.camel@innu> On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: > On 13.03.2012 20:16, wrote Timo Sirainen: > > On 13.3.2012, at 20.42, e-frog wrote: > > > >> This is what I have done: > >> 1. Create the directory /tmp/backup which is empty > >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > > .. > >> Then I see the following errors: > >> > >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > > > Try without mailbox_list_index=yes > > Yes, after disabling list indexes it works. With latest hg version it should work. From tss at iki.fi Wed Mar 14 16:09:49 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:09:49 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: <1331734189.2081.137.camel@innu> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync > keeps them in sync (cron job every 5 minutes). > > The problem comes when I start using the master-master model: emails > starts getting duplicate with different ids. I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign), but anyway: Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers. Each time dsync notices that both sides have had new mails, it needs to reassign new IMAP UIDs for the messages, which means that IMAP clients may need to redownload the mails. Better to give one MX a higher priority so mails typically are delivered through it. From tss at iki.fi Wed Mar 14 16:29:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:29:15 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: Message-ID: <1331735355.2081.140.camel@innu> On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: > Heya, > > We are expiriencing issues with dovecot 2.1.1 on Linux with weird > filenames in home directory of username. We are using mbox IMAP > folders, with no special changes (mail_location = mbox:~/:INBOX=% > h/.mailbox). > > Mar 6 13:37:17 machine dovecot: imap(username): Panic: file > mail-storage.c: line 628 (mailbox_alloc): assertion failed: > (uni_utf8_str_is_valid(vname)) .. > AFAIK, the problem lies in processing the file list of home folder, > which can contain filenames that do not have proper UTF-8 encoding of > filenames, which causes dovecot to crash. Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This should fix it by renaming such mailboxes: http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 > On the other hand, UTF-8 filenames created on the system by hand > (using touch), are not displayed in IMAP LIST command (sample is > included in the folder structure; single letter file). This is a bit trickier problem. The mailbox names are currently stored in filesystem as IMAP's modified-UTF7. So it's not really even currently supposed to work, although it's not very nice that the mailboxes aren't visible either. Maybe I'll do something smart in future for this, like allowing both mUTF-7 and UTF-8 and remembering per-mailbox which formatting it is in. From tss at iki.fi Wed Mar 14 16:40:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:40:05 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F55BA0C.5090606@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> <4F55BA0C.5090606@one.com> Message-ID: <1331736005.2081.144.camel@innu> On Tue, 2012-03-06 at 08:17 +0100, Peter Mogensen wrote: > On 2012-03-05 16:36, Timo Sirainen wrote: > >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > > > Neither behavior is wrong, just different. :) > > Ok... I were in doubt if I had missed something from the RFC. > However... for testing, I tried to create "INBOX.INBOX" on dovecot. > But then dovecot answers NO and complains that the folder already > exists. Though it's still not on disk and dovecot still doesn't list it > with "*". This is a bit problematic when you have prefix="INBOX." namespace. There if you access "INBOX.INBOX", its internal storage name is "INBOX". And INBOX's internal storage name is also "INBOX". So in some parts of the code they are treated as if they were both the same mailbox. Maybe I can get this fixed for v2.2. From campbell at cnpapers.com Wed Mar 14 16:46:58 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 10:46:58 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions Message-ID: <4F60AF62.80702@cnpapers.com> I've mostly finished a conversion from an old Centos 3 UW-Imap server to a new Centos 6 dovecot server. I did not copy the old ~/.mailboxlist file to ~/mail/.subscriptions file, but notice some users have the latter file now. These are all mbox folders on the old and new server. I'm getting ready to do the same to another old/new pair of servers and I'm wondering if there is an advantage of doing the copy. I'm assuming the .subscription files are created when they access their account through our webmail application, but I'm not sure if it was automatic or due to a "subscribe" action done manually. There are 49 accounts with a .mailboxlist file and only 4 with the new .subscriptions file. So either our webmail application isn't being used a lot or there's a problem with it due to the missing .subscriptions file, but the phone usually rings pretty quickly when problems arise. One last question, please. Over the years, some imap accounts had their folders directly in their home directory and the contents of the .mailboxlist file would have an entry with just the name of the folder in it (Trash, eg), and most had the folders in their ~/mail folder with an entry like "mail/Trash". Our webmail app, Horde/Imp, always seemed to take care of this. If I create the .subscription file for the users during the move to the new server, should I move the folders to the mail directory and amend their .subscriptions file to reflect that change on these odd ball accounts, and will that affect how their client is seeing these? The first server conversion was a bear due to my lack of dovecot knowledge. I've since learned a little more, and mostly found out that dovecot is a more complex application than the old imap application. There's so much more that can be done with dovecot, whereas the old imap server was mostly just load-and-go. Seems like no matter how much I read, the more I discovered I didn't know. Anyway, thanks for all the past help and any opinions anyone might decide to offer on this post. steve campbell From marcio.merlone at a1.ind.br Wed Mar 14 17:00:49 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 12:00:49 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <1331727571.2081.126.camel@innu> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> <1331727571.2081.126.camel@innu> Message-ID: <4F60B2A1.7060104@a1.ind.br> Em 14-03-2012 09:19, Timo Sirainen escreveu: > On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: >> Em 13-03-2012 15:58, Marcio Merlone escreveu: >>> Em 13-03-2012 13:49, Timo Sirainen escreveu: >>>> A perfectly working solution would be to (upgrade to v2.x and) switch >>>> to sdbox or mdbox format with alt storage enabled, then you could >>>> simply do: doveadm altmove -A mailbox 'Archives*' all >> Should this command be run every time a new folder (.Archives.2012 for >> example) is created or can this be automated, something like instructing >> deliver to do this to any Archives* is created? In other words, to cron >> or not to cron? > It's not deliver that creates it, it's the IMAP client. And it would I first though about deliver since this seems the guy who could do that, but don't matter. > have to be done when IMAP client copies mails there. But there's > currently no easy way to automate that, so you'd need to do it in cron. Just imagine that: protocol imap{ ... x_alternate_storage_always = Archives, Spam, Trash x_alternate_storage_size = 20MB x_alternate_storage_age = 1y ... } The client (Thunderbird?) sends imap commands to dovecot create such folder or to move a message to such folder. Dovecot obeys and check the config to see if the folder in question is one of those listed on x_alternate_storage_always. If the folder is not on alternate storage yet, create/move it there and them proceed what was asked to do. I imagine that an age based action would depend of a cron job in order to not overload the server each time it performs any imap command and check old messages age... > Another possibility could be to add a new feature (plugin) that always > immediately saves mails in Archives* mailboxes to alt storage. I don't know if the duck is male, I want the egg! :) I know nothing about the inner workings of dovecot, and very little about the outer working. A well designed and integrated plugin could be, but seems to be a nice core feature for dovecot, based on folder name, size or age. Many thanks for your time. Please be kind considering those ideas, I am just a poor man's server admin. :) Best regards. -- *Marcio Merlone* From campbell at cnpapers.com Wed Mar 14 17:48:43 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 11:48:43 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60BDDB.8000103@cnpapers.com> On 3/14/2012 10:46 AM, Steve Campbell wrote: > > > One last question, please. > > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". > Our webmail app, Horde/Imp, always seemed to take care of this. If I > create the .subscription file for the users during the move to the new > server, should I move the folders to the mail directory and amend > their .subscriptions file to reflect that change on these odd ball > accounts, and will that affect how their client is seeing these? > > The first server conversion was a bear due to my lack of dovecot > knowledge. I've since learned a little more, and mostly found out that > dovecot is a more complex application than the old imap application. > There's so much more that can be done with dovecot, whereas the old > imap server was mostly just load-and-go. Seems like no matter how much > I read, the more I discovered I didn't know. > > Anyway, thanks for all the past help and any opinions anyone might > decide to offer on this post. > > steve campbell I've discovered another situation. This may not be a problem, but I've got to deal with it at any rate. I find that some users have a .mailboxlist which points to folders in their home directory, and have folders in their mail directory as well. For the most part, this situation involved horde/imp "sent-mail" folders which are created when users send mail through our webmail but they more than likely have a client on either their phone or desktop that is configured as imap. The horde/imp "sent-mail" is not listed in their .mailboxlist file. So I'm guessing this will be all right to leave as is or to modify the resultant .subscription file to point to moved folders. So many things to consider for so many different situations. Thanks steve > > From terry at cnysupport.com Wed Mar 14 17:59:26 2012 From: terry at cnysupport.com (Terry Carmen) Date: Wed, 14 Mar 2012 11:59:26 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F60C05E.1090803@cnysupport.com> On 03/14/2012 06:58 AM, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. Looks interesting. I have currently have horde/imp/kronolith running with postfix/dovecot/mysql on the back end and it's been working nicely with all the clients and devices except for outlook. I'll have to take a look at sogo, because I'd really like to keep outlook for the users that want it, to cut down on support and complaints. Thanks! Terry From arnaud.abelard at univ-nantes.fr Wed Mar 14 18:06:32 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Wed, 14 Mar 2012 17:06:32 +0100 Subject: [Dovecot] RECENT status always 0 Message-ID: <4F60C208.6010304@univ-nantes.fr> Hello, we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) then 2 minutes later: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) How can the UNSEEN count change without the RECENT count changing accordingly? Thanks in advance, Arnaud -- Arnaud Ab?lard (jabber: arnaud.abelard at univ-nantes.fr) Administrateur Syst?me - Responsable Services Web Direction des Syst?mes d'Informations Universit? de Nantes - ne pas utiliser: trapemail at univ-nantes.fr From tss at iki.fi Wed Mar 14 18:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:27:09 +0200 Subject: [Dovecot] RECENT status always 0 In-Reply-To: <4F60C208.6010304@univ-nantes.fr> References: <4F60C208.6010304@univ-nantes.fr> Message-ID: <1AE988E0-F799-45AA-A098-A3462DC13340@iki.fi> On 14.3.2012, at 18.06, Arnaud Ab?lard wrote: > Hello, > > we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) > > then 2 minutes later: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) > > How can the UNSEEN count change without the RECENT count changing accordingly? If any client has the INBOX opened, the recent count for other connections stays at 0. I guess you're expecting recent flags to work differently than how IMAP RFC specifies them. (It's entirely possible that Courier implemented them in the wrong way.) From jom at grosjo.net Wed Mar 14 18:36:55 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 22:21:55 +0545 Subject: [Dovecot] FTS crash Message-ID: Hi Timo, I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. When I remove the plugin from the dovecot.conf, all works fine. No debug available here, but let me know how can I help you. Joan From andrei.michescu at miau.ca Wed Mar 14 18:45:07 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 12:45:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Hello, Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment. Servers are having different priorities, but this was a basic scenario to test the master-master synchronization. Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster... One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options: - the DB model = each server has a set of ids that can give (either ranges, either increment with step different then 1) - the vpopmail/qmail model = append the server name (as you saw in the previous email in the listings the email files contained the hostname mx1.a and mx2.a) Thank you, Andrei > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync >> keeps them in sync (cron job every 5 minutes). >> >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Each time dsync > notices that both sides have had new mails, it needs to reassign new > IMAP UIDs for the messages, which means that IMAP clients may need to > redownload the mails. Better to give one MX a higher priority so mails > typically are delivered through it. > > > > !DSPAM:4f60a6b137151972926802! > > From tss at iki.fi Wed Mar 14 18:56:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:56:58 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: On 14.3.2012, at 18.45, Michescu Andrei wrote: > Nope dsync was not running during the email delivery on that account. I've > simulated in a controlled environment. How? You mean simply deliver mail to server A and to server B and run dsync and it duplicates it? I can't reproduce it that way, only if I run dsync during a flood of new mails. > Think that for incoming SMTP I can even restrict which server is the > master (forcing all other to redeliver to this one). BUT, for a > distributed IMAP cluster there is no way to restrict users to perform > changes on only one server. This would defeat the model and the purpose of > a distributed cluster... For IMAP it's not much of a problem, because user typically still uses only one client actively, so clients aren't uploading mails to multiple servers at the same time. > One idea might be to have the IDs dependent on server where they appear > first time so that they keep the ID once they get replicated. Here there > are many options: The messages have GUIDs that stay the same always, but IMAP UIDs are required to be ascending from client's point of view, and several clients rely on that, so when UID conflict happens the only safe thing to do is to assign new UIDs for all of the conflicting mails. From tss at iki.fi Wed Mar 14 18:58:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:58:36 +0200 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: On 14.3.2012, at 18.36, Joan Moreau wrote: > I updated my 2.1 from HG, and now the FTS (Squat) plugin > makes a segmentation fault. To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > No debug available here, but let me know > how can I help you. gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From CMarcus at Media-Brokers.com Wed Mar 14 19:00:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 13:00:25 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60CEA9.3080008@Media-Brokers.com> On 2012-03-14 10:46 AM, Steve Campbell wrote: > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes... dovecot doesn't like it when stuff other than mail is in the home folder: http://wiki2.dovecot.org/VirtualUsers/Home -- Best regards, Charles From andrei.michescu at miau.ca Wed Mar 14 19:26:42 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 13:26:42 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: <2c95e6565a8b3783b05b2eafc2d4833e.squirrel@web.miau.ca> > On 14.3.2012, at 18.45, Michescu Andrei wrote: > >> Nope dsync was not running during the email delivery on that account. >> I've >> simulated in a controlled environment. > > How? You mean simply deliver mail to server A and to server B and run > dsync and it duplicates it? I can't reproduce it that way, only if I run > dsync during a flood of new mails. > YES. simply deliver mail to server A and then to server B (to the same user_1). After run dsync and you get exactly what you saw in my previous email. That's why I included the ls for the both servers, so that you can see the email files too. because each server duplicates only its own email (so brings the email from the other server and duplicates its own email). >> Think that for incoming SMTP I can even restrict which server is the >> master (forcing all other to redeliver to this one). BUT, for a >> distributed IMAP cluster there is no way to restrict users to perform >> changes on only one server. This would defeat the model and the purpose >> of >> a distributed cluster... > > For IMAP it's not much of a problem, because user typically still uses > only one client actively, so clients aren't uploading mails to multiple > servers at the same time. > hehe... one would think so, but when you have road-warriors that roam you can not insure that the server where they connect for IMAP (closest based on geo-ip) is the same as the server that you have picked for inbound SMTP. So you already have 2 servers that mess-up with user's mailbox. The second case where you can not control this is for mobile devices that flip-in/out of wi-fi (my iPhone is in Canada when it is on 3G and in Europe when it is WiFi due to vpn tunneling, and this can change every couple of minutes... :( ) >> One idea might be to have the IDs dependent on server where they appear >> first time so that they keep the ID once they get replicated. Here there >> are many options: > > The messages have GUIDs that stay the same always, but IMAP UIDs are > required to be ascending from client's point of view, and several clients > rely on that, so when UID conflict happens the only safe thing to do is to > assign new UIDs for all of the conflicting mails. well I don't know much about IMAP standard (you guys are the experts :)! here). If the GUID stays the same then this can be used to prevent the duplication error. Also, as you can detect if the email is new or not (a client has already seen it or not): in the case that no one has seen it then it is safe to assign any UIDs that fits. In case that on only one server it has been seen then you can give it that UIDs on all servers, and reassign all the unseen ones. So the only messed-up case is if on both servers the message has been seen with different UIDs :( Thank you very much for your time and patience. I know that our setup is pretty atypical. And think that this model with only 2 servers I'm showing you is only for simplicity as the real deployment has multiple servers geographically sparse connected by slow intercontinental internet links... :)) Otherwise we'd use a distribute file system and have only a unified storage :P Best regards. Andrei From jom at grosjo.net Wed Mar 14 19:31:47 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 23:16:47 +0545 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: I have been unlucky in deed. Problem solved with recent changes Le 14/03/2012 22:43, Timo Sirainen a ?crit : > On 14.3.2012, at 18.36, Joan Moreau wrote: > >> I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. > > To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > >> No debug available here, but let me know how can I help you. > > gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From e-frog at gmx.de Wed Mar 14 21:33:49 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 14 Mar 2012 20:33:49 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <1331732490.2081.127.camel@innu> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> Message-ID: <4F60F29D.2010409@gmx.de> On 14.03.2012 14:41, wrote Timo Sirainen: > On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: >> On 13.03.2012 20:16, wrote Timo Sirainen: >>> On 13.3.2012, at 20.42, e-frog wrote: >>> >>>> This is what I have done: >>>> 1. Create the directory /tmp/backup which is empty >>>> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>> .. >>>> Then I see the following errors: >>>> >>>> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>>> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. >>> >>> Try without mailbox_list_index=yes >> >> Yes, after disabling list indexes it works. > > With latest hg version it should work. > Hi Timo, The "can't delete mailbox INBOX" error is gone now with changeset c077ca9bc306 and it's working successfully on the account from yesterday where it also worked with mailbox_list_index=no. However using a different account (more mail and mailboxes) I'm seeing dbox corruption errors. I have tested with mailbox_list_index=yes and no and it's the same for both. So this might be unrelated to this setting. Attached are logs from doveadm backup runs. First to an empty directory and 2 consecutive runs. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: log.txt URL: From campbell at cnpapers.com Wed Mar 14 21:53:00 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 15:53:00 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60CEA9.3080008@Media-Brokers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> Message-ID: <4F60F71C.8090306@cnpapers.com> On 3/14/2012 1:00 PM, Charles Marcus wrote: > On 2012-03-14 10:46 AM, Steve Campbell wrote: >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > > Yes... dovecot doesn't like it when stuff other than mail is in the > home folder: > > http://wiki2.dovecot.org/VirtualUsers/Home I'm not sure these are virtual users, so that link may have confused me. All accounts on these servers have real unix accounts. Their inbox is /var/spool/mail/unix-user-name. Their imap folders, the ones that they create using an imap client or webmail, are either in ~ or ~/mail. Their original .mailboxlist is always in ~. Based on that, I should probably copy any imap folders not in ~/mail to that folder, duplicate ~/.mailboxlist to the file ~/mail/.subscriptions, and amend any .subscriptions file contents to just have the name of the folders (without any "mail/folder" reference in it). My example would then be as follows /home/steve = folder /home/steve/Drafts = original folder /home/steve/AnyFolder = original folder /home/steve/.mailboxlist = original file /home/steve/mail = folder (either original or created) /home/steve/mail/.subscriptions = copied contents of .mailboxlist file /home/steve/mail/Drafts = copied folder of original /home/steve/mail/AnyFolder = copied folder of original Contents of original .mailboxlist and new .subscriptions: Drafts AnyFolder If the imap folders were in ~/mail, then the original .mailboxlist would have been mail/Drafts mail/AnyFolder but after the corrections to the .subscriptions file, they would be as above (without reference to the mail folder). Is this correct? thanks for the help steve From trashcan at odo.in-berlin.de Wed Mar 14 22:36:30 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 21:36:30 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Hi -- On 14.03.2012, at 15:09, Timo Sirainen wrote: > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), Good to hear ;-) > but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ... > Better to give one MX a higher priority so mails typically are delivered > through it. That's what I did. Now dsync/replicator is performing great, if the mail volume is rather low. I'm very satisfied, because this is the best performance ever. (Before I was running unison and dsync 2.0.) But, whenever the high priority server will show delays during stress situations like huge mail loads, the low priority server will receive loads of mails as well. A dsync/replicator setup will then most probably produce duplicates (and multiples). That is a rather unrealistic scenario for my little severs, but others might have more difficulties. And spammers don't care about mx priorities at all :-( Regards, Michael From tss at iki.fi Wed Mar 14 23:14:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 23:14:10 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: On 14.3.2012, at 22.36, Michael Grimm wrote: > And spammers don't care about mx priorities at all :-( But spams go to spam mailbox where duplicates don't really matter. :) From trashcan at odo.in-berlin.de Wed Mar 14 23:26:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:26:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Hi - On 14.03.2012, at 22:14, Timo Sirainen wrote: > On 14.3.2012, at 22.36, Michael Grimm wrote: >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) True ;-) But spam mails might interfere with syncing of legitimate mail arriving at the same time. Regards, Michael From trashcan at odo.in-berlin.de Wed Mar 14 23:30:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:30:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Message-ID: <5EE60F3E-A25F-4E6D-98E9-2246B75ACA10@odo.in-berlin.de> Hi -- On 14.03.2012, at 22:26, Michael Grimm wrote: > But spam mails might interfere with syncing of legitimate > mail arriving at the same time. Forget about that part, I was wrong because duplicates are produced in corresponding mailboxes, only. Sorry for the noise, Michael From andrei.michescu at miau.ca Wed Mar 14 23:32:40 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 17:32:40 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: > On 14.3.2012, at 22.36, Michael Grimm wrote: > >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) In an ideal world yes... or no. In our deployment spam is simply header tagged and left in INBOX. Each user can decide after what they want to do with it (client side rules). And, in the end, it is the same discussion, because the spam mailbox get replicated too and if the spam gets duplicated we are in the worst scenarios... knowing that spam represents 95% of all email traffic (in a real-world public-facing system). ;) hehe... in the meanwhile I looked a little on the ietf and there are different RFCs out there on disconnected clients and UIDPLUS and other nice features ;) let me know if you are interested to get some help in implementing it :D From andrei at lctax.ro Wed Mar 14 22:58:09 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Wed, 14 Mar 2012 16:58:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> > Hi -- > > On 14.03.2012, at 15:09, Timo Sirainen wrote: >> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > >>> The problem comes when I start using the master-master model: emails >>> starts getting duplicate with different ids. > >> I was testing this a bit, and I guess in your tests dsync was running >> during a mail delivery, which seems to make it duplicate mails >> sometimes. I'll probably fix this at some point (I've actually been >> thinking about a larger dsync redesign), > > Good to hear ;-) > >> but anyway: >> >> Even if dsync worked perfectly and didn't duplicate mails, it's not a >> great idea to do deliver mails randomly to both servers. > > Sometimes croncobs are running on both servers at the same time > producing locally delivered mails simultaneously, though. Ok, one > can modify run times accordingly ... Why do you run the crontab on all the servers? You can run a start-based system where only one ("main"-master) syncs all the other masters. And like this you avoid the time synch'ing of crontabs (especially if you don't always know how longer it will take for a dsync to finish). > >> Better to give one MX a higher priority so mails typically are delivered >> through it. > > That's what I did. Now dsync/replicator is performing great, if the > mail volume is rather low. I'm very satisfied, because this is the > best performance ever. (Before I was running unison and dsync 2.0.) > > But, whenever the high priority server will show delays during stress > situations like huge mail loads, the low priority server will receive > loads of mails as well. A dsync/replicator setup will then most probably > produce duplicates (and multiples). That is a rather unrealistic > scenario for my little severs, but others might have more difficulties. This is another nice case to "motivate" Timo to look for solutions ;) I tried to push 2 ideas in the same direction earlier :P Especially that he confirmed that every single email has a unique GID (which should help prevent duplication/multiplication)... > > And spammers don't care about mx priorities at all :-( Actually, statistically speaking, spammers select the low priority ones. > > Regards, > Michael > Nice to hear that we are not the only ones out there to try to run something like this over dovecot :P Thnx, Andrei From trashcan at odo.in-berlin.de Wed Mar 14 23:51:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:51:22 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> Message-ID: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Hi -- On 14.03.2012, at 21:58, Michescu Andrei wrote: >> Sometimes croncobs are running on both servers at the same time >> producing locally delivered mails simultaneously, though. Ok, one >> can modify run times accordingly ... > > Why do you run the crontab on all the servers? You can run a start-based > system where only one ("main"-master) syncs all the other masters. You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times. >> And spammers don't care about mx priorities at all :-( > > Actually, statistically speaking, spammers select the low priority ones. Actually: you are right ;-) > Nice to hear that we are not the only ones out there to try to run > something like this over dovecot :P Yes. I never loved the idea of a clusterfs for my small mail servers, I always considered such clusterfs an overkill. Well, my servers do reside in the same housing building, thus it could be done without performance loss. But a scenario of worldwide distributed mail servers desires a dsync/replicator scheme, IMHO ;-) Regards, Michael From jtam.home at gmail.com Thu Mar 15 01:33:23 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 14 Mar 2012 16:33:23 -0700 (PDT) Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: Steve Campbell writes: > Their imap folders, the ones that they create using an imap client or > webmail, are either in ~ or ~/mail. Their original .mailboxlist is > always in ~. Based on that, I should probably copy any imap folders not > in ~/mail to that folder, duplicate ~/.mailboxlist to the file > ~/mail/.subscriptions, and amend any .subscriptions file contents to > just have the name of the folders (without any "mail/folder" reference > in it). > > My example would then be as follows > > /home/steve = folder > /home/steve/Drafts = original folder > /home/steve/AnyFolder = original folder > /home/steve/.mailboxlist = original file > /home/steve/mail = folder (either original or created) > /home/steve/mail/.subscriptions = copied contents of .mailboxlist > file > /home/steve/mail/Drafts = copied folder of original > /home/steve/mail/AnyFolder = copied folder of original > > Contents of original .mailboxlist and new .subscriptions: > > Drafts > AnyFolder > > If the imap folders were in ~/mail, then the original .mailboxlist would > have been > > mail/Drafts > mail/AnyFolder > > but after the corrections to the .subscriptions file, they would be as > above (without reference to the mail folder). > > Is this correct? That depends -- are you aliasing namespaces so that prefix={"", "mail/", etc.} all map to a user's ~/mail folder? You may be creating a confusing situation where a client with a null IMAP prefix has 2 copies of a mailbox. Joseph Tam From kgc at corp.sonic.net Thu Mar 15 01:51:38 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Wed, 14 Mar 2012 16:51:38 -0700 Subject: [Dovecot] Just in time AV scanning Message-ID: <20120314235138.GE39671@corp.sonic.net> I'm curious if anyone has any plugins for AV integration directly into dovecot. Our old pop servers have been scanning messges as they're moved from new->cur in the inbox and, at least where user's aren't poping every few seconds, there is occasionally enough time between scanning through the MXs to message retreval to snag a few more virues with updated definitions before they reach customers. Anyone doing anything similar? -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtl+dovecot at uvm.edu Thu Mar 15 03:24:34 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 14 Mar 2012 21:24:34 -0400 Subject: [Dovecot] director lmtp -> smtp problem Message-ID: <4F6144D2.2080900@uvm.edu> Hi Timo & Dovecot users, We have a 2-node director setup which front-ends for 4 nodes which share a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. Approximately 40k users, but typically only a few thousand active at any time. The director nodes run sendmail, which deliver mail "locally" using LMTP to the director, which then feeds to SMTP on the real servers (also sendmail.) Why sendmail? Because procmail is used for mail filtering and as the delivery agent. Here's the problem, on the director: Mar 14 20:40:08 imapdir2 dovecot: lmtp(10692): Connect from local Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input ->eof) Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3d99a) [0x7f79156c499a] -> /usr/lib/doveco t/libdovecot.so.0(+0x3d9e6) [0x7f79156c49e6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f791569df8f] -> dovecot/lmtp() [0x406e77] -> /usr/l ib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f79156d0044] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f79156d 0c3b] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f79156cfca8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7915 6bdfc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7914ef8cdd] -> dovecot/lmtp() [0x403d69] Mar 14 20:40:38 imapdir2 sendmail[6905]: q2D8KodI018432: SYSERR(root): timeout writing message to localhost: Broken pipe Most mail goes through OK, but some messages do not and end up queued until they run into the queue time limit. So far as I have been able to tell, all of the messages have this failure when the following conversation takes place between sendmail (on director), the Dovecot LMTP proxy, and sendmail on the backend node (SMTP): (names mangled to protect the guilty) (first, sendmail -> director LMTP) > [root at imapdir2 ~]# sendmail -qIq2EFZt1p004708 -v > > Running /var/spool/mqueue/qd2/q2EFZt1p004708 (sequence 1 of 1) > ... Connecting to > /var/lib/dovecot/lmtp-socket via cyrusv2... > 220 imapdir2.uvm.edu Dovecot LMTP ready > >>> LHLO imapdir2.uvm.edu > 250-imapdir2.uvm.edu > 250-8BITMIME > 250-ENHANCEDSTATUSCODES > 250 PIPELINING > >>> MAIL From: > 250 2.1.0 OK > >>> RCPT To: > >>> DATA > 250 2.1.5 OK > 354 OK > timeout writing message to localhost: Broken pipe > ... Deferred > Closing connection to localhost The conversation between the director (LMTP) and the backend (sendmail SMTP) goes like this: > 250-penguinc.uvm.edu Hello imapdir2.uvm.edu [132.198.100.150], pleased > to meet you > 250-ENHANCEDSTATUSCODES > 250-PIPELINING > 250-8BITMIME > 250-SIZE 10485760 > 250-ETRN > 250-AUTH DIGEST-MD5 CRAM-MD5 > 250-DELIVERBY > 250 HELP > MAIL FROM: > 250 2.1.0 ... > Sender ok > RCPT TO: > 552 5.2.2 User ntssdfwe mailbox is full At this point Dovecot should return the failed RCPT TO: status back to sendmail over LMTP, but instead it sits there (waiting for a timeout to expire?) and eventually dies. doveconf -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.2.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.2 (Santiago) base_dir = /var/run/dovecot/ default_client_limit = 6000 default_process_limit = 10240 director_mail_servers = penguina.uvm.edu penguinb.uvm.edu penguinc.uvm.edu penguind.uvm.edu director_servers = imapdir1.uvm.edu imapdir2.uvm.edu lmtp_proxy = yes login_trusted_networks = [REDACTED] passdb { args = proxy=y nopassword=y protocol=smtp driver = static } service anvil { client_limit = 40000 } service auth { client_limit = 45960 unix_listener auth-userdb { group = mail mode = 0660 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director service_count = 0 } service imap { process_limit = 10240 vsz_limit = 1 G } service lmtp { client_limit = 1 inet_listener lmtp { port = 24 } unix_listener /var/lib/dovecot/lmtp-socket { group = root mode = 0600 user = root } } service pop3-login { executable = pop3-login director service_count = 0 } service pop3 { process_limit = 5000 } shutdown_clients = no ssl_cert = <[REDACTED].pem ssl_key = <[REDACTED].key userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol lmtp { auth_socket_path = director-userdb } protocol imap { mail_max_userip_connections = 100 } Hope you can help, Jim Lawson From alexis.lelion at gmail.com Thu Mar 15 11:48:52 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 10:48:52 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules Message-ID: Hello, In my current setup, I have a spam filter upstream that adds a specific header - X-Spam-Level on every incoming mail. Based on this level, the mail will be moved to the user spam folder using sieve by doing "fileinto :create 'spam';" Unfortunately, some legitimate email may end up in this spam folder, so I have kind of a whitelist that performs an explicit keep over specific trusted domains. So, my complete spam filtering rule is : if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ]{ ??? keep; elseif header :contains "X-Spam-Level" ["0","1","2"] { ??? fileinto :create "__spam__"; } This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf This works as expected except that it doesn't take into account users filtering for domains that were matched for the explicit keep. For example, I have the following rule : if address :domain "From" "trusted.tld" { ??? fileinto "trusted" } But mail coming from that domain are still delivered in my mailbox. Is there something I'm missing here? I guess yes, otherwise it would work as I want ^_^ Any help/comment is appreciated Thanks! Alexis From tss at iki.fi Thu Mar 15 12:02:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:02:16 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F6144D2.2080900@uvm.edu> References: <4F6144D2.2080900@uvm.edu> Message-ID: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Hi, On 15.3.2012, at 3.24, Jim Lawson wrote: > We have a 2-node director setup which front-ends for 4 nodes which share > a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. .. > Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > (proxy->data_input > ->eof) I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. From paul at actionlans.com Thu Mar 15 02:29:19 2012 From: paul at actionlans.com (paul) Date: Thu, 15 Mar 2012 10:59:19 +1030 Subject: [Dovecot] firefox dovecot-sieve Message-ID: <1331771362.1955.1.camel@paul15.localdomain> Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org On 14.3.2012, at 4.45, paul wrote: > HI. I have just started to play with sieve and everything seems ok when > logging on using telnet localhost 4190 and an encoded username/password. > If I try to connect with firefox at localhost:4190 I get > "IMPLEMENTATION" "Dovecot Pigeonhole" > "SIEVE" "fileinto reject envelope encoded-character vacation subaddress > comparator-i;ascii-numeric relational regex imap4flags copy include > variables body enotify environment mailbox date ihave" > "NOTIFY" "mailto" > "SASL" "PLAIN LOGIN" > "STARTTLS" > "VERSION" "1.0" > OK "Dovecot ready." > NO "Error in MANAGESIEVE command received by server." > NO "Error in MANAGESIEVE command received by server." > NO "Invalid characters in atom" > BYE "Too many invalid MANAGESIEVE commands." > my dovecot -n shows > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.42.9-1.fc15.i686.PAE i686 Fedora release 15 (Lovelock) > auth_mechanisms = plain login > disable_plaintext_auth = no > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mail_privileged_group = mail > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date ihave > mbox_write_locks = fcntl > passdb { > driver = pam > } > plugin { > sieve = ~/.dovecot.sieve > sieve_dir = ~/sieve > } > protocols = imap pop3 lmtp sieve > service managesieve-login { > inet_listener sieve { > port = 4190 > } > } > ssl_cert = ssl_key = userdb { > driver = passwd > } > Have I missed something obvious or does Firefox clash with Managesieve? > Thanks. Paul > From aydin.demirel at endersys.com Thu Mar 15 12:09:11 2012 From: aydin.demirel at endersys.com (=?UTF-8?B?QXlkxLFuIERlbWlyZWw=?=) Date: Thu, 15 Mar 2012 12:09:11 +0200 Subject: [Dovecot] Login Failed Message-ID: <4F61BFC7.1000607@endersys.com> Hi; We are using scripts for login successes. Is there a feature for login failed status or can it be developed? Regards -- *Ayd?n Demirel Endersys Ltd. Sistem Destek M?hendisi/ System Support Engineer* * *<> Phone : +90 216 470 9423 | GSM : +90 530 401 8203 Fax : +90 216 470 9508 | Web : http://www.endersys.com Blog : http://blog.endersys.com Twitter : http://www.twitter.com/endersys LPI : The #1 Linux Certification for IT Professionals LPI (Linux Professional Institute) Turkey http://www.lpi-turkey.com From CMarcus at Media-Brokers.com Thu Mar 15 12:20:38 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:20:38 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60F71C.8090306@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> <4F60F71C.8090306@cnpapers.com> Message-ID: <4F61C276.90309@Media-Brokers.com> On 2012-03-14 3:53 PM, Steve Campbell wrote: > I'm not sure these are virtual users, so that link may have confused me. > All accounts on these servers have real unix accounts. Their inbox is > /var/spool/mail/unix-user-name. Doesn't matter, the same thing applies... don't put mail directly in their 'home' folder, put it in a subfolder (ie, /home/user/mail)... You *will* have problems if you leave those as they are... -- Best regards, Charles From tss at iki.fi Thu Mar 15 12:29:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:29:55 +0200 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <1331807395.10319.3.camel@innu> On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: > I've mostly finished a conversion from an old Centos 3 UW-Imap server to > a new Centos 6 dovecot server. This is messy stuff to do. There are ways you could make Dovecot behave identically to UW-IMAP (mail_full_filesystem_access=yes), but for future and for security it's better if you don't do that. > I did not copy the old ~/.mailboxlist > file to ~/mail/.subscriptions file, but notice some users have the > latter file now. These are all mbox folders on the old and new server. Copying it for users who haven't already readded their subscriptions would be a good idea. > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes, move all of the mboxes to mail/ directory. With the compatibility namespaces it should work so that clients don't notice changes: http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility There are also a few old mailing list threads detailing all kinds of issues and solutions related to UW-IMAP -> Dovecot migration.. From tss at iki.fi Thu Mar 15 12:31:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:31:40 +0200 Subject: [Dovecot] Login Failed In-Reply-To: <4F61BFC7.1000607@endersys.com> References: <4F61BFC7.1000607@endersys.com> Message-ID: <1331807500.10319.4.camel@innu> On Thu, 2012-03-15 at 12:09 +0200, Ayd?n Demirel wrote: > We are using scripts for login successes. Is there a feature for login > failed status or can it be developed? Login failures are only visible in auth and login processes. Probably better to implement it in auth process. And there it depends on what passdb you use. You could for example switch to passdb checkpassword, which allows you to easily run scripts for both success and failure. From tss at iki.fi Thu Mar 15 12:33:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:33:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <20120314235138.GE39671@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> Message-ID: <1331807624.10319.6.camel@innu> On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: > I'm curious if anyone has any plugins for AV integration directly into > dovecot. > > Our old pop servers have been scanning messges as they're moved from > new->cur in the inbox and, at least where user's aren't poping every > few seconds, there is occasionally enough time between scanning through > the MXs to message retreval to snag a few more virues with updated > definitions before they reach customers. > > Anyone doing anything similar? http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a script that modifies a mail while it's being read. You could make it run a virus check, and if that happens you could change the virus MIME part to be full of spaces (better not to change message size, line count or MIME structure). From stephan at rename-it.nl Thu Mar 15 12:43:05 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 11:43:05 +0100 Subject: [Dovecot] firefox dovecot-sieve In-Reply-To: <1331771362.1955.1.camel@paul15.localdomain> References: <1331771362.1955.1.camel@paul15.localdomain> Message-ID: <4F61C7B9.9050605@rename-it.nl> Op 3/15/2012 1:29 AM, paul schreef: > Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org > > On 14.3.2012, at 4.45, paul wrote: > >> HI. I have just started to play with sieve and everything seems ok when logging on using telnet localhost 4190 and an encoded username/password. If I try to connect with firefox at localhost:4190 I get >> "IMPLEMENTATION" "Dovecot Pigeonhole" >> "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave" >> "NOTIFY" "mailto" >> "SASL" "PLAIN LOGIN" >> "STARTTLS" >> "VERSION" "1.0" >> OK "Dovecot ready." >> NO "Error in MANAGESIEVE command received by server." >> NO "Error in MANAGESIEVE command received by server." >> NO "Invalid characters in atom" >> BYE "Too many invalid MANAGESIEVE commands." >> Have I missed something obvious or does Firefox clash with Managesieve? Yes you have :). Firefox speaks HTTP (and quite a few other protocols), but not ManageSieve. You'll need to run a Sieve editor on your webserver if you want to edit Sieve scripts using your browser. Regards, Stephan. From Ralf.Hildebrandt at charite.de Thu Mar 15 12:44:21 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 11:44:21 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? Message-ID: <20120315104421.GW21113@charite.de> Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:47:26 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:47:26 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:51:01 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:51:01 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 during that time, I wasn't able to access the mailbox using imap. I then issued doveadm kick username and all over sudden the mailbox was accessible (via IMAP) # doveadm kick username kicked connections from the following users: username # /usr/local/scripts/find_abnormal_imap Mar 15 11:38:48 postamt dovecot: imap: Warning: Killed with signal 15 (by pid=24545 uid=0 code=kill) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 12:47:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:47:07 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315104421.GW21113@charite.de> References: <20120315104421.GW21113@charite.de> Message-ID: <1331808427.10319.7.camel@innu> On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > during that time, I wasn't able to access the mailbox using imap. I > then issued Maildir? > doveadm kick username > > and all over sudden the mailbox was accessible (via IMAP) So one of them had the INBOX locked. Do you have pop3_lock_session=yes? From CMarcus at Media-Brokers.com Thu Mar 15 12:51:11 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:51:11 -0400 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F502485.9070503@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> Message-ID: <4F61C99F.2040409@Media-Brokers.com> On 2012-03-01 8:38 PM, Stan Hoeppner wrote: > Get yourself a qualified network architect. Pay for a full network > traffic analysis. He'll attach sniffers at multiple points in your > network to gather traffic/error/etc data. Then you'll discuss the new > office, which employees/types with move there, and you'll be able to > know almost precisely the average and peak bandwidth needs over the MAN > link. He'll very likely tell you the same thing I have, that a single > gigabit MAN link is plenty. If you hire him to do the work, he'll > program the proper QOS setup to match the traffic patterns gleaned from > the sniffers. Finally had time to properly review your answers here Stan. The time you took for the in-depth reply is very much appreciated - and I'm sure you got a kick out of the level of my ignorance... ;) As for hiring a network architect, I will absolutely be doing as you recommend (was already planning on it), but with the information I'm now armed with, at least I'll have a better chance of knowing if they know what they are doing/talking about... I'm still planning for the two physical servers (one at each location), but you have convinced me that trying to run two live mail systems is an unnecessary and even unwanted level of complexity. The DC VM will still be hot (it is always best to have two DCs in a windows domain environment anyway) so I'll get automatic real time off site backup of all of the users data (since it will all be on DFS), but for the mail services, I'll just designate one as live, and one as the hot/standby that is kept in sync using dsync. This way I'll automatically get off site back up for each site for the users data stored in the DFS, and have a second mail system ready to go if something happens to the primary. Again, thanks Stan... I am constantly amazed at the level of expertise and quality of advice available *for free* in the open source world, as is available on these lists. -- Best regards, Charles From Ralf.Hildebrandt at charite.de Thu Mar 15 13:01:18 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 12:01:18 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331808427.10319.7.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> Message-ID: <20120315110118.GX21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > > during that time, I wasn't able to access the mailbox using imap. I > > then issued > > Maildir? Yep. > > doveadm kick username > > > > and all over sudden the mailbox was accessible (via IMAP) > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > Yes. # makes Dovecot lock the mailbox for the whole session pop3_lock_session = yes Shouldn't be doing this I guess? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 13:04:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 13:04:43 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315110118.GX21113@charite.de> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> Message-ID: <1331809483.10319.9.camel@innu> On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > Yes. > # makes Dovecot lock the mailbox for the whole session > pop3_lock_session = yes > > Shouldn't be doing this I guess? If you do it then a single POP3 session can keep the mailbox locked pretty much forever. If you don't do it, you're violating POP3 RFC, but I don't think anyone really cares about that.. I guess this setting should really use a separate POP3-only lock when it's enabled. From stephan at rename-it.nl Thu Mar 15 13:11:16 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 12:11:16 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: Message-ID: <4F61CE54.4010607@rename-it.nl> Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > Hello, > > In my current setup, I have a spam filter upstream that adds a > specific header - X-Spam-Level on every incoming mail. Based on this > level, the mail will be moved to the user spam folder using sieve by > doing "fileinto :create 'spam';" > Unfortunately, some legitimate email may end up in this spam folder, > so I have kind of a whitelist that performs an explicit keep over > specific trusted domains. So, my complete spam filtering rule is : require ["fileinto", "mailbox"]; if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { keep; } elsif header :contains "X-Spam-Level" ["0","1","2"] { fileinto :create "__spam__"; } Fixed a few syntax issues there before I could test this. > This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is > my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf What version are you using? The above statement hints that it is recent, probably Dovecot v2.1 with matching Pigeonhole. > This works as expected except that it doesn't take into account users > filtering for domains that were matched for the explicit keep. For > example, I have the following rule : > if address :domain "From" "trusted.tld" { > fileinto "trusted" > } > But mail coming from that domain are still delivered in my mailbox. At my end, this is correctly delivered in the "trusted" folder, provided that this folder exists. Are you sure that the user's personal script even executes correctly? For example, the above script omits a ';'. The script also fails when there is n no "trusted" folder. Check the log files for errors. The default action in the event of an error is to store the message in INBOX, which may well be what you're seeing here. Regards, Stephan. From CMarcus at Media-Brokers.com Thu Mar 15 13:21:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 07:21:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Message-ID: <4F61D0A3.7070503@Media-Brokers.com> On 2012-03-14 5:51 PM, Michael Grimm wrote: > You misunderstood. I was referring to system cronjob's mail reports > from cron.daily jobs like security reports et al. Those reports > normally run at identical times. But are these really 'duplicate' mails? It sounds to me like they are individual to each system. I'm also confused - are you actually delivering the exact *same* mail to two (or multiple) *different* servers simultaneously? If only one copy of the mail gets delivered, regardless of which server it gets delivered to, when dsync runs, there would be no duplicates, right? I'm asking for clarification because I was considering a similar setup. -- Best regards, Charles From campbell at cnpapers.com Thu Mar 15 13:21:17 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 07:21:17 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: <4F61D0AD.1050705@cnpapers.com> On 3/14/2012 7:33 PM, Joseph Tam wrote: > Steve Campbell writes: > >> Their imap folders, the ones that they create using an imap client or >> webmail, are either in ~ or ~/mail. Their original .mailboxlist is >> always in ~. Based on that, I should probably copy any imap folders not >> in ~/mail to that folder, duplicate ~/.mailboxlist to the file >> ~/mail/.subscriptions, and amend any .subscriptions file contents to >> just have the name of the folders (without any "mail/folder" reference >> in it). >> >> My example would then be as follows >> >> /home/steve = folder >> /home/steve/Drafts = original folder >> /home/steve/AnyFolder = original folder >> /home/steve/.mailboxlist = original file >> /home/steve/mail = folder (either original or created) >> /home/steve/mail/.subscriptions = copied contents of .mailboxlist >> file >> /home/steve/mail/Drafts = copied folder of original >> /home/steve/mail/AnyFolder = copied folder of original >> >> Contents of original .mailboxlist and new .subscriptions: >> >> Drafts >> AnyFolder >> >> If the imap folders were in ~/mail, then the original .mailboxlist would >> have been >> >> mail/Drafts >> mail/AnyFolder >> >> but after the corrections to the .subscriptions file, they would be as >> above (without reference to the mail folder). >> >> Is this correct? > > That depends -- are you aliasing namespaces so that prefix={"", > "mail/", etc.} all map to a user's ~/mail folder? You may be creating a > confusing situation where a client with a null IMAP prefix has 2 copies > of a mailbox. > > Joseph Tam I have the following set: mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u namespace { type = private separator = / prefix = "#mbox/" location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } namespace { type = private separator = / prefix = mail/ hidden = yes list = no # for v1.1+ } namespace { type = private separator = / prefix = ~/mail/ hidden = yes list = yes # for v1.1+ location = mbox:~/mail:INBOX=/var/mail/%u } namespace { type = private separator = / prefix = ~%u/mail/ hidden = yes list = no # for v1.1+ } These are mostly what's defined as the "Backward Compatability" namespaces in the wiki. Are you saying that I should probably have something like the following then: namespace { type = private separator = / prefix = location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } And is the multiple "inbox = yes" in the differing namespaces a no-no? Based on the comments in the 10-mail.conf file, it seems to say it is a problem, but if a user has any prefix defined, even the blank prefix, wouldn't that mean they use only that set of parameters defined in the namespace being used? So far, I've only changed one prefix in the building to the #mbox prefix and that was because of the weird layout of files they had. I'm hoping one day to understand all of this. Dovecot, as I stated before, is much more complex that the imap server used previously. It allows one to use all of the facilities of the imap protocol, and much more, but unfortunately, for admins like me that are just moving to these new imap servers, most of those extras were either unknown to me or unused. Again, thanks all for the patience and help. steve From alexis.lelion at gmail.com Thu Mar 15 13:42:14 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 12:42:14 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61CE54.4010607@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> Message-ID: Hello Stephan, Thanks for your answer, and sorry for forgetting to specify which dovecot version I was using :-/ I'm using Dovecot 2.0.15, with PigeonHole. The syntax issues are some typos I made while writing this email, I double checked, and indeed, my production script was slightly different from what I wrote in the first place. I can confirm that the scripts compile properly with sievec, and also that the folder does exist, but just to be sure this is not an issue, I added the ":create" option to the user's fileinto. I have no errors in my logs, the only thing displayed is tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' Is there any way to increase verbosity for sieve only? Thanks On Thu, Mar 15, 2012 at 12:11 PM, Stephan Bosch wrote: > Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > >> Hello, >> >> In my current setup, I have a spam filter upstream that adds a >> specific header - X-Spam-Level on every incoming mail. Based on this >> level, the mail will be moved to the user spam folder using sieve by >> doing "fileinto :create 'spam';" >> Unfortunately, some legitimate email may end up in this spam folder, >> so I have kind of a whitelist that performs an explicit keep over >> specific trusted domains. So, my complete spam filtering rule is : > > require ["fileinto", "mailbox"]; > > if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { > ? ?keep; > } elsif header :contains "X-Spam-Level" ["0","1","2"] { > ? ?fileinto :create "__spam__"; > } > > Fixed a few syntax issues there before I could test this. > > >> This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is >> my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf > > What version are you using? The above statement hints that it is recent, > probably Dovecot v2.1 with matching Pigeonhole. > > >> This works as expected except that it doesn't take into account users >> filtering for domains that were matched for the explicit keep. For >> example, I have the following rule : >> if address :domain "From" "trusted.tld" { >> ? ? fileinto "trusted" >> } >> But mail coming from that domain are still delivered in my mailbox. > > > At my end, this is correctly delivered in the "trusted" folder, provided > that this folder exists. Are you sure that the user's personal script even > executes correctly? For example, the above script omits a ';'. The script > also fails when there is n no "trusted" folder. Check the log files for > errors. The default action in the event of an error is to store the message > in INBOX, which may well be what you're seeing here. > > Regards, > > Stephan. From jtl+dovecot at uvm.edu Thu Mar 15 13:50:22 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 07:50:22 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Message-ID: <4F61D77E.8020805@uvm.edu> On 3/15/12 6:02 AM, Timo Sirainen wrote: > Hi, > > On 15.3.2012, at 3.24, Jim Lawson wrote: >> We have a 2-node director setup which front-ends for 4 nodes which share >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > .. >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >> (proxy->data_input >> ->eof) > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > I'll give it a shot. For the purposes of doing a rolling upgrade, is it reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for the duration, or should I split-brain them during the upgrade? Jim From Ralf.Hildebrandt at charite.de Thu Mar 15 14:00:40 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 13:00:40 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331809483.10319.9.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> <1331809483.10319.9.camel@innu> Message-ID: <20120315120040.GA21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > > > Yes. > > # makes Dovecot lock the mailbox for the whole session > > pop3_lock_session = yes > > > > Shouldn't be doing this I guess? > > If you do it then a single POP3 session can keep the mailbox locked > pretty much forever. If you don't do it, you're violating POP3 RFC, but > I don't think anyone really cares about that.. Indeed. All I care about is that the user gets his/her mail. Which he didn't. I disabled it. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From campbell at cnpapers.com Thu Mar 15 14:15:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 08:15:39 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <1331807395.10319.3.camel@innu> References: <4F60AF62.80702@cnpapers.com> <1331807395.10319.3.camel@innu> Message-ID: <4F61DD6B.2020606@cnpapers.com> On 3/15/2012 6:29 AM, Timo Sirainen wrote: > On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: >> I've mostly finished a conversion from an old Centos 3 UW-Imap server to >> a new Centos 6 dovecot server. > This is messy stuff to do. There are ways you could make Dovecot behave > identically to UW-IMAP (mail_full_filesystem_access=yes), but for future > and for security it's better if you don't do that. > >> I did not copy the old ~/.mailboxlist >> file to ~/mail/.subscriptions file, but notice some users have the >> latter file now. These are all mbox folders on the old and new server. > Copying it for users who haven't already readded their subscriptions > would be a good idea. > >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > Yes, move all of the mboxes to mail/ directory. With the compatibility > namespaces it should work so that clients don't notice changes: > > http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility > > There are also a few old mailing list threads detailing all kinds of > issues and solutions related to UW-IMAP -> Dovecot migration.. I'd replied to an earlier thread, and in it, I'd asked a question about a "blank" prefix namespace and the backward compatability namespaces. I'm not sure whether my "mail_location" takes precedence over namespaces (with or without a "location" parm), especially since I don't define a "blank" prefix defined. It's been working, or at least I'm not getting calls, so maybe I'm OK. In any event, I believe if I move all of these folders to ~/mail, ensure the .subscriptions file is matching, that at least people using Thunderbird will re-read the file and set their folders properly. Not sure about other clients. Thanks for the help. steve > > From tss at iki.fi Thu Mar 15 14:22:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:22:11 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: <1331814131.10319.15.camel@innu> On Thu, 2012-03-08 at 21:36 +0200, Timo Sirainen wrote: > On 8.3.2012, at 21.18, Markus Petri wrote: > > > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > > once per user sharing an folder in LIST "" "%". > > > > I also noticed, that with 2.1 the user folder (Shared/) is no > > longer tagged as \NoSelect. > > > > Is this the intended behaviour and mutt simply cannot cope with it or > > is it a dovecot problem? > > Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even > if it did. This is a bit difficult to fix. I'll probably leave it until v2.2. > Also Dovecot probably should add \Noselect, especially if the mailbox > isn't really selectable (there's some weirdness between shared/user > being equal to shared/user/INBOX, but I'm not sure what to do about > it). These should fix this: http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 From tss at iki.fi Thu Mar 15 14:25:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:25:12 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F61D77E.8020805@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> Message-ID: <1331814312.10319.18.camel@innu> On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: > On 3/15/12 6:02 AM, Timo Sirainen wrote: > > Hi, > > > > On 15.3.2012, at 3.24, Jim Lawson wrote: > >> We have a 2-node director setup which front-ends for 4 nodes which share > >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > > .. > >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > >> (proxy->data_input > >> ->eof) > > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > > > I'll give it a shot. For the purposes of doing a rolling upgrade, is it > reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for > the duration, or should I split-brain them during the upgrade? I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The current hg version has some extra features, but it doesn't use them until all of the directors have upgraded to the new version. From tss at iki.fi Thu Mar 15 14:58:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:58:06 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: Message-ID: <1331816286.10319.23.camel@innu> On Mon, 2012-03-05 at 00:32 +0000, Andreas M. Kirchwitz wrote: > Thanks for this patch. I've applied it to the dovecot-20120303 > nightly snapshot. The good news is, compilation works fine. > The bad news is, the libraries and binaries don't work because > they don't find the custom SSL libraries. .. > $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch > $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install You would have needed to run autogen.sh again. It works with me now that I tried in a test server with OpenSSL in non-standard dir. From mhlavink at redhat.com Thu Mar 15 15:34:32 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Thu, 15 Mar 2012 14:34:32 +0100 Subject: [Dovecot] dovecot and systemd Message-ID: <4F61EFE8.1000901@redhat.com> Hi all, dovecot supports systemd socket activation. Together with standard unit activation (like old sysv init script), there are two ways how to configure dovecot(only interface:port, not whole configuration). This can result in situation where those configurations does not say the same. Question is what should happen then? For example, lets have dovecot configured to listen for imap(s) and lets have systemd dovecot socket configured to listen for all protocols - pop3(s) and imap(s). When dovecot is configured to start on boot, systemd will start it and dovecot will listen on imap(s) ports. But when dovecot.socket is enabled, it'll listen on pop3(s) too and when new pop3 connection comes, it'll pass it to dovecot and dovecot will serve it. The question is: Should this happen? What exactly should happen when dovecot.conf does not match dovecot.socket configuration? Michal From markus at mpetri.org Thu Mar 15 15:46:27 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 15 Mar 2012 14:46:27 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <1331814131.10319.15.camel@innu> References: <20120308201812.2932e90c@legolas.home.ceotex.de> <1331814131.10319.15.camel@innu> Message-ID: <20120315144627.6173dc44@legolas.home.ceotex.de> On Thu, 15 Mar 2012 14:22:11 +0200 Timo Sirainen wrote: > > Also Dovecot probably should add \Noselect, especially if the > > mailbox isn't really selectable (there's some weirdness between > > shared/user being equal to shared/user/INBOX, but I'm not sure what > > to do about it). > > These should fix this: > > http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c > http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 > Yes, those fix the problem. Thanks. From trashcan at odo.in-berlin.de Thu Mar 15 15:46:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 14:46:56 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61D0A3.7070503@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> Message-ID: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Hi -- On 15.03.2012 12:21, Charles Marcus wrote: > On 2012-03-14 5:51 PM, Michael Grimm > wrote: >> You misunderstood. I was referring to system cronjob's mail reports >> from cron.daily jobs like security reports et al. Those reports >> normally run at identical times. > > But are these really 'duplicate' mails? It sounds to me like they are > individual to each system. > > I'm also confused - are you actually delivering the exact *same* mail > to two (or multiple) *different* servers simultaneously? If only one > copy of the mail gets delivered, regardless of which server it gets > delivered to, when dsync runs, there would be no duplicates, right? Well, let me explain it in more detail: Given there are two servers called mx1 and mx2. They both have cron.daily jobs running, and let's say those cronjobs are meant to create at 3:00 a postfix-logwatch report on every server. Thus, the cronjob at mx1 sends his final report to the admin of mx1, and the one at mx2 to the admin of mx2. I happen to be the one who will finally receive those reports, and therefore I did tell sieve to drop them into some folder of mine, let's say REPORTS. Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report". I had dsync running every minute. Thus at 3:00 the final sync has been initiated, and at 3:01 dsync will find two mails to sync in REPORTS. In 99.9% of all synchronizations the final result at both server's REPORTS mailbox is as expected and as follows: mx1-report 3:01 mx2-report 3:01 But occasionally, and what I refer to as duplicates, I did find either ... mx1-report 3:01 mx1-report 3:01 mx2-report 3:01 ... or ... mx1-report 3:01 mx2-report 3:01 mx2-report 3:01 Actually, that was when I started to investigate how dsync will behave when many mails arrive at two servers simultaneously with identical final mailboxes. The day I switched to the new replicator/dsync technique, those duplicates are history, but I'm still able to produce duplicates (and multiples) if I simultaneously produce *many* mails at every server with identical mailbox destinations in a minute (see my other report a couple of days ago). Timo is suspecting the combination of arriving mails while running dsync to be a possible cause of such duplicates, if I didn't get him wrong. Again, if your servers aren't receiving loads of mails for the very same mailboxes within very short time, the current dsync/replicator works great. HTH and regards, Michael From tss at iki.fi Thu Mar 15 15:59:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 15:59:38 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <1331819978.10319.29.camel@innu> On Mon, 2012-03-05 at 18:01 -0800, Joseph Tam wrote: > On Sun, 4 Mar 2012, Timo Sirainen writes: > > > > I would like to run various doveadm commands that involves all (mail) users like > > > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > > > but any doveadm command that uses "-A" to iterate through all users will > > > stop processing at the first account with UID > > > What userdb are you using? userdb passwd should already skip users that > > aren't in the valid range. And what Dovecot version are you using? > > passwd-file under dovecot 2.0.16. Ah. The skipping only works in v2.1. Also you mean you're using passwd-file for /etc/passwd? You shouldn't really be doing that. From CMarcus at Media-Brokers.com Thu Mar 15 16:04:01 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 10:04:01 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4F61F6D1.9010703@Media-Brokers.com> On 2012-03-15 9:46 AM, Michael Grimm wrote: > Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder > REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the > mailfolder REPORTS. Important: both mails are different but they arrive > in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. > And, let's call the report from mx1 cronjob "mx1-report" and that from > mx2 "mx2-report". so these are LOCAL mails delivered to local user accounts? The easiest thing to do for this is simply alias the local address(es) so that they all go to one single server/account (I would use only virtual, but you can do it with system accounts too). I see lots of potential problems doing it the way you are doing it. -- Best regards, Charles From tss at iki.fi Thu Mar 15 16:05:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:05:29 +0200 Subject: [Dovecot] dovecot and systemd In-Reply-To: <4F61EFE8.1000901@redhat.com> References: <4F61EFE8.1000901@redhat.com> Message-ID: <1331820329.10319.32.camel@innu> On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: > What exactly should happen when > dovecot.conf does not match dovecot.socket configuration? Dovecot's systemd code was written by one of you Redhat guys. I had some similar thoughts when I applied the patch, but didn't really know what to do about it, so I didn't do anything. So: I don't know. Maybe some other project has solved this somehow already? Dovecot anyway needs its own internal UNIX listeners. Should all internal inet listeners be disabled? Could Dovecot somehow talk to systemd and ask what listeners it's using for Dovecot and log warnings if they don't match? From stephan at rename-it.nl Thu Mar 15 16:17:55 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 15:17:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: <4F61CE54.4010607@rename-it.nl> Message-ID: <4F61FA13.5060204@rename-it.nl> On 3/15/2012 12:42 PM, Alexis Lelion wrote: > Hello Stephan, > > Thanks for your answer, and sorry for forgetting to specify which > dovecot version I was using :-/ > I'm using Dovecot 2.0.15, with PigeonHole. > > The syntax issues are some typos I made while writing this email, I > double checked, and indeed, my production script was slightly > different from what I wrote in the first place. I can confirm that the > scripts compile properly with sievec, and also that the folder does > exist, but just to be sure this is not an issue, I added the ":create" > option to the user's fileinto. > > I have no errors in my logs, the only thing displayed is > tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into > mailbox 'INBOX' > > Is there any way to increase verbosity for sieve only? You can test Sieve outside normal delivery using the sieve-test tool; include the global sieve_before script using a -s argument. Alternatively, you can use the vnd.dovecot.debug extension as follows: require ["fileinto", "mailbox", "vnd.dovecot.debug"]; if address :domain "From" "trusted.tld" { fileinto :create "trusted"; debug_log "Tried to save in \"trusted\""; } You need to add the vnd.dovecot.debug extension to sieve_extensions in your 90-sieve.conf, e.g.: sieve_extensions = +vnd.dovecot.debug This will produce the following output in the user's personal sieve log (typically ~/.dovecot.sieve.log): sieve: info: started log at Mar 15 15:13:29. main_script: line 5: info: DEBUG: Tried to save in "trusted". info: msgid=unspecified: stored mail into mailbox 'trusted'. If the DEBUG line is missing at your end, the fileinto is not executed at all. If it is, and things are still delivered in INBOX, something else is going on. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 15 16:24:01 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 15:24:01 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61F6D1.9010703@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4F61F6D1.9010703@Media-Brokers.com> Message-ID: Hi -- On 15.03.2012 15:04, Charles Marcus wrote: > On 2012-03-15 9:46 AM, Michael Grimm > wrote: >> Thus, at 3:01 one report from mx1 will be delivered at mx1 into >> mailfolder >> REPORTS and at 3:01 one report from mx2 will be delivered at mx2 >> into the >> mailfolder REPORTS. Important: both mails are different but they >> arrive >> in the mailfolder REPORTS at the same time, one at mx1 the other at >> mx2. >> And, let's call the report from mx1 cronjob "mx1-report" and that >> from >> mx2 "mx2-report". > > so these are LOCAL mails delivered to local user accounts? All locally produced mails are aliased to the very same virtual user, namely myself. > The easiest thing to do for this is simply alias the local > address(es) > so that they all go to one single server/account (I would use only > virtual, but you can do it with system accounts too). That is exactly what I'm doing, I'm running virtual, only. No local user accounts here. Every locally produced system mail end in virtual mailboxes of myself. In the given example "mx1-report" is delivered to REPORTS at mx1 and "mx2-report" to REPORTS at mx2. Now, I want to access them via IMAP for instance at my mx1 mail account. Without dsync I would only be able to access "mx1-report", thus I do need to sync REPORTS to see both at mx1. > I see lots of potential problems doing it the way you are doing it. Hmm, now, I don't understand you. Regards, Michael From alexis.lelion at gmail.com Thu Mar 15 16:38:55 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 15:38:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61FA13.5060204@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> <4F61FA13.5060204@rename-it.nl> Message-ID: Thanks for this useful information, I will give it a try On Thu, Mar 15, 2012 at 3:17 PM, Stephan Bosch wrote: > On 3/15/2012 12:42 PM, Alexis Lelion wrote: >> >> Hello Stephan, >> >> Thanks for your answer, and sorry for forgetting to specify which >> dovecot version I was using :-/ >> I'm using Dovecot 2.0.15, with PigeonHole. >> >> The syntax issues are some typos I made while writing this email, I >> double checked, and indeed, my production script was slightly >> different from what I wrote in the first place. I can confirm that the >> scripts compile properly with sievec, and also that the folder does >> exist, but just to be sure this is not an issue, I added the ":create" >> option ?to the user's fileinto. >> >> I have no errors in my logs, the only thing displayed is >> tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into >> mailbox 'INBOX' >> >> Is there any way to increase verbosity for sieve only? > > > You can test Sieve outside normal delivery using the sieve-test tool; > include the global sieve_before script using a -s argument. > > Alternatively, you can use the vnd.dovecot.debug extension as follows: > > require ["fileinto", "mailbox", "vnd.dovecot.debug"]; > > > if address :domain "From" "trusted.tld" { > ?fileinto :create "trusted"; > ?debug_log "Tried to save in \"trusted\""; > } > > You need to add the vnd.dovecot.debug extension to sieve_extensions in your > 90-sieve.conf, e.g.: > > sieve_extensions = +vnd.dovecot.debug > > This will produce the following output in the user's personal sieve log > (typically ~/.dovecot.sieve.log): > > sieve: info: started log at Mar 15 15:13:29. > main_script: line 5: info: DEBUG: Tried to save in "trusted". > info: msgid=unspecified: stored mail into mailbox 'trusted'. > > If the DEBUG line is missing at your end, the fileinto is not executed at > all. If it is, and things are still delivered in INBOX, something else is > going on. > > Regards, > > Stephan. > > From tss at iki.fi Thu Mar 15 16:53:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:53:53 +0200 Subject: [Dovecot] v2.1.2 released Message-ID: <1331823233.10319.40.camel@innu> http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig There are a ton of proxying related improvements in this release. You should now be able to do pretty much anything you want with Dovecot proxy/director. This release also includes the initial version of dsync-based replication. I'm already successfully using it for @dovecot.fi mails, but it still has some problems. See http://dovecot.org/list/dovecot/2012-March/064243.html for some details how to configure it. + Initial implementation of dsync-based replication. For now this should be used only on non-critical systems. + Proxying: POP3 now supports sending remote IP+port from proxy to backend server via Dovecot-specific XCLIENT extension. + Proxying: proxy_maybe=yes with host= (instead of IP) works now properly. + Proxying: Added auth_proxy_self setting + Proxying: Added proxy_always extra field (see wiki docs) + Added director_username_hash setting to specify what part of the username is hashed. This can be used to implement per-domain backends (which allows safely accessing shared mailboxes within domain). + Added a "session ID" string for imap/pop3 connections, available in %{session} variable. The session ID passes through Dovecot IMAP/POP3 proxying to backend server. The same session ID is can be reused after a long time (currently a bit under 9 years). + passdb checkpassword: Support "credentials lookups" (for non-plaintext auth and for lmtp_proxy lookups) + fts: Added fts_index_timeout setting to abort search if indexing hasn't finished by then (default is to wait forever). - doveadm sync: If mailbox was expunged empty, messages may have become back instead of also being expunged in the other side. - director: If user logged into two directors while near user expiration, the directors might have redirected the user to two different backends. - imap_id_* settings were ignored before login. - Several fixes to mailbox_list_index=yes - Previous v2.1.x didn't log all messages at shutdown. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tss at iki.fi Thu Mar 15 17:23:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 17:23:50 +0200 Subject: [Dovecot] v2.0.19 released Message-ID: <1331825030.10319.42.camel@innu> http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz.sig Hopefully one of the last v2.0.x releases. - IMAP: ENABLE CONDSTORE/QRESYNC + STATUS for a mailbox might not have seen latest external changes to it, like new mails. - imap_id_* settings were ignored before login. - doveadm altmove did too much work sometimes, retrying moves it had already done. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tom at talpey.com Thu Mar 15 18:04:44 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 12:04:44 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 Message-ID: <4F62131C.2090008@talpey.com> I'm seeing a few warnings emitted when building for x86. They're pretty obvious, but if you want the configure options etc, I can provide those. In Dovecot 2.1.2 (I also see some of these in 2.1.1): 1) src/lib-index/mail-cache-fields.c (comparison between two last_used fields) mail-cache-fields.c: In function 'mail_cache_header_fields_read': mail-cache-fields.c:406: warning: comparison between signed and unsigned 2) src/director/user-directory.c (comparison with ioloop_time) user-directory.c: In function 'user_directory_user_is_recently_updated': user-directory.c:147: warning: comparison between signed and unsigned 3) src/replication/replicator/replicator-brain.c (comparison with ioloop_time) replicator-brain.c: In function 'doveadm_replicate': replicator-brain.c:113: warning: comparison between signed and unsigned 4) src/replication/replicator/replicator-queue.c (comparison with ioloop_time) replicator-queue.c: In function 'replicator_queue_pop': replicator-queue.c:201: warning: comparison between signed and unsigned In Pigeonhole 0.3.0: 5) src/managesieve-login/client-authenticate.c (passing size_t * not uoff_t *) client-authenticate.c: In function 'managesieve_client_auth_read_response': client-authenticate.c:214: warning: passing argument 3 of 'i_stream_get_size' from incompatible pointer type From tss at iki.fi Thu Mar 15 18:25:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 18:25:21 +0200 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F62131C.2090008@talpey.com> References: <4F62131C.2090008@talpey.com> Message-ID: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> On 15.3.2012, at 18.04, Tom Talpey wrote: > I'm seeing a few warnings emitted when building for x86. They're pretty > obvious, but if you want the configure options etc, I can provide those. > > In Dovecot 2.1.2 (I also see some of these in 2.1.1): Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. From dluke at geeklair.net Thu Mar 15 18:33:20 2012 From: dluke at geeklair.net (Daniel J. Luke) Date: Thu, 15 Mar 2012 12:33:20 -0400 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: On Mac OS X 10.5.8 / darwin 9.8.0, I'm getting this error on startup again: dovecot[74267]: master: Fatal: kevent(EV_ADD, READ, 19) failed: Invalid argument dovecot.conf contains: service stats { fifo_listener stats-mail { mode = 0 } } which fixed the issue with 2.1.1 adding the following seems to have fixed things: service aggregator { fifo_listener replication-notify-fifo { mode = 0 } } from looking at config/all-settings.c it looks like I should maybe also add the following (but I have not tried it). service director { fifo_listener login/proxy-notify { mode = 0 } } It would be really nice if this failed more gracefully so the config tweaks weren't necessary. (I can work on a patch if it's something that would be accepted and if someone can point me in the right direction). -- Daniel J. Luke +========================================================+ | *---------------- dluke at geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ From mcazzador at gmail.com Thu Mar 15 18:42:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 17:42:03 +0100 Subject: [Dovecot] replication howto Message-ID: Hello, excuse me but there is some documentation about replication now? I dont' understand where i must put the lines below (dovecot.conf? , 20-imap?) Excuse but it's not so clear for me cause i'm a new dovecot user. Another question, i use virtual users on mysql backend , so for replication i need to give ssh at every virtual users? Or i can use a only use a system ssh user? Thank's service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From trashcan at odo.in-berlin.de Thu Mar 15 19:09:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:09:21 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi -- On 15.03.2012, at 17:42, Matteo Cazzador wrote: > Hello, excuse me but there is some documentation about replication now? Not that I'm aware of. > I dont' understand where i must put the lines below (dovecot.conf? , > 20-imap?) You can put them wherever you wish, as long as you include that part of your configuration. Myself, I'm still using a single dovecot.conf, only. > Another question, i use virtual users on mysql backend , so for > replication i need to give ssh at every virtual users? > Or i can use a only use a system ssh user? If I'm not mistaken, you can use a single ssh user, and you could use the vmail user for instance. That's what I do, and I'm using sqlite for userdb. Here's my configuration: ----------------------------------------------------------------------- If you choose to run ssh on a different port from the default one, you need: ## ssh command line used in dsync replication (ssh port added) # dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} If not, you can start here: ## --- DSYNC REPLICATION ---------------------------------------- # # aggregator, replicator, doveadm, and config needed, and # dsync_remote_cmd if running ssh via non-default port # service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } The following part is for server 1, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server2.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } The following part is for server 2, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server1.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } HTH, Michael From mcazzador at gmail.com Thu Mar 15 19:16:17 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:16:17 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi, thank's a lot! for your detailed answer. About ssh (excuse for my english) i think you correctly understand what is "my problem" with virtual user (i have no system user ) and there are not ssh account. So i must use a dedicate account for replication (ssh) that must act sync for all virtual mail account. Thank' s i try you suggest now! Il 15 marzo 2012 18:09, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 17:42, Matteo Cazzador wrote: > >> Hello, excuse me but there is some documentation about replication now? > > Not that I'm aware of. > >> I dont' understand where i must put the lines below (dovecot.conf? , >> 20-imap?) > > You can put them wherever you wish, as long as you include that part > of your configuration. Myself, I'm still using a single dovecot.conf, > only. > >> Another question, i use virtual users on mysql backend , so for >> replication i need to give ssh at every virtual users? >> Or i can use a only use a system ssh user? > > If I'm not mistaken, you can use a single ssh user, and you could use > the vmail user for instance. That's what I do, and I'm using sqlite for > userdb. > > Here's my configuration: > ----------------------------------------------------------------------- > > If you choose to run ssh on a different port from the default one, you need: > > ? ## ssh command line used in dsync replication (ssh port added) > ? # > ? dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > > > If not, you can start here: > > ? ## --- DSYNC REPLICATION ---------------------------------------- > ? # > ? # aggregator, replicator, doveadm, and config needed, and > ? # dsync_remote_cmd if running ssh via non-default port > ? # > ? service aggregator { > ? ? ? ?# give enough permissions for mail processes > ? ? ? ?# > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? } > ? service replicator { > ? ? ? ?# start replication at startup > ? ? ? ?# > ? ? ? ?process_min_avail = 1 > ? } > ? service doveadm { > ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail > ? ? ? ?# (not root) > ? ? ? ?# > ? ? ? ?user = vmail > ? } > ? service config { > ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm > ? ? ? ?# > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > ? } > > > > The following part is for server 1, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server2.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > > > The following part is for server 2, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server1.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > HTH, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jtl+dovecot at uvm.edu Thu Mar 15 19:23:01 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:23:01 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <1331814312.10319.18.camel@innu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> Message-ID: <4F622575.7050405@uvm.edu> On 3/15/12 8:25 AM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: >> On 3/15/12 6:02 AM, Timo Sirainen wrote: >>> Hi, >>> >>> On 15.3.2012, at 3.24, Jim Lawson wrote: >>>> We have a 2-node director setup which front-ends for 4 nodes which share >>>> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. >>> .. >>>> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >>>> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >>>> (proxy->data_input >>>> ->eof) >>> I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. >>> >> I'll give it a shot. For the purposes of doing a rolling upgrade, is it >> reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for >> the duration, or should I split-brain them during the upgrade? > I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The > current hg version has some extra features, but it doesn't use them > until all of the directors have upgraded to the new version. > Trying with v2.1.2 (peer is v2.0.18): Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left == NULL && dir->right == NULL)) Mar 15 13:15:53 imapdir2 dovecot: director: Fatal: master: service(director): child 513 killed with signal 6 (core not dumped) Mar 15 13:15:53 imapdir2 dovecot: director: Error: Director 132.198.100.149:9090/right disconnected Which is OK, I can run them split-brained (rules in iptables to prevent directors from talking) while I move users around. It'll mean poor performance for GFS for the duration, but that's better than an outage. The good news is, the lmtp problem I wrote about above appears to be fixed. Thanks !!! Jim From trashcan at odo.in-berlin.de Thu Mar 15 19:28:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:28:37 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:16, Matteo Cazzador wrote: > with virtual user (i have no system user ) and there are not ssh > account. So i must use a dedicate account for replication (ssh) > that must act sync for all virtual mail account. Yes, that's what I use. I did create a dedicated account for vmail with all the necessary ssh stuff in ~vmail/.ssh One remark I forgot to mention in my last mail: >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> # >> user = vmail >> } This part is only needed, if you choose to run device doveadm as user vmail like I do. >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> # >> unix_listener config { >> user = vmail >> } >> } Regards, Michael From andrei at lctax.ro Thu Mar 15 19:49:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 13:49:58 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: > The day I switched to the new replicator/dsync technique, those > duplicates > are history, but I'm still able to produce duplicates (and multiples) > if Hello, Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated. I only do replication using the doveadm sync command. My servers are geographically distributed as you might remember from previous posts so I run doveadm every 5 minutes, and only 1 instance of doveadm runs at any given times (so let's say that due to a HUGE volume the doveamd take 30 minutes to complete, then all in-between 5minutes are skipped). Thnx, Andrei From tss at iki.fi Thu Mar 15 19:52:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:52:58 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F622575.7050405@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> Message-ID: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> On 15.3.2012, at 19.23, Jim Lawson wrote: >> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >> current hg version has some extra features, but it doesn't use them >> until all of the directors have upgraded to the new version. >> > Trying with v2.1.2 (peer is v2.0.18): > > Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line > 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left > == NULL && dir->right == NULL)) This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? From tss at iki.fi Thu Mar 15 19:53:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:53:57 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: On 15.3.2012, at 19.49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. > > I only do replication using the doveadm sync command. Try at least v2.1.2 first, since it has some fixes. Also post your doveconf -n output. From jtl+dovecot at uvm.edu Thu Mar 15 19:55:57 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:55:57 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> Message-ID: <4F622D2D.80802@uvm.edu> On 3/15/12 1:52 PM, Timo Sirainen wrote: > On 15.3.2012, at 19.23, Jim Lawson wrote: > >>> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >>> current hg version has some extra features, but it doesn't use them >>> until all of the directors have upgraded to the new version. >>> >> Trying with v2.1.2 (peer is v2.0.18): >> >> Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line >> 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left >> == NULL && dir->right == NULL)) > This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? > That's correct. Configs are the same between directors (same as I sent in the original msg) Jim From mcazzador at gmail.com Thu Mar 15 19:57:22 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:57:22 +0100 Subject: [Dovecot] replication howto In-Reply-To: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi, yes it'a good idea but i'm using now root i hope this not invalid all I obtain this error but maybe i need some pause Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF Thank's a lot! Il 15 marzo 2012 18:28, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 18:16, Matteo Cazzador wrote: > >> with virtual user (i have no system user ) and there are not ssh >> account. So i must use a dedicate account for replication (ssh) >> that must act sync for all virtual mail account. > > Yes, that's what I use. I did create a dedicated account for vmail > with all the necessary ssh stuff in ~vmail/.ssh > > One remark I forgot to mention in my last mail: > >>> ? service doveadm { >>> ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail >>> ? ? ? ?# (not root) >>> ? ? ? ?# >>> ? ? ? ?user = vmail >>> ? } > > This part is only needed, if you choose to run device doveadm as user > vmail like I do. > >>> ? service config { >>> ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm >>> ? ? ? ?# >>> ? ? ? ?unix_listener config { >>> ? ? ? ? ? ? ? ?user = vmail >>> ? ? ? ?} >>> ? } > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From campbell at cnpapers.com Thu Mar 15 21:06:31 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 15:06:31 -0400 Subject: [Dovecot] Lack of external documentation? Message-ID: <4F623DB7.9060707@cnpapers.com> Firstly, this isn't meant to be critical, and I realize the subject line probably suggest criticism, so... I was sort of forced into using dovecot as my imap/pop server due to upgrading 3 versions of OS on my mail servers. So far, that's not bad. What surprises me is that one of the first things I usually do whenever I start using different software is to purchase a book that seems to suit me. Searching all of the common places like amazon, ebay, etc for manuals turned up little to nothing on dovecot. I'm wondering why and is this so new that people just haven't written books about it yet? The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Thanks for all the help I've received so far and I think I'm really going to like dovecot. Once I get the hang of it, I'll probably reduce the amount of noise on the list by half. steve campbell From terry at cnysupport.com Thu Mar 15 21:27:37 2012 From: terry at cnysupport.com (Terry Carmen) Date: Thu, 15 Mar 2012 15:27:37 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <4F6242A9.6090209@cnysupport.com> On 03/15/2012 03:06 PM, Steve Campbell wrote: > Firstly, this isn't meant to be critical, and I realize the subject > line probably suggest criticism, so... > > I was sort of forced into using dovecot as my imap/pop server due to > upgrading 3 versions of OS on my mail servers. So far, that's not bad. > What surprises me is that one of the first things I usually do > whenever I start using different software is to purchase a book that > seems to suit me. Searching all of the common places like amazon, > ebay, etc for manuals turned up little to nothing on dovecot. > > I'm wondering why and is this so new that people just haven't written > books about it yet? > > The one thing I'm a little critical of, though, is that trying to make > heads or tails of dovecot by following the online documentation is a > little problematic. I'm constantly jumping to another page and then > back to the original page, and for the most part, I just don't know > enough about it all yet to know what I'm looking for. The best docs are on the wiki and this mailing list. If you find the information in the wiki to be lacking, the best thing you can do is find the solution yourself and/or on this mailing list, and then make a wiki entry so the next person will know how to solve the same problem you had. Dovecot is a complex piece of software, and understanding some functionality requires reading the wiki, asking on the mailing list and/or examining the source code. You can also obtain paid support from these companies: http://dovecot.org/support.html I'll be the first to admit that complex and specialized configurations are sometimes difficult to figure out, however this list has always been a tremendous amount of help. Terry From stan at hardwarefreak.com Thu Mar 15 21:44:57 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 15 Mar 2012 14:44:57 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F61C99F.2040409@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> <4F61C99F.2040409@Media-Brokers.com> Message-ID: <4F6246B9.5080309@hardwarefreak.com> On 3/15/2012 5:51 AM, Charles Marcus wrote: > On 2012-03-01 8:38 PM, Stan Hoeppner wrote: >> Get yourself a qualified network architect. Pay for a full network >> traffic analysis. He'll attach sniffers at multiple points in your >> network to gather traffic/error/etc data. Then you'll discuss the new >> office, which employees/types with move there, and you'll be able to >> know almost precisely the average and peak bandwidth needs over the MAN >> link. He'll very likely tell you the same thing I have, that a single >> gigabit MAN link is plenty. If you hire him to do the work, he'll >> program the proper QOS setup to match the traffic patterns gleaned from >> the sniffers. > > Finally had time to properly review your answers here Stan. > > The time you took for the in-depth reply is very much appreciated - and Multi-site setups can be tricky as they often temp folks to do unnecessary things they otherwise would not. Just trying to help keep your sails pointed in the right direction. :) #1 rule when building a multi-site network: only duplicate hardware and services at the remote site(s) when absolutely necessary. > I'm sure you got a kick out of the level of my ignorance... ;) Not at all. I'm sure there is some subject or another where you would demonstrate my ignorance. From another perspective, if there was no ignorance left on the planet then there would be nothing left for anyone to learn. That would make for a boring world. > As for hiring a network architect, I will absolutely be doing as you > recommend (was already planning on it), but with the information I'm now > armed with, at least I'll have a better chance of knowing if they know > what they are doing/talking about... Now that you are aware of network analysis using sniffers, allow me to throw you a curve ball. For a network of your size, less than 70 users IIRC, with a typical application mix but with SMB/NFS traffic/file sizes a little above 'average', a qualified engineer probably won't need to plug sniffers into your network to determine the size MAN pipe and what traffic shaping you'll need. He'll have already done a near identical setup dozens of times. The good news is this saves you a few grand. Analysis with sniffers ain't cheap, even for small networks. And sniffers are normally only deployed to identify the cause of network problems, not very often for architectural or capacity planning. But, asking him about doing a full analysis using sniffers, and hearing his response, may lead to a valuable discussion nonetheless. Have your MAN and internet providers' (if not the same company) pricing sheet(s) in hand when you meet with the engineer. Depending on fast ethernet MAN, GbE MAN, and internet pipe pricing, he may have some compelling options/recommendations for you, possibly quite different, less costly, and more redundant than what you have been considering up to this point. > I'm still planning for the two physical servers (one at each location), Again, if you don't _need_ hardware and services at the 2nd site to achieve the current service level at the primary site, do not add these things to the 2nd site. I really want to put a bunch of exclamation points here but I hate exclamation points in technical emails--actually I just hate them, period. ;) > but you have convinced me that trying to run two live mail systems is an > unnecessary and even unwanted level of complexity. Running an active/active Dovecot cluster doesn't guarantee an unnecessary nor unwanted additional complexity. The need for clustering should go through a justification process just like anything else: what's the benefit, total 'cost', what's the ROI, etc. Lots of people here do active/active clustering every day with great success. Connecting the cluster nodes over a MAN link, however, does introduce unnecessary complexity. Locating one node in another building many blocks away is unnecessary. Putting the nodes in the same rack/room is smart, and easily accomplished in your environment, gives you the redundancy above, but without the potentially problematic MAN link as the cluster interconnect. Granted you'll need to build two new (preferably identical) systems from scratch and setup shared storage (DRBD or a SAN array) and GFS2 or OCFS, etc. Given your environment, there are only two valid reasons for locating equipment and duplicating data and services at a remote site: 1. Unrecoverable network failure (due to single MAN link) 2. Unrecoverable primary site failure (natural or man made disaster) #1 is taken care of by redundant MAN links #2 you've never planned for to this date (probability is *low*) and you need _everything_ duplicated at the remote site Duplicating servers for high(er) user throughput/lower latency to/from servers isn't a valid reason for remote site duplication in your case because you are able to afford plenty of bandwidth and link redundancy between the sites. The relative low cost and high bandwidth of the MAN link outweighs any benefit of service replication due to the latter's complexity level. Here are some other 'rules': 1. Don't duplicate servers at remote sites to mitigate network link failure when sites are close and redundant bandwidth is afforadable 2. Do duplicate network links to mitigate link failure when sites are close and bandwidth is affordable 3. Implement and test a true disaster avoidance and recovery plan > The DC VM will still > be hot (it is always best to have two DCs in a windows domain > environment anyway) so I'll get automatic real time off site backup of > all of the users data (since it will all be on DFS), but for the mail > services, I'll just designate one as live, and one as the hot/standby > that is kept in sync using dsync. This way I'll automatically get off > site back up for each site for the users data stored in the DFS, and > have a second mail system ready to go if something happens to the primary. Again, you're not looking at this network design from the proper perspective. See rules 1-3 above. Off site backups/replication are used exclusively to mitigate data loss due to catastrophic facility failure, not server failure, enabling rapid system recovery when new equipment has arrived. Many business insurers have catastrophic IT equipment replacement plans and relationships with the big 5 hardware vendors, enabling you to get new new equipment racked and begin your restore from offsite tape, within as little as 24 hours of notification. Think of how FEMA stages emergency supplies all around the country. Now think 10 times better, faster. Such services increase your premiums, but if you're serious about disaster avoidance and recovery, this is the only way to go. IBM, HP, maybe Dell, Sun (used to anyway), have dedicated account reps for disaster recovery. They work with you to keep an inventory of all of your systems and storage. Your records are constantly updated when your products are EOL'd or superseded or you replace or add hardware, and a list is maintained of current hardware best matched to replace all of your now burned, flooded, tornado shredded, hurricane blasted equipment, right down to bare metal restore capability, if possible/applicable. You plan to replicate filesystem user data and mailbox data to a 2nd site to mitigate single server failures. Why does that need to be done to an offsite location/system? It doesn't. There is no benefit whatsoever. You can accomplish this in the same rack/room and get by with a smaller MAN pipe saving time, money, and administrative burden. The restore procedure will be faster if all machines are in the same rack/room and you're using tape, and you won't slow users down with restore traffic going over the MAN link. If you really want off-site backup, for what it's meant to accomplish, get a network attached tape library/silo, or a speedy high cap LTO-4/5 tape drive in each server, put a real backup rotation and restore plan in place, and store backup tapes in a secure facility. A remote "hot site" is great when it's in a different city, better yet region, or in a hardened facility in any locale. Your hot site is only a few blocks away. If your primary site it taken out by anything other than fire, such as a tornado, earthquake, hurricane being more likely in your case, chances are your hot site may go down soon after the primary. If you want/need a real off site backup solution, rotate tapes to an everything-proof facility. Here are 3 companies in the Atlanta area that offer media rotation storage services. Watch the Offsite Tape Vaulting video at IronMountain: http://www.ironmountain.com/Knowledge-Center/Reference-Library/View-by-Document-Type/Demonstrations-Videos/Tours/Offsite-Tape-Vaulting.aspx http://www.askads.net/media-rotation/ http://www.adamsdatamanagement.com/tape-rotation-atlanta-ga.htm > Again, thanks Stan... I am constantly amazed at the level of expertise > and quality of advice available *for free* in the open source world, as > is available on these lists. Always glad to assist my brethren in this digital kingdom. Whichever architecture/topology you choose, remote replicated systems or not, I hope my input has given you some good information on which to base your decisions. -- Stan From list at airstreamcomm.net Thu Mar 15 21:48:57 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 15 Mar 2012 14:48:57 -0500 Subject: [Dovecot] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> On Thu, 15 Mar 2012 16:53:53 +0200, Timo Sirainen wrote: > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. Are there any performance metrics around dsync replication, such as how many users this has been tested on, or how long the replication take to occur? Also I have not been able to determine from reading the mailinglist whether or not dsync replication works with different types of mailboxes (maildir, dbox, mbox), what is supported? From tss at iki.fi Thu Mar 15 21:55:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 21:55:57 +0200 Subject: [Dovecot] v2.1.2 released In-Reply-To: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> References: <1331823233.10319.40.camel@innu> <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> Message-ID: <6489C385-E8C3-425E-8D2D-B3A242A6E0AF@iki.fi> On 15.3.2012, at 21.48, wrote: > Are there any performance metrics around dsync replication, such as how > many users this has been tested on, or how long the replication take to > occur? The performance isn't optimal yet. You can probably replicate some hundreds of users ok, maybe thousands, but depends. > Also I have not been able to determine from reading the mailinglist > whether or not dsync replication works with different types of mailboxes > (maildir, dbox, mbox), what is supported? Maildir and dbox is supported, mbox probably works okayish but since it doesn't have proper message GUIDs you could run into trouble. From tom at talpey.com Thu Mar 15 22:23:19 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 16:23:19 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> Message-ID: <4F624FB7.9000408@talpey.com> On 3/15/2012 12:25 PM, Timo Sirainen wrote: > On 15.3.2012, at 18.04, Tom Talpey wrote: > >> I'm seeing a few warnings emitted when building for x86. They're pretty >> obvious, but if you want the configure options etc, I can provide those. >> >> In Dovecot 2.1.2 (I also see some of these in 2.1.1): > > Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. Confirmed, Dovecot builds cleanly for me now. Thanks Timo! The pigeonhole warning appears to be harmless and I'll wait for Stefan to confirm/address. From giles at coochey.net Thu Mar 15 22:30:19 2012 From: giles at coochey.net (Giles Coochey) Date: Thu, 15 Mar 2012 20:30:19 +0000 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F62515B.1050207@coochey.net> On 15/03/2012 19:27, Terry Carmen wrote: > On 03/15/2012 03:06 PM, Steve Campbell wrote: >> Firstly, this isn't meant to be critical, and I realize the subject >> line probably suggest criticism, so... >> >> I was sort of forced into using dovecot as my imap/pop server due to >> upgrading 3 versions of OS on my mail servers. So far, that's not >> bad. What surprises me is that one of the first things I usually do >> whenever I start using different software is to purchase a book that >> seems to suit me. Searching all of the common places like amazon, >> ebay, etc for manuals turned up little to nothing on dovecot. >> >> I'm wondering why and is this so new that people just haven't written >> books about it yet? >> >> The one thing I'm a little critical of, though, is that trying to >> make heads or tails of dovecot by following the online documentation >> is a little problematic. I'm constantly jumping to another page and >> then back to the original page, and for the most part, I just don't >> know enough about it all yet to know what I'm looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always > been a tremendous amount of help. > > Terry > > > > > > What he said +1. I don't want to be-little IMAP software or the work that Timo has done to get dovecot to the IMAP server world, but IMAP in general is a small enough subject to only really warrant two maybe three books - the most recent of which was written 5-7 years ago. The original release of dovecot was around 2002, but I don't think it became as widely adopted as Courier / Cyrus until around 2010. I wouldn't be surprised that if there is a next edition of "The Book of IMAP" or the O'reilly "Managing IMAP" that there would probably be an equal share section on dovecot than any other server out there. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From jerry at seibercom.net Thu Mar 15 22:46:18 2012 From: jerry at seibercom.net (Jerry) Date: Thu, 15 Mar 2012 16:46:18 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <20120315164618.705ca356@scorpio> On Thu, 15 Mar 2012 15:27:37 -0400 Terry Carmen articulated: > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > Firstly, this isn't meant to be critical, and I realize the subject > > line probably suggest criticism, so... > > > > I was sort of forced into using dovecot as my imap/pop server due > > to upgrading 3 versions of OS on my mail servers. So far, that's > > not bad. What surprises me is that one of the first things I > > usually do whenever I start using different software is to purchase > > a book that seems to suit me. Searching all of the common places > > like amazon, ebay, etc for manuals turned up little to nothing on > > dovecot. > > > > I'm wondering why and is this so new that people just haven't > > written books about it yet? > > > > The one thing I'm a little critical of, though, is that trying to > > make heads or tails of dovecot by following the online > > documentation is a little problematic. I'm constantly jumping to > > another page and then back to the original page, and for the most > > part, I just don't know enough about it all yet to know what I'm > > looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized > configurations are sometimes difficult to figure out, however this > list has always been a tremendous amount of help. The lack of truly informative documentation has been the Achilles' heel of open-source software since its inception. I feel your pain. I have always loved a hard copy, i.e. book documenting the subject I am studying. Jumping from screen to screen sucks, plus how do I highlight a passage on the monitor for future reference? There have been a few books written to document Postfix, but to the best of my knowledge, none exist for Dovecot. -- Jerry ? Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __________________________________________________________________ From e-frog at gmx.de Thu Mar 15 22:46:22 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 15 Mar 2012 21:46:22 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F60F29D.2010409@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> <4F60F29D.2010409@gmx.de> Message-ID: <4F62551E.1000102@gmx.de> On 14.03.2012 20:33, wrote e-frog: > On 14.03.2012 14:41, wrote Timo Sirainen: >> >> With latest hg version it should work. >> > > Hi Timo, > > The "can't delete mailbox INBOX" error is gone now with changeset > c077ca9bc306 and it's working successfully on the account from yesterday > where it also worked with mailbox_list_index=no. > > However using a different account (more mail and mailboxes) I'm seeing > dbox corruption errors. I have tested with mailbox_list_index=yes and no > and it's the same for both. So this might be unrelated to this setting. > Attached are logs from doveadm backup runs. First to an empty directory > and 2 consecutive runs. > Further testing (now with 2.1.2) shows it only seems to work for a single mailbox. e.g. doveadm -v backup -u testuser at ubuntu-test.localdomain -m 'INBOX' mdbox:/tmp/backup dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=c63f581c030b774b572a0000ec8d17cd) -> no errors This works for every single mailbox in this account. The errors only occur without -m 'mailbox'. Using maildir as destination format however seems to work fine on the whole account. doveadm -v backup -u testuser at ubuntu-test.localdomain maildir:/tmp/backup -> no errors Thanks, e-frog From trashcan at odo.in-berlin.de Thu Mar 15 22:48:31 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:48:31 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi -- On 15.03.2012, at 18:57, Matteo Cazzador wrote: > Hi, yes it'a good idea but i'm using now root i hope this not > invalid all Actually it's a bad idea to use root for ssh from a security point of view. A hacked root account isn't fun. Thus, normally one needs to explicitly change the config of the sshd daemon to allow root logins (at least with FreeBSD what I'm using). Thus, I do recommend to use an unprivileged user like vmail. > I obtain this error but maybe i need some pause ;-) > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found root doesn't not find doveadm at the remote server. As mentioned above you better create an account for vmail and allow that user to find doveadm in its path. > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF That's an error due to not finding doveadm at the remote site. Regards, Michael From trashcan at odo.in-berlin.de Thu Mar 15 22:55:17 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:55:17 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html and http://www.dovecot.org/img/dsync-director-replication-ssh.png helped me a lot understand the idea behind it. > As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. As Timo recommended already, you better upgrade to 2.1.2 first. I can confirm that he fixed a lot compared to older dsync versions. Regards, Michael From tss at iki.fi Thu Mar 15 23:01:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:01:39 +0200 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <67E4C4F8-A9CE-4912-9B3F-05770041C383@iki.fi> On 15.3.2012, at 21.06, Steve Campbell wrote: > The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Perhaps it would be helpful to have some more talkative howtos for some of the typical configurations, that don't only list the options that are given but actually talks about why things are done the way they are? I've tried to avoid duplication of text in wiki, because if something changes it's difficult to update it everywhere, but in howtos I guess it wouldn't be too bad. Or maybe the wiki could be restructured in some way to make it easier to follow. I think I'm the worst possible person to figure out anything like that, because I don't know what the difficult parts are. I'd think the Dovecot wiki is good if you know what you want to do and just want to know some specifics, but I guess it can be difficult to figure things out otherwise. > Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? A few people have talked about writing a Dovecot book and I've promised to help them, but no one's actually written one as far as I know. > So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Features that aren't yet even fully implemented don't really have documentation for them. From tss at iki.fi Thu Mar 15 23:05:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:05:54 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> On 15.3.2012, at 22.48, Michael Grimm wrote: > On 15.03.2012, at 18:57, Matteo Cazzador wrote: > >> Hi, yes it'a good idea but i'm using now root i hope this not >> invalid all > > Actually it's a bad idea to use root for ssh from a security point > of view. A hacked root account isn't fun. Thus, normally one needs > to explicitly change the config of the sshd daemon to allow root > logins (at least with FreeBSD what I'm using). Thus, I do recommend > to use an unprivileged user like vmail. Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. From hoogendyk at bio.umass.edu Thu Mar 15 23:13:34 2012 From: hoogendyk at bio.umass.edu (Chris Hoogendyk) Date: Thu, 15 Mar 2012 17:13:34 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <4F625B7E.5060902@bio.umass.edu> On 3/15/12 4:46 PM, Jerry wrote: > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > >> On 03/15/2012 03:06 PM, Steve Campbell wrote: >>> Firstly, this isn't meant to be critical, and I realize the subject >>> line probably suggest criticism, so... >>> >>> I was sort of forced into using dovecot as my imap/pop server due >>> to upgrading 3 versions of OS on my mail servers. So far, that's >>> not bad. What surprises me is that one of the first things I >>> usually do whenever I start using different software is to purchase >>> a book that seems to suit me. Searching all of the common places >>> like amazon, ebay, etc for manuals turned up little to nothing on >>> dovecot. >>> >>> I'm wondering why and is this so new that people just haven't >>> written books about it yet? >>> >>> The one thing I'm a little critical of, though, is that trying to >>> make heads or tails of dovecot by following the online >>> documentation is a little problematic. I'm constantly jumping to >>> another page and then back to the original page, and for the most >>> part, I just don't know enough about it all yet to know what I'm >>> looking for. >> The best docs are on the wiki and this mailing list. If you find the >> information in the wiki to be lacking, the best thing you can do is >> find the solution yourself and/or on this mailing list, and then make >> a wiki entry so the next person will know how to solve the same >> problem you had. >> >> Dovecot is a complex piece of software, and understanding some >> functionality requires reading the wiki, asking on the mailing list >> and/or examining the source code. You can also obtain paid support >> from these companies: http://dovecot.org/support.html >> >> I'll be the first to admit that complex and specialized >> configurations are sometimes difficult to figure out, however this >> list has always been a tremendous amount of help. > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. I like books, but, especially in the case of actively developed software such as Dovecot, they become outdated very quickly. I have two editions of the Unix System Administration Handbook (can't remember the last time I looked at them), and two editions of Backup & Recovery. I use Amanda for backup. It has been developed actively over the last several years, and the Backup & Recovery chapter on Amanda is sorely out of date. The wiki, the users mailing list, and the man pages are the only way to really be up-to-date. With the book, you won't know anything about any changes or additions since the book was written, which would have been at least many months before it was published. I'm into online documentation every day. I'm a Solaris admin, but I've been jumping from Solaris 10 to Ubuntu without any books, and I've been jumping from ZFS to LVM without any books. That's a significant transition. But it seems I can find almost everything online. Sometimes another admin gives me an explanation and a link. It's just the way things are. The digital world is moving too fast to be frozen in print. -- --------------- Chris Hoogendyk - O__ ---- Systems Administrator c/ /'_ --- Biology& Geology Departments (*) \(*) -- 140 Morrill Science Center ~~~~~~~~~~ - University of Massachusetts, Amherst --------------- Erd?s 4 From amk at spamfence.net Thu Mar 15 23:43:02 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Thu, 15 Mar 2012 21:43:02 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: <1331816286.10319.23.camel@innu.invalid> Message-ID: Timo Sirainen wrote: >> $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch >> $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install > > You would have needed to run autogen.sh again. It works with me now that > I tried in a test server with OpenSSL in non-standard dir. Sorry, I didn't know that with "autogen.sh". Just grabbed Dovecot 2.1.2 (which is all properly set up - so I couldn't do anything wrong ;-) and compiled it. Compilation works. Great! The binaries find all their libraries. But two libraries are not quite okay. They don't find their SSL libs: libdovecot-lda.so libdovecot-storage.so Since libdovecot-lda.so doesn't contain the words libssl or libcrypto, I guess that ldd just complains because it uses libdovecot-storage.so. Thus, libdovecot-storage.so is the (only) one left with an incomplete library search path. Luckily, all binaries use some additional libraries which come with a proper library path. So the whole things works, but it's more like some kind of magic. It would be great if libdovecot-storage.so could be fixed as well to make things finally perfect. Thanks for all your effort. (I know this isn't top priority as most people use precompiled stuff and never run into such kind of things.) Greetings, Andreas From p at state-of-mind.de Thu Mar 15 23:46:59 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Thu, 15 Mar 2012 22:46:59 +0100 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <20120315214658.GC3750@state-of-mind.de> * Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. Dovecot is a moving target and it is hard to produce any print that represents what Dovecot can do when the print finally will be released. I know, because I am one of the two authors who wrote "The Book of Postfix" and we found it hard if almost impossible to keep up with Wietse's pace when he wrote major parts of Postfix. For now, I believe, the wiki and the mailing list is as good as it gets. Later when Dovecot settles a book might be something to write and something to spend money on because it lasts for a while. man pages would be a good thing, but given Dovecots configuration syntax and flexibility this might be an even harder task. Its probably easier to describe certain aspects of configuration or use cases than list all options and their possible occurences. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 From david at blue-labs.org Thu Mar 15 23:49:54 2012 From: david at blue-labs.org (David Ford) Date: Thu, 15 Mar 2012 17:49:54 -0400 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <4F626402.4030606@blue-labs.org> in ~privilgeduser/.ssh/authorized keys: from= cmd=dsync.sh pubkey... On 03/15/2012 05:05 PM, Timo Sirainen wrote: > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. And if you allow ssh > login only with public key authentication I don't think there are much > security issues. And finally, it would be possible to write a small > wrapper that allows the root's public key auth to only execute > dsync-user.sh script that can't do anything except sync a specified > user's mails. From andrei at lctax.ro Thu Mar 15 23:52:39 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 17:52:39 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <64c9b1a4813862ad254f591c6a5ffc02.squirrel@web.miau.ca> Hello Timo, I have update the repository with hg pull -u, recompiled and redeployed and somehow the dovecot -n still shows 2.1.1... :( I ran exactly the same test: starting for 1 clean user1, I create 2 emails, one on mx1.a and one on mx2.a and I sync them with doveadm. The output is exactly as previously sent :( Here is my conf: # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } Thank you, Andrei > On 15.3.2012, at 19.49, Michescu Andrei wrote: > >> Can you get a little bit more in details about this replicator/dsync >> techique? As my main problem is that EVERYTHING (that gets created on >> different servers in the same time) gets duplicated. >> >> I only do replication using the doveadm sync command. > > Try at least v2.1.2 first, since it has some fixes. Also post your > doveconf -n output. > > > !DSPAM:4f622cb881591647615726! > > From tss at iki.fi Thu Mar 15 23:55:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:55:26 +0200 Subject: [Dovecot] replication howto In-Reply-To: <4F626402.4030606@blue-labs.org> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Plus the scripts that 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) and 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. On 15.3.2012, at 23.49, David Ford wrote: > in ~privilgeduser/.ssh/authorized keys: > > from= cmd=dsync.sh pubkey... > > On 03/15/2012 05:05 PM, Timo Sirainen wrote: >> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. > From stephan at rename-it.nl Fri Mar 16 00:17:40 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 23:17:40 +0100 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F624FB7.9000408@talpey.com> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> <4F624FB7.9000408@talpey.com> Message-ID: <4F626A84.1070705@rename-it.nl> On 3/15/2012 9:23 PM, Tom Talpey wrote: > On 3/15/2012 12:25 PM, Timo Sirainen wrote: >> On 15.3.2012, at 18.04, Tom Talpey wrote: >> >>> I'm seeing a few warnings emitted when building for x86. They're pretty >>> obvious, but if you want the configure options etc, I can provide >>> those. >>> >>> In Dovecot 2.1.2 (I also see some of these in 2.1.1): >> >> Thanks, fixed in hg. I guess I should add x86 vm building these >> nightly as well.. > > Confirmed, Dovecot builds cleanly for me now. Thanks Timo! > > The pigeonhole warning appears to be harmless and I'll wait for Stefan > to confirm/address. Thanks, fixed: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/75c1a2fd9b26 Regards, Stephan. From andrei at lctax.ro Fri Mar 16 01:20:00 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 19:20:00 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Message-ID: hello, So I upgraded to 2.1.2 (not from repository because that one still says 2.1.1, but from the release). I ran exactly the same test with exactly the same behaviour. (new account, synced successfully on 2 servers, deliver 1 email to each server, run doveadm sync)... Please find below the dovecot-uidlists: on mx1.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852540.19862.mx2,S=272 2 G1331852540.19862.mx2,S=272 :1331852573.M89342P19877.mx2,S=272 3 :1331852488.30409.mx1,S=268 on mx2.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852488.30409.mx1,S=268 2 :1331852540.19862.mx2,S=272 3 G1331852488.30409.mx1,S=268 :1331852572.M622052P30410.mx1,S=268 As you can see both servers duplicated the email that was delivered first to them (1 in both cases, because the user1 is a clean account). There is the same effect in the folders: initial there is only one file on each server and after sync there are 3 files instead of only 2... Also, after the sync, there should be 2 new emails (N2 if I interpret correctly that N1 means only one new). Thank you. Andrei PS: also I need to run dsync twice, because first time I receive: dsync-local(user1 at a): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at a): Info: INBOX: Couldn't keep all uids dsync-local(user1 at a): Warning: Mailbox changes caused a desync. You may want to run dsync again. The config is below: # 2.1.2: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From gedalya at gedalya.net Fri Mar 16 01:55:09 2012 From: gedalya at gedalya.net (Gedalya) Date: Thu, 15 Mar 2012 19:55:09 -0400 Subject: [Dovecot] IMAP to Maildir Migration preserving UIDs? In-Reply-To: <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> Message-ID: <4F62815D.7020002@gedalya.net> On 01/28/2012 12:45 PM, Timo Sirainen wrote: > On 27.1.2012, at 2.00, Gedalya wrote: > >> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> >> Program received signal SIGSEGV, Segmentation fault. >> mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 >> 213 mailbox-log.c: No such file or directory. >> in mailbox-log.c > This crash is now fixed, so there's no need to give /tmp/imapc path anymore: > http://hg.dovecot.org/dovecot-2.1/rev/7b94d1c8a6e7 > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0. From ml at smtp.fakessh.eu Fri Mar 16 02:31:59 2012 From: ml at smtp.fakessh.eu (ml) Date: Fri, 16 Mar 2012 01:31:59 +0100 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <1331857919.16694.5.camel@localhost> Le jeudi 15 mars 2012 ? 16:53 +0200, Timo Sirainen a ?crit : > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. > > i build with succes the last release for centos 5 work fine and best ns.fakessh.eu/rpms/dovecot-2.1.2-1.centme.el5.src.rpm thanks Timo -- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC2626742 gpg --keyserver pgp.mit.edu --recv-key C2626742 http://urlshort.eu fakessh @ http://gplus.to/sshfake http://gplus.to/sshswilting http://gplus.to/john.swilting https://lists.fakessh.eu/mailman/ This list is moderated by me, but all applications will be accepted provided they receive a note of presentation -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From campbell at cnpapers.com Fri Mar 16 03:08:15 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 21:08:15 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <1331860095.4f62927f9acd0@perdition.cnpapers.net> Quoting Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. > > -- > Jerry ??? > > Disclaimer: off-list followups get on-list replies or get ignored. > Please do not ignore the Reply-To header. > __________________________________________________________________ > So many great replies, but I'll pick this one to use as my reply-to since it mirrors mostly how I feel about my experiences so far when it comes to learning Dovecot. I installed a new server, going from Centos 3 to Centos 6. I found that Postfix was the preferred SMTP server and Dovecot was the preferred imap/pop server. I gave Postfix my best shot, but didn't really have it tested well enough to stick with it, so I dropped back to Sendmail, something I'm somewhat familiar with. I've read multiple versions of O'Reilly's Sendmail books along with the Sendmail Cookbook. I have to admit that it was these books that made me realize the power of Sendmail. Post l website to further learn, but I had to get the basics first to do what needed to be done to get the job into a working server. Dovecot is an application that probably would work out of the box for me if I didn't have to use data from the previous server. So I had to use more than the standard options to make this work. Finding those options was the main gripe I had with the wiki - there are just so many options to make Dovecot the complete server. That's a good thing. Just remember, us noobies-to-Dovecot have to discover all of those options. I mentioned that I was happy with the wiki and the list when it comes to answering my questions. But I'm sure the list will get tired of me asking what must appear to be redundant, simple, obnoxious questions. The index-like wiki page is most helpful. I knew dovecot has been around for a while, but didn't know how mature it was. The fact that Centos/Red Hat uses it as a default says quite a bit about it's reliability, so I'll stick with it. One of the the things I was planning on doing was combining two servers, which services one domain on one server and services two other domains on the other, into one server, and have the other as a server-in-waiting. So along comes this dsync thread, and now it appears that Dovecot might make that all easier. I see all the potential Dovecot has, but learning it is a little difficult for us new users. Once I get the hang of it, I'm sure I want need to search for the things I need to find, but for now, a good book would have been nice and a lot easier. I give all the praise to Timo that he deserves. (I'm guessing he's either the developer, the lead guru on the list or something of that stature). I like what I'm seeing, I'm just not always seeing what I need. Again, this is not critical in nature. I'm just stating what this particular rookie is conveying to the list about my experience (and lack of experience) in getting where I need to be with Dovecot. Thanks for such a great application, all the great replies and help so far. steve ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From lists at wiesinger.com Fri Mar 16 08:02:10 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Fri, 16 Mar 2012 07:02:10 +0100 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage Message-ID: <4F62D762.7080607@wiesinger.com> Hello, After fixing configuration and other issues I'm still having one problem with imap executable and pine: less .pinerc # Changed config: #rsh-command=/usr/sbin/dovecot --exec-mail imap rsh-command=/usr/local/bin/imap Calling imap still fails as non root: imap /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied collect2: ld returned 1 exit statusn Any ideas to fix it? BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Thnx. Ciao, Gerhard From tss at iki.fi Fri Mar 16 10:05:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 10:05:20 +0200 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <4F62D762.7080607@wiesinger.com> References: <4F62D762.7080607@wiesinger.com> Message-ID: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: > After fixing configuration and other issues I'm still having one problem with imap executable and pine: > less .pinerc > # Changed config: > #rsh-command=/usr/sbin/dovecot --exec-mail imap > rsh-command=/usr/local/bin/imap That's correct. > Calling imap still fails as non root: > imap > /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied > collect2: ld returned 1 exit statusn Huh? That looks like imap is running ld to link something. It shouldn't be doing that. > BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Maildir for reliability, sdbox/mdbox for performance. From mcazzador at gmail.com Fri Mar 16 10:37:47 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 09:37:47 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, thank's everybody, today afternoon i apply the suggest and i test solution. I post the actual configuration that i will test: vmail users is present too, i create ssh-keygen for users vmail and relative home directory and permit ssh with no password with user vmail on two servers. Then i use the configuration below i leave comment the line below or i need to active it excuse but i don't understand clear cause my terrible english? #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} and apply this on two servers service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } Thank's everyboy Il 15 marzo 2012 22:55, Timo Sirainen ha scritto: > Plus the scripts that > > 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) > > and > > 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself > > Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. > > On 15.3.2012, at 23.49, David Ford wrote: > >> in ~privilgeduser/.ssh/authorized keys: >> >> from= cmd=dsync.sh pubkey... >> >> On 03/15/2012 05:05 PM, Timo Sirainen wrote: >>> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. >> > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jernej.porenta at arnes.si Fri Mar 16 11:09:07 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Fri, 16 Mar 2012 10:09:07 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1331735355.2081.140.camel@innu> References: <1331735355.2081.140.camel@innu> Message-ID: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> On Mar 14, 2012, at 3:29 PM, Timo Sirainen wrote: > On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: >> Heya, >> >> We are expiriencing issues with dovecot 2.1.1 on Linux with weird >> filenames in home directory of username. We are using mbox IMAP >> folders, with no special changes (mail_location = mbox:~/:INBOX=% >> h/.mailbox). >> >> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >> (uni_utf8_str_is_valid(vname)) > .. >> AFAIK, the problem lies in processing the file list of home folder, >> which can contain filenames that do not have proper UTF-8 encoding of >> filenames, which causes dovecot to crash. > > Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This > should fix it by renaming such mailboxes: > http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. Whenever . LIST "" "*" is issued, dovecot crashes: Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b5466f3119c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Any clues? > >> On the other hand, UTF-8 filenames created on the system by hand >> (using touch), are not displayed in IMAP LIST command (sample is >> included in the folder structure; single letter file). > > This is a bit trickier problem. The mailbox names are currently stored > in filesystem as IMAP's modified-UTF7. So it's not really even currently > supposed to work, although it's not very nice that the mailboxes aren't > visible either. Maybe I'll do something smart in future for this, like > allowing both mUTF-7 and UTF-8 and remembering per-mailbox which > formatting it is in. I think we can leave this issue out, since I don't believe the users will be creating folders directly from interactive SSH sessions and rather use IMAP to create folders. So, this use-case is not very likely to occur. Thank you for your help... Cheers, Jernej From nmilas at noa.gr Fri Mar 16 11:26:45 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 11:26:45 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x Message-ID: <4F630755.7070909@noa.gr> Hi, A quick question: Are there any incompatibilities in config settings among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require changes of any config settings (as upgrading from v1.x to 2.0.x did)? Thanks, Nick From mstevens at imt-systems.com Fri Mar 16 11:39:53 2012 From: mstevens at imt-systems.com (Morten Stevens) Date: Fri, 16 Mar 2012 10:39:53 +0100 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <4F630755.7070909@noa.gr> References: <4F630755.7070909@noa.gr> Message-ID: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> On 16.03.2012 10:26, Nikolaos Milas wrote: > Hi, > > A quick question: Are there any incompatibilities in config settings > among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? > > That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require > changes of any config settings (as upgrading from v1.x to 2.0.x did)? Hi, See: http://wiki2.dovecot.org/Upgrading/2.1 Best regards, Morten From nicku at nicku.org Fri Mar 16 12:08:36 2012 From: nicku at nicku.org (Nick Urbanik) Date: Fri, 16 Mar 2012 21:08:36 +1100 Subject: [Dovecot] imaptest: performance testing Message-ID: <20120316100836.GA12049@nicku.org> Dear Folks, Using head of imaptest with dovecot 2.1, I am attempting to put a heavy load on the server, with insufficient success. I made 15000 user accounts, put them in to a file, one per line. Then I ran imaptest with ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 pass=SECRETPASSWORD But CPU load is only reaching 20. I put an mbox containing 67 messages into ~/mail/dovecot-crlf So what is the best way to put a really heavy load on an imap server? The documentation describes testing correctness of the server without really emphasising performance. I would like to do something like dnsperf, which adds a linearly increasing load until the server is unable to cope. This machine has 24G RAM and 8 cores. I'll be grateful for any constructive suggestions. -- Nick Urbanik http://nicku.org nicku at nicku.org GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24 From odhiambo at gmail.com Fri Mar 16 12:51:46 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 13:51:46 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 Message-ID: I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall. I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook pops a screen saying "IMAP session state is inconsistent, please relogin". Dovecot says: Mar 16 13:30:26 jaribu dovecot: master: Warning: Killed with signal 15 (by pid=72242 uid=0 code=kill) Mar 16 13:30:29 master: Info: Dovecot v2.1.2 starting up Mar 16 13:31:15 auth-worker(72594): Info: mysql(localhost): Connected to database exim4u Mar 16 13:31:15 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72595 Mar 16 13:31:15 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:15 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72646 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid=-1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72647 Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around offset=894): msg header has bad magic value Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage: rebuilding indexes Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: /var/spool/virtual/kictanet.or.ke/wash/mdbox/mailboxes/SentItems/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 16 13:33:23 imap(wash at kictanet.or.ke): Info: Disconnected: IMAP session state is inconsistent, please relogin. bytes=13816863/907529 My doveconf output is here -> http://pastebin.com/6yNP5ygt -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From mlopez at gtdinternet.com Fri Mar 16 13:07:15 2012 From: mlopez at gtdinternet.com (=?ISO-8859-1?Q?Mauricio_L=F3pez_Riffo?=) Date: Fri, 16 Mar 2012 08:07:15 -0300 Subject: [Dovecot] POP3 Performance Message-ID: <4F631EE3.40806@gtdinternet.com> Hi, We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith) Attach of dovecot -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail at mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert = References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F631F27.2050506@Media-Brokers.com> On 2012-03-15 3:27 PM, Terry Carmen wrote: > On 2012-03-15 3:06 PM, Steve Campbell wrote: >> Does anyone know of any manuals/books that have been written that >> might introduce me to most of the stuff in dovecot? > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always been > a tremendous amount of help. I agree completely. The ability to come to places like this and get answers directly from the software developer(s) is one of the main reasons I love open source software. And I will also say that Timo (yes, Steve, he is *the* dovecot developer, although he has had some excellent help for a while now) and this list is one of the most civil & respectful of any list I've been on, and the quality of support/answers is second to none. The postfix list is imo just as good as far as the quality of support, but they are very strict on 'form' - ie, no top-posting, you're expected (and often reminded) to read the instructions in the welcome message as to 'How to report a problem' and to actually follow those instructions - and quite often their replies seem harsh and unfriendly. I'd actually like to see dovecot have a similarly detailed welcome message (complete with a link to a detailed wiki page on 'How to Report a Problem' along with some helpful troubleshooting tips), but as much as I dislike top-posters (especially those who blindly quote the entire message they are replying to), I'm glad that this list is a bit less strict on form, and just seems more friendly. I for one would *love* to see some kind of 'The Book of Dovecot' (like 'The Book of Postfix'), but one reason I can see that would keep someone from wanting to write one is that dovecot (like most popular open source software) is still a very fast moving target as compared to the useful life of a book. Maybe his commercial support company can provide the resources for writing one once the target slows down a bit - or maybe even start off writing [a][some] smaller 'Basic Configuration' guide[s] for the things that aren't such fast moving targets that could eventually become chapters in a more comprehensive book. That would I think be a (admittedly probably fairly small) revenue generator, but hopefully at least enough to pay for itself and maybe provide a small profit. Another option I can think of would be for Timo to provide a method for people to pay a small fee for his support company to write up a custom 'How-To' for someone based on a list of requirements. I would imagine this as a web page that is put together with the appropriate questions, the answers for which are necessary to accomplish the goal. Of course, the other option is for other people to step up and 'fix the wiki' or 'write the Book' (or How-Tos), instead of just complaining about the lack (no offense, your 'complaint' wasn't all that bad). Yeah, I know this is the standard answer on free/open source software support lists, but it is the standard answer for a reason. On 2012-03-15 9:08 PM, Steve Campbell wrote: > I found that Postfix was the preferred SMTP server and Dovecot was > the preferred imap/pop server. I gave Postfix my best shot, but > didn't really have it tested well enough to stick with it, so I > dropped back to Sendmail, something I'm somewhat familiar with. I understand the argument for sticking with something you're familiar with, but I don't think you gave postfix a fair shot either - and it *does* have a number of excellent books written for it, so you don't have that excuse for postfix... ;). It is *much* easier to configure and run than sendmail, is much more performant and supposedly much more secure (just going by what I've read), and can do most anything that sendmail does (even supports milters). > Dovecot is an application that probably would work out of the box for > me if I didn't have to use data from the previous server. So I had to > use more than the standard options to make this work. Finding those > options was the main gripe I had with the wiki - there are just so > many options to make Dovecot the complete server. That's a good > thing. Just remember, us noobies-to-Dovecot have to discover all of > those options. You always have the option to get commercial support for fast resolutions to complex problems like this... ;) -- Best regards, Charles From robert at schetterer.org Fri Mar 16 14:10:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Fri, 16 Mar 2012 13:10:49 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F632DC9.4070108@schetterer.org> Am 16.03.2012 12:07, schrieb Mauricio L?pez Riffo: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see that > the performance is very poor and receive complaint about delays of > autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection (betwen > mtas and nfs storage) All account information work in LDAP plataform > (two servers in replicated mode, no high average or delays detected in > this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all of > service suffer a high load average (about 8 - 20 load average in each > dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, > autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, like > a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english > looks like you need to play with some config stuff and do more debug on your possible bottlenecks, what did you allready played with dovecot to high performance guess Timo will help about config settings after all for short to read http://wiki.dovecot.org/Authentication/Caching http://wiki.dovecot.org/LoginProcess http://wiki2.dovecot.org/Services any reasons for that mail_full_filesystem_access = yes ? general nfs may not optimal, but that a long story also virtual machines have their pros and contras maildir is not so good in performance -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From janfrode at tanso.net Fri Mar 16 14:11:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 16 Mar 2012 13:11:07 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <20120316121107.GA23566@dibs.tanso.net> One quick fix to try, if it's the login-time that's killing you, is to enable auth caching: http://wiki2.dovecot.org/Authentication/Caching that should offload your backend LDAP-servers from doing bind() on ever login, had a huge login performance impact for us. We use "auth_cache_size = 100 M", which gives us 99% cache hits: dovecot: auth: Authentication cache hits 3654591/3669119 (99%) dovecot: auth: Authentication cache inserts: positive: 588030 80931909B, negative: 912 49888B -jf From mcazzador at gmail.com Fri Mar 16 14:12:40 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 13:12:40 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, i obtain the same error Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF i've create vmail users (i've virtual domain netlite.locale (postfix), mysql backend i receive ana send mail correctly i use imap protocol), get ssh connection with publick key, i verify that with su - vmail , vmail find doveadm, i post my dovecot.conf file because i don't know what is wrong vmail exist on every server with publick key (server one => 10.0.0.118 server two => 10.0.0.122) dovecot.conf -> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm unix_listener config { user = vmail } } plugin { # this host replicates to remote host # mail_replica = remote:vmail at 10.0.0.122 # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hour } idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 thank's -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Fri Mar 16 14:49:55 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 14:49:55 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> References: <4F630755.7070909@noa.gr> <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> Message-ID: <4F6336F3.6040601@noa.gr> On 16/3/2012 11:39 ??, Morten Stevens wrote: > See: http://wiki2.dovecot.org/Upgrading/2.1 Thank you Morten. This was exactly what I was looking for. By the way, searching in the wiki2 for "Upgrade" does not locate the upgrade pages, except "Upgrading/1.0". The pages are found when searching for "upgrading". I am wondering whether it would be possible to add keyword(s) to the respective articles, in order to provide better search results. Thanks again, Nick From campbell at cnpapers.com Fri Mar 16 14:54:21 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Fri, 16 Mar 2012 08:54:21 -0400 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F6337FD.4070404@cnpapers.com> On 3/16/2012 7:07 AM, Mauricio L?pez Riffo wrote: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see > that the performance is very poor and receive complaint about delays > of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection > (betwen mtas and nfs storage) All account information work in LDAP > plataform (two servers in replicated mode, no high average or delays > detected in this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all > of service suffer a high load average (about 8 - 20 load average in > each dovecot) and authenticacion takes about 2 -10 seconds (in low > traffic, autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, > like a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = > maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english It doesn't seem to matter what type of hardware you might have, NFS can cause real bottlenecks, even to the point that your machine may report disk errors. Unfortunately, it's an evil necessity in some shops, but any way to eliminate NFS when large throughput is occurring will definitely help. Make sure you're running the latest version of NFS on all machines since V3 and V4 don't always like each other. I don't have a solution for it's replacement other than expensive hardware solutions. steve From mcazzador at gmail.com Fri Mar 16 15:02:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:02:03 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, with this changes first step is passed: I decomment this #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} i active and add absolute path of doveadm dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} But now from server 1 obtain Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no home directory Note: if i send a mail from server2 mail goes correcly in local (server 2) home virtual directory I note that when i launch manually from server1 sync, mysql on server 2 make correct sql to find home user dir On server 2 SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE username = 'matteo at netlite.locale' result are: +------------------------+------+------+------------------------------------------+ | maildir | uid | gid | mail | +------------------------+------+------+------------------------------------------+ | netlite.locale/matteo/ | 1000 | 1000 | /home/domini-posta/netlite.locale/matteo | +------------------------+------+------+------------------------------------------+ this configuration is teh same for server 1 and 2 thank's Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: > Hi, i obtain the same error > > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF > > > i've create vmail users (i've virtual domain netlite.locale (postfix), > mysql backend i receive ana send mail correctly i use imap protocol), > get ssh connection with publick key, > i verify that with su - vmail , vmail find doveadm, i post my > dovecot.conf file because i don't know what is wrong > > vmail exist on every server with publick key > > (server one => 10.0.0.118 > > server two => 10.0.0.122) > > dovecot.conf -> > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > service aggregator { > # give enough permissions for mail processes > # > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > } > service replicator { > # start replication at startup > # > ? ? ? ?process_min_avail = 1 > } > > > service doveadm { > # if you're using a single virtual user, set this to start ssh as vmail > # (not root) > > ? ? ? ?user = vmail > } > > service config { > # needed to grant access to /var/run/dovecot/config for service doveadm > > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > } > > plugin { > ? ? ? # this host replicates to remote host > # > ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 > > # run full synchronization mode every other hour > # (default is every 24 hours) > # > ? ? ? ?replication_full_sync_interval = 1 hour > } > > > idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 > > thank's > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:07:24 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:07:24 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) What does the CPU usage and NFS IOPS usage look like during those times? Meaning is the problem related to disk usage or something else? Note that for POP3 connections you don't get the "OK Logged in" reply until all of the message sizes have been read into memory. > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Are you randomly redirecting users to different Dovecot servers? Dovecot director would work better: http://wiki2.dovecot.org/Director > auth_worker_max_count = 100 Auth workers are irrelevant with LDAP. > lock_method = dotlock fcntl would be faster, if your NFS setup can handle it. > mail_cache_min_mail_count = 5 I'm not really sure if it's a good idea to ever set this anything else than 0. Of course if you have detected that this actually decreases disk IO I'd be interested to see numbers. > mail_full_filesystem_access = yes If your users are sharing the same UID, this means all the users can access each others' mails now! Even if they have different UIDs this is unlikely to be helpful. > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes Is /data also on NFS? Or does each server have its own local indexes? > maildir_copy_with_hardlinks = no This makes IMAP COPY command slower. Is there a reason why you've disabled it? > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } post-login script makes logins slower. What do you do in it? Why only for IMAP, not POP3? > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } These also slow logins down a little bit. v2.1 fixes that. > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } pop3_uidl_format=%u is a little bad, since it doesn't include %v. And you can improve pop3 performance with: pop3_no_flag_changes=yes And if the maildir filenames don't contain S=1234 sizes, this also makes a huge difference: pop3_fast_size_lookups=yes From mcazzador at gmail.com Fri Mar 16 15:10:04 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:10:04 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, Solved! i add at my sql SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail, '/home/domini-posta/netlite.locale/matteo' as home FROM mailbox WHERE username = 'matteo at netlite.locale' Now i've see first replication going!!! thank's everybody I hope my test help someone. Now i proceedd at use the replication system. Il 16 marzo 2012 14:02, Matteo Cazzador ha scritto: > Hi, with this changes first step is passed: > > I decomment this > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > i active and add absolute path of doveadm > > dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm > dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > But now ?from server 1 obtain > > Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no > home directory > > Note: if i send a mail from server2 mail goes correcly in local > (server 2) home virtual directory > > I note that when i launch manually from server1 sync, mysql on server > 2 make correct sql to find home user dir > > On server 2 > > SELECT maildir, 1000 AS uid, 1000 AS gid, > '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE > username = 'matteo at netlite.locale' > > result are: > > +------------------------+------+------+------------------------------------------+ > | maildir ? ? ? ? ? ? ? ?| uid ?| gid ?| mail > ? ? ? ? ? | > +------------------------+------+------+------------------------------------------+ > | netlite.locale/matteo/ | 1000 | 1000 | > /home/domini-posta/netlite.locale/matteo | > +------------------------+------+------+------------------------------------------+ > > this configuration is teh same for server 1 and 2 > > thank's > > Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: >> Hi, i obtain the same error >> >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: >> command not found >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: read() from worker server >> failed: EOF >> >> >> i've create vmail users (i've virtual domain netlite.locale (postfix), >> mysql backend i receive ana send mail correctly i use imap protocol), >> get ssh connection with publick key, >> i verify that with su - vmail , vmail find doveadm, i post my >> dovecot.conf file because i don't know what is wrong >> >> vmail exist on every server with publick key >> >> (server one => 10.0.0.118 >> >> server two => 10.0.0.122) >> >> dovecot.conf -> >> >> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server >> -u%u -l%{lock_timeout} -n%{namespace} >> >> service aggregator { >> # give enough permissions for mail processes >> # >> ? ? ? ?fifo_listener replication-notify-fifo { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> ? ? ? ?unix_listener replication-notify { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> } >> service replicator { >> # start replication at startup >> # >> ? ? ? ?process_min_avail = 1 >> } >> >> >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> >> ? ? ? ?user = vmail >> } >> >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> >> ? ? ? ?unix_listener config { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ?} >> } >> >> plugin { >> ? ? ? # this host replicates to remote host >> # >> ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 >> >> # run full synchronization mode every other hour >> # (default is every 24 hours) >> # >> ? ? ? ?replication_full_sync_interval = 1 hour >> } >> >> >> idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 >> >> thank's >> >> >> -- >> Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. >> ****************************************** >> Ing. Matteo Cazzador >> Email: mcazzador at gmail.com >> ****************************************** > > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:14:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:14:12 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> Message-ID: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> On 16.3.2012, at 11.09, Jernej Porenta wrote: >>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>> (uni_utf8_str_is_valid(vname)) >> .. > We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. > > Whenever . LIST "" "*" is issued, dovecot crashes: > Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot I don't think this is the same Panic as the original one? What is the Panic message now? From tss at iki.fi Fri Mar 16 15:17:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:17:50 +0200 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <20120316100836.GA12049@nicku.org> References: <20120316100836.GA12049@nicku.org> Message-ID: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> On 16.3.2012, at 12.08, Nick Urbanik wrote: > Using head of imaptest with dovecot 2.1, I am attempting to put a > heavy load on the server, with insufficient success. > > I made 15000 user accounts, put them in to a file, one per line. Then > I ran imaptest with > > ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 > pass=SECRETPASSWORD A single imaptest process can't handle that many simultaneous clients. You'd need to run multiple imaptests in parallel. > So what is the best way to put a really heavy load on an imap server? > The documentation describes testing correctness of the server without > really emphasising performance. Yes, imaptest is mainly meant to test server correctness (i.e. for me to test that Dovecot is bugfree). It spends a lot of time checking and tracking things that is irrelevant when you simply want to load the server. You could add no_tracking parameter to get rid of some of it. From tss at iki.fi Fri Mar 16 15:24:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:24:53 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: Message-ID: <1331904293.26095.2.camel@innu> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) > to Apple Mail. Having looked at all options, I have resorted to the, > perhaps, most difficult way: Create folders on the IMAP server, copy > e-mails into them from Outlook, connect Apple Mail and do the reverse. > However, I have hit a wall. > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items > folder, but the process dies at some point. When that happens, Outlook pops > a screen saying "IMAP session state is inconsistent, please relogin". .. > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around > offset=894): msg header has bad magic value Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)? What happens if you now run: doveadm force-resync -u wash at kictanet.or.ke INBOX Does it show any errors? If not, and if you try to copy the mails again, does it still fail? It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage. From nmilas at noa.gr Fri Mar 16 15:39:38 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 15:39:38 +0200 Subject: [Dovecot] ldapi support Message-ID: <4F63429A.6040304@noa.gr> Hi, Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) connections for LDAP lookups (user, password, auth etc.) or only ldap/ldaps (over TCP)? If yes, how do we specify ldapi://localhost in Dovecot configuration files? Thanks, Nick From odhiambo at gmail.com Fri Mar 16 15:51:18 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 16:51:18 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > Yes, it was pretty much empty..actually, I just configured the Dovecot instance yesterday evening and only tested for "correct operation" by sending a test mail to myself and login to POP3/IMAP. Pretty much pristine. And yes, if is FreeBSD ufs. [wash at jaribu ~]$ mount /dev/label/rootfs0 on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) procfs on /proc (procfs, local) linprocfs on /compat/linux/proc (linprocfs, local) /dev/ada1s1a on /disk2 (ufs, local, noatime, soft-updates) > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > Unfortunately, the user has taken the MacBook now so I am unable to test copying again. > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > I should be able to test this again at some point next week, but just to add, I changed the storage to Maildir and I was able to copy all the mails to the IMAP folder without any issue so I think it's something with SIS. PS: I wish I could test this with Outlook running on Windows, but I guess that introduces a completely different environment than what I had on the MacBook, right? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From odhiambo at gmail.com Fri Mar 16 16:00:41 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:00:41 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > > Ok. I have been able to reproduce it anyway. Environment: Windows 8 Consumer Preview, Outlook 2010. I had 415 e-mails in the Inbox. While copying, Outlook popped up an error: IMAP session state is inconsistent, please relogin. Protocol: IMAP Server: 192.168.40.252 Port: 143 Error Code: 0x800CCCDD ...and dovecot.log details at that time can be found here - http://196.200.26.114/~wash/dovecot.log.txt And the output of the force-resync command is: [root at jaribu] /usr/home/wash# cp /var/log/dovecot.log ~wash/public_html/dovecot.log.txt [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.5 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.5 doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.10 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.10 -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From CMarcus at Media-Brokers.com Fri Mar 16 16:03:12 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 10:03:12 -0400 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> References: <20120316100836.GA12049@nicku.org> <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> Message-ID: <4F634820.5040605@Media-Brokers.com> On 2012-03-16 9:17 AM, Timo Sirainen wrote: > imaptest is mainly meant to test server correctness (i.e. for me to > test that Dovecot is bugfree). It spends a lot of time checking and > tracking things that is irrelevant when you simply want to load the > server. You could add no_tracking parameter to get rid of some of it. Maybe imaptest could be duped+modified somehow to produce a new imap_load_test utility...? -- Best regards, Charles From giles at coochey.net Fri Mar 16 16:11:40 2012 From: giles at coochey.net (Giles Coochey) Date: Fri, 16 Mar 2012 14:11:40 +0000 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <4F634A1C.8060501@coochey.net> On 16/03/2012 14:00, Odhiambo Washington wrote: > On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> I have a situation where I need to migrate e-mails from Outlook 2011 >> (Mac) >> Personally I would just use readpst to export the standard Outlook personal storage folders to mbox format... -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From mhlavink at redhat.com Fri Mar 16 16:48:00 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Fri, 16 Mar 2012 15:48:00 +0100 Subject: [Dovecot] dovecot and systemd In-Reply-To: <1331820329.10319.32.camel@innu> References: <4F61EFE8.1000901@redhat.com> <1331820329.10319.32.camel@innu> Message-ID: <4F6352A0.5020200@redhat.com> On 03/15/2012 03:05 PM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: >> What exactly should happen when >> dovecot.conf does not match dovecot.socket configuration? > > Dovecot's systemd code was written by one of you Redhat guys. I had some > similar thoughts when I applied the patch, but didn't really know what > to do about it, so I didn't do anything. So: I don't know. Maybe some > other project has solved this somehow already? > > Dovecot anyway needs its own internal UNIX listeners. Should all > internal inet listeners be disabled? Could Dovecot somehow talk to > systemd and ask what listeners it's using for Dovecot and log warnings > if they don't match? I don't know that match about systemd. I'll forward this to systemd mailing list and I will let you know once I know more. From odhiambo at gmail.com Fri Mar 16 16:50:50 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:50:50 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F634A1C.8060501@coochey.net> References: <1331904293.26095.2.camel@innu> <4F634A1C.8060501@coochey.net> Message-ID: On Fri, Mar 16, 2012 at 17:11, Giles Coochey wrote: > On 16/03/2012 14:00, Odhiambo Washington wrote: > >> On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: >> >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> >>>> I have a situation where I need to migrate e-mails from Outlook 2011 >>>> >>> (Mac) >>> >>> Personally I would just use readpst to export the standard Outlook > personal storage folders to mbox format... > > Outlook 2011 (Mac OS X - Lion) can export everything into (!pst) .olm I haven't no clue whether .olm and .pst are one and the same, but I highly doubt. With Outlook 2011, the guys at Redmond intended to lock the user to Outlook! I have seen Outlook->Apple Mail migration procedures fraught with e-mail loses. Well, not quite a loss because you still have the e-mail inside Outlook, but that idea of running Outlook side-by-side with Apple Mail is not what I want to subject the user to. I also don't want to make them start searching for the e-mails that might not have been migrated and forward them to themselves. Migration should be complete & safe - no losses. There are commercial software out there that's said to be good at this process, but I wanted the "free"method. If there is *readpst *that can migrate from .olm to mbox, then I am willing to give it a shot, but I also want to see if Dovecot gets a fix for whatever problem I am facing. -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 17:05:42 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:05:42 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <1331910342.26095.34.camel@innu> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > Corrupted > > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > > offset=894): msg header has bad magic value > > > Ok. I have been able to reproduce it anyway. It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails. Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest Simply run it for an empty test account as: imaptest host=localhost user=testuser pass=testpass Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful.. From tss at iki.fi Fri Mar 16 17:09:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:09:14 +0200 Subject: [Dovecot] ldapi support In-Reply-To: <4F63429A.6040304@noa.gr> References: <4F63429A.6040304@noa.gr> Message-ID: <1331910554.26095.35.camel@innu> On Fri, 2012-03-16 at 15:39 +0200, Nikolaos Milas wrote: > Hi, > > Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) > connections for LDAP lookups (user, password, auth etc.) or only > ldap/ldaps (over TCP)? > > If yes, how do we specify ldapi://localhost in Dovecot configuration files? OpenLDAP library handles the connections internally. It probably works the same way in Dovecot as in other software that uses OpenLDAP. So I don't know, try ldapi://. From lists at wildgooses.com Fri Mar 16 17:22:07 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:22:07 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F635A9F.2020406@wildgooses.com> On 14/03/2012 10:58, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. > > I also learned something very interesting yesterday concerning SOGo > and dovecot during a sales call with a SOGo rep, but I'll wait and see > if Timo cares to chime in on this one... ;) > If the answer is that he will write a Z-Push/Activesync module for SOGo then I'm all ears! I have been watching SOGo for some time and the main thing I would miss is that every phone I have ever owned has largely limited/broken Funambol based sync and annoyingly working Activesync capability (I own a stream of Nokias...). It seems that although I don't like it, I need activesync support if I want my contacts/calendar on my phone... (I think I can do caldav on some of them, but not cardav on my N9) Apart from that it's a very neat system! Ed W From tss at iki.fi Fri Mar 16 17:26:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:26:33 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <1331911593.26095.47.camel@innu> On Fri, 2012-03-16 at 15:22 +0000, Ed W wrote: > > I also learned something very interesting yesterday concerning SOGo > > and dovecot during a sales call with a SOGo rep, but I'll wait and see > > if Timo cares to chime in on this one... ;) > > > > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) We're also very much wishing for SOGo Activesync, but I'm not planning on writing it myself (but maybe we'll hire someone who will). Annoyingly Microsoft has patented Activesync, so I guess it can't be legally used at least in USA without paying MS. From lists at wildgooses.com Fri Mar 16 17:30:42 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:30:42 +0000 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <1331807624.10319.6.camel@innu> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> Message-ID: <4F635CA2.8080401@wildgooses.com> On 15/03/2012 10:33, Timo Sirainen wrote: > On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: >> I'm curious if anyone has any plugins for AV integration directly into >> dovecot. >> >> Our old pop servers have been scanning messges as they're moved from >> new->cur in the inbox and, at least where user's aren't poping every >> few seconds, there is occasionally enough time between scanning through >> the MXs to message retreval to snag a few more virues with updated >> definitions before they reach customers. >> >> Anyone doing anything similar? > http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a > script that modifies a mail while it's being read. You could make it run > a virus check, and if that happens you could change the virus MIME part > to be full of spaces (better not to change message size, line count or > MIME structure). > > Couple of other ideas: 1) Could use one of the (buggy and variously unsupported) on access virus scanners. I think Dazuko is now abandoned, but this is a new one mentioned via the Clamav site: http://www.fsl.cs.sunysb.edu/docs/avfs-security04/index.html 2) Extremely racey, but if you were on maildir you could use some kind of pre-login scripting to kick off a scan on login. Touch some lock file so that you can tell when last scanned and only scan if the definitions have been updated since you last scanned? 3) There are some POP proxies which offer inline virus scanning. Could place one in front of your mail server. Presumably this will expose you to all the bugs in that proxy... Good luck Ed W From CMarcus at Media-Brokers.com Fri Mar 16 17:45:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 11:45:58 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <4F636036.4020604@Media-Brokers.com> On 2012-03-16 11:22 AM, Ed W wrote: > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) While I agree it would be nice, why not just switch to a supported phone and be done with it? ;) When we roll out SOGo, we'll only be supporting the officially supported mobile clients (android, iphone/ipad, blackberry and windows mobile)... -- Best regards, Charles From odhiambo at gmail.com Fri Mar 16 17:57:29 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 18:57:29 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 18:05, Timo Sirainen wrote: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > > Corrupted > > > > dbox file /var/spool/virtual/ > kictanet.or.ke/wash/mdbox/storage/m.4(around > > > > offset=894): msg header has bad magic value > > > > > Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > > > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I had to do that, but things don't appear good at all. [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest host=localhost user=wash at kictanet.or.ke pass=XXX Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox You are welcome to access my box if you like, but I must warn you, things will be slow. Internationa links are slow as a result of fiber cuts in MSA, KE! -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 18:09:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:09:58 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <9AD845E2-7922-4E4A-A7ED-F1C2EE55F31A@iki.fi> On 16.3.2012, at 17.57, Odhiambo Washington wrote: >> imaptest host=localhost user=testuser pass=testpass >> >> Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else >> can report that they can reproduce this problem that would be helpful.. >> > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I > had to do that, but things don't appear good at all. No, you can also specify it as mbox=path parameter. > [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest > host=localhost user=wash at kictanet.or.ke pass=XXX > Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox But don't point the mbox there! You'll need to download http://www.dovecot.org/tmp/dovecot-crlf file and point the mbox to that. It's used to upload mails. From e-frog at gmx.de Fri Mar 16 18:16:49 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 17:16:49 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <4F636771.1000308@gmx.de> On 16.03.2012 16:05, wrote Timo Sirainen: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>> Corrupted >>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>> offset=894): msg header has bad magic value >>> This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related. http://www.dovecot.org/list/dovecot/2012-March/064462.html >> Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > From lists at wildgooses.com Fri Mar 16 18:17:19 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 16:17:19 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F636036.4020604@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> <4F636036.4020604@Media-Brokers.com> Message-ID: <4F63678F.9080104@wildgooses.com> On 16/03/2012 15:45, Charles Marcus wrote: > On 2012-03-16 11:22 AM, Ed W wrote: >> If the answer is that he will write a Z-Push/Activesync module for SOGo >> then I'm all ears! I have been watching SOGo for some time and the main >> thing I would miss is that every phone I have ever owned has largely >> limited/broken Funambol based sync and annoyingly working Activesync >> capability (I own a stream of Nokias...). It seems that although I >> don't like it, I need activesync support if I want my contacts/calendar >> on my phone... (I think I can do caldav on some of them, but not cardav >> on my N9) > > While I agree it would be nice, why not just switch to a supported > phone and be done with it? ;) > > When we roll out SOGo, we'll only be supporting the officially > supported mobile clients (android, iphone/ipad, blackberry and windows > mobile)... > That implies you will be using cardav/caldav on those phones? I thought Android support was quite weak for those? I definitely don't like the idea of supporting activesync, but it seems like the only widely supported solution to pushing calendar and contacts updates to clients? Caldav gets you part of the way there, but cardav seems badly supported and there is no push support with either... Out of curiousity, what kind of performance are you getting out of the web interface and any tricks you used to improve "perceived" performance? My quick testing gave something circa 150-200ms response times from SOGo (forget exactly now) and as a result it was perceivable and just very slightly laggy (versus a desktop mail program!!). I get slightly better perceived performance from Roundcube (which also seems more amenable to building extension plugins) Seems a bit of a surprise that a compiled language delivers results slightly less quickly than PHP... Did you find any magic knobs to twist to get performance up there with gmail? Cheers Ed W From tss at iki.fi Fri Mar 16 18:57:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:57:57 +0200 Subject: [Dovecot] v2.1.3 released Message-ID: <1235E4DC-130A-4CE7-9C22-C6180062D914@iki.fi> http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz.sig Do not use v2.1.2 with multi-dbox format, it's broken. I didn't notice that a small optimization I did a few days ago broke mdbox in some situations (when mdbox first created a new m.X file, and later in same session saved another message to it). It's quite a high priority for me to run some automated testing before releasing new versions, for example a small imaptest run with mdbox would have caught this. Perhaps the next release will already have the automated testing. From tss at iki.fi Fri Mar 16 19:00:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:00:57 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F636771.1000308@gmx.de> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: On 16.3.2012, at 18.16, e-frog wrote: > On 16.03.2012 16:05, wrote Timo Sirainen: >> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>> Corrupted >>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>> offset=894): msg header has bad magic value >>>> > > This is kind of the same error message I'm getting with doveadm backup. > I can reproduce this at will. Not sure however if this is related. > > http://www.dovecot.org/list/dovecot/2012-March/064462.html Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. From e-frog at gmx.de Fri Mar 16 19:24:38 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 18:24:38 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: <4F637756.3020707@gmx.de> On 16.03.2012 18:00, wrote Timo Sirainen: > On 16.3.2012, at 18.16, e-frog wrote: > >> On 16.03.2012 16:05, wrote Timo Sirainen: >>> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>>> Corrupted >>>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>>> offset=894): msg header has bad magic value >>>>> >> >> This is kind of the same error message I'm getting with doveadm backup. >> I can reproduce this at will. Not sure however if this is related. >> >> http://www.dovecot.org/list/dovecot/2012-March/064462.html > > Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. Ok, just tested with 2.1.3 and it works again. Thanks Timo! From kgc at corp.sonic.net Fri Mar 16 19:49:46 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:49:46 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <4F637D3A.1000301@corp.sonic.net> On 03/16/12 06:07, Timo Sirainen wrote: > Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. Timo, can you explain why Maildir isn't a good for POP3 in this context? Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. -K From kgc at corp.sonic.net Fri Mar 16 19:52:58 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:52:58 -0700 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F635CA2.8080401@wildgooses.com> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> Message-ID: <4F637DFA.9070800@corp.sonic.net> On 03/16/12 08:30, Ed W wrote: > 2) Extremely racey, but if you were on maildir you could use some kind > of pre-login scripting to kick off a scan on login. Touch some lock file > so that you can tell when last scanned and only scan if the definitions > have been updated since you last scanned? I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Still need to take a look at Timo's patch set. -K From tss at iki.fi Fri Mar 16 19:54:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:54:06 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F637D3A.1000301@corp.sonic.net> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> Message-ID: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> On 16.3.2012, at 19.49, Kelsey Cummings wrote: > On 03/16/12 06:07, Timo Sirainen wrote: >> Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > > Timo, can you explain why Maildir isn't a good for POP3 in this context? Compared to mbox/mdbox: It needs to read and delete multiple separate files, which is typically much slower than reading and deleting a single file. > Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. You mean deleting the messages after +OK, instead of before? Does it really make a difference?.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. From tss at iki.fi Fri Mar 16 19:55:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:55:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F637DFA.9070800@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> <4F637DFA.9070800@corp.sonic.net> Message-ID: <044DBE43-7DEA-45D5-A34A-41F7E0FB8345@iki.fi> On 16.3.2012, at 19.52, Kelsey Cummings wrote: > On 03/16/12 08:30, Ed W wrote: >> 2) Extremely racey, but if you were on maildir you could use some kind >> of pre-login scripting to kick off a scan on login. Touch some lock file >> so that you can tell when last scanned and only scan if the definitions >> have been updated since you last scanned? > > I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Whatever you do: Don't modify existing message files (without renaming them so they appear as new mails). IMAP (and Dovecot) require that messages never change. From kgc at corp.sonic.net Fri Mar 16 21:04:19 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 12:04:19 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> Message-ID: <4F638EB3.6040802@corp.sonic.net> On 03/16/12 10:54, Timo Sirainen wrote: >> Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. > > You mean deleting the messages after +OK, instead of before? Does it really make a difference? In the context of a clients "send and receive" phase taking a (small) fraction of a second less time, perhaps, but it is a small difference in any case. It was one of many small changes we made to try to improve interactive performance. >.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. True, we decided that loosing that ability didn't really matter. (Like not counting newlines as two bytes in the message size.) -K From alex.handle at gmail.com Fri Mar 16 22:00:05 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:00:05 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection Message-ID: Hi Dovecot-list! My setup consists of a dovecot server with lmtp delivery and 3 postfix mta servers in front. Previously the mtas authenticated (SASL) through the courier-authdaemond software to our mysql database. To get support for more password formats i migrated to dovecot for SASL authentification. Our postfix mtas connect to dovecot through a tcp-socket smtpd_sasl_auth_enable = yes smtpd_sasl_type = dovecot smtpd_sasl_path = inet:10.11.100.230:12345 smtpd_sasl_security_options = noanonymous smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = broken_sasl_auth_clients = yes 10.11.100.230 is our dovecot server. Please look at the attached doveconf.log to see my auth service configuration. I did the configuration according to the postfix SASL README. http://www.postfix.org/SASL_README.html#server_dovecot I tested the setup and everything worked fine but after 2 days i noticed these error messages in my mail.log: dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of existing connection and also these messages from postfix: SASL LOGIN authentication failed: Connection lost to authentication server I get the dovecot error message about 3000 times a day and postfix message about 270 times. Please see my attached mail.log for a detailed trace. Thank you for your help :) Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: doveconf.log Type: application/octet-stream Size: 14009 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mail.log Type: application/octet-stream Size: 3120 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 22:14:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 22:14:28 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: Message-ID: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> On 16.3.2012, at 22.00, Alex Ha wrote: > dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of > existing connection Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. -------------- next part -------------- A non-text attachment was scrubbed... Name: tcp-auth.diff Type: application/octet-stream Size: 718 bytes Desc: not available URL: From alex.handle at gmail.com Fri Mar 16 22:39:42 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:39:42 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 22.00, Alex Ha wrote: > >> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >> existing connection > > Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. > Thanks Timo! I will try the patch and report to you. Alex From lists at wiesinger.com Sat Mar 17 08:31:12 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Sat, 17 Mar 2012 07:31:12 +0100 (CET) Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> References: <4F62D762.7080607@wiesinger.com> <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> Message-ID: On Fri, 16 Mar 2012, Timo Sirainen wrote: > On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: >> Calling imap still fails as non root: >> imap >> /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied >> collect2: ld returned 1 exit statusn > > Huh? That looks like imap is running ld to link something. It shouldn't be doing that. After starting it once as root the following files are created and it works also as non root: ls -l /usr/local/bin/.libs/ total 1160 -rwxr-xr-x 1 root root 235848 Aug 25 2010 lt-doveconf -rwxr-xr-x 1 root root 938454 Mar 16 07:03 lt-imap Before only one of these files was generated (I think lt-doveconf). Strange. Any ideas? Ciao, Gerhard -- http://www.wiesinger.com/ From hsn at filez.com Sat Mar 17 08:36:22 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 07:36:22 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs Message-ID: <4F6430E6.6040100@filez.com> Is there way to import old plain mboxes via dsync? It complains about lack of index files: ponto:(admin)~>dsync mirror mbox:~/mail dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files From tss at iki.fi Sat Mar 17 12:32:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 12:32:06 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F6430E6.6040100@filez.com> References: <4F6430E6.6040100@filez.com> Message-ID: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> On 17.3.2012, at 8.36, Radim Kolar wrote: > Is there way to import old plain mboxes via dsync? It complains about lack of index files: > > ponto:(admin)~>dsync mirror mbox:~/mail > dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? From varia at e-healthexpert.org Sat Mar 17 16:14:24 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Sat, 17 Mar 2012 14:14:24 +0000 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <20120317141424.1c629e46@e-healthexpert.org> On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: > On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > pop3_no_flag_changes=yes Is it the same as pop3_no_flag_updates=yes ? M. From tss at iki.fi Sat Mar 17 16:40:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 16:40:59 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <20120317141424.1c629e46@e-healthexpert.org> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <20120317141424.1c629e46@e-healthexpert.org> Message-ID: <1FE9581E-498E-4AE7-800D-4038DA32BB73@iki.fi> On 17.3.2012, at 16.14, Mark Alan wrote: > On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: >> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > >> pop3_no_flag_changes=yes > > Is it the same as pop3_no_flag_updates=yes ? Yeah. I wrote it from my memory. From hsn at filez.com Sat Mar 17 19:29:07 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 18:29:07 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> Message-ID: <4F64C9E3.7080102@filez.com> > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? indexes never existed because these mboxes were never used by dovecot, its not conversion from one format to another, its import. Maybe open bug to add feature "dsync import" which will not depend on existing indexes? From kayasaman at gmail.com Sat Mar 17 20:31:33 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 18:31:33 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice Message-ID: <4F64D885.8000501@gmail.com> Hi, I am currently in the process of setting up an IMAP repository for round 100 users.... Currently the user authentication method is being handled via a Windows Domain Controller. The host OS for Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? Can anyone give me a hand with this? Regards, Kaya From sven at svenhartge.de Sat Mar 17 21:36:15 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 20:36:15 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> Message-ID: <88l5d3on0ev8@mids.svenhartge.de> Kaya Saman wrote: > I am currently in the process of setting up an IMAP repository for > round 100 users.... > Currently the user authentication method is being handled via a > Windows Domain Controller. > The host OS for Dovecot will either be FreeBSD or CentOS. > Would Dovecot be able to authenticate to either the DC directly or > would we need to go through LDAP?? Why not join the server to the domain and simply use PAM? Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you. > Additionally what would be the best method to store the **mail** > information? - as in MySQL database or Maildir format; coinciding with > this what is the best backup method in order to be able to do 'dump' > backups or restore single emails?? Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress. To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots. Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore. Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself. Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory. Some, like mbox or mdbox are a little bit more complex, but with the correct doveadm command you are nevertheless able to restore single mails. Gr??e, Sven. -- Sigmentation fault. Core dumped. From pstm.spain at gmail.com Sat Mar 17 21:42:27 2012 From: pstm.spain at gmail.com (PSTM) Date: Sat, 17 Mar 2012 20:42:27 +0100 Subject: [Dovecot] Problem managing mbox Message-ID: <4F64E923.5060401@gmail.com> Hello, I have a problem with dovecot. seems that do not erase mail that mail client request to be erased. And I have this errors: > Error: Next message unexpectedly corrupted in mbox file Info: > dovecot-2.1.1-2.0.cf.fc16.i686 > root 5979 0.0 0.1 3208 1260 ? Ss 20:18 0:00 > /usr/sbin/dovecot -F > dovenull 5985 0.0 0.2 7060 2280 ? S 20:18 0:00 > dovecot/imap-login > vmail 5988 0.0 0.1 7888 1848 ? S 20:18 0:00 > dovecot/imap permissions on mail dir: > total 4 > drwxr-xr-x 9 vmail mail 4096 ene 21 21:43 vmail Any suggestion? Regards, -- -- http://www.0pc.eu/ From kayasaman at gmail.com Sat Mar 17 21:55:35 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 19:55:35 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <88l5d3on0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> Message-ID: <4F64EC37.5010309@gmail.com> On 03/17/2012 07:36 PM, Sven Hartge wrote: > Kaya Saman wrote: > >> I am currently in the process of setting up an IMAP repository for >> round 100 users.... >> Currently the user authentication method is being handled via a >> Windows Domain Controller. >> The host OS for Dovecot will either be FreeBSD or CentOS. >> Would Dovecot be able to authenticate to either the DC directly or >> would we need to go through LDAP?? > Why not join the server to the domain and simply use PAM? > > Using ActiveDirectory through LDAP is a bit of a pain so I would avoid > this if I where you. Danke Sven :-) I don't actually have much AD/LDAP integration experience so I will try your method! >> Additionally what would be the best method to store the **mail** >> information? - as in MySQL database or Maildir format; coinciding with >> this what is the best backup method in order to be able to do 'dump' >> backups or restore single emails?? > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. > > To have a consistent backup, your mail storage should be able to > snapshot the volume the mail is stored on, so use LVM or an external > storage unit capable of snapshots. Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. > > Then backup the content of the snapshot using any program you like. > I use Bacula for long-term offsite storage and a local rsnapshot to keep > 7 days worth of mail for a quick restore. To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server. > > Whether you are able to restore single mails or the complete storage is > no property or feature of the mailbox format itself. > > Some formats are simpler to handle, like Maildir++, where you just drop > the file containing a mail into a directory. You mention Maildir++... is this Maildir format or something new which I haven't heard about yet? > > Some, like mbox or mdbox are a little bit more complex, but with the > correct doveadm command you are nevertheless able to restore single > mails. > > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sat Mar 17 23:03:21 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 22:03:21 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> Message-ID: <98l5i9bn0ev8@mids.svenhartge.de> Kaya Saman wrote: > On 03/17/2012 07:36 PM, Sven Hartge wrote: >> Kaya Saman wrote: >>> I am currently in the process of setting up an IMAP repository for >>> round 100 users.... Currently the user authentication method is >>> being handled via a Windows Domain Controller. The host OS for >>> Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to >>> authenticate to either the DC directly or would we need to go >>> through LDAP?? >> Why not join the server to the domain and simply use PAM? >> Using ActiveDirectory through LDAP is a bit of a pain so I would >> avoid this if I where you. > I don't actually have much AD/LDAP integration experience so I will > try your method! Question: do you need public or shared folders? Using samba and winbindd to join a domain creates real users on your server and as far as I know configuring shared folders with real users is a bit of a pain, especially of you need shared flags (like Seen, Replied, etc.) (Someone [Timo?] please correct me.) >>> Additionally what would be the best method to store the **mail** >>> information? - as in MySQL database or Maildir format; coinciding >>> with this what is the best backup method in order to be able to do >>> 'dump' backups or restore single emails?? >> Storing mails inside SQL? Not supported by dovecot and not very wise, >> IMHO. DBmail does this, but to be honest, I never heard any good >> feedback from admins using that product. From what I have been told, you >> need quite the beefy server to get a decent performance out of DBmail, >> compared to the needs of a "traditional" setup like with dovecot or >> courier-mail, but I digress. >> >> To have a consistent backup, your mail storage should be able to >> snapshot the volume the mail is stored on, so use LVM or an external >> storage unit capable of snapshots. > Hmm..... so FreeBSD coupled together with a ZFS repo for mail should > take care of 'Snapshot' issues. Yes. Or using LVM on Linux. >> Then backup the content of the snapshot using any program you like. >> I use Bacula for long-term offsite storage and a local rsnapshot to >> keep 7 days worth of mail for a quick restore. > To be honest I was considering rsync'ing the dir containing users > mailboxes to either another storage pool or server. No need to rsync, if you use ZFS. Just create a new snapshot and you are done. Bet thing about ZFS: you get deduplication for free, so the needed space to store the backups will not grow as fast. But you still may want to store the mails offsite/offserver for desaster recovery. Either use doveadm backup for that purpose or use rsnapshot, again gaining you deduplication on the target server. >> Whether you are able to restore single mails or the complete storage is >> no property or feature of the mailbox format itself. >> >> Some formats are simpler to handle, like Maildir++, where you just drop >> the file containing a mail into a directory. > You mention Maildir++... is this Maildir format or something new which I > haven't heard about yet? Maildir++ extends the original Maildir with things like Quota and ACLs and was first implemented in Courier. http://www.courier-mta.org/imap/README.maildirquota.html All current MTAs and POP3/IMAP servers implement this variant. Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are). In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands. I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever". Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sat Mar 17 23:24:25 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:24:25 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <98l5i9bn0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> Message-ID: <4F650109.7090702@gmail.com> Thanks so much Sven for your indepth and complete responses! > Question: do you need public or shared folders? I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage. Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange. To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc. > > Using samba and winbindd to join a domain creates real users on your > server and as far as I know configuring shared folders with real users > is a bit of a pain, especially of you need shared flags (like Seen, > Replied, etc.) (Someone [Timo?] please correct me.) Actually we might have an LDAP server already taking care of the AD<->UNIX integration..... I don't know yet it's only my first week :-) > >>>> Additionally what would be the best method to store the **mail** >>>> information? - as in MySQL database or Maildir format; coinciding >>>> with this what is the best backup method in order to be able to do >>>> 'dump' backups or restore single emails?? >>> Storing mails inside SQL? Not supported by dovecot and not very wise, >>> IMHO. DBmail does this, but to be honest, I never heard any good >>> feedback from admins using that product. From what I have been told, you >>> need quite the beefy server to get a decent performance out of DBmail, >>> compared to the needs of a "traditional" setup like with dovecot or >>> courier-mail, but I digress. >>> >>> To have a consistent backup, your mail storage should be able to >>> snapshot the volume the mail is stored on, so use LVM or an external >>> storage unit capable of snapshots. >> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >> take care of 'Snapshot' issues. > Yes. Or using LVM on Linux. Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too...... > >>> Then backup the content of the snapshot using any program you like. >>> I use Bacula for long-term offsite storage and a local rsnapshot to >>> keep 7 days worth of mail for a quick restore. >> To be honest I was considering rsync'ing the dir containing users >> mailboxes to either another storage pool or server. > No need to rsync, if you use ZFS. Just create a new snapshot and you are > done. Bet thing about ZFS: you get deduplication for free, so the needed > space to store the backups will not grow as fast. Ok so that solves that! :-) > But you still may want to store the mails offsite/offserver for desaster > recovery. They are currently being stored on the parent company mail server so this will be the/off-site/ disaster recovery system in a way :-P > > Either use doveadm backup for that purpose or use rsnapshot, again > gaining you deduplication on the target server. I will research this - thank you for that info :-) > >>> Whether you are able to restore single mails or the complete storage is >>> no property or feature of the mailbox format itself. >>> >>> Some formats are simpler to handle, like Maildir++, where you just drop >>> the file containing a mail into a directory. >> You mention Maildir++... is this Maildir format or something new which I >> haven't heard about yet? > Maildir++ extends the original Maildir with things like Quota and ACLs > and was first implemented in Courier. > http://www.courier-mta.org/imap/README.maildirquota.html > > All current MTAs and POP3/IMAP servers implement this variant. > > Depending on the amount of mail a user collects inside a folder, Maildir > is not the best storage format. You may want to check into mdbox, if > your users are kind of "mail hoarders" (like some of my users are). > > In my opinion, Maildir has outlived its usefullnes. It was fine when > users had 1,000 mails in some 10 folders, but today, users collect over > 100,000 mails a year and Maildir is causing serious I/O trouble and the > need to heavily fine tune your storage and filesystems to cope with > those demands. > > I cannot thank Timo enough for inventing mdbox, as this format breaks > this viciuos cycle and, as someone else said "it ends the battle at the > I/O front forever". So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!} Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability. However, if like you say mdbox is the way to go then I will put a strong case together! > > Gr??e, > Sven. > Regards, Kaya From stsiol at yahoo.co.uk Sat Mar 17 23:51:28 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sat, 17 Mar 2012 21:51:28 +0000 (GMT) Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F64D885.8000501@gmail.com> References: <4F64D885.8000501@gmail.com> Message-ID: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> >Hi, > >I am currently in the process of setting up an IMAP repository for round 100 users.... > >Currently the user authentication method is being handled via a Windows Domain Controller. > >The host OS for Dovecot will either be FreeBSD or CentOS. > > >Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? > > >Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? > > >Can anyone give me a hand with this? > > >Regards, > > >Kaya Hi Kaya, I can't force you to follow a specific path. All I can do, is tell you my experience on this. Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for Webmail, OpenLDAP for LDAP (no windows software there) and CentOS v5.5 32-bit onwards. User base is about 30 users. System uptime without a glitch reached at some point (had to reboot the server for maintenance reasons) about 200 days. I am sure it would go beyond 365 days. Hope this helps, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From kayasaman at gmail.com Sat Mar 17 23:53:44 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:53:44 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> References: <4F64D885.8000501@gmail.com> <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> Message-ID: <4F6507E8.4060204@gmail.com> On 03/17/2012 09:51 PM, Spyros Tsiolis wrote: >> Hi, >> >> I am currently in the process of setting up an IMAP repository for round 100 users.... >> >> Currently the user authentication method is being handled via a Windows Domain Controller. >> >> The host OS for Dovecot will either be FreeBSD or CentOS. >> >> >> Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? >> >> >> Additionally > what would be the best method to store the **mail** information? - as > in MySQL database or Maildir format; coinciding with this what is the > best backup method in order to be able to do 'dump' backups or restore > single emails?? >> >> Can anyone give me a hand with this? >> >> >> Regards, >> >> >> Kaya > > > > Hi Kaya, > > I can't force you to follow a specific path. > All I can do, is tell you my experience on this. > > Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for > Webmail, OpenLDAP for LDAP (no windows software > there) and CentOS v5.5 32-bit onwards. > User base is about 30 users. > System uptime without a glitch reached at some point > (had to reboot the server for maintenance reasons) about > 200 days. I am sure it would go beyond 365 days. > > Hope this helps, > > spyros > > > > ---- > "I merely function as a channel that filters > music through the chaos of noise" > - Vangelis Thanks for that Spyros! Regards, Kaya From sven at svenhartge.de Sun Mar 18 00:28:04 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 23:28:04 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: Kaya Saman wrote: > >> Question: do you need public or shared folders? > I don't need anything apart from an IMAP storage solution. I don't > intend to tie in Dovecot with an MTA either as I will simply be using > this for storage. > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. So, how do you plan to get the mails into this storage? offlineimap? imapsync? mbsync? fetchmail? >>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>> take care of 'Snapshot' issues. >> Yes. Or using LVM on Linux. > Yeah.... true but I specified ZFS as I'm a fan and also am quite > comfortable with Solaris/*BSD too...... If you know ZFS and are familiar with it, then, by all means, go for it. >> Depending on the amount of mail a user collects inside a folder, >> Maildir is not the best storage format. You may want to check into >> mdbox, if your users are kind of "mail hoarders" (like some of my >> users are). >> >> In my opinion, Maildir has outlived its usefullnes. It was fine when >> users had 1,000 mails in some 10 folders, but today, users collect >> over 100,000 mails a year and Maildir is causing serious I/O trouble >> and the need to heavily fine tune your storage and filesystems to >> cope with those demands. >> >> I cannot thank Timo enough for inventing mdbox, as this format breaks >> this viciuos cycle and, as someone else said "it ends the battle at >> the I/O front forever". > So mdbox is a 'new' mailbox standard? ie. one can replace Maildir > format with this and use mdbox instead. {Note to self: time to > browse!} mdbox is a format invented by Timo for dovecot. But dovecot can use nearly all common mailbox formats (except MH, but no one uses that one today). > Since where I'm implementing this is mainly an MS based environment > they are concerned about /flat/ files.... which MS seems to typically > do (although never used MS before so I wouldn't know). So there is > some concern over performance, efficiency and manageability. Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no doubt about this. But even with this crappy format, dovecot is able to deliver astounding performance by use of separete index files which allow it to access the storage in an efficient manner. mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten. mdbox solves this problem by a) storing all meta-data in the index and b) by only ever appending to a mdbox storage file, c) never truncating an existing mdbox storage file and d) using more than one mdbox storage file. Max size and TTL are configurable. But this also means deleted mails are still inside a mdbox storage file and need to be finally removed by copying all remaining files into a new file. This process has to be manually run during low traffic hours, for example using a cronjob. You can say, mdbox is like mbox on steroids. ;) Flat files are not evil or bad or slow per se, but you have to use them the right way. > However, if like you say mdbox is the way to go then I will put a > strong case together! You may want to start with something familiar and convert later, which is no problem with dovecot. Gr??e, Sven -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 00:35:37 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 22:35:37 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F6511B9.1020801@gmail.com> On 03/17/2012 10:28 PM, Sven Hartge wrote: > Kaya Saman wrote: >> >>> Question: do you need public or shared folders? >> I don't need anything apart from an IMAP storage solution. I don't >> intend to tie in Dovecot with an MTA either as I will simply be using >> this for storage. >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > So, how do you plan to get the mails into this storage? offlineimap? > imapsync? mbsync? fetchmail? Since everything is blocked at the Exchange end, users will have to manually transfer for now through MS Outlook. Currently that's what they're doing to their PST's.... > >>>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>>> take care of 'Snapshot' issues. >>> Yes. Or using LVM on Linux. >> Yeah.... true but I specified ZFS as I'm a fan and also am quite >> comfortable with Solaris/*BSD too...... > If you know ZFS and are familiar with it, then, by all means, go for it. :-) > >>> Depending on the amount of mail a user collects inside a folder, >>> Maildir is not the best storage format. You may want to check into >>> mdbox, if your users are kind of "mail hoarders" (like some of my >>> users are). >>> >>> In my opinion, Maildir has outlived its usefullnes. It was fine when >>> users had 1,000 mails in some 10 folders, but today, users collect >>> over 100,000 mails a year and Maildir is causing serious I/O trouble >>> and the need to heavily fine tune your storage and filesystems to >>> cope with those demands. >>> >>> I cannot thank Timo enough for inventing mdbox, as this format breaks >>> this viciuos cycle and, as someone else said "it ends the battle at >>> the I/O front forever". >> So mdbox is a 'new' mailbox standard? ie. one can replace Maildir >> format with this and use mdbox instead. {Note to self: time to >> browse!} > mdbox is a format invented by Timo for dovecot. But dovecot can use > nearly all common mailbox formats (except MH, but no one uses that one > today). Ok so if you claim that mdbox is the 'best' mailbox storage solution then I'll look at implementing this. > >> Since where I'm implementing this is mainly an MS based environment >> they are concerned about /flat/ files.... which MS seems to typically >> do (although never used MS before so I wouldn't know). So there is >> some concern over performance, efficiency and manageability. > Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no > doubt about this. > > But even with this crappy format, dovecot is able to deliver astounding > performance by use of separete index files which allow it to access the > storage in an efficient manner. > > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. > > mdbox solves this problem by a) storing all meta-data in the index and > b) by only ever appending to a mdbox storage file, c) never > truncating an existing mdbox storage file and d) using more than one > mdbox storage file. Max size and TTL are configurable. > > But this also means deleted mails are still inside a mdbox storage file > and need to be finally removed by copying all remaining files into a new > file. This process has to be manually run during low traffic hours, for > example using a cronjob. > > You can say, mdbox is like mbox on steroids. ;) > > Flat files are not evil or bad or slow per se, but you have to use them > the right way. Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-( Need to make a strong case! > >> However, if like you say mdbox is the way to go then I will put a >> strong case together! > You may want to start with something familiar and convert later, which > is no problem with dovecot. Maildir is what I'm familiar with currently and mbox format - though only use mbox as an unfortunate side product of /system mail/ accounts. Works well with Alpine client though! > > Gr??e, > Sven > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:04:22 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:04:22 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: Kaya Saman wrote: >> Flat files are not evil or bad or slow per se, but you have to use >> them the right way. > Thanks a lot for that info. I will research more into this but I maybe > overridden at some point :-( > Need to make a strong case! Hmm. Just because Microsofts way of usage of flat file database sucks does not mean any usage of flat files is bad or evil or slow, if done right. Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox But as I wrote before, it is quite easy to convert from one format to the other: http://wiki2.dovecot.org/Migration/MailFormat Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:18:30 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:18:30 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: <4F6529D6.60609@gmail.com> On 03/18/2012 12:04 AM, Sven Hartge wrote: > Kaya Saman wrote: > >>> Flat files are not evil or bad or slow per se, but you have to use >>> them the right way. >> Thanks a lot for that info. I will research more into this but I maybe >> overridden at some point :-( >> Need to make a strong case! > Hmm. > > Just because Microsofts way of usage of flat file database sucks does > not mean any usage of flat files is bad or evil or slow, if done right. Coming from a UNIX background I deal quite a lot with this kind of stuff so there's not problem for me. However, where I'm trying to deploy this system is a primarily MS based enterprise meaning that as the only UNIX engineer onsite and the newest addition to the team I have to convince people of working with UNIX technologies or somehow increase UNIX awareness. As a bi-product I know nothing about MS tech. only what it told to me by my colleagues :-) > > Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox I checked that out after your last email... I started Google'ing a little. :-) Looks like it would be a good solution! > > But as I wrote before, it is quite easy to convert from one format to > the other: http://wiki2.dovecot.org/Migration/MailFormat Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system??? To be honest, I run Zimbra @home for my OpenSource work and really enjoy it; in conjunction with Dovecot on FreeBSD which I run imapsync to backup **all** emails to. It works really well...... :-) I have messed around with Postfix, Dovecot and Horde3 in the past which also was really nice. > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:32:41 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:32:41 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: Kaya Saman wrote: > Once we get setup this may come in quite handy! Not sure what's going > on currently as everyone above me is still quite set in using an SQL > DB as a mail storage system??? RDBMS where not designed for such a task. Using a relational database as a storage method for big chunks of data is very unwise, in my opinion. It degrades them to just being some sort of filing cabinet. Now, wouldn't it be nice, if we had something like that, a filing cabinet where we can store large chunks of data and randomly read and write them in a fast manner? Oh yes, I remember, it is called a "filesystem". Let's use some of those to store the mail data. It will be soooo awesome! ;-) Ok, back being serious: there is nothing wrong with using a RDBMS in the way it was intented, to store user credentials, quota values, account settings, forwarding addresses, address book data, bookmarks, etc. Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:42:29 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:42:29 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F652F75.7060901@gmail.com> On 03/18/2012 12:32 AM, Sven Hartge wrote: > Kaya Saman wrote: > >> Once we get setup this may come in quite handy! Not sure what's going >> on currently as everyone above me is still quite set in using an SQL >> DB as a mail storage system??? > RDBMS where not designed for such a task. Using a relational database > as a storage method for big chunks of data is very unwise, in my > opinion. It degrades them to just being some sort of filing cabinet. > > Now, wouldn't it be nice, if we had something like that, a filing > cabinet where we can store large chunks of data and randomly read and > write them in a fast manner? > > Oh yes, I remember, it is called a "filesystem". Let's use some of those > to store the mail data. It will be soooo awesome! ;-) I think for the serious engineer there's Linux if even more serious there's UNIX and for the rest there's MS..... Actually as a medical term MS is something not that great to have; why does that also equate to IT/Computing too ;-P > > > Ok, back being serious: there is nothing wrong with using a RDBMS in the > way it was intented, to store user credentials, quota values, account > settings, forwarding addresses, address book data, bookmarks, etc. I agree! My humble opinion for a personal preference setup in this instance: FreeBSD 8.2 x64 as base OS UFS2 running on root drive Create ZFS pools for storage Have users mailboxes on the ZFS pools Enable ZFS caching and snapshots Dovecot to manage IMAPv4 --- Get rid of MS altogether! ....Then start working a really cool implementation of UNIX/Linux only infrastructure :-) > > > Gr??e, > Sven. > Regards, Kaya From stan at hardwarefreak.com Sun Mar 18 11:16:55 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 18 Mar 2012 04:16:55 -0500 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F650109.7090702@gmail.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F65A807.9020206@hardwarefreak.com> On 3/17/2012 4:24 PM, Kaya Saman wrote: > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. It's been many years since I used, or supported, MS Outlook. That said, for the 10+ years I did support it, ~1996-2006, the corporate version of Outlook, not to be confused with Outlook Express, did not store any mail in local .PST files unless specifically configured to do so. By default it keeps all mail in the user account in the Exchange server store. Thus I would assume these Outlook clients have been manually configured to use .PST files to keep copies of mail locally, for faster access and to keep inefficient MS Exchange (MAPI) traffic off the WAN link? Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off? If the problem isn't with the .PST format for storing the emails, why not simply setup a local Samba server and configure the Outlook clients to store users' PSTs on Samba shares? Better yet, if you already have a file server for home directories, simply use a folder redirection policy to put the PST files in folders in users' home directories. This is an extremely common practice in the MS world because all Microsoft Windows apps store everything in the user profile directory by default, which again, causes big problems with roaming profiles, which many/most enterprises use. -- Stan From kayasaman at gmail.com Sun Mar 18 11:46:13 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 09:46:13 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F65AEE5.30208@gmail.com> On 03/18/2012 09:16 AM, Stan Hoeppner wrote: > On 3/17/2012 4:24 PM, Kaya Saman wrote: > >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > It's been many years since I used, or supported, MS Outlook. That said, > for the 10+ years I did support it, ~1996-2006, the corporate version of > Outlook, not to be confused with Outlook Express, did not store any mail > in local .PST files unless specifically configured to do so. By default > it keeps all mail in the user account in the Exchange server store. > > Thus I would assume these Outlook clients have been manually configured > to use .PST files to keep copies of mail locally, for faster access and > to keep inefficient MS Exchange (MAPI) traffic off the WAN link? > > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? > > If the problem isn't with the .PST format for storing the emails, why > not simply setup a local Samba server and configure the Outlook clients > to store users' PSTs on Samba shares? > > Better yet, if you already have a file server for home directories, > simply use a folder redirection policy to put the PST files in folders > in users' home directories. This is an extremely common practice in the > MS world because all Microsoft Windows apps store everything in the user > profile directory by default, which again, causes big problems with > roaming profiles, which many/most enterprises use. > Thanks Stan for this, currently our users have about 270MB space located on the Exchange server which we have no control over. Therefor users are currently manually backing up their information to .pst. Since all contact/calendar/other information is already stored on the server the IMAP solution is a better one. It also means that a more UNIX/Linux centric approach is being ask for/tolerated which is where I come in being the only full-bread UNIX engineer on site. I couldn't comment on the MS side of things as I have never really used MS stuff before but my plan using ZFS and FreeBSD should be the best. Again it's going to be **scalable** storage which is perfect! -Also easy to maintain; otherwise I don't think there would be anyone left who will be able to admin the SMB/local directory method (I can't as I don't understand MS) - additionally there isn't much space available and bandwidth either meaning we would purchase a dedicated server or build a dedicated server for this (well I would :-) ). Regards, Kaya From tss at iki.fi Sun Mar 18 17:15:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 17:15:46 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> On 18.3.2012, at 0.28, Sven Hartge wrote: > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. Dovecot moves only minimal amount of data within mbox. A flag change writes only a few bytes to mbox file, it doesn't rewrite it the whole file. Only time when the entire mbox file is rewritten is when you expunge the first message. From sven at svenhartge.de Sun Mar 18 17:19:38 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 16:19:38 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: Timo Sirainen wrote: > On 18.3.2012, at 0.28, Sven Hartge wrote: >> mbox has big problems with concurrent writes, the bigger the mbox is, >> the more problems you get. This is mainly caused by the meta-data of >> a message (meaning flags, status, etc.) which is stored inside the >> mbox file itself. Flagging a message as read or replied causes the >> whole mbox file to be rewritten. > Dovecot moves only minimal amount of data within mbox. A flag change > writes only a few bytes to mbox file, it doesn't rewrite it the whole > file. Only time when the entire mbox file is rewritten is when you > expunge the first message. OK, then my memory was clouded by my exposure to the brokenness of uw-imapd and uw-popd, who rewrite an mbox file very often, resulting in abysmal performance. Gr??e, Sven. -- Sigmentation fault. Core dumped. From CMarcus at Media-Brokers.com Sun Mar 18 17:32:21 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:32:21 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F660005.8070902@Media-Brokers.com> On 2012-03-18 5:16 AM, Stan Hoeppner wrote: > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? If so, using redirected folders (if you're not using them, you should be) would alleviate this problem nicely, even in a large environment. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 18 17:36:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:36:25 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: <4F6600F9.1010604@Media-Brokers.com> On 2012-03-18 11:15 AM, Timo Sirainen wrote: > Only time when the entire mbox file is rewritten is when you > expunge the first message. Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Although I have no desire to use mbox (planning on using mdbox), so it isn't important to me... ;) -- Best regards, Charles From janfrode at tanso.net Sun Mar 18 19:52:49 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 18 Mar 2012 18:52:49 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <20120318175249.GA15524@dibs.tanso.net> On Sun, Mar 18, 2012 at 11:36:25AM -0400, Charles Marcus wrote: > > Hmmm... wonder if there would be a way to add some kind of 'dummy' > first message that dovecot would simply ignore (not show to the > user), that would prevent that bevaior? That's what uw-imap does. It creates a message with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA", which is very annoying if your users has direct access to the mbox's... http://www.washington.edu/imap/IMAP-FAQs/index.html#6.14 -jf From tss at iki.fi Sun Mar 18 20:07:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 20:07:34 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <50F4C47F-EE04-4384-9465-89AFCF5C8B67@iki.fi> On 18.3.2012, at 17.36, Charles Marcus wrote: > On 2012-03-18 11:15 AM, Timo Sirainen wrote: >> Only time when the entire mbox file is rewritten is when you >> expunge the first message. > > Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Dovecot has such a dummy first message exactly like UW-IMAP. But what I meant is that if you expunge the first message and you want to free the space used by it, there's no other choice than to rewrite all of the messages after it. (And of course the first message isn't special in any way, there's just about as much of rewriting if you delete the 2nd or 3rd or other messages from the beginning of the mbox.) From arekm at maven.pl Sun Mar 18 23:00:35 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:00:35 +0100 Subject: [Dovecot] mdbox and filesystem quota Message-ID: <201203182200.36011.arekm@maven.pl> http://wiki2.dovecot.org/MailboxFormat/dbox "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. " What happens if there is filesystem hard quota that is exceeded? Will dovecot allow to delete mails to free space without a need to wait for cronjob to do the purge? -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:45:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:45:54 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182200.36011.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> Message-ID: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > http://wiki2.dovecot.org/MailboxFormat/dbox > > "Expunging a message only decreases the message's refcount. The space is later > freed in "purge" step. This is typically done in a nightly cronjob when > there's less disk I/O activity. " > > What happens if there is filesystem hard quota that is exceeded? Will dovecot > allow to delete mails to free space without a need to wait for cronjob to do > the purge? No. Also the purging itself won't work, because it needs to write new data first before it can delete old data. Don't run out of disk space! From arekm at maven.pl Sun Mar 18 23:52:38 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:52:38 +0100 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> Message-ID: <201203182252.38995.arekm@maven.pl> On Sunday 18 of March 2012, Timo Sirainen wrote: > On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > > http://wiki2.dovecot.org/MailboxFormat/dbox > > > > "Expunging a message only decreases the message's refcount. The space is > > later freed in "purge" step. This is typically done in a nightly cronjob > > when there's less disk I/O activity. " > > > > What happens if there is filesystem hard quota that is exceeded? Will > > dovecot allow to delete mails to free space without a need to wait for > > cronjob to do the purge? > > No. Also the purging itself won't work, because it needs to write new data > first before it can delete old data. Don't run out of disk space! Can dovecot treat soft quota like hard quota for user then? Or better enforce quota based on filesystem quot information. With xfs I can set quota but turn enforcement off. All fs quota counters work but no enforcement is being made by xfs itself. -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:56:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:56:48 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182252.38995.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> <201203182252.38995.arekm@maven.pl> Message-ID: <4BC1C8AE-4AD5-4A51-8954-FFAA4B84F35D@iki.fi> On 18.3.2012, at 23.52, Arkadiusz Mi?kiewicz wrote: >>> "Expunging a message only decreases the message's refcount. The space is >>> later freed in "purge" step. This is typically done in a nightly cronjob >>> when there's less disk I/O activity. " >>> >>> What happens if there is filesystem hard quota that is exceeded? Will >>> dovecot allow to delete mails to free space without a need to wait for >>> cronjob to do the purge? >> >> No. Also the purging itself won't work, because it needs to write new data >> first before it can delete old data. Don't run out of disk space! > > Can dovecot treat soft quota like hard quota for user then? > > Or better enforce quota based on filesystem quot information. With xfs I can > set quota but turn enforcement off. All fs quota counters work but no > enforcement is being made by xfs itself. Yes, Dovecot does the quota enforcement itself. I'm not entirely sure if it uses soft or hard quota though. From trashcan at odo.in-berlin.de Mon Mar 19 10:35:34 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Mon, 19 Mar 2012 09:35:34 +0100 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi -- On 15.03.2012 22:05, Timo Sirainen wrote: > On 15.3.2012, at 22.48, Michael Grimm wrote: >> Actually it's a bad idea to use root for ssh from a security point >> of view. A hacked root account isn't fun. Thus, normally one needs >> to explicitly change the config of the sshd daemon to allow root >> logins (at least with FreeBSD what I'm using). Thus, I do recommend >> to use an unprivileged user like vmail. > > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. Root has access to everyone's mail as well. > And if you allow ssh login only with public key authentication I > don't think there are much security issues. And finally, it would > be possible to write a small wrapper that allows the root's public > key auth to only execute dsync-user.sh script that can't do anything > except sync a specified user's mails. All those safety measures can be applied for the vmail user as well. Actually, that's what I did in my case, plus allowing ssh only between both mail servers (firewall rule). Regards, Michael From lcaron at unix-scripts.info Mon Mar 19 11:06:20 2012 From: lcaron at unix-scripts.info (Laurent CARON) Date: Mon, 19 Mar 2012 10:06:20 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot Message-ID: <20120319095939.maneexuo@trusted.unix-scripts.info> Hi, I'm currently having a fairly simple setup: - users (real, not virtual) - Maildir storage (over NFS) - 1 namespace I'm currently trying to render the storage snapshots available through dovecot (to allow my users to browse their mail history). dovecot.conf: namespace { inbox = yes location = prefix = INBOX. type = private } I did the following modifications: dovecot.conf: namespace snaps-h0 { prefix = INBOX.EmailBackup.h0. hidden = no list = yes inbox = no location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u type = private } Problem: I don't see the content of the inbox folder contained in the snapshots (subfolders are perfectly viewed). Do any of you have a clue on how to render it visible ? Thanks Laurent From jernej.porenta at arnes.si Mon Mar 19 11:58:56 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 10:58:56 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> Message-ID: <3974AB53-476A-4945-A828-11425C667165@arnes.si> On Mar 16, 2012, at 2:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 11.09, Jernej Porenta wrote: > >>>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>>> (uni_utf8_str_is_valid(vname)) >>> .. >> We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. >> >> Whenever . LIST "" "*" is issued, dovecot crashes: >> Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot > > I don't think this is the same Panic as the original one? What is the Panic message now? Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) It is the same. We will try 2.1.3 today and report the results... Regards, Jernej From mcazzador at gmail.com Mon Mar 19 12:50:39 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Mon, 19 Mar 2012 11:50:39 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi, i've a simple question, what do you mean for dovecot director setup? 'i've a doubt. The solution that i'm testing is using 3 mail server in different geoghrapic locations. An user can travel in varius location, and i want his imap mail reside on mail server in every locations. S? i use you solution about replication. First server (by dns record) that receive mail sync it on the other servers, and when user consult is mail by imap protocol everything is sync on all servers. Do you suggest to use a horizontal structure for it like i explain or is better to have a single node external mail server and customer locations server like slave? Thank's Il 19 marzo 2012 09:35, Michael Grimm ha scritto: > Hi -- > > > On 15.03.2012 22:05, Timo Sirainen wrote: >> >> On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>> Actually it's a bad idea to use root for ssh from a security point >>> of view. A hacked root account isn't fun. Thus, normally one needs >>> to explicitly change the config of the sshd daemon to allow root >>> logins (at least with FreeBSD what I'm using). Thus, I do recommend >>> to use an unprivileged user like vmail. >> >> >> Then again it's safer to use system user accounts than a single vmail >> account that has access to everyone's emails. > > > Root has access to everyone's mail as well. > > >> And if you allow ssh login only with public key authentication I >> don't think there are much security issues. And finally, it would >> be possible to write a small wrapper that allows the root's public >> key auth to only execute dsync-user.sh script that can't do anything >> except sync a specified user's mails. > > > All those safety measures can be applied for the vmail user as well. > Actually, that's what I did in my case, plus allowing ssh only between > both mail servers (firewall rule). > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Mon Mar 19 13:20:22 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 13:20:22 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages Message-ID: <4F671676.3060809@noa.gr> Hi, We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've been using LTB OpenLDAP packages (http://ltb-project.org/wiki/download#openldap), which get installed in non-standard file system locations. So, I would like to re-build Dovecot packages based on these OpenLDAP libraries, esp. because I see that dovecot RPM packages are built using OpenLDAP v2.3 libraries. I am not much experienced in building RPMs and preparing spec files. In http://dl.atrpms.net/all/dovecot.spec I see: ------------------------------------------------ BuildRequires: openldap-devel, cyrus-sasl-devel ... Obsoletes: %{name}-pgsql < %{epoch}:%{version}-%{release}, %{name}-mysql < %{epoch}:%{version}-%{release}, %{name}-sqlite < %{epoch}:%{version}-%{release}, %{name}-ldap < %{epoch}:%{version}-%{release}, $ Conflicts: %{name}-pgsql > %{epoch}:%{version}-%{release}, %{name}-mysql > %{epoch}:%{version}-%{release}, %{name}-sqlite > %{epoch}:%{version}-%{release}, %{name}-ldap > %{epoch}:%{version}-%{release}, $ ------------------------------------------------ So, I can change the former reference (openldap-devel) to: openldap-ltb-debuginfo, cyrus-sasl-devel Question 1: What other changes should we make in order to specify that we will be using LDAP libraries from: /usr/local/openldap/lib64 and include files from: /usr/local/openldap/include (rather than from /usr/lib64 and /usr/include, respectively, which are the standard file paths used in openldap-devel) Question 2: How the Obsoletes and Conflicts lines should be changed? Question 3: It seems to me (by reading the spec file) that the final Dovecot RPM (and the included executables) does not need any LDAP dynamic library in order to run with LDAP support (because I don't see any dependencies on openldap package). Can somebody please confirm? Any other associated info would be appreciated. Thanks, Nick From rob0 at gmx.co.uk Mon Mar 19 14:20:50 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:20:50 -0500 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <20120319122050.GM4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 09:35:34AM +0100, Michael Grimm wrote: > On 15.03.2012 22:05, Timo Sirainen wrote: > >On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>Actually it's a bad idea to use root for ssh from a security > >>point of view. A hacked root account isn't fun. Thus, normally > >>one needs to explicitly change the config of the sshd daemon to > >>to allow root logins (at least with FreeBSD what I'm using). > >>Thus, I do recommend to use an unprivileged user like vmail. > > > >Then again it's safer to use system user accounts than a single > >vmail account that has access to everyone's emails. > > Root has access to everyone's mail as well. I think you are missing the point, that being: if all your mail are belong to vmail, somebody set up us the bomb if the vmail account is compromised. (Obviously that's true with a root compromise as well, but that is unavoidable. Effects of a root compromise can be limited with technologies like Apparmor and SELinux, but that is difficult to configure properly and only provides limited benefit: compromised root can do everything real root was allowed to do.) The point is: vmail has added a SECOND vulnerable point from which disaster can ensue. If mailbox ownership is distributed among multiple UID/GID, compromise of any one of those only endangers the mails to which it had access. > >And if you allow ssh login only with public key authentication I > >don't think there are much security issues. And finally, it would > >be possible to write a small wrapper that allows the root's public > >key auth to only execute dsync-user.sh script that can't do > >anything except sync a specified user's mails. > > All those safety measures can be applied for the vmail user as > well. Actually, that's what I did in my case, plus allowing ssh > only between both mail servers (firewall rule). Sure, but there too, all your email eggs are in the vmail basket. No, disaster is not imminent nor even likely to ensue, but the fact stands that you and millions of other virtual-only sites do have this additional potential vulnerability. It is well supported in Dovecot to be able to use a unique UID and GID for every virtual mailbox, but management of such a system presents more challenges than the single-vmail-user approach. Consequently the popular virtual frontends don't support it. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From cjeanneret at internux.ch Mon Mar 19 12:28:34 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Mon, 19 Mar 2012 12:28:34 +0200 Subject: [Dovecot] Problem with sieve Message-ID: Hello List! I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA doesn't run sieve, and thus doesn't filter my emails. Here's the sieve configuration: plugin { # Used by both the Sieve plugin and the ManageSieve protocol sieve=/var/local/vmail/%n/dovecot.sieve sieve_dir=/var/local/vmail/%n/sieve sieve_extensions = +notify +imapflags } The managesiege: protocol managesieve { # Specify an alternative address:port the daemon must listen on # (default: *:2000) listen = localhost:2000 managesieve_logout_format = bytes ( in=%i : out=%o ) } (this one is working fine, I can edit the filters through roundcube webmail, and the correct file (/var/local/vmail/%n/dovecot.sieve) is edited) the lda part: protocol lda { postmaster_address = foo at bar.com mail_plugins = sieve } I think all is in place to allow dovecot to use sieve... ? One more thing: dovecot --version 1.2.15 Any help will be welcomed :). Thanks in advance ! Cheers, C. From ngu.antoine at gmail.com Mon Mar 19 14:27:08 2012 From: ngu.antoine at gmail.com (Antoine Nguyen) Date: Mon, 19 Mar 2012 13:27:08 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: 2012/3/19 C?dric Jeanneret > Hello List! > > I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > doesn't run sieve, and thus doesn't filter my emails. > > Here's the sieve configuration: > > plugin { > # Used by both the Sieve plugin and the ManageSieve protocol > sieve=/var/local/vmail/%n/**dovecot.sieve > sieve_dir=/var/local/vmail/%n/**sieve > sieve_extensions = +notify +imapflags > } > > The managesiege: > > protocol managesieve { > # Specify an alternative address:port the daemon must listen on > # (default: *:2000) > listen = localhost:2000 > managesieve_logout_format = bytes ( in=%i : out=%o ) > } > (this one is working fine, I can edit the filters through roundcube > webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > edited) > > the lda part: > > protocol lda { > postmaster_address = foo at bar.com > mail_plugins = sieve > } > > I think all is in place to allow dovecot to use sieve... ? > > One more thing: > > dovecot --version > 1.2.15 > > > Any help will be welcomed :). > > Thanks in advance ! > > Cheers, > > C. > Have you checked the MTA configuration. Does it use dovecot's LDA ? Antoine From rob0 at gmx.co.uk Mon Mar 19 14:32:14 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:32:14 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F671676.3060809@noa.gr> References: <4F671676.3060809@noa.gr> Message-ID: <20120319123213.GN4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 01:20:22PM +0200, Nikolaos Milas wrote: > We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / > RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've > been using LTB OpenLDAP packages > (http://ltb-project.org/wiki/download#openldap), which get > installed in non-standard file system locations. ISTM that herein lies the whole problem. Why did you not rpmbuild your OpenLDAP? That would have avoided all further fuss. Another observation I can offer, unwelcome as it may be: your OS choice was not a good one when you want the features of recent software. Perhaps you should rethink that choice. You have invested much effort in this task. > So, I would like to re-build Dovecot packages based on these > OpenLDAP libraries, esp. because I see that dovecot RPM packages > are built using OpenLDAP v2.3 libraries. > > I am not much experienced in building RPMs and preparing spec > files. And that is really more a question for a CentOS forum than here. > In http://dl.atrpms.net/all/dovecot.spec I see: > > ------------------------------------------------ > BuildRequires: openldap-devel, cyrus-sasl-devel The latter requirement seems curious to me. In what way does Dovecot use Cyrus SASL? -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From jernej.porenta at arnes.si Mon Mar 19 15:27:06 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 14:27:06 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <3974AB53-476A-4945-A828-11425C667165@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: On Mar 19, 2012, at 10:58 AM, Jernej Porenta wrote: > Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt > Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) > > It is the same. We will try 2.1.3 today and report the results... Same thing with 2.1.3 (. LIST "" "*"): Mar 19 14:08:59 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=28438, secured Mar 19 14:09:04 server dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 14:09:04 server dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071811870] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae0718118c6] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071810d83] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae0715500c5] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d6ac] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d826] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2ae07155e221] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2ae07155717c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 14:09:04 server dovecot: imap(username): Fatal: master: service(imap): child 28438 killed with signal 6 (core dumps disabled) The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz # dovecot -n # 2.1.3: /opt/dovecot-2.1.3/etc/dovecot/dovecot.conf # OS: Linux 2.6.18-274.17.1.el5 x86_64 CentOS release 5.7 (Final) default_login_user = nobody disable_plaintext_auth = no login_greeting = Server ready. login_trusted_networks = x.y.z.p/32 mail_location = mbox:~/:INBOX=%h/.mailbox:INDEX=/opt/dovecot2-indexes/%1u/%u mail_plugins = quota mail_privileged_group = dovecot mbox_write_locks = fcntl namespace inbox { inbox = yes location = prefix = separator = / type = private } passdb { args = session=yes dovecot driver = pam } plugin { quota = fs } protocols = imap pop3 service imap-login { inet_listener imap { port = 143 } inet_listener imaps { ssl = no } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { ssl = no } } ssl = no userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_max_userip_connections = 15 mail_plugins = quota imap_quota } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } Regards, Jernej From nmilas at noa.gr Mon Mar 19 15:47:24 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 15:47:24 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319123213.GN4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> Message-ID: <4F6738EC.5050200@noa.gr> On 19/3/2012 2:32 ??, /dev/rob0 wrote: > ISTM that herein lies the whole problem. Why did you not rpmbuild > your OpenLDAP? That would have avoided all further fuss. Thanks for the reply. First, how would I rpmbuild my openldap v2.4.x as a standard CentOS 5 package (i.e. replacing native openldap-2.3.43-25)? If I were more experienced, I could have tried to engineer openldap-2.3.43-25.el5.src.rpm to upgrade the system to use 2.4.x... But still, I haven't seen any OpenLDAP packages attempting to do so, probably because of the tight integration of CentOS with some openldap v2.3 libraries. I think it's good that third-party packages (even of the same software) give the ability to not mess with standard system. The same is true for reputable Symas OpenLDAP packages. So, I simply use LTB OpenLDAP, even though it's installed at non-standard locations. (This has an added benefit of easy migration. You can setup any/all of those on the same system and decide which one to enable at any time.) > Another observation I can offer, unwelcome as it may be: your OS > choice was not a good one when you want the features of recent > software. Perhaps you should rethink that choice. You have invested > much effort in this task. I like CentOS from many aspects as an enterprise server OS. I wouldn't change it. Yet, it's important to me to be able to build/combine non-standard packages. Even with CentOS 6, I would still continue to use LTB OpenLDAP for a number of reasons. It's true that I've invested much effort in this task, but mostly because my knowledge on this subject is very basic. Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. We just try to make things better (and make our life a bit more difficult) :-) > > And that is really more a question for a CentOS forum than here. > True, but I am hoping that there might be some Dovecot RHEL/CentOS packagers in this list, and that would help resolve issues more effectively, as it is a Dovecot-specific (even if for a package thereof) question. So, any help will be appreciated! > The latter requirement seems curious to me. In what way does Dovecot > use Cyrus SASL? Hmm, I can't tell. I hope atrpm packager(s), if present on this list, can provide some feedback. Thanks again, Nick From tss at iki.fi Mon Mar 19 15:53:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:53:40 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: <1332165220.26095.71.camel@innu> On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: > > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > > > > It is the same. We will try 2.1.3 today and report the results... > The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d From tss at iki.fi Mon Mar 19 15:57:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:57:58 +0200 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <20120319095939.maneexuo@trusted.unix-scripts.info> References: <20120319095939.maneexuo@trusted.unix-scripts.info> Message-ID: <1332165478.26095.73.camel@innu> On Mon, 2012-03-19 at 10:06 +0100, Laurent CARON wrote: > I did the following modifications: > dovecot.conf: > namespace snaps-h0 { > prefix = INBOX.EmailBackup.h0. > hidden = no > list = yes > inbox = no > location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u > type = private > } > > Problem: > I don't see the content of the inbox folder contained in the snapshots > (subfolders are perfectly viewed). > > Do any of you have a clue on how to render it visible ? So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| new} directories? The INBOX should be accessible via the "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a newer Dovecot version. From tss at iki.fi Mon Mar 19 16:00:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:00:00 +0200 Subject: [Dovecot] Problem managing mbox In-Reply-To: <4F64E923.5060401@gmail.com> References: <4F64E923.5060401@gmail.com> Message-ID: <1332165600.26095.75.camel@innu> On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: > Hello, > > I have a problem with dovecot. seems that do not erase mail that mail > client request to be erased. Are you sure the clients have actually issued the EXPUNGE command, rather than simply marked the mail with \Deleted flag? > And I have this errors: > > Error: Next message unexpectedly corrupted in mbox file > Info: mbox code isn't perfect, but if this doesn't happen often it shouldn't matter much. doveconf -n output might have been helpful in giving more suggestions. From tss at iki.fi Mon Mar 19 16:02:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:02:43 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F64C9E3.7080102@filez.com> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> <4F64C9E3.7080102@filez.com> Message-ID: <1332165763.26095.77.camel@innu> On Sat, 2012-03-17 at 18:29 +0100, Radim Kolar wrote: > > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? > indexes never existed because these mboxes were never used by dovecot, > its not conversion from one format to another, its import. > > Maybe open bug to add feature "dsync import" which will not depend on > existing indexes? dsync doesn't need existing indexes, it wants to create indexes. If it can't because of e.g. permission issues, you should be able to work around it with: dsync mirror mbox:~/mail:INDEX=/tmp/indexes I might change dsync at some point to work even without permanent mailbox GUIDs, but there are many other more important things to do. From lcaron at lncsa.com Mon Mar 19 16:05:38 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 15:05:38 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <1332165478.26095.73.camel@innu> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> Message-ID: <4F673D32.2060300@lncsa.com> On 19/03/2012 14:57, Timo Sirainen wrote: > So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| > new} directories? The INBOX should be accessible via the > "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a > newer Dovecot version. > It's not actually. Sorry for the obvoius info I didn't give. I'm currently using dovecot 2.0.7 Regards, Laurent From rob0 at gmx.co.uk Mon Mar 19 17:28:20 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 10:28:20 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F6738EC.5050200@noa.gr> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> Message-ID: <20120319152820.GP4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 03:47:24PM +0200, Nikolaos Milas wrote: > On 19/3/2012 2:32 ??, /dev/rob0 wrote: > > >ISTM that herein lies the whole problem. Why did you not rpmbuild > >your OpenLDAP? That would have avoided all further fuss. > > Thanks for the reply. > > First, how would I rpmbuild my openldap v2.4.x as a standard CentOS > 5 package (i.e. replacing native openldap-2.3.43-25)? If I were > more experienced, I could have tried to engineer > openldap-2.3.43-25.el5.src.rpm to upgrade the system to use That's what I would have tried. > 2.4.x... But still, I haven't seen any OpenLDAP packages attempting > to do so, probably because of the tight integration of CentOS with > some openldap v2.3 libraries. I don't have anything to tell you there, and I note that we are now fully off-topic. :) > I think it's good that third-party packages (even of the same > software) give the ability to not mess with standard system. The same > is true for reputable Symas OpenLDAP packages. > > So, I simply use LTB OpenLDAP, even though it's installed at > non-standard locations. Failing the SRPM translation, why not just install into the CentOS standard locations? ... oops, I typed too fast ... > (This has an added benefit of easy migration. You can setup any/all > of those on the same system and decide which one to enable at any > time.) So you are in fact using both the CentOS OpenLDAP and your own version? This does not sound good at all. :( > >Another observation I can offer, unwelcome as it may be: your > >OS choice was not a good one when you want the features of > >recent software. Perhaps you should rethink that choice. You > >have invested much effort in this task. > > I like CentOS from many aspects as an enterprise server OS. I > wouldn't change it. I don't doubt that CentOS/RHEL offers many benefits, but my point here is that in this endeavor you are seeing the drawbacks. > Yet, it's important to me to be able to build/combine non-standard > packages. Even with CentOS 6, I would still continue to use LTB > OpenLDAP for a number of reasons. > > It's true that I've invested much effort in this task, but mostly > because my knowledge on this subject is very basic. And there too, the better forum, with more of the skills you need, would be the CentOS one. :) > Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), > i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. > We just try to make things better (and make our life a bit more > difficult) :-) > > > > >And that is really more a question for a CentOS forum than here. > > > > True, but I am hoping that there might be some Dovecot RHEL/CentOS > packagers in this list, and that would help resolve issues more > effectively, as it is a Dovecot-specific (even if for a package > thereof) question. > > So, any help will be appreciated! > > >The latter requirement seems curious to me. In what way does > >Dovecot use Cyrus SASL? > > Hmm, I can't tell. I hope atrpm packager(s), if present on this > list, can provide some feedback. I was thinking maybe Timo would know. As far as I can tell it doesn't. I do see in configure.in's check for LDAP, a search for sasl.h or sasl/sasl.h, so it appears that Cyrus SASL might be required to build Dovecot's LDAP support. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From nmilas at noa.gr Mon Mar 19 18:01:01 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 18:01:01 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319152820.GP4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> <20120319152820.GP4603@harrier.slackbuilds.org> Message-ID: <4F67583D.2000309@noa.gr> On 19/3/2012 5:28 ??, /dev/rob0 wrote: > So you are in fact using both the CentOS OpenLDAP and your own > version? This does not sound good at all. :( I talked about migration, didn't I? (Helps in test environments too!) > And there too, the better forum, with more of the skills you need, > would be the CentOS one. :) > > ... > > And that is really more a question for a CentOS forum than here. > I guess I might have to subscribe to CentOS forum/mailing list... (I haven't needed to yet.) Thanks anyway, Nick From hsn at filez.com Mon Mar 19 18:01:18 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:01:18 +0100 Subject: [Dovecot] INBOX cant be created Message-ID: <4F67584E.7030309@filez.com> Inbox does not exists on disk, but following command sequence will not create it. 2 select INBOX 2 NO Mailbox doesn't exist: INBOX 3 create INBOX 3 NO [ALREADYEXISTS] Mailbox already exists: INBOX i think its bug From lcaron at lncsa.com Mon Mar 19 18:11:04 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 17:11:04 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <4F673D32.2060300@lncsa.com> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> <4F673D32.2060300@lncsa.com> Message-ID: <4F675A98.10302@lncsa.com> On 19/03/2012 15:05, Laurent CARON wrote: > On 19/03/2012 14:57, Timo Sirainen wrote: >> So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| >> new} directories? The INBOX should be accessible via the >> "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a >> newer Dovecot version. >> > > It's not actually. > > Sorry for the obvoius info I didn't give. > > I'm currently using dovecot 2.0.7 > > Regards, > > Laurent Upgrading did the trick. Thanks From hsn at filez.com Mon Mar 19 18:22:44 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:22:44 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67584E.7030309@filez.com> References: <4F67584E.7030309@filez.com> Message-ID: <4F675D54.4020203@filez.com> doveadm does not works too: sudo doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied sudo doveadm mailbox create -u admin INBOX.2 (works) From patrickdk at patrickdk.com Mon Mar 19 18:37:28 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Mon, 19 Mar 2012 12:37:28 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> I'm having this problem also, with a very very few users. But in my case the email isn't double gzip, just single like normal. Error: read(.../.Deleted Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output error (uid=250) All I have to do is rename the file to add back the lost S= part and all is fine. This has happened in the inbox, deleted, and trash folders so far. and always after a change, the S= exists for new emails. It's like it's loosing it on adding the read flag, and mailbox moves But out of millions of emails, only a very few are like this, that I know of, around 6 emails. I manually fixed them, will be looking to see if this issue comes back. Quoting Timo Sirainen : > On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: > >>> Alternatively you can just tell Dovecot not to care about it: >>> maildir_broken_filename_sizes=yes. Although you probably can't do >>> that if you have compressed mails. >> >> In the case above that mail was gzipped twice :( > > Yes, looks like Dovecot can't correctly fix the wrong S size for > gzipped mails. I don't know if I should bother fixing it, especially > since in your case the doubly-gzipped mails will look corrupted to > user.. From alexwbaule at gmail.com Mon Mar 19 19:04:12 2012 From: alexwbaule at gmail.com (Alex Baule) Date: Mon, 19 Mar 2012 14:04:12 -0300 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F675D54.4020203@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: doveadm(admin): Error: Can't create mailbox INBOX: Permission denied The INBOX exists but has a wrong owner. Em 19 de mar?o de 2012 13:22, Radim Kolar escreveu: > doveadm does not works too: > > sudo doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > sudo doveadm mailbox create -u admin INBOX.2 > (works) > From hsn at filez.com Mon Mar 19 20:23:04 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 19:23:04 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: <4F677988.9080403@filez.com> > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > > The INBOX exists but has a wrong owner. nope ponto# cd /var/mail ponto# mv admin/ admin.X ponto# doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied but it might be that ordinary user admin cant create directories in /var/mail message from IMAP reply is wrong for sure because mailbox does not exists: ponto# cd /var/mail ponto# mv admin admin.x ponto# telnet localhost imap 3 select inbox 3 NO Mailbox doesn't exist: INBOX 4 create INBOX 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX From Ralf.Hildebrandt at charite.de Mon Mar 19 20:35:47 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Mon, 19 Mar 2012 19:35:47 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> Message-ID: <20120319183547.GA28363@charite.de> * Patrick Domack : > I'm having this problem also, with a very very few users. > > But in my case the email isn't double gzip, just single like normal. > > Error: read(.../.Deleted > Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output > error (uid=250) > > All I have to do is rename the file to add back the lost S= part and > all is fine. > This has happened in the inbox, deleted, and trash folders so far. > and always after a change, the S= exists for new emails. It's like > it's loosing it on adding the read flag, and mailbox moves Yes, I'm also seeing it now with mailboxes where no mail is doubly gzipped. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From alex.handle at gmail.com Mon Mar 19 21:16:58 2012 From: alex.handle at gmail.com (Alex Ha) Date: Mon, 19 Mar 2012 20:16:58 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:39 PM, Alex Ha wrote: > On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: >> On 16.3.2012, at 22.00, Alex Ha wrote: >> >>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>> existing connection >> >> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >> > > Thanks Timo! I will try the patch and report to you. > Hi Timo! I tried the patch with 2.0.19 and the dovecot error messages disappeared. I still get a lot of this postfix warnings: SASL LOGIN authentication failed: Connection lost to authentication server but only for ips which tried a sasl brute force attack. "Connection lost to authentication server" could this be because of the dovecot auth penalties? so far i did not get any complaints from users. Thanks for your help! Alex From dovecot at r.paypc.com Mon Mar 19 23:11:25 2012 From: dovecot at r.paypc.com (Robin) Date: Mon, 19 Mar 2012 14:11:25 -0700 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F67A0FD.8050900@r.paypc.com> On 3/17/2012 12:36 PM, Sven Hartge wrote: > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. Ugh, I've tried the product. It works pretty well, until you move more than a small handful of users and email hives to it, and you hit some hard walls pretty fast with how many inbound emails/second it can handle for even burly server configurations. Those hard walls occur at too low a threshold for me. The product's mailing list is supportive and there are many dedicated DBMail users who step in an answer questions, but be prepared for "BUY MORE RAM" as the answer to concerns about performance. When 128GB of RAM is needed for a small organisation's email setup to perform well, I am strongly inclined to move on to the next product. Best practices for it seem to revolve around being able to have your ENTIRE email + index content resident in RAM. Well, gosh. Why didn't I think of that before instead of wasting all of this time worrying about design and efficiency? And if you're hoping that it will make text searches "automagically" fast, think again. Timo's FTS_SQUAT blows it out of the water by orders of magnitude, even with mailbox sizes of around 300K emails (20GB), let alone something like Lucene or Solr. I understand why it seems like a great idea to store email this way, but realise that the bulk of email is NOT structured or inherently relational. =R= From jsimmons at goblin.punk.net Tue Mar 20 01:04:29 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:04:29 -0700 Subject: [Dovecot] Using plaintext auth and SSL Message-ID: <201203191604.29407.jsimmons@goblin.punk.net> I'm working with a company that presently has a Linux mailserver which all users have (no shell) accounts on. Mail is accessed via pop3 with plaintext authentication. They want to move to a system using imap with SSL. I'm building them a new server. I'd like to offer both for a while so we can work the bugs out and migrate users over to SSL imap over time. It appears that in order to limit the imap connections to SSL I will need to run two separate instances of Dovecot. Is this correct? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From jsimmons at goblin.punk.net Tue Mar 20 01:37:05 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:37:05 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67BE5E.4000501@knutejohnson.com> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> Message-ID: <201203191637.05129.jsimmons@goblin.punk.net> On Monday, March 19, 2012 04:16:46 pm you wrote: > On 3/19/2012 4:04 PM, Jeff Simmons wrote: > > I'm working with a company that presently has a Linux mailserver which > > all users have (no shell) accounts on. Mail is accessed via pop3 with > > plaintext authentication. They want to move to a system using imap with > > SSL. I'm building them a new server. I'd like to offer both for a while > > so we can work the bugs out and migrate users over to SSL imap over > > time. It appears that in order to limit the imap connections to SSL I > > will need to run two separate instances of Dovecot. Is this correct? > > I only have SSL or TLS connections enabled and I only have one copy of > Dovecot running. Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls only authentication imap. The 'allow plaintext authentication' configuration directive appears to be global, meaning I will need to run two instances of dovecot for a while. Is that correct, or can this be done on a single instance of dovecot? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From dovecot at knutejohnson.com Tue Mar 20 01:56:01 2012 From: dovecot at knutejohnson.com (Knute Johnson) Date: Mon, 19 Mar 2012 16:56:01 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67C791.2000609@knutejohnson.com> On 3/19/2012 4:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > I'm pretty sure if you set disable_plain_text_auth = no that you can log in to the appropriate ports with SSL or without. Sorry I sent the first reply to you, wasn't paying attention. -- Knute Johnson From eliezer at ngtech.co.il Tue Mar 20 02:16:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:16:21 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67CC55.3070706@ngtech.co.il> On 20/03/2012 01:37, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > there is no connection between the plaintext auth to the ssl\tls layer. you can just change the in the service section of the 10-master.conf file of the imap to no imap at all and use only imaps listener with port for your choose such as 143 or 993 and you will have a only imap over ssl. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:18:39 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:18:39 +0200 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F677988.9080403@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> Message-ID: <4F67CCDF.2010309@ngtech.co.il> On 19/03/2012 20:23, Radim Kolar wrote: > >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >> >> The INBOX exists but has a wrong owner. > nope > ponto# cd /var/mail > ponto# mv admin/ admin.X > ponto# doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied get into the maildir folder and use: ls -la to see all the directories and permissions. it might be with a starting "." what will make it "invisible" to regular ls. Regards, Eliezer > > but it might be that ordinary user admin cant create directories in > /var/mail > message from IMAP reply is wrong for sure because mailbox does not exists: > > ponto# cd /var/mail > ponto# mv admin admin.x > ponto# telnet localhost imap > 3 select inbox > 3 NO Mailbox doesn't exist: INBOX > 4 create INBOX > 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:45:16 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:45:16 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67D31C.2030302@ngtech.co.il> On 20/03/2012 02:16, Eliezer Croitoru wrote: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. one mistake, change the imap service to port 0 and port 143 will be disabled with regular imap service > > Regards, > Eliezer > -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From florob at babelmonkeys.de Tue Mar 20 04:26:12 2012 From: florob at babelmonkeys.de (Florian Zeitz) Date: Tue, 20 Mar 2012 03:26:12 +0100 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67EAC4.1050208@babelmonkeys.de> Am 20.03.2012 01:16, schrieb Eliezer Croitoru: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. > Because it is going to drive me insane if I don't ask: Is there really no way to archive this with a modern (aka. STARTTLS based) IMAP setup? From gedalya at gedalya.net Tue Mar 20 05:19:42 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:19:42 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67F74E.10403@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > This is all you have to do: protocol imap { ssl=required } See: http://wiki2.dovecot.org/SSL Globally, you can leave disable_plaintext_auth = no, and leave protocol pop3 {} alone. Your clients will be able to log in to pop3 with any authentication mechanism you have enabled, and imap will be accessible only with SSL/TLS, either over port 143 with STARTTLS or over port 993 with implicit SSL. I actually took the trouble to verify this on my local server before posting, and it turns out the wiki didn't lie. From gedalya at gedalya.net Tue Mar 20 05:42:38 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:42:38 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67FCAE.9060205@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > Turns out you can also use the disable_plaintext_auth = yes directive under protocol imap {}, but as noted by others previously, this is related specifically to plaintext authentication methods, and is not the same as requiring SSL/TLS for the entire session. If my understanding is correct, disable_plaintext_auth means your clients can authenticate with non-plaintext e.g. with CRAM-MD5 and proceed with an unsecured session. From jeetuindian at gmail.com Tue Mar 20 06:33:47 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 10:03:47 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hi guys, I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix in 2 system both system is having same configuration and os. I want to use dsync in mirror mode via ssh but I am not able to do it. When I followed the dsync wiki then I an unable to find the username : *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com dsync -u username* In the above command what will b user name and id_dsa.dovecot I am unable to understand. -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andrei.michescu at miau.ca Tue Mar 20 07:19:47 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 20 Mar 2012 01:19:47 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: <679071689c2e93f66654f318d277e8fc.squirrel@web.miau.ca> Hello Jitendra, I'm trying to do the same thing but I hit a wall as the sync seems to be doubling some emails in my case. My issues is still open with Timo (I HOPE!) As you seem to be stuck in an earlier stage here are my advices: 1) add the id_rsa to the .ssh folder to the user that runs the sync to test that this step is ok you should be able to $ ssh mailuser at example.com without being asked for a certificate / password (if you wonder how to achieve this here is a sample tutorial http://jaybyjayfresh.com/2009/02/04/logging-in-without-a-password-certificates-ssh/) 2) username is the user that has the email (if you work in a virtual environment generally is user at domain). 3) the syntax that I found to be working for me is the following: doveadm -Dv sync -u user1 at dom1 -f ssh mx1.a doveadm dsync-server -u user1 at dom1 To explain a little bit clearer the setup: - you start with 2 server mx1.a and mx2.a. On both servers you have vpopmail as the virtual user management for the virtual domain dom1 - from mx1.a you can ssh vpopmail at mx2.a directly without being prompted for a certificate or password) - user1 at dom1 is a virtual user defined both on mx1.a and on mx2.a (which means that you can deliver emails to this user both at mx1.a and mx2.a and you can also read them through imap on both servers). Hope this makes it a little bit more clear. Have fun, Andrei > Hi guys, > > I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix > in 2 system both system is having same configuration and os. I want to use > dsync in mirror mode via ssh but I am not able to do it. When I followed > the dsync wiki then I an unable to find the username : > > *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com > dsync -u username* > > In the above command what will b user name and id_dsa.dovecot I am unable > to understand. > > > > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > From achekalin at lazurit.com Tue Mar 20 07:28:30 2012 From: achekalin at lazurit.com (Alexander Chekalin) Date: Tue, 20 Mar 2012 08:28:30 +0300 Subject: [Dovecot] Per-user IMAP enable - is it possible? Message-ID: <4F68157E.5090806@lazurit.com> Just wonder if it is possible to enable/disable IMAP4 on Dovecot (2.0.x as far) on per-user basis? The deal is simple: our policy is not to store a lot of mailing on mailserver (the user should store it locally), thus the 'use POP3' approach, but for a vary few users it is permitted to use IMAP4. But users sometimes simple miss the point that some mail clients (e.g. TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of mailings and no local store of it on user's workstation. Sound too complicated, but setting up two Dovecots is not something I'd love to do as well. Thank you for any ideas, Alexander From gedalya at gedalya.net Tue Mar 20 07:43:10 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 01:43:10 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F68157E.5090806@lazurit.com> References: <4F68157E.5090806@lazurit.com> Message-ID: <4F6818EE.6090801@gedalya.net> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: > Just wonder if it is possible to enable/disable IMAP4 on Dovecot > (2.0.x as far) on per-user basis? > > The deal is simple: our policy is not to store a lot of mailing on > mailserver (the user should store it locally), thus the 'use POP3' > approach, but for a vary few users it is permitted to use IMAP4. But > users sometimes simple miss the point that some mail clients (e.g. TB) > 'prefer' to use IMAP4 first, and afterward I see mailbox full of > mailings and no local store of it on user's workstation. > > Sound too complicated, but setting up two Dovecots is not something > I'd love to do as well. > > Thank you for any ideas, > Alexander There would be various ways to do this, the specifics would depend on what kind of passdb you use. If you happen to be using a SQL database, you could do something like this: Add an allow_imap column, and change the password_query in dovecot-sql.conf.ext to something like this: password_query = SELECT password FROM user WHERE username = '%n' AND domain = '%d' \ AND ('%s' != 'imap' or allow_imap=1) This would make the user appear to not exist when trying to log in via IMAP. http://wiki2.dovecot.org/Variables From gedalya at gedalya.net Tue Mar 20 08:18:12 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 02:18:12 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6818EE.6090801@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> Message-ID: <4F682124.4010406@gedalya.net> On 3/20/2012 1:43 AM, Gedalya wrote: > On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >> (2.0.x as far) on per-user basis? >> >> The deal is simple: our policy is not to store a lot of mailing on >> mailserver (the user should store it locally), thus the 'use POP3' >> approach, but for a vary few users it is permitted to use IMAP4. But >> users sometimes simple miss the point that some mail clients (e.g. >> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >> mailings and no local store of it on user's workstation. >> >> Sound too complicated, but setting up two Dovecots is not something >> I'd love to do as well. >> >> Thank you for any ideas, >> Alexander > There would be various ways to do this, the specifics would depend on > what kind of passdb you use. > > If you happen to be using a SQL database, you could do something like > this: Add an allow_imap column, and change the password_query in > dovecot-sql.conf.ext to something like this: > > password_query = SELECT password FROM user WHERE username = '%n' AND > domain = '%d' \ > AND ('%s' != 'imap' or allow_imap=1) > > This would make the user appear to not exist when trying to log in via > IMAP. > > http://wiki2.dovecot.org/Variables > Or like this, might be more appropriate. password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, NULL, 'y') as nologin \ FROM user WHERE username = '%n' AND domain = '%d' http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin From cjeanneret at internux.ch Tue Mar 20 08:49:10 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Tue, 20 Mar 2012 08:49:10 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: On 19.03.2012 14:27, Antoine Nguyen wrote: > 2012/3/19 C?dric Jeanneret > >> Hello List! >> >> I have a tiny-teeny problem with dovecot + sieve: it seems that the >> LDA >> doesn't run sieve, and thus doesn't filter my emails. >> >> Here's the sieve configuration: >> >> plugin { >> # Used by both the Sieve plugin and the ManageSieve protocol >> sieve=/var/local/vmail/%n/**dovecot.sieve >> sieve_dir=/var/local/vmail/%n/**sieve >> sieve_extensions = +notify +imapflags >> } >> >> The managesiege: >> >> protocol managesieve { >> # Specify an alternative address:port the daemon must listen on >> # (default: *:2000) >> listen = localhost:2000 >> managesieve_logout_format = bytes ( in=%i : out=%o ) >> } >> (this one is working fine, I can edit the filters through roundcube >> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) >> is >> edited) >> >> the lda part: >> >> protocol lda { >> postmaster_address = foo at bar.com >> mail_plugins = sieve >> } >> >> I think all is in place to allow dovecot to use sieve... ? >> >> One more thing: >> >> dovecot --version >> 1.2.15 >> >> >> Any help will be welcomed :). >> >> Thanks in advance ! >> >> Cheers, >> >> C. >> > > Have you checked the MTA configuration. Does it use dovecot's LDA ? > > Antoine Hello Antoine (and List), Well, it should use dovecot, as the mails are delivered to the user inbox - and I don't think postfix knows about them.. How may I be sure otherwise that postfix really uses dovecot? Cheers, C. From evocage at gmail.com Tue Mar 20 09:06:42 2012 From: evocage at gmail.com (evolution age) Date: Tue, 20 Mar 2012 12:36:42 +0530 Subject: [Dovecot] Dovecot with postfix setup Message-ID: Hey frnds, Could you suggest me any ebook or documentation for the setup of dovecot with postfix on centos 5.7 . I need it. -- Warm Regards Jitendra Kumar Bhaskar cell :- +91-8886742555 From gedalya at gedalya.net Tue Mar 20 09:12:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:12:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: <4F682DF3.2030409@gedalya.net> On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > On 19.03.2012 14:27, Antoine Nguyen wrote: >> 2012/3/19 C?dric Jeanneret >> >>> Hello List! >>> >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA >>> doesn't run sieve, and thus doesn't filter my emails. >>> >>> Here's the sieve configuration: >>> >>> plugin { >>> # Used by both the Sieve plugin and the ManageSieve protocol >>> sieve=/var/local/vmail/%n/**dovecot.sieve >>> sieve_dir=/var/local/vmail/%n/**sieve >>> sieve_extensions = +notify +imapflags >>> } >>> >>> The managesiege: >>> >>> protocol managesieve { >>> # Specify an alternative address:port the daemon must listen on >>> # (default: *:2000) >>> listen = localhost:2000 >>> managesieve_logout_format = bytes ( in=%i : out=%o ) >>> } >>> (this one is working fine, I can edit the filters through roundcube >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is >>> edited) >>> >>> the lda part: >>> >>> protocol lda { >>> postmaster_address = foo at bar.com >>> mail_plugins = sieve >>> } >>> >>> I think all is in place to allow dovecot to use sieve... ? >>> >>> One more thing: >>> >>> dovecot --version >>> 1.2.15 >>> >>> >>> Any help will be welcomed :). >>> >>> Thanks in advance ! >>> >>> Cheers, >>> >>> C. >>> >> >> Have you checked the MTA configuration. Does it use dovecot's LDA ? >> >> Antoine > > Hello Antoine (and List), > > Well, it should use dovecot, as the mails are delivered to the user > inbox - and I don't think postfix knows about them.. > How may I be sure otherwise that postfix really uses dovecot? > > Cheers, > > C. If you are using dovecot 1.2, I think lda should show lines like the following in your log. Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: msgid=: stored mail into mailbox 'INBOX' 'deliver' refers to lda. From cjeanneret at internux.ch Tue Mar 20 09:20:41 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:20:41 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F682DF3.2030409@gedalya.net> References: <4F682DF3.2030409@gedalya.net> Message-ID: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:12:51 -0400 Gedalya wrote: > On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > > On 19.03.2012 14:27, Antoine Nguyen wrote: > >> 2012/3/19 C?dric Jeanneret > >> > >>> Hello List! > >>> > >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > >>> doesn't run sieve, and thus doesn't filter my emails. > >>> > >>> Here's the sieve configuration: > >>> > >>> plugin { > >>> # Used by both the Sieve plugin and the ManageSieve protocol > >>> sieve=/var/local/vmail/%n/**dovecot.sieve > >>> sieve_dir=/var/local/vmail/%n/**sieve > >>> sieve_extensions = +notify +imapflags > >>> } > >>> > >>> The managesiege: > >>> > >>> protocol managesieve { > >>> # Specify an alternative address:port the daemon must listen on > >>> # (default: *:2000) > >>> listen = localhost:2000 > >>> managesieve_logout_format = bytes ( in=%i : out=%o ) > >>> } > >>> (this one is working fine, I can edit the filters through roundcube > >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > >>> edited) > >>> > >>> the lda part: > >>> > >>> protocol lda { > >>> postmaster_address = foo at bar.com > >>> mail_plugins = sieve > >>> } > >>> > >>> I think all is in place to allow dovecot to use sieve... ? > >>> > >>> One more thing: > >>> > >>> dovecot --version > >>> 1.2.15 > >>> > >>> > >>> Any help will be welcomed :). > >>> > >>> Thanks in advance ! > >>> > >>> Cheers, > >>> > >>> C. > >>> > >> > >> Have you checked the MTA configuration. Does it use dovecot's LDA ? > >> > >> Antoine > > > > Hello Antoine (and List), > > > > Well, it should use dovecot, as the mails are delivered to the user > > inbox - and I don't think postfix knows about them.. > > How may I be sure otherwise that postfix really uses dovecot? > > > > Cheers, > > > > C. > > If you are using dovecot 1.2, I think lda should show lines like the > following in your log. > > Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: > msgid=: stored mail into mailbox 'INBOX' > > 'deliver' refers to lda. > Hmm... strange, doesn't show up like that in logs: Mar 20 08:14:54 sqdf3 postfix/smtpd[27509]: connect from host.foo.bar[...] Mar 20 08:15:16 sqdf3 postfix/smtpd[27509]: 892335659F4: client=host.foo.bar[...] Mar 20 08:15:35 sqdf3 postfix/cleanup[27516]: 892335659F4: message-id=<> Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 08:15:35 sqdf3 postfix/virtual[27518]: 892335659F4: to=, relay=virtual, delay=27, delays=27/0.03/0/0.14, dsn=2.0.0, status=sent (delivered to maildir) Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: removed Maybe the problem is there (postfix main.cf): virtual_transport = virtual I have virtual users and a mailman running on the same domain :/. If postfix doesn't use dovecot, how comes that mails are put in the right place ? From 24x7server at 24x7server.net Tue Mar 20 09:35:56 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Tue, 20 Mar 2012 13:05:56 +0530 (Asi) Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 Message-ID: <.120.61.90.33.1332228956.squirrel@24x7server.net> hi my system is a centos 5 with qmailtoaster i migrated my email server with around 5000 users from dovecot version 1.2 to version 2 i have two separate 2 tb hdd's storing webmail data of these users. the load on the server goes very high over 100 during peak load times and the imap connections get dropped frequently, webmail becomes very slow. however pop3 download works ok in the dovecot log file i get errors as such Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 expunge attempts) i had configured dovecot 1.2 using source but i had installed version 2 using qmailtoaster rpm i am a bit confused as to what settings are to be done for a very busy server note that the cpu system usage and ram usage are less but IO wait goes to around 80-90 percent can you please guide me or post some dovecot version 2 config file settings that are relevant to a busy server. rajesh From gedalya at gedalya.net Tue Mar 20 09:36:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:36:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683393.60403@gedalya.net> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > status=sent (delivered to maildir) Your log clearly says postfix is delivering directly to the maildir. Postfix must be configured to find the user's maildir using things like virtual_mailbox_base and / or virtual_mailbox_maps ? From cjeanneret at internux.ch Tue Mar 20 09:44:58 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:44:58 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683393.60403@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> Message-ID: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:36:51 -0400 Gedalya wrote: > On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > > status=sent (delivered to maildir) > Your log clearly says postfix is delivering directly to the maildir. > Postfix must be configured to find the user's maildir using things like > virtual_mailbox_base and / or virtual_mailbox_maps ? > Geez.. right - taking it from ldap... should only comment out the following lines: virtual_mailbox_base = / virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf ? From gedalya at gedalya.net Tue Mar 20 10:01:08 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:01:08 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683944.2030408@gedalya.net> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 03:36:51 -0400 > Gedalya wrote: > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>> status=sent (delivered to maildir) >> Your log clearly says postfix is delivering directly to the maildir. >> Postfix must be configured to find the user's maildir using things like >> virtual_mailbox_base and / or virtual_mailbox_maps ? >> > Geez.. right - taking it from ldap... should only comment out the following lines: > virtual_mailbox_base = / > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > ? If you want postfix to use the dovecot LDA then you have to set it up, you can get a general idea here http://wiki.dovecot.org/LDA/Postfix and adapt it to your circumstances. Basically you have to add the appropriate lines to master.cf and put virtual_transport = dovecot in main.cf. From cjeanneret at internux.ch Tue Mar 20 10:23:30 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 09:23:30 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683944.2030408@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> Message-ID: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 04:01:08 -0400 Gedalya wrote: > On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > > On Tue, 20 Mar 2012 03:36:51 -0400 > > Gedalya wrote: > > > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > >>> status=sent (delivered to maildir) > >> Your log clearly says postfix is delivering directly to the maildir. > >> Postfix must be configured to find the user's maildir using things like > >> virtual_mailbox_base and / or virtual_mailbox_maps ? > >> > > Geez.. right - taking it from ldap... should only comment out the following lines: > > virtual_mailbox_base = / > > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > > > ? > If you want postfix to use the dovecot LDA then you have to set it up, > you can get a general idea here > http://wiki.dovecot.org/LDA/Postfix > and adapt it to your circumstances. > > Basically you have to add the appropriate lines to master.cf and put > virtual_transport = dovecot in main.cf. > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. It _should_ take transport_maps = hash:/etc/postfix/transport which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( From gedalya at gedalya.net Tue Mar 20 10:50:33 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:50:33 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6844D9.7050504@gedalya.net> On 3/20/2012 4:23 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 04:01:08 -0400 > Gedalya wrote: > >> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: >>> On Tue, 20 Mar 2012 03:36:51 -0400 >>> Gedalya wrote: >>> >>>> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>>>> status=sent (delivered to maildir) >>>> Your log clearly says postfix is delivering directly to the maildir. >>>> Postfix must be configured to find the user's maildir using things like >>>> virtual_mailbox_base and / or virtual_mailbox_maps ? >>>> >>> Geez.. right - taking it from ldap... should only comment out the following lines: >>> virtual_mailbox_base = / >>> virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf >>> >>> ? >> If you want postfix to use the dovecot LDA then you have to set it up, >> you can get a general idea here >> http://wiki.dovecot.org/LDA/Postfix >> and adapt it to your circumstances. >> >> Basically you have to add the appropriate lines to master.cf and put >> virtual_transport = dovecot in main.cf. >> > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > It _should_ take > transport_maps = hash:/etc/postfix/transport > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( Once you have dovecot set up in your master.cf you should be able to put as the last line simply domain.com dovecot What do your logs look like when this is set? From cjeanneret at internux.ch Tue Mar 20 11:08:01 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 10:08:01 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6844D9.7050504@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> Message-ID: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> > > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > > It _should_ take > > transport_maps = hash:/etc/postfix/transport > > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( > Once you have dovecot set up in your master.cf you should be able to put > as the last line simply > domain.com dovecot > What do your logs look like when this is set? > I guess it may be easier if I paste my whole config in here: postfix main.cf: smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases mydestination = public_fqdn, localhost mynetworks = 127.0.0.0/8 inet_interfaces = all recipient_delimiter = + smtpd_sasl_security_options = noanonymous myorigin = domain.ltd smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_sasl_path = private/auth smtpd_sasl_type = dovecot owner_request_special = no smtpd_tls_CAfile = /etc/postfix/ssl/ca.crt smtpd_tls_cert_file = /etc/postfix/ssl/cert.crt smtpd_tls_key_file = /etc/postfix/ssl/keyForApache2.key smtpd_use_tls = yes smtpd_sasl_tls_security_options = $smtpd_sasl_security_options smtpd_sasl_auth_enable = yes default_process_limit = 5 smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/access, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unlisted_recipient smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache broken_sasl_auth_clients = yes transport_maps = hash:/etc/postfix/transport virtual_transport = dovecot transport_maps = hash:/etc/postfix/transport virtual_alias_maps = ldap:/etc/postfix/ldap-aliases.cf, hash:/var/lib/mailman/data/virtual-mailman virtual_gid_maps = static:104 virtual_minimum_uid = 8 virtual_uid_maps = static:8 virtual_mailbox_base = / mailman_destination_recipient_limit = 1 virtual_mailbox_domains = avocats-ch.ch virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf With this configuration, here's what I get: sending to a virtual user: Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: DC9285659F4: client=remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/cleanup[1528]: DC9285659F4: message-id=<> Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 10:02:49 sqdf3 dovecot: deliver(camptocamp): msgid=: saved mail to INBOX Mar 20 10:02:49 sqdf3 postfix/pipe[1529]: DC9285659F4: to=, relay=dovecot, delay=0.45, delays=0.18/0.01/0/0.26, dsn=2.0.0, status=sent (delivered via dovecot service) Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: removed so it uses dovecot. but mail is not filtered as it should :(( And now, sending a mail to a mailman list: Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: NOQUEUE: reject: RCPT from remote.host.ltd[...]: 550 5.1.1 : Recipient address rejected: User unknown in virtual mailbox table; from= to= proto=ESMTP helo= Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: warning: non-SMTP command from remote.host.ltd[...]: Subject: testing mailman Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: disconnect from remote.host.ltd[...] may I cry ? :( From luca.palazzo at unict.it Tue Mar 20 11:09:12 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 20 Mar 2012 10:09:12 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash Message-ID: <4F684938.9000208@unict.it> Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. Log lines are like: Mar 20 10:05:45 mailgw-lb dovecot: pop3-login: Fatal: master: service(pop3-login): child 27764 killed with signal 11 (core dumps disabled) Mar 20 10:06:17 mailgw-lb dovecot: imap-login: Fatal: master: service(imap-login): child 28468 killed with signal 11 (core dumps disabled) Stack trace of a crashed process is like: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1220163904 (LWP 27764)] 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 #4 0xb77ca9b5 in server_input (proxy=0x0) at login-proxy.c:93 #5 0xb7793762 in io_loop_call_io (io=0x8094180) at ioloop.c:380 #6 0xb7794cc9 in io_loop_handler_run (ioloop=0x8055480) at ioloop-epoll.c:213 #7 0xb77936f9 in io_loop_run (ioloop=0x8055480) at ioloop.c:399 #8 0xb777e4c8 in master_service_run (service=0x80553b0, callback=0xb77cc110 ) at master-service.c:544 #9 0xb77cbcee in login_binary_run (binary=0x804ad80, argc=2, argv=0x80551c0) at main.c:406 #10 0x08049812 in main (argc=0, argv=0x0) at client.c:303 The strange part of the story is that not all process crash. I'm trying to figure out if only TLS/SSL process crash. Any idea? Thanks Luca From eliezer at ngtech.co.il Tue Mar 20 11:42:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:42:21 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6850FD.9010602@ngtech.co.il> On 20/03/2012 11:08, Cedric Jeanneret wrote: > I guess it may be easier if I paste my whole config in here: you didnt sent the virtual_transport file content. i will quote from the man pages of the transport: [quote] user at domain transport:nexthop Deliver mail for user at domain through transport to nexthop. [\quote] means you can specify specific transport such as maliman to specific user. but because you are using the virtual maps table\lookup you also must have a vaild ldap user with the same name for the list. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 11:45:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:45:21 +0200 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F682124.4010406@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> Message-ID: <4F6851B1.4030509@ngtech.co.il> On 20/03/2012 08:18, Gedalya wrote: > On 3/20/2012 1:43 AM, Gedalya wrote: >> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>> (2.0.x as far) on per-user basis? >>> >>> The deal is simple: our policy is not to store a lot of mailing on >>> mailserver (the user should store it locally), thus the 'use POP3' >>> approach, but for a vary few users it is permitted to use IMAP4. But >>> users sometimes simple miss the point that some mail clients (e.g. >>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>> mailings and no local store of it on user's workstation. >>> >>> Sound too complicated, but setting up two Dovecots is not something >>> I'd love to do as well. >>> >>> Thank you for any ideas, >>> Alexander >> There would be various ways to do this, the specifics would depend on >> what kind of passdb you use. >> >> If you happen to be using a SQL database, you could do something like >> this: Add an allow_imap column, and change the password_query in >> dovecot-sql.conf.ext to something like this: >> >> password_query = SELECT password FROM user WHERE username = '%n' AND >> domain = '%d' \ >> AND ('%s' != 'imap' or allow_imap=1) >> >> This would make the user appear to not exist when trying to log in via >> IMAP. >> >> http://wiki2.dovecot.org/Variables >> > Or like this, might be more appropriate. > > password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, > NULL, 'y') as nologin \ > FROM user WHERE username = '%n' AND domain = '%d' > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin > but this will disallow also pop3... Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From gedalya at gedalya.net Tue Mar 20 11:49:45 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 05:49:45 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6851B1.4030509@ngtech.co.il> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> <4F6851B1.4030509@ngtech.co.il> Message-ID: <4F6852B9.1050809@gedalya.net> On 3/20/2012 5:45 AM, Eliezer Croitoru wrote: > On 20/03/2012 08:18, Gedalya wrote: >> On 3/20/2012 1:43 AM, Gedalya wrote: >>> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>>> (2.0.x as far) on per-user basis? >>>> >>>> The deal is simple: our policy is not to store a lot of mailing on >>>> mailserver (the user should store it locally), thus the 'use POP3' >>>> approach, but for a vary few users it is permitted to use IMAP4. But >>>> users sometimes simple miss the point that some mail clients (e.g. >>>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>>> mailings and no local store of it on user's workstation. >>>> >>>> Sound too complicated, but setting up two Dovecots is not something >>>> I'd love to do as well. >>>> >>>> Thank you for any ideas, >>>> Alexander >>> There would be various ways to do this, the specifics would depend on >>> what kind of passdb you use. >>> >>> If you happen to be using a SQL database, you could do something like >>> this: Add an allow_imap column, and change the password_query in >>> dovecot-sql.conf.ext to something like this: >>> >>> password_query = SELECT password FROM user WHERE username = '%n' AND >>> domain = '%d' \ >>> AND ('%s' != 'imap' or allow_imap=1) >>> >>> This would make the user appear to not exist when trying to log in via >>> IMAP. >>> >>> http://wiki2.dovecot.org/Variables >>> >> Or like this, might be more appropriate. >> >> password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, >> NULL, 'y') as nologin \ >> FROM user WHERE username = '%n' AND domain = '%d' >> >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin >> > but this will disallow also pop3... > > > Eliezer > No. It will return NULL unless the service is 'imap' and allow_imap != 1. nologin=NULL has no effect, so everything is allowed. From cjeanneret at internux.ch Tue Mar 20 12:00:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 11:00:21 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6850FD.9010602@ngtech.co.il> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> <4F6850FD.9010602@ngtech.co.il> Message-ID: <20120320110021.529fba41@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 11:42:21 +0200 Eliezer Croitoru wrote: > On 20/03/2012 11:08, Cedric Jeanneret wrote: > > I guess it may be easier if I paste my whole config in here: > you didnt sent the virtual_transport file content. > i will quote from the man pages of the transport: > [quote] > user at domain transport:nexthop > Deliver mail for user at domain through transport to nexthop. > [\quote] > means you can specify specific transport such as maliman to specific user. > but because you are using the virtual maps table\lookup you also must > have a vaild ldap user with the same name for the list. > > Regards, > Eliezer > Hello, while trying to remove/add options to my postfix, this part is now working now - it uses dovecot for virtual users, and mailman for lists :). Now that's good, I'll go back to sieve and ensure there's no missing configuration in dovecot. I had to remove "virtual_transport" option, and remove an "@" in my transport map. I'll come back on this thread if I still have problem with sieve. Cheers, C. From cjeanneret at internux.ch Tue Mar 20 13:05:28 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 12:05:28 +0100 Subject: [Dovecot] dovecot, sieve and vacation Message-ID: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Hi there ! Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to after searching a bit on the net, I stumbled on another (old) thread: http://www.mail-archive.com/dovecot at dovecot.org/msg25955.html I'm not really sure it's the same problem, as I'm sending the mail from another host (via telnet, for testing purpose). Other sieve rules (such as flagging, moving and so on) work fine. Only vacation is crapy. Here's the roundcube generated rule: require ["vacation"]; if true { vacation :days 2 "on holidays!"; } Any help welcome :) Cheers, C. Informations: # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-6-pve i686 Debian 6.0.4 simfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): localhost:2000 ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login first_valid_uid: 8 mail_privileged_group: mail mail_uid: mail mail_gid: mail mail_location: maildir:/var/local/vmail/%n mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve managesieve_logout_format(default): bytes=%i/%o managesieve_logout_format(imap): bytes=%i/%o managesieve_logout_format(pop3): bytes=%i/%o managesieve_logout_format(managesieve): bytes ( in=%i : out=%o ) lda: postmaster_address: foo at bar.com mail_plugins: sieve auth default: mechanisms: plain login user: mail passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 432 user: postfix group: mail plugin: home: /var/local/vmail/%u sieve: /var/local/vmail/%n/.dovecot.sieve sieve_dir: /var/local/vmail/%n/sieve sieve_extensions: +notify +imapflags From jeetuindian at gmail.com Tue Mar 20 13:10:46 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 16:40:46 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hey Frnds, Could you tell me about the error : # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com doveadm(root): Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, home=/home/example1.com/jitendra.b doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:/home/ example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b doveadm(jitendra.b at example.com): Debug: fs: root=/home/ example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ example.com/jitendra.b, alt= dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 doveadm(jitendra.b): Fatal: User doesn't exist dsync-local(jitendra.b at example.com): Error: read() from worker server failed: EOF -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andreas.a.lamprecht at atos.net Tue Mar 20 13:16:33 2012 From: andreas.a.lamprecht at atos.net (Lamprecht, Andreas) Date: Tue, 20 Mar 2012 11:16:33 +0000 Subject: [Dovecot] IMAP and POP3 per SSL Message-ID: Hi! I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set SSLHonorCipherOrder On in apache config. This results in the following C-Code being executed: SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. Is there a way to implement such a setting into Dovecot, too? I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 *************** *** 924,930 **** X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); --- 924,930 ---- X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... So, maybe you have the time to look over it and implement a final solution for the BEAST problem. Greetings Andreas lamprecht From support at palatineweb.com Tue Mar 20 13:26:56 2012 From: support at palatineweb.com (Palatine Support) Date: Tue, 20 Mar 2012 11:26:56 +0000 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67CCDF.2010309@ngtech.co.il> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> <4F67CCDF.2010309@ngtech.co.il> Message-ID: <4F686980.5040600@palatineweb.com> I have tried to unsubscribe from this mailing list 10 times now. Remove my email address please asap. Thanks Paul On 20/03/2012 00:18, Eliezer Croitoru wrote: > On 19/03/2012 20:23, Radim Kolar wrote: >> >>> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >>> >>> The INBOX exists but has a wrong owner. >> nope >> ponto# cd /var/mail >> ponto# mv admin/ admin.X >> ponto# doveadm mailbox create -u admin INBOX >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > get into the maildir folder and use: > ls -la to see all the directories and permissions. > it might be with a starting "." what will make it "invisible" to > regular ls. > > Regards, > Eliezer > >> >> but it might be that ordinary user admin cant create directories in >> /var/mail >> message from IMAP reply is wrong for sure because mailbox does not >> exists: >> >> ponto# cd /var/mail >> ponto# mv admin admin.x >> ponto# telnet localhost imap >> 3 select inbox >> 3 NO Mailbox doesn't exist: INBOX >> 4 create INBOX >> 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX > > From robert at schetterer.org Tue Mar 20 13:32:04 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:32:04 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: References: Message-ID: <4F686AB4.3070506@schetterer.org> Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: > Hi! > > I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. > > Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. > The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 > > "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set > > SSLHonorCipherOrder On > > in apache config. This results in the following C-Code being executed: > > SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); > > This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. > > Is there a way to implement such a setting into Dovecot, too? > > I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. > This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: > > *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 > --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 > *************** > *** 924,930 **** > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > --- 924,930 ---- > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > > > Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... > > So, maybe you have the time to look over it and implement a final solution for the BEAST problem. > > Greetings > Andreas lamprecht > perhaps look at http://wiki2.dovecot.org/SSL/DovecotConfiguration -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From robert at schetterer.org Tue Mar 20 13:34:58 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:34:58 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: <4F686AB4.3070506@schetterer.org> References: <4F686AB4.3070506@schetterer.org> Message-ID: <4F686B62.2050205@schetterer.org> Am 20.03.2012 12:32, schrieb Robert Schetterer: > Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: >> Hi! >> >> I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. >> >> Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. >> The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 >> >> "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set >> >> SSLHonorCipherOrder On >> >> in apache config. This results in the following C-Code being executed: >> >> SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); >> >> This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. >> >> Is there a way to implement such a setting into Dovecot, too? >> >> I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. >> This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: >> >> *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 >> --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 >> *************** >> *** 924,930 **** >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> --- 924,930 ---- >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> >> >> Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... >> >> So, maybe you have the time to look over it and implement a final solution for the BEAST problem. >> >> Greetings >> Andreas lamprecht >> > > perhaps look at > > http://wiki2.dovecot.org/SSL/DovecotConfiguration > and perhaps have a look at http://hg.dovecot.org/dovecot-2.0/rev/e3d46fd04105 and upgrade your dove version to dovecot 2.0.18 -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From CMarcus at Media-Brokers.com Tue Mar 20 14:22:59 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 20 Mar 2012 08:22:59 -0400 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F6876A3.2040409@Media-Brokers.com> On 2012-03-20 3:06 AM, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. Distro specific questions should be directed to your distro support lists. -- Best regards, Charles From nmilas at noa.gr Tue Mar 20 15:12:31 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 20 Mar 2012 15:12:31 +0200 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F68823F.4040905@noa.gr> On 20/3/2012 9:06 ??, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. You have not provided any info on your requirements, so it's hard to provide assistance. If you are only now starting the design (you should devote at least some time to it - design is the cornerstone) and you want LDAP, you may want to check projects like: GOsa (https://oss.gonicus.de/labs/gosa/) Or use a packaged solution, if it's OK with your requirements: http://www.iredmail.org/ (I prefer to install/control packages personally.) If you go manually, it shouldn't be difficult to find one of the many guides on the web. First, you need to find packages supporting the features you need (because the CentOS standard packages are very old). Of course you can start with CentOS standard Postfix package (supports ldap, pcre, SASL, TLS), but make sure you upgrade soon esp. if it's a production system! For Postfix, check that the package offers support for whatever you want (e.g. LDAP, mysql, SASL auth, TLS, pcre etc.). Otherwise, you may need to build your own RPM. You may want to read: http://tech.groups.yahoo.com/group/postfix-users/message/284530 http://tech.groups.yahoo.com/group/postfix-users/message/284359 Dovecot RPMs from here: http://packages.atrpms.net/dist/el5/dovecot/ are fine (I think they support anything you would possibly want). Then, find a guide like: http://www.howtoforge.com/linux_postfix_virtual_hosting or http://wanderingbarque.com/howtos/mailserver/mailserver.html depending on the type of setup you want (e.g. multiple domains, local vs virtual users, etc.) Good luck! Nick From nicolas.kowalski at gmail.com Tue Mar 20 15:55:12 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Tue, 20 Mar 2012 14:55:12 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? Message-ID: <20120320135512.GD28951@petole.demisel.net> Hello, Does dovecot support the subject Alternative Name email value [1] as ssl_cert_username_field? If so, how should it be specified in the configuration? Thanks. [1] http://www.openssl.org/docs/apps/x509v3_config.html#Subject_Alternative_Name_ -- Nicolas From stephan at rename-it.nl Tue Mar 20 16:16:21 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 20 Mar 2012 15:16:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F689135.6020602@rename-it.nl> Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > Hi there ! > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to The vacation action will not send a response when the envelope-to address (in your case virtual_user at hostname) is not contained in the To: or Cc: headers of the message itself; the message needs to be explicitly addressed to the recipient. For the version you are using this needs to match the final recipient as passed to Dovecot. In newer versions of the Pigeonhole Sieve implementation the original SMTP envelope recipient (i.e. before local rewrites) can also be used instead. Alternatively, new versions allow disabling this behavior entirely, although this is not recommended. Regards, Stephan. From andrei at lctax.ro Tue Mar 20 16:46:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Tue, 20 Mar 2012 10:46:58 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: Hello, As log as example.com resolves to 192.0.43.10 (which I suppose it is a host that you don't own) this will not work. dsync seems to be resolving example.com and trying to connect there via ssh using the current user. It is better to test on domains that you own, on on domains that don't exists and you adjust your hosts file accordingly. Also seems that you have some issues with the configuration file. Can you post your dovecot -n output. Thnx, Andrei > Hey Frnds, > > Could you tell me about the error : > > # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com > > > > doveadm(root): Debug: Loading modules from directory: > /usr/local/lib/dovecot/doveadm > doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: > undefined symbol: acl_user_module (this is usually intentional, so just > ignore this message) > doveadm(root): Debug: Skipping module doveadm_expire_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined > symbol: expire_set_lookup (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_quota_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined > symbol: quota_user_module (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because > dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: > undefined symbol: i_stream_create_deflate (this is usually intentional, so > just ignore this message) > doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: > undefined symbol: fts_list_backend (this is usually intentional, so just > ignore this message) > doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, > home=/home/example1.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, > prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes > location=mbox:/home/ > example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: fs: root=/home/ > example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ > example.com/jitendra.b, alt= > dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions > from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 > doveadm(jitendra.b): Fatal: User doesn't exist > dsync-local(jitendra.b at example.com): Error: read() from worker server > failed: EOF > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > > !DSPAM:4f6865bf72822789337279! > From cjeanneret at internux.ch Tue Mar 20 16:48:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 15:48:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <4F689135.6020602@rename-it.nl> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> <4F689135.6020602@rename-it.nl> Message-ID: <20120320154821.11c80a31@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 15:16:21 +0100 Stephan Bosch wrote: > Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > > Hi there ! > > > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to > > The vacation action will not send a response when the envelope-to > address (in your case virtual_user at hostname) is not contained in the To: > or Cc: headers of the message itself; the message needs to be explicitly > addressed to the recipient. For the version you are using this needs to > match the final recipient as passed to Dovecot. In newer versions of the > Pigeonhole Sieve implementation the original SMTP envelope recipient > (i.e. before local rewrites) can also be used instead. Alternatively, > new versions allow disabling this behavior entirely, although this is > not recommended. > > Regards, > > Stephan. Hello, thanks for the hint. In fact, I have to add the address aliases in the rule, and it works. Not really cool, but it works like that :). Cheers, C. From patrickdk at patrickdk.com Tue Mar 20 16:55:38 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Tue, 20 Mar 2012 10:55:38 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319183547.GA28363@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> Message-ID: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> And found two more users with this issue, but while looking at it, I see another related issue, but it's not throwing an error. all email in the INBOX/new and /cur are correct but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa This is happening for all folder moves. the Sent folder isn't affected, but I assume cause an email wasn't moved in that case. Quoting Ralf Hildebrandt : > * Patrick Domack : >> I'm having this problem also, with a very very few users. >> >> But in my case the email isn't double gzip, just single like normal. >> >> Error: read(.../.Deleted >> Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output >> error (uid=250) >> >> All I have to do is rename the file to add back the lost S= part and >> all is fine. >> This has happened in the inbox, deleted, and trash folders so far. >> and always after a change, the S= exists for new emails. It's like >> it's loosing it on adding the read flag, and mailbox moves > > Yes, I'm also seeing it now with mailboxes where no mail is doubly > gzipped. > > -- > Ralf Hildebrandt > Gesch?ftsbereich IT | Abteilung Netzwerk > Charit? - Universit?tsmedizin Berlin > Campus Benjamin Franklin > Hindenburgdamm 30 | D-12203 Berlin > Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 > ralf.hildebrandt at charite.de | http://www.charite.de From ka at pacific.net Tue Mar 20 17:26:01 2012 From: ka at pacific.net (Ken A) Date: Tue, 20 Mar 2012 10:26:01 -0500 Subject: [Dovecot] mdbox and pop3 locking Message-ID: <4F68A189.2010800@pacific.net> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Thanks, Ken -- Ken Anderson Pacific Internet - http://www.pacific.net From Ralf.Hildebrandt at charite.de Tue Mar 20 17:33:20 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Tue, 20 Mar 2012 16:33:20 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120320153320.GD26616@charite.de> * Patrick Domack : > And found two more users with this issue, but while looking at it, I > see another related issue, but it's not throwing an error. > > all email in the INBOX/new and /cur are correct > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have > double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Yes indeed: postamt:/home/h/a/happel/Maildir/.Trash/cur# ll total 16 -rw------- 1 happel users 7541 Mar 20 15:23 1332253428.M342974P5666.postamt.charite.de,S=37641,W=38197,S=37641,W=38197:2,Se -rw------- 1 happel users 6378 Mar 20 15:42 1332254568.M9552P591.postamt.charite.de,S=27486,W=28188,S=27486,W=28188:2,Se -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From micah at riseup.net Tue Mar 20 17:40:43 2012 From: micah at riseup.net (Micah Anderson) Date: Tue, 20 Mar 2012 11:40:43 -0400 Subject: [Dovecot] Antispam plugin not compatible with Dovecot 2.1 References: <4F155670.6010905@gmail.com> <1326897258.11500.53.camel@innu> <1326904309.11500.83.camel@innu> Message-ID: <877gyfp9fo.fsf@algae.riseup.net> "Eugene Paskevich" writes: > On Wed, 18 Jan 2012 18:31:49 +0200, Timo Sirainen wrote: > >> On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: >>> >> mailbox.c: In function 'antispam_save_begin': >>> >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >>> >> 'copying' >>> > >>> > The "copying" should be changed to "copying_via_save". >>> >>> Thank you, Timo. >>> Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? >> >> Where do you expect to find such macro? ;) Hm. Perhaps I should try to >> add one. > > Heh. That's Johannes' package private macro... :) I notice that Johannes hasn't made a 2.1 version of the anti-spam plugin, Eugene were you able to build one successfully? If so, would you be willing to share your changes that were required to make it work? thanks, micah From jernej.porenta at arnes.si Tue Mar 20 19:44:26 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 20 Mar 2012 18:44:26 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1332165220.26095.71.camel@innu> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> <1332165220.26095.71.camel@innu> Message-ID: <79D375C1-1009-46B3-A383-A33DD0A699E8@arnes.si> On Mar 19, 2012, at 2:53 PM, Timo Sirainen wrote: > On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: >>> Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) >>> >>> It is the same. We will try 2.1.3 today and report the results... > >> The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz > > Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d > Confirmed working? Thank you again, cheers, Jernej From mjeghers at Brocade.com Tue Mar 20 20:29:56 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 11:29:56 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> All, Below is my config. When I run dovecot from xinetd, I get these errors in the log: Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ total 248656 drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root users 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim Any ideas what is wrong? What is different running under xinetd? All the process run under the same user ids... Thanks, /Mark My config --------------------------------------------------------------- # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.7.1.el6.centos.plus.i686 i686 CentOS release 6.2 (Final) ext4 auth_debug = yes auth_verbose = yes auth_verbose_passwords = plain disable_plaintext_auth = no doveadm_worker_count = 4 mail_debug = yes mail_gid = users mail_location = mbox:/var/spool/mailpop3:INBOX=/var/spool/mailpop3/%u mail_uid = root mbox_write_locks = fcntl passdb { args = /etc/passwd.dovecot driver = passwd-file } passdb { driver = shadow } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size from subject vsize flags mail_log_group_events = yes } protocols = pop3 ssl_cert = References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F6943D6.1000600@hardwarefreak.com> On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From mjeghers at Brocade.com Wed Mar 21 06:26:23 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 21:26:23 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <4F6943D6.1000600@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Hi Stan Afraid it did not help. Here is what I got: *** entered into a telnet session... user ann +OK pass ******** -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Connection closed by foreign host. [root at t4pserver2 mailpop3]# *** resulted in maillog... Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 *** file permissions... [root at t4pserver2 mailpop3]# ls -al total 248652 drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. Any other ideas? /Mark -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Stan Hoeppner Sent: Tuesday, March 20, 2012 7:59 PM To: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From nicolas.kowalski at gmail.com Wed Mar 21 08:50:49 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Wed, 21 Mar 2012 07:50:49 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? In-Reply-To: <20120320135512.GD28951@petole.demisel.net> References: <20120320135512.GD28951@petole.demisel.net> Message-ID: <20120321065049.GE28951@petole.demisel.net> On Tue, Mar 20, 2012 at 02:55:12PM +0100, Nicolas KOWALSKI wrote: > Does dovecot support the subject Alternative Name email value [1] as > ssl_cert_username_field? If so, how should it be specified in the > configuration? Well, I just found the wiki states no: "The text is looked up from subject DN's specified field" (http://wiki2.dovecot.org/SSL/DovecotConfiguration) Sorry for the noise, -- Nicolas From stan at hardwarefreak.com Wed Mar 21 10:41:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 03:41:39 -0500 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F699443.1090704@hardwarefreak.com> On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From nmilas at noa.gr Wed Mar 21 11:00:10 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Wed, 21 Mar 2012 11:00:10 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 Message-ID: <4F69989A.3000106@noa.gr> Hi, I've upgraded from 2.0.13 to 2.1.1 and when I started the service, I got the following warning: Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) I didn't see such warnings in 2.0.13. I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? (I could also set "userdb_warning_disable=yes" as advised, but I'm trying to figure out what's the real cause of the warning.) The config follows below. Thanks, Nick ============================================================= protocols = imap pop3 mail_location = maildir:~/Maildir/ mail_gid = 502 mail_uid = 502 auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes disable_plaintext_auth = no mail_plugins = quota protocol imap { imap_client_workarounds = "delay-newmail " mail_plugins = quota imap_quota } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota postmaster_address = sysadmin at example.com sendmail_path = /usr/lib/sendmail } userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap } passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u } service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root } service imap-login { service_count = 1 vsz_limit = 64 M } service pop3-login { service_count = 1 vsz_limit = 64 M } ssl_ca = References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> On 19.3.2012, at 21.16, Alex Ha wrote: >>>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>>> existing connection >>> >>> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >>> >> >> Thanks Timo! I will try the patch and report to you. >> > > Hi Timo! > > I tried the patch with 2.0.19 and the dovecot error messages disappeared. OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever gets released). > I still get a lot of this postfix warnings: > > SASL LOGIN authentication failed: Connection lost to authentication server > > but only for ips which tried a sasl brute force attack. > > "Connection lost to authentication server" could this be because of > the dovecot auth penalties? > so far i did not get any complaints from users. The auth penalties wait for max. 17 seconds I think. Looks like Postfix has a timeout of 10 seconds. You could disable auth penalties, or perhaps Postfix should use 20 second limit. From CMarcus at Media-Brokers.com Wed Mar 21 13:55:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 07:55:19 -0400 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> Message-ID: <4F69C1A7.2040601@Media-Brokers.com> On 2012-03-21 7:48 AM, Timo Sirainen wrote: > On 19.3.2012, at 21.16, Alex Ha wrote: >>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>> 7542 of existing connection >>>> Oh, right, PIDs of course aren't unique when you're using >>>> mulitiple servers. Try if the attached patch fixes your >>>> troubles. If it does, I'll commit it to hg. >>> Thanks Timo! I will try the patch and report to you. >> I tried the patch with 2.0.19 and the dovecot error messages >> disappeared. > OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever > gets released). Presumably you mean 2.1.4 (since 2.1.3 is already released)? -- Best regards, Charles From tss at iki.fi Wed Mar 21 13:57:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:57:45 +0200 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: On 20.3.2012, at 20.29, Mark Jeghers wrote: > Below is my config. When I run dovecot from xinetd, I get these errors in the log: You can't run Dovecot v2.x via inetd. You could run it via systemd though. From tss at iki.fi Wed Mar 21 13:58:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:58:09 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <4F69C1A7.2040601@Media-Brokers.com> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> <4F69C1A7.2040601@Media-Brokers.com> Message-ID: <38A53BE8-A53F-4906-996F-6CC863E537CC@iki.fi> On 21.3.2012, at 13.55, Charles Marcus wrote: > On 2012-03-21 7:48 AM, Timo Sirainen wrote: >> On 19.3.2012, at 21.16, Alex Ha wrote: >>>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>>> 7542 of existing connection > >>>>> Oh, right, PIDs of course aren't unique when you're using >>>>> mulitiple servers. Try if the attached patch fixes your >>>>> troubles. If it does, I'll commit it to hg. > >>>> Thanks Timo! I will try the patch and report to you. > >>> I tried the patch with 2.0.19 and the dovecot error messages >>> disappeared. > >> OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever >> gets released). > > Presumably you mean 2.1.4 (since 2.1.3 is already released)? Ah, yes. :) From tss at iki.fi Wed Mar 21 13:59:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:59:50 +0200 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <4F68A189.2010800@pacific.net> References: <4F68A189.2010800@pacific.net> Message-ID: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> On 20.3.2012, at 17.26, Ken A wrote: > With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? > > Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. From tss at iki.fi Wed Mar 21 14:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:06:53 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 In-Reply-To: <4F69989A.3000106@noa.gr> References: <4F69989A.3000106@noa.gr> Message-ID: On 21.3.2012, at 11.00, Nikolaos Milas wrote: > Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) > Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) > > I didn't see such warnings in 2.0.13. > > I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? Hmm. Yes, if dovecot-usrdb-ldap.conf is a separate file from dovecot-passdb-ldap.conf you can just remove it. But this reminds me that in several places I've suggested to make one of them a symlink to the other, and you can't really do it then. Perhaps I'll need to remove this warning, or maybe make it recognize the symlink case. Anyway I added it for both LDAP and SQL hoping that it would reduce questions like: "I changed user_attrs, but it doesn't do anything!" From tss at iki.fi Wed Mar 21 14:26:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:26:05 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: On 20.3.2012, at 16.55, Patrick Domack wrote: > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From patrickdk at patrickdk.com Wed Mar 21 14:47:56 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 08:47:56 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321084756.Horde.xxWqdZLnE6FPac38iyLGWYA@kishi.patrickdk.com> Thanks, applied it to 2.1.3 and going to test. You didn't even give me enough time to look at the source myself to find the issue. Quoting Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > >> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have >> double S and W tags. >> >> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >> >> This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From noel.butler at ausics.net Wed Mar 21 15:26:19 2012 From: noel.butler at ausics.net (Noel Butler) Date: Wed, 21 Mar 2012 23:26:19 +1000 Subject: [Dovecot] sysconfdir depreacted Message-ID: <1332336379.10474.5.camel@tardis> The purpose of any build scripts --sysconfdir is to tell the configuration to build in a path for its binaries configuration file(s). Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, is this a bug? if not, then I see no point of sysconfdir any more and it should be removed, if dovecot deliberately ignores what it is told to use. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Wed Mar 21 15:46:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 15:46:44 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332336379.10474.5.camel@tardis> References: <1332336379.10474.5.camel@tardis> Message-ID: On 21.3.2012, at 15.26, Noel Butler wrote: > The purpose of any build scripts --sysconfdir is to tell the > configuration to build in a path for its binaries configuration file(s). > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > is this a bug? if not, then I see no point of sysconfdir any more and it > should be removed, if dovecot deliberately ignores what it is told to > use. --sysconfdir=/etc uses /etc/dovecot/ --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ There is now always the dovecot/ suffix, but the the /etc part is still configurable. From jtl+dovecot at uvm.edu Wed Mar 21 15:53:50 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 09:53:50 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) Message-ID: <4F69DD6E.1090502@uvm.edu> Had a user who couldn't access his INBOX: > Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file > mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai > led: (offset < (uint16_t)-1) > Mar 21 09:21:17 penguina dovecot: imap([USER]): Error: Raw backtrace: > /usr/lib/dovecot/libdovecot.so.0 [0x342683c660] -> /usr > /lib/dovecot/libdovecot.so.0 [0x342683c6b6] -> > /usr/lib/dovecot/libdovecot.so.0 [0x342683bb73] -> > /usr/lib/dovecot/libdovecot > -storage.so.0 [0x3426c966a8] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_ext_intro+0x240) > [0x3426c979c0] -> / > usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_record+0x401) > [0x3426c99151] -> /usr/lib/dovecot/libdovecot-storage.s > o.0(mail_index_sync_map+0x245) [0x3426c99c55] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_map+0x71b) > [0x3426c8afbb > ] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c85d8b] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_open+0x1c > e) [0x3426c8617e] -> > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_mailbox_open+0xb5) > [0x3426c4d865] -> /usr/lib/dov > ecot/libdovecot-storage.so.0 [0x3426c75eab] -> > /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c31006] -> > dovecot/imap [hdtod > d 10.245.30.58 SELECT](cmd_ Stack trace made it look like it was the INBOX, so I deleted the index files for his INBOX and everything was OK. doveconf -n: > # OS: Linux 2.6.18-274.18.1.el5 x86_64 Red Hat Enterprise Linux Server > release 5.8 (Tikanga) > auth_gssapi_hostname = penguina.uvm.edu > auth_krb5_keytab = /etc/krb5.keytab.dovecot > auth_master_user_separator = * > auth_mechanisms = plain login gssapi > base_dir = /var/run/dovecot/ > default_process_limit = 250 > first_valid_uid = 50 > lock_method = flock > login_trusted_networks = [REDACTED] > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%1u/%1.1u/%u > mail_max_lock_timeout = 30 secs > mail_max_userip_connections = 100 > mbox_read_locks = flock > mbox_write_locks = flock > mmap_disable = yes > namespace { > inbox = yes > location = > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > args = /etc/dovecot/passwd.masterusers > driver = passwd-file > master = yes > } > passdb { > driver = pam > } > service imap { > process_limit = 4096 > } > service lmtp { > client_limit = 1 > inet_listener lmtp { > port = 24 > } > } > ssl_cert = <[REDACTED] > ssl_key = < [REDACTED] > userdb { > driver = passwd > } > verbose_proctitle = yes Any questions/suggestions welcome. Jim From tss at iki.fi Wed Mar 21 16:02:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 16:02:47 +0200 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <4F69DD6E.1090502@uvm.edu> References: <4F69DD6E.1090502@uvm.edu> Message-ID: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> On 21.3.2012, at 15.53, Jim Lawson wrote: > Had a user who couldn't access his INBOX: > >> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >> led: (offset < (uint16_t)-1) I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. > Stack trace made it look like it was the INBOX, so I deleted the index > files for his INBOX and everything was OK. If it happens again, get a copy of the indexes. From CMarcus at Media-Brokers.com Wed Mar 21 16:26:29 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 10:26:29 -0400 Subject: [Dovecot] squat not working in 2.1 In-Reply-To: <20120229143038.GX13045@charite.de> References: <6A93411B-4058-4A7D-9F94-452403AE83ED@iki.fi> <4F4DF7F7.8020405@in.tum.de> <20120229100957.GX13045@charite.de> <20120229102250.GY13045@charite.de> <20120229132718.GN13045@charite.de> <92D75C5F-46E8-4EE4-B43D-60A3261E071C@iki.fi> <46a03b5105c847df7f7491f0889ef7ec@imt-systems.com> <20120229135851.GU13045@charite.de> <1be342370509d17ae81682aede00f016@imt-systems.com> <5febb8861c0cc824b0446cb2fec98d19@imt-systems.com> <20120229143038.GX13045@charite.de> Message-ID: <4F69E515.9080904@Media-Brokers.com> On 2012-02-29 9:30 AM, Ralf Hildebrandt wrote: > * Morten Stevens: > >> This is a Fedora-specific problem, because clucene (build >> requirement) is not correctly packaged. > > Well, debian showed the same packaging (wrong place). I just attempted to update to 2.1.3 on gentoo and received the same error: /usr/include/CLucene/SharedHeader.h:18:36: fatal error: CLucene/clucene-config.h: No such file or directory So, is this also a packaging error that I need to report to gentoo? -- Best regards, Charles From Ralf.Hildebrandt at charite.de Wed Mar 21 16:33:48 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 15:33:48 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321143348.GR2789@charite.de> * Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > > > This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 That doesn't seem to work: Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename has wrong S value, renamed the file from /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S to /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) failed: Input/output error (uid=5270) It's renaming itself to itself again? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From luca at lm-net.it Wed Mar 21 16:43:14 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Wed, 21 Mar 2012 15:43:14 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup Message-ID: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Hello list. I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. It would be desirable for multiple reasons: - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) - be ready to serve traffic over IPv6 without touching our precious mailbox servers - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). I'd like to know if that plan sounds reasonable or if there's something stupid in it. Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? thanks, -- Luca Lesinigo From tss at iki.fi Wed Mar 21 17:00:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:00:26 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120321143348.GR2789@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: On 21.3.2012, at 16.33, Ralf Hildebrandt wrote: >>> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. >>> >>> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >>> >>> This is happening for all folder moves. >> >> Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 > > That doesn't seem to work: It fixed only the duplicate S= and W= values. > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename > has wrong S value, renamed the file from > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > to > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) > failed: Input/output error (uid=5270) > > It's renaming itself to itself again? Hmm. Yeah, this is a bit problematic for compressed mails. If the S=size isn't correct, Dovecot fixes it by stat()ing the file and using it as the size. And that's of course wrong. Also Dovecot can't simply remove the S=size, because the current Maildir code assumes that it always exists for compressed mails. There's no easy and efficient way to fix this.. Maybe you could just manually rename the files to have correct S=size? :) zcat file | wc should give the right size. From tss at iki.fi Wed Mar 21 17:17:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:17:56 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F684938.9000208@unict.it> References: <4F684938.9000208@unict.it> Message-ID: Hi, On 20.3.2012, at 11.09, Luca Palazzo wrote: > Hi Timo, hi all, > after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > > 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > 710 { > (gdb) bt > #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 526 bytes Desc: not available URL: From ka at pacific.net Wed Mar 21 17:19:16 2012 From: ka at pacific.net (Ken Anderson) Date: Wed, 21 Mar 2012 10:19:16 -0500 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> References: <4F68A189.2010800@pacific.net> <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> Message-ID: <4F69F174.9000501@pacific.net> On 3/21/2012 6:59 AM, Timo Sirainen wrote: > On 20.3.2012, at 17.26, Ken A wrote: > >> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? >> >> Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? > > Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. > Awesome! I haven't migrated to mdbox yet, but in testing with it on a dev server, it looks like it will solve a huge problem. Users seem to want ever larger mailboxes, and mdbox gives them that, without asking more than additional disk space. Fixing the pop locking would be an additional benefit! Thanks, Ken Pacific.Net From tss at iki.fi Wed Mar 21 17:21:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:21:43 +0200 Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 In-Reply-To: <.120.61.90.33.1332228956.squirrel@24x7server.net> References: <.120.61.90.33.1332228956.squirrel@24x7server.net> Message-ID: <7D494B74-E138-415F-8010-F1208604E246@iki.fi> On 20.3.2012, at 9.35, Rajesh M wrote: > i migrated my email server with around 5000 users from dovecot version 1.2 > to version 2 > > i have two separate 2 tb hdd's storing webmail data of these users. You mean you simply upgraded the Dovecot version, the server is exactly the same? > the load on the server goes very high over 100 during peak load times and > the imap connections get dropped frequently, webmail becomes very slow. There shouldn't be much performance difference between v1.2 and v2.x. > in the dovecot log file i get errors as such > > Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: > Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 > expunge attempts) This simply means that the disk IO usage is very high. > i am a bit confused as to what settings are to be done for a very busy server Show dovecot -n output of the new server, and if you have the old configuration available that could be helpful also to compare their differences. From tss at iki.fi Wed Mar 21 17:25:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:25:46 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: On 19.3.2012, at 12.50, Matteo Cazzador wrote: > Hi, i've a simple question, what do you mean for dovecot director setup? > 'i've a doubt. > The solution that i'm testing is using 3 mail server in different > geoghrapic locations. > An user can travel in varius location, and i want his imap mail reside > on mail server in every locations. > S? i use you solution about replication. First server (by dns record) > that receive mail sync it on the other servers, and when > user consult is mail by imap protocol everything is sync on all servers. > Do you suggest to use a horizontal structure for it like i explain or > is better to have a single node external mail server > and customer locations server like slave? Dovecot director isn't really meant to be used for geographic user distribution. Also the replication doesn't yet support more than two servers. A master-slave setup wouldn't have the UID conflict problems that multi-master dsync replication has, but the UID conflicts probably won't be a big problem. Anyway, difficult to give recommendations about an unfinished feature.. From lukas.mueller at newmedia.ch Wed Mar 21 17:45:09 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Wed, 21 Mar 2012 15:45:09 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS Message-ID: Hi, I'm stuck with a problem we have with dovecot. My suspicion is, that it has to do with accessing the same mailbox/mail stored on a NFS-share from two machines at the same time. setup We have to mail servers running, both run a Ubuntu 10.04, Postfix 2.70 and Dovecot 1.2.9. The mailboxes are stored in maildir format on a NFS-Share. In front of those to mail servers we have a load balancer. Unfortunately it can't be set up to use the same server for each domain, but it uses the same server for the same source-ip for at least 1 hour. Here is the output of dovecot -n: # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-33-server x86_64 Ubuntu 10.04.3 LTS nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/ca-bundle/SSL123_CA_Bundle.pem ssl_cert_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.crt ssl_key_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.key ssl_verify_client_cert: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_max_userip_connections: 25 mail_privileged_group: mail mail_location: maildir:/data/vmail/%d/%n:INDEX=/data/vmail/%d/%n/indexes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): auth default: passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf plugin: quota: maildir:storage=409600 sieve_global_path: /data/vmail/globalsieverc dict: quotadict: mysql:/etc/dovecot-dict-quota.conf problem the problem happens with a client's mailbox that is used by multiple users. >From time to time he cannot see any Emails in the mailbox, neither with his mail clients (Apple Mail) nor with in the webmail (Roundcube). Around this time I get the following entries in the log files: Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1524, file=1327500903.V15I5722c8M210039.mail01:2,Se) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f37602b8b5a] -> imap(+0xaebc7) [0x7f37602b8bc7] -> imap(+0xae238) [0x7f37602b8238] -> imap(+0x497d7) [0x7f37602537d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f37602545c2] -> imap(+0x4bb06) [0x7f3760255b06] -> imap(maildir_uidlist_sync_init+0x4d) [0x7f376025652d] -> imap(+0x46ed4) [0x7f3760250ed4] -> imap(maildir_storage_sync_init+0x147) [0x7f3760251557] -> imap(imap_sync_init+0x70) [0x7f376023b190] -> imap(+0x2411e) [0x7f376022e11e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f37602c069c] -> imap(io_loop_handler_run+0x60) [0x7f37602c1000] -> imap(io_loop_run+0x18) [0x7f37602c0448] -> imap(main+0x58e) [0x7f376023bc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f375f877c4d] -> imap(+0x21979) [0x7f376022b979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 16934 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7ff81b415b5a] -> imap(+0xaebc7) [0x7ff81b415bc7] -> imap(+0xae238) [0x7ff81b415238] -> imap(+0x497d7) [0x7ff81b3b07d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7ff81b3b15c2] -> imap(maildir_uidlist_sync_init+0x105) [0x7ff81b3b35e5] -> imap(+0x46ed4) [0x7ff81b3aded4] -> imap(maildir_storage_sync_force+0x52) [0x7ff81b3ae392] -> imap(maildir_file_do+0x99) [0x7ff81b3b3cb9] -> imap(+0x4d944) [0x7ff81b3b4944] -> imap(index_mail_set_seq+0x148) [0x7ff81b3c8ed8] -> imap(index_storage_search_next_nonblock+0x162) [0x7ff81b3cd622] -> imap(mailbox_search_next_nonblock+0x20) [0x7ff81b3db2c0] -> imap(mailbox_search_next+0x26) [0x7ff81b3db316] -> imap(imap_fetch_more+0x2bf) [0x7ff81b39295f] -> imap(cmd_fetch+0x36c) [0x7ff81b38a9ec] -> imap(+0x28fad) [0x7ff81b38ffad] -> imap(+0x2908d) [0x7ff81b39008d] -> imap(client_handle_input+0x135) [0x7ff81b3902c5] -> imap(client_input+0x5f) [0x7ff81b390baf] -> imap(io_loop_handler_run+0xbd) [0x7ff81b41e05d] -> imap(io_loop_run+0x18) [0x7ff81b41d448] -> Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): imap(main+0x58e) [0x7ff81b398c5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ff81a9d4c4d] -> imap(+0x21979) [0x7ff81b388979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 13712 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f4a311fcb5a] -> imap(+0xaebc7) [0x7f4a311fcbc7] -> imap(+0xae238) [0x7f4a311fc238] -> imap(+0x497d7) [0x7f4a311977d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f4a311985c2] -> imap(+0x47023) [0x7f4a31195023] -> imap(maildir_storage_sync_init+0x147) [0x7f4a31195557] -> imap(imap_sync_init+0x70) [0x7f4a3117f190] -> imap(+0x2411e) [0x7f4a3117211e] -> imap(+0x64c0e) [0x7f4a311b2c0e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f4a3120469c] -> imap(io_loop_handler_run+0x60) [0x7f4a31205000] -> imap(io_loop_run+0x18) [0x7f4a31204448] -> imap(main+0x58e) [0x7f4a3117fc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f4a307bbc4d] -> imap(+0x21979) [0x7f4a3116f979] Mar 6 08:42:31 mail01 dovecot: dovecot: child 24257 (imap) killed with signal 6 (core dumps disabled) Note: the first part is on a mail server, while the smaller second part is on the other one. Unfortunately I'm not able to reproduce this error. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. I have no idea how to solve this problem and any help is greatly appreciated. If you need further information, please say so. Mit freundlichen Gr?ssen Lukas M?ller Systems Engineer _______________________________________________ NEWMEDIA S?dostschweiz Newmedia AG Kasernenstrasse 1 Postfach 508, CH-7007 Chur http://www.newmedia.ch _______________________________________________ TYPO3 & Drupal - Wir wissen wie. Ihre professionelle Web Agentur in Chur, Ilanz, Glarus und Z?rich. From tss at iki.fi Wed Mar 21 17:50:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:50:29 +0200 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: Message-ID: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> On 21.3.2012, at 17.45, M?ller Lukas wrote: > Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. > My suspicion/speculation what happens is the following: > Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. > Somehow this leads to problems with Locks on NFS, which leads to the crash. Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > I have no idea how to solve this problem and any help is greatly appreciated. The only way to fully fix this is: http://wiki2.dovecot.org/Director From Ralf.Hildebrandt at charite.de Wed Mar 21 17:52:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 16:52:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: <20120321155245.GS2789@charite.de> * Timo Sirainen : > > It's renaming itself to itself again? > > Hmm. Yeah, this is a bit problematic for compressed mails. If the > S=size isn't correct, Dovecot fixes it by stat()ing the file and using > it as the size. And that's of course wrong. Also Dovecot can't simply > remove the S=size, because the current Maildir code assumes that it > always exists for compressed mails. There's no easy and efficient way > to fix this.. Maybe you could just manually rename the files to have > correct S=size? :) zcat file | wc should give the right size. Right now the whole system is down because nobody can acces his/her mails due to this. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From fxmulder at gmail.com Wed Mar 21 17:56:12 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 09:56:12 -0600 Subject: [Dovecot] distributed mdbox Message-ID: Anyone know how to setup dovecot with mdbox so that it can be used through shared storage from multiple hosts? I've setup a gluster volume and am sharing it between 2 test clients. I'm using postfix/dovecot LDA for delivery and I'm using postal to send mail between 40 users. In doing this, I'm seeing these errors in the logs Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count 272 -> 271 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=3768 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but next_uid = 517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=4220 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=5088 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index This is my dovecot config currently: jdevine at test-gluster-client2:~> dovecot -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 lock_method = dotlock mail_fsync = always mail_location = mdbox:~/mdbox mail_nfs_index = yes mail_nfs_storage = yes mmap_disable = yes passdb { driver = pam } protocols = " imap" ssl_cert = References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> Message-ID: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > * Timo Sirainen : > >>> It's renaming itself to itself again? >> >> Hmm. Yeah, this is a bit problematic for compressed mails. If the >> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >> it as the size. And that's of course wrong. Also Dovecot can't simply >> remove the S=size, because the current Maildir code assumes that it >> always exists for compressed mails. There's no easy and efficient way >> to fix this.. Maybe you could just manually rename the files to have >> correct S=size? :) zcat file | wc should give the right size. > > Right now the whole system is down because nobody can acces his/her > mails due to this. All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 385 bytes Desc: not available URL: From luca.palazzo at unict.it Wed Mar 21 18:04:00 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Wed, 21 Mar 2012 17:04:00 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: References: <4F684938.9000208@unict.it> Message-ID: <4F69FBF0.6090003@unict.it> It worked. We have no more sigsegv on *-login process. Thanks Luca Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > Hi, > > On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> Hi Timo, hi all, >> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >> >> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> 710 { >> (gdb) bt >> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > > Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > From tss at iki.fi Wed Mar 21 18:05:52 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:05:52 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> On 21.3.2012, at 17.56, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs Dovecot assumes that the filesystem behaves the same way as regular local filesystems. > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 Looks like gluster doesn't fit that assumption. So, the solution is the same as with NFS: http://wiki2.dovecot.org/Director From tss at iki.fi Wed Mar 21 18:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:08:07 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F69FBF0.6090003@unict.it> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> Message-ID: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e I'll figure out a proper fix soon. On 21.3.2012, at 18.04, Luca Palazzo wrote: > It worked. We have no more sigsegv on *-login process. > > Thanks > > Luca > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >> Hi, >> >> On 20.3.2012, at 11.09, Luca Palazzo wrote: >> >>> Hi Timo, hi all, >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>> >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> 710 { >>> (gdb) bt >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >> >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >> > From fxmulder at gmail.com Wed Mar 21 18:25:14 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:25:14 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > On 21.3.2012, at 17.56, James Devine wrote: > > > Anyone know how to setup dovecot with mdbox so that it can be used > through > > shared storage from multiple hosts? I've setup a gluster volume and am > > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > > delivery and I'm using postal to send mail between 40 users. In doing > > this, I'm seeing these errors in the logs > > Dovecot assumes that the filesystem behaves the same way as regular local > filesystems. > > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: > Fixed > > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: > messages_count > > 272 -> 271 > > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > > synchronization error at seq=4,offset=3768 for > > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > > next_uid = 517 > > Looks like gluster doesn't fit that assumption. So, the solution is the > same as with NFS: http://wiki2.dovecot.org/Director > > What filesystem mechanisms might not be working in this case? From fxmulder at gmail.com Wed Mar 21 18:47:53 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:47:53 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: Also I don't seem to get these errors with a single dovecot machine using the shared storage and it looks like there are multiple simultaneous delivery processes running On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > > > On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > >> On 21.3.2012, at 17.56, James Devine wrote: >> >> > Anyone know how to setup dovecot with mdbox so that it can be used >> through >> > shared storage from multiple hosts? I've setup a gluster volume and am >> > sharing it between 2 test clients. I'm using postfix/dovecot LDA for >> > delivery and I'm using postal to send mail between 40 users. In doing >> > this, I'm seeing these errors in the logs >> >> Dovecot assumes that the filesystem behaves the same way as regular local >> filesystems. >> >> > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >> Fixed >> > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >> messages_count >> > 272 -> 271 >> > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >> Log >> > synchronization error at seq=4,offset=3768 for >> > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >> but >> > next_uid = 517 >> >> Looks like gluster doesn't fit that assumption. So, the solution is the >> same as with NFS: http://wiki2.dovecot.org/Director >> >> > What filesystem mechanisms might not be working in this case? > From tss at iki.fi Wed Mar 21 19:04:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 19:04:36 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. On 21.3.2012, at 18.47, James Devine wrote: > Also I don't seem to get these errors with a single dovecot machine using > the shared storage and it looks like there are multiple simultaneous > delivery processes running > > On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > >> >> >> On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: >> >>> On 21.3.2012, at 17.56, James Devine wrote: >>> >>>> Anyone know how to setup dovecot with mdbox so that it can be used >>> through >>>> shared storage from multiple hosts? I've setup a gluster volume and am >>>> sharing it between 2 test clients. I'm using postfix/dovecot LDA for >>>> delivery and I'm using postal to send mail between 40 users. In doing >>>> this, I'm seeing these errors in the logs >>> >>> Dovecot assumes that the filesystem behaves the same way as regular local >>> filesystems. >>> >>>> Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >>> Fixed >>>> index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >>> messages_count >>>> 272 -> 271 >>>> Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >>> Log >>>> synchronization error at seq=4,offset=3768 for >>>> /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >>> but >>>> next_uid = 517 >>> >>> Looks like gluster doesn't fit that assumption. So, the solution is the >>> same as with NFS: http://wiki2.dovecot.org/Director >>> >>> >> What filesystem mechanisms might not be working in this case? >> From jtl+dovecot at uvm.edu Wed Mar 21 20:19:19 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 14:19:19 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> References: <4F69DD6E.1090502@uvm.edu> <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> Message-ID: <4F6A1BA7.7030208@uvm.edu> On 3/21/12 10:02 AM, Timo Sirainen wrote: > On 21.3.2012, at 15.53, Jim Lawson wrote: > >> Had a user who couldn't access his INBOX: >> >>> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >>> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >>> led: (offset < (uint16_t)-1) > I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. This applies cleanly against 2.0.19; should I try it on that version, or not recommended? >> Stack trace made it look like it was the INBOX, so I deleted the index >> files for his INBOX and everything was OK. > If it happens again, get a copy of the indexes. > I sent them, encrypted, to your email address/GPG key 0x40558AC9. Jim From mjeghers at Brocade.com Wed Mar 21 20:59:39 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Wed, 21 Mar 2012 11:59:39 -0700 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <4F699443.1090704@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> All, I was mistaken in how I described my problem, please forgive this dovecot newbie for describing the problem incorrectly! It is not under xinitd, it is trying to run as an init.d service. Ok, let's try again... I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? Thanks, hope this clarifies things, /Mark -----Original Message----- From: Stan Hoeppner [mailto:stan at hardwarefreak.com] Sent: Wednesday, March 21, 2012 1:42 AM To: Mark Jeghers Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From tss at iki.fi Wed Mar 21 21:16:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 21:16:51 +0200 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> Message-ID: On 21.3.2012, at 20.59, Mark Jeghers wrote: > I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? .. >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Permission errors point to SELinux being the problem. Try disabling it. From bear at rwhartzell.net Wed Mar 21 22:54:09 2012 From: bear at rwhartzell.net (Robert Hartzell) Date: Wed, 21 Mar 2012 13:54:09 -0700 Subject: [Dovecot] Dovecot 2.1.3 on solaris with mysql - make fails Message-ID: <95DD93BE-F841-4BEB-A96C-059FFF0ACF2F@rwhartzell.net> I'm trying to build 2.1.3 on solaris 11 11/11 with gcc 4.5.2 & sun studio 12.2 & 12.3 CPPFLAGS="-I/opt/openssl/include -I/usr/mysql/include/mysql" \ LDFLAGS="-L/opt/openssl/lib -L/usr/mysql/lib/mysql -R/opt/openssl/lib:/usr/mysql/lib/mysql" \ ./configure --prefix=/opt/dovecot \ --sysconfdir=/etc/opt \ --with-ssl=openssl \ --with-mysql make fails with both solaris standard openssl and my build of openssl. I'm also getting the same error using sunstudio mysql version is 5.1.37 The relevant output of make is on pastebin http://pastebin.com/aALHG0yL I have seen some reference to this with google but nothing thats very recent and no solutions. Anyone know how to get past this? Any tips on building dovecot on solaris? Pointers would be much appreciated. -- Robert From patrickdk at patrickdk.com Wed Mar 21 23:52:39 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 17:52:39 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120321175239.Horde.6cigCZLnE6FPak2nibEXzWA@kishi.patrickdk.com> Quoting Timo Sirainen : > On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > >> * Timo Sirainen : >> >>>> It's renaming itself to itself again? >>> >>> Hmm. Yeah, this is a bit problematic for compressed mails. If the >>> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >>> it as the size. And that's of course wrong. Also Dovecot can't simply >>> remove the S=size, because the current Maildir code assumes that it >>> always exists for compressed mails. There's no easy and efficient way >>> to fix this.. Maybe you could just manually rename the files to have >>> correct S=size? :) zcat file | wc should give the right size. >> >> Right now the whole system is down because nobody can acces his/her >> mails due to this. > > All of your mails are compressed and have wrong S=size in the > filename? You can disable the check with the attached patch, but I'm > not sure if there are other places where it fails. At least quota > calculations won't be correct. The issue only started happening since I upgraded to 2.1.1, it didn't exist before then, I have check my system, and files before the date of upgrade are fine, only files/emails moved after upgrading to 2.1.1 have lost the S= value. I have made something that can pretty easily fix the issue, but it only stays fixed till another email gets moved and looses it's S= value. Sorry, I haven't had time to test out 2.1.3 yet. This will print out the commands needed to fix the files though. find . -name '*hostname:*' -exec 'gzip' '-l' '{}' ';' | awk '/hostname/ {for(x=4;x References: <1332336379.10474.5.camel@tardis> Message-ID: <1332381356.4112.9.camel@tardis> On Wed, 2012-03-21 at 15:46 +0200, Timo Sirainen wrote: > On 21.3.2012, at 15.26, Noel Butler wrote: > > > The purpose of any build scripts --sysconfdir is to tell the > > configuration to build in a path for its binaries configuration file(s). > > > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > > is this a bug? if not, then I see no point of sysconfdir any more and it > > should be removed, if dovecot deliberately ignores what it is told to > > use. > > > --sysconfdir=/etc uses /etc/dovecot/ > > --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ > > There is now always the dovecot/ suffix, but the the /etc part is still configurable. > perhaps it should be renamed then, given it violates the known normal for SYSCONF dir, you've just created another form of --datadir from gnu.org: "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) "datadir" The directory for installing idiosyncratic read-only architecture-independent data files for this program. This is usually the same place as ?datarootdir?, but we use the two separate variables so that you can move these program-specific files without altering the location for Info files, man pages, etc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From gedalya at gedalya.net Thu Mar 22 04:46:20 2012 From: gedalya at gedalya.net (Gedalya) Date: Wed, 21 Mar 2012 22:46:20 -0400 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A8BAC.4000002@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> Message-ID: <4F6A927C.6010003@gedalya.net> On 3/21/2012 10:17 PM, Martin Schitter wrote: > Am 16.3.2011 20:59, schrieb Gedalya: >>> >>>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at >>>> example.com -o imapc_password=**** backup -u jedi at example.com -R >>>> imapc: >>> >> Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current >> versions are putting the body of the last message in "Sent Items" in >> place of every single email in INBOX. >> In other words, for every email that sits in INBOX in the source, I get >> a copy of the last email in "Sent Items" instead. >> This happens for every account I try to migrate. >> Very strange. I noticed this only now, and the last package I have left >> in the local apt cache which still works is 2.1.rc7-0~auto+0. > > i see the same regression (2.1.3-0~auto+4) :( > > doveadm sync/backup via impac puts the same message all over the place... Thanks Martin, I've set up a test platform to investigate this further but I've been short on time... From stan at hardwarefreak.com Thu Mar 22 06:11:19 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 23:11:19 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> Message-ID: <4F6AA667.1080908@hardwarefreak.com> On 3/21/2012 12:04 PM, Timo Sirainen wrote: > The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. > > With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could use a true cluster file system such as OCFS or GFS. Both will eliminate this problem, and without requiring Dovecot director. And you'll get better performance than with Gluster, which, BTW, isn't really suitable as a transactional filesystem, was not designed for such a use case. -- Stan From ruskie at codemages.net Thu Mar 22 08:28:40 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Thu, 22 Mar 2012 07:28:40 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332381356.4112.9.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: :2012-03-22T11:55:Noel Butler: > perhaps it should be renamed then, given it violates the known normal > for SYSCONF dir, you've just created another form of --datadir Not really. The way I see it works as expected. The sysconf dir is the root of the configuration dir. Then if the app so chooses uses it's own directory structure under that. Considering that by default dovecot uses dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Quis custodiet ipsos custodes? From noel.butler at ausics.net Thu Mar 22 10:30:27 2012 From: noel.butler at ausics.net (Noel Butler) Date: Thu, 22 Mar 2012 18:30:27 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: <1332405027.6792.7.camel@tardis> On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > :2012-03-22T11:55:Noel Butler: > > > perhaps it should be renamed then, given it violates the known normal > > for SYSCONF dir, you've just created another form of --datadir > > Not really. The way I see it works as expected. The sysconf dir is the Then you and I and a few other devs involved in other very well known bits of software that everyone likely uses, will have to agree to disagree "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) > root of the configuration dir. Then if the app so chooses uses it's own > directory structure under that. Considering that by default dovecot uses > dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. > By default as of only 2.something, not in 0.x not in 1.0.x not in 1.1.x and not in 1.2.x I've said all I'm going to say on the mater, I got three emails offlist from others here agreeing with me, shame they didn't do it on-list, but I respect their right to remain silent so as not to endure the wrath of Timo and certain other cretins well known for having nothing else better to do. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From Ralf.Hildebrandt at charite.de Thu Mar 22 10:46:10 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 22 Mar 2012 09:46:10 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120322084609.GF28323@charite.de> * Timo Sirainen : > > Right now the whole system is down because nobody can acces his/her > > mails due to this. > > All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. That patch totally saved my ass. I rolled it out today and the Mar 22 09:33:00 postamt dovecot: imap(stoffelm): Error: Maildir filename has wrong S value, renamed the file from /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd to /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd errors subsided. At the same time the users CAN access the affected folder. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From amateo at um.es Thu Mar 22 11:55:58 2012 From: amateo at um.es (Angel L. Mateo) Date: Thu, 22 Mar 2012 10:55:58 +0100 Subject: [Dovecot] dovecot-auth restaring and caching Message-ID: <4F6AF72E.9030206@um.es> Hello, I'm trying to configure dovecot (2.0.13) to cache user and pass dbs. This a mail server whose purpose is only to deliver messages through dovecot lda. My users are in a ldap server. So I have configure auth_cache_size (with 20MB) and auth_cache_ttl (with 1 day). I have checked that caching is being done, and it is. If a send a message to a user, dovecot looks for it in my ldap server. If then I send another, then it uses cache information. The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. This is the auth log for the first message: Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:29:41 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:29:41 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXX Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXXX#011gid=XXX#011home=XXXXXXXXXXXX And this is the second one, just after a few minutes: Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:41:03 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:41:03 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXXX Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXX#011gid=XXX#011home=XXXXXXXXXX This is my configuration: root at lynx10:/etc/dovecot/conf.d# doveconf -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-4-amd64 x86_64 Ubuntu 10.04.4 LTS auth_cache_size = 20 M auth_cache_ttl = 1 days auth_debug = yes auth_verbose = yes hostname = lynx10 passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +imapflags sieve_max_redirects = 15 } postmaster_address = postmaster at um.es protocols = " imap lmtp pop3" service auth { unix_listener auth-userdb { mode = 0666 } } ssl_cert = References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> Message-ID: <4F6A8BAC.4000002@mur.at> Am 16.3.2011 20:59, schrieb Gedalya: >> >>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current > versions are putting the body of the last message in "Sent Items" in > place of every single email in INBOX. > In other words, for every email that sits in INBOX in the source, I get > a copy of the last email in "Sent Items" instead. > This happens for every account I try to migrate. > Very strange. I noticed this only now, and the last package I have left > in the local apt cache which still works is 2.1.rc7-0~auto+0. i see the same regression (2.1.3-0~auto+4) :( doveadm sync/backup via impac puts the same message all over the place... From tss at iki.fi Thu Mar 22 16:05:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 16:05:34 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> Message-ID: <1332425134.26095.88.camel@innu> Hi, These should fix it properly: http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: > The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e > > I'll figure out a proper fix soon. > > On 21.3.2012, at 18.04, Luca Palazzo wrote: > > > It worked. We have no more sigsegv on *-login process. > > > > Thanks > > > > Luca > > > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > >> Hi, > >> > >> On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> > >>> Hi Timo, hi all, > >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > >>> > >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> 710 { > >>> (gdb) bt > >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > >> > >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > >> > > > From luca.palazzo at unict.it Thu Mar 22 16:16:34 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Thu, 22 Mar 2012 15:16:34 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <1332425134.26095.88.camel@innu> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> <1332425134.26095.88.camel@innu> Message-ID: <4F6B3442.8020000@unict.it> I've applied both and reverted previous one. Everything seems to run flowless. Thanks Luca On 03/22/2012 03:05 PM, Timo Sirainen wrote: > Hi, > > These should fix it properly: > > http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 > http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d > > On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: >> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e >> >> I'll figure out a proper fix soon. >> >> On 21.3.2012, at 18.04, Luca Palazzo wrote: >> >>> It worked. We have no more sigsegv on *-login process. >>> >>> Thanks >>> >>> Luca >>> >>> Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >>>> Hi, >>>> >>>> On 20.3.2012, at 11.09, Luca Palazzo wrote: >>>> >>>>> Hi Timo, hi all, >>>>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>>>> >>>>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> 710 { >>>>> (gdb) bt >>>>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>>>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>>>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >>>> >>>> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >>>> >>> >> > > From micah at riseup.net Thu Mar 22 16:38:55 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 22 Mar 2012 10:38:55 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply Message-ID: <87bonon1j4.fsf@algae.riseup.net> I've been moving users from one system to another by doing a dsync mirror operation. The first dsync mirror takes some time, because of the number of users involved, so I am doing an initial sync, and then I direct the users to the new location and do a final 'freshening' sync to get any changes that happened during the longer sync. The problem seems to be with this freshening sync, it seems quite fragile breaking with things like: dsync-local(user at example.com): Error: msg-get failed: box=Spam uid=3034 guid=1ad456015ae9694f083b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69096 guid=c22b541a71e4694fc93700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69097 guid=4b6d6b13d0e9694f505700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69098 guid=175b1c2e4aea694fc97100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69099 guid=bfb08c1b3bee694f133e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69100 guid=fa5d630c17ef694fa75f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69101 guid=7ca96011dcef694f3f0400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69102 guid=ef547107eff1694ff96700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69103 guid=5597bc0519f2694f2e7000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69104 guid=8336a53a54f5694fb21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69105 guid=96169d13c8fd694f831800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69106 guid=af21a5183f036a4f263200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69107 guid=d0fde3348e036a4ff44000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69108 guid=4ce01d1a59056a4fee2200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69109 guid=497f96066e056a4f322700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69110 guid=ef34f505c0066a4fc26b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69111 guid=81adcb2c6e076a4f751100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69112 guid=a110841e8a076a4fa21500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69113 guid=60d8e70a970d6a4fae2100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69114 guid=7c6cb41572106a4ff13c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69115 guid=aaf4d32b2f126a4ff21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69116 guid=ab52f43a58126a4ffd1800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69117 guid=eb543a2179186a4fe45800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69118 guid=cd7cb408a12a6a4f272100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69119 guid=2ec02e2ef2326a4f9e1100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69120 guid=e7a4552ff8336a4f7f6700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69121 guid=0724b023d33a6a4f1b3300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69122 guid=9985c91afe3b6a4f127100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69123 guid=9300751b913d6a4f7a4000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69124 guid=822ff806ae3f6a4f293b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69125 guid=eac8ed1f2b426a4f164200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69126 guid=4109561ae3426a4ff26700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69127 guid=30bc832e5e496a4f563600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69128 guid=c0b36410bd4b6a4f102b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69129 guid=38a9d41a534d6a4ff40200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69130 guid=b8e84d239b4d6a4fd11000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69131 guid=f060ef22154f6a4f2b5c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69132 guid=e2999c107c4f6a4f5a7600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69133 guid=8d09280aae506a4f073500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69134 guid=43d7ec3aa6556a4f963a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69135 guid=c5800130d2556a4f594200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69136 guid=83a91e08b4566a4f197100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69137 guid=50cf9721f95e6a4f7e4400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69138 guid=fda2a82886606a4f881700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69139 guid=97ee1d1ad1636a4fc94d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69140 guid=4f50671f85666a4f306100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69141 guid=0fea590fb4666a4f7a6b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69142 guid=f3210b02a5676a4ffa1f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69143 guid=8d325a06686a6a4f2b3600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69144 guid=0cbf1839f1756a4f8f6800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69145 guid=d6209a2898796a4f671a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69146 guid=576ade31da7d6a4f5f5700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69147 guid=70a15b34247e6a4f445a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69148 guid=3ff92631cd886a4ffd6300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69149 guid=3ff2081568916a4f134d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69150 guid=d1a67b0907ab6a4f546000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69151 guid=3d4cb1197ee96a4fbf5f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69152 guid=aae2542818266b4f7d1e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2077 guid=c5a0930248e8694fb77c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2078 guid=dce63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2079 guid=dde63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2080 guid=dee63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2081 guid=dfe63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2082 guid=a110b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2083 guid=a210b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2084 guid=93afeb1f7d0b6a4f694e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2085 guid=ef2bb1098a2e6a4fcf1b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2086 guid=d8ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2087 guid=d9ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2088 guid=7e5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2089 guid=7f5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2090 guid=b08c5205ba646a4f106700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2091 guid=805b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2092 guid=815b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. those final "Unexpected finish reply" and "Unexpected reply from server" are a bit surprising results. Running dsync again seems to resolve things, but there seems to be a bug here that is causing unexpected results to leak through to the dsync process? thanks, micah From jtl+dovecot at uvm.edu Thu Mar 22 18:17:10 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 22 Mar 2012 12:17:10 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6AA667.1080908@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> Message-ID: <4F6B5086.4030001@uvm.edu> On 03/22/2012 12:11 AM, Stan Hoeppner wrote: > On 3/21/2012 12:04 PM, Timo Sirainen wrote: >> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >> >> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. > If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could > use a true cluster file system such as OCFS or GFS. Both will eliminate > this problem, and without requiring Dovecot director. And you'll get > better performance than with Gluster, which, BTW, isn't really suitable > as a transactional filesystem, was not designed for such a use case. Speaking as an admin who has run Dovecot on top of GFS both with and without the director, I would never go back to a cluster without the director. The cluster performs *so* much better when glocks can be cached on a single node, and this can't happen if a single user has IMAP processes on separate nodes. No, you don't strictly need the director if you have GFS, but if you can manage it, you'll be a lot happier. Jim From ms at mur.at Thu Mar 22 19:09:13 2012 From: ms at mur.at (Martin Schitter) Date: Thu, 22 Mar 2012 18:09:13 +0100 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A927C.6010003@gedalya.net> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> Message-ID: <4F6B5CB9.9080204@mur.at> Am 2012-03-22 03:46, schrieb Gedalya: >> >> doveadm sync/backup via impac puts the same message all over the place... > > Thanks Martin, I've set up a test platform to investigate this further > but I've been short on time... after some debugging a few more remarks about this problem: the bug only appears on recursive folder hierarchies. if you specity option "-m INBOX" everything works fine. for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( has anyone a clue how to fix this problem in the source code? From tss at iki.fi Thu Mar 22 20:57:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 20:57:21 +0200 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <4F6AF72E.9030206@um.es> References: <4F6AF72E.9030206@um.es> Message-ID: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> On 22.3.2012, at 11.55, Angel L. Mateo wrote: > The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. service auth { idle_kill = 0 } From ednitido at gmail.com Thu Mar 22 23:18:12 2012 From: ednitido at gmail.com (Ed Nitido) Date: Thu, 22 Mar 2012 17:18:12 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy Message-ID: Hey all, I've upgraded from a working Dovecot 2.0.17 Proxy with a master user setup to Dovecot 2.1.3 and I've merged my conf settings from 2.0.17 into 2.1.3. I'm able to start up dovecot proxy and telnet localhost, however it creates the users home director on the proxy server instead of going to the backend dovecot server (which has already been successfully upgraded to 2.1.3 from 2.0.17). In my old 2.0.17, I had the entire namespace section commented out in 10-mail.conf. To achieve the same in 2.1.3 I had to remove the file 15-mailboxes.conf, otherwise my error log would have the following if I just commented out the namespace section and left that file in place: *Error: user edward at dev.domain.com: Initialization failed: namespace configuration error: inbox=yes namespace missing* *Error: Invalid user settings. Refer to server log for more information.* So now I'm trying to figure out why the proxy doesn't get the mail server IP from director, when doveadm has the backend server listed *# doveadm director status* *mail server ip vhosts users* *192.168.12.205 100 0* Can anyone see something I've missed? Here's my doveconf output: # 2.1.3: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-openvz-amd64 i686 Debian 6.0.4 auth_debug = yes auth_debug_passwords = yes auth_master_user_separator = * auth_mechanisms = plain login auth_socket_path = /usr/local/var/run/dovecot/auth-master auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log director_doveadm_port = 542 director_mail_servers = 192.168.12.205 director_servers = 192.168.12.209 disable_plaintext_auth = no info_log_path = /var/log/dovecot-info.log listen = * lmtp_proxy = yes log_path = /var/log/dovecot-err.log mail_debug = yes mail_gid = vmail mail_location = maildir:%h/Maildir mail_plugins = " quota" mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/home/%d/shared-mailboxes quota = maildir:User quota quota_rule = *:storage=1G sieve = %h/.dovecot.sieve sieve_dir = ~/sieve sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M sieve_quota_max_scripts = 0 sieve_quota_max_storage = 0 } postmaster_address = postmaster at dev.domain.com protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0666 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 541 } inet_listener { port = 542 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { user = postfix } } service managesieve-login { executable = managesieve-login director inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service pop3-login { executable = pop3-login director inet_listener pop3 { port = 110 } } ssl_cert = Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root at n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M /home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root at n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real 0m2.677s user 0m3.184s sys 0m1.513s [root at n24 bu]# time dsync backup -u testuser at domain.com \ mdbox:/home/bu/testuser real 1m9.519s user 1m7.592s sys 0m1.126s [root at n24 bu]# time dsync backup -u testuser at domain.com \ sdbox:/home/bu/testuser2 real 1m2.164s user 1m0.882s sys 0m0.993s [root at n24 bu]# From list at airstreamcomm.net Fri Mar 23 04:36:44 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 22 Mar 2012 21:36:44 -0500 Subject: [Dovecot] Dovecot and scalable database storage Message-ID: I saw some interesting mails from TImo back in 2009 talking about the idea of using something like Cassandra db or similar as a storage platform for both email and index/logs. I was wondering if this has been discussed since then, and if there are any plans to support something like this in the future? I have been playing with Cassandra and found that their RackAwareStrategy gives you the ability to replicate writes to as many nodes as you would like, but more importantly what nodes and one of those nodes could be defined by what rack it lives in or what data center it lives in. This means multiple sites high available storage clusters, seemingly a system that dovecot could benefit from in terms of performance and redundancy and simplicity. Any takers? From stan at hardwarefreak.com Fri Mar 23 09:13:18 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 02:13:18 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6B5086.4030001@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> Message-ID: <4F6C228E.5060902@hardwarefreak.com> On 3/22/2012 11:17 AM, Jim Lawson wrote: > On 03/22/2012 12:11 AM, Stan Hoeppner wrote: >> On 3/21/2012 12:04 PM, Timo Sirainen wrote: >>> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >>> >>> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. >> If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could >> use a true cluster file system such as OCFS or GFS. Both will eliminate >> this problem, and without requiring Dovecot director. And you'll get >> better performance than with Gluster, which, BTW, isn't really suitable >> as a transactional filesystem, was not designed for such a use case. > > Speaking as an admin who has run Dovecot on top of GFS both with and > without the director, I would never go back to a cluster without the > director. The cluster performs *so* much better when glocks can be > cached on a single node, and this can't happen if a single user has IMAP > processes on separate nodes. > > No, you don't strictly need the director if you have GFS, but if you can > manage it, you'll be a lot happier. Did/do you see the Director/glock benefit with both maildir and mdbox Jim? Do you see any noteworthy performance differences between the two formats on GFS, with and without Director? BTW, are you hitting FC or iSCSI LUNs? -- Stan From tlx at leuxner.net Fri Mar 23 11:40:52 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Fri, 23 Mar 2012 10:40:52 +0100 Subject: [Dovecot] Dovecot v2.1.3 (f30437ed63dc) Auth/Login Issues Message-ID: <20120323094052.GA9851@nihlus.leuxner.net> Hi, some change between ff5c341f8838 and f30437ed63dc seems to have broken auth: => Bad Login Mar 23 09:01:46 spectre dovecot: master: Dovecot v2.1.3 (f30437ed63dc) starting up [...] Mar 23 10:25:44 spectre dovecot: auth: Debug: auth client connected (pid=7266) Mar 23 10:25:45 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=62388#011resp= Mar 23 10:25:45 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): hit: #011userdb_quota_rule=*:storage=5G#011userdb_acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:25:45 spectre dovecot: auth: Debug: master in: REQUEST#0113958898689#0117266#0111#011bfc44f32051961b909e2b458440d645f Mar 23 10:25:45 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): hit: tlx at leuxner.net#011uid=5000#011gid=5000#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: master out: USER#0113958898689#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=7267, TLS Mar 23 10:25:45 spectre dovecot: imap(tlx at leuxner.net): Connection closed in=0 out=319uthentication/login: => Good Login Mar 23 10:26:37 spectre dovecot: master: Dovecot v2.1.3 (ff5c341f8838) starting up [...] Mar 23 10:27:18 spectre dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 10:27:18 spectre dovecot: auth: Debug: auth client connected (pid=9832) Mar 23 10:27:19 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=51647#011resp= Mar 23 10:27:19 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file /var/vmail/auth.d/leuxner.net/passwd: Read 1 users in 0 secs Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:27:19 spectre dovecot: auth: Debug: master in: REQUEST#0113656384513#0119832#0111#0114782efcbd0324b228bb85aaae916cfe6 Mar 23 10:27:19 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: master out: USER#0113656384513#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:27:19 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=9835, TLS Regards Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From tss at iki.fi Fri Mar 23 11:51:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 11:51:43 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332405027.6792.7.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> Message-ID: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> On 22.3.2012, at 10.30, Noel Butler wrote: > On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > >> :2012-03-22T11:55:Noel Butler: >> >>> perhaps it should be renamed then, given it violates the known normal >>> for SYSCONF dir, you've just created another form of --datadir >> >> Not really. The way I see it works as expected. The sysconf dir is the > > > Then you and I and a few other devs involved in other very well known > bits of software that everyone likely uses, will have to agree to > disagree A ton of software installs into /etc// directory. Most Linux distributions installed Dovecot v1.x that way as well. And of course everyone expects configuration to be under /etc. The default of sysconfdir is PREFIX/etc/. Dovecot v2.0 really shouldn't install its stuff into PREFIX/etc/ but into PREFIX/etc/dovecot/. So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? From alain.defrance at univ-evry.fr Fri Mar 23 12:20:01 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:20:01 +0100 Subject: [Dovecot] quota ldap Message-ID: <4F6C4E51.7010603@univ-evry.fr> hello all, i'm using quota + ldap with dovecot 2 in dovecot-ldap.conf.ext file i have the line : user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid,quota=quota_rule=*:storage=%$B how can i add "Trash:storage= " to have more place for deleting messages like in 90-quota.conf file ? quota_rule2 = Trash:storage thanks for help regards -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From mlists at edicom.eu Fri Mar 23 12:38:18 2012 From: mlists at edicom.eu (Miguel Tormo) Date: Fri, 23 Mar 2012 11:38:18 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <201203231138.18338.mlists@edicom.eu> El Mi?rcoles, 21 de Marzo de 2012 15:43:14 Luca Lesinigo escribi?: > Hello list. Hello, > > I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. > I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. > > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. > It would be desirable for multiple reasons: I have recently deployed a very similar setup: imap proxy, mailbox sharding... Although not exactly like yours. Comments below: > - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes This is reasonable. For example, I did this to seamless migrate lots of users from one server to another, migrating just a few of them at a time. > - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) You can do the exact same thing in the future, of course. > - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) This is fairly easy to accomplish with imap proxying. > - be ready to serve traffic over IPv6 without touching our precious mailbox servers This is doable. > - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > > Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). I didn't implement a lemonade profile nor quotas in my setup. However, I can confirm you that IMAP IDLE does work with imap proxy. > > In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). No problem with that, but this is related to the MTA configuration, not dovecot. > > I'd like to know if that plan sounds reasonable or if there's something stupid in it. > Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. From dovecot-l at fu-berlin.de Fri Mar 23 12:44:32 2012 From: dovecot-l at fu-berlin.de (Heiko Schlichting) Date: Fri, 23 Mar 2012 11:44:32 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <20120323104432.GB1353054@CIS.FU-Berlin.DE> Timo wrote: > So the only way I can think of how to change this is to add another > option to optionally remove the dovecot/ suffix from the directory, but > is this really worth the trouble? I would appreciate such option too. For large dedicated installations other schemes than /etc/dovecot are common. See http://dovecot.org/list/dovecot/2009-January/036131.html Heiko Heiko Schlichting Freie Universit?t Berlin heiko.schlichting at fu-berlin.de Zentraleinrichtung f?r Datenverarbeitung Telefon +49 30 838-54327 Fabeckstra?e 32 Telefax +49 30 838454327 14195 Berlin From nmilas at noa.gr Fri Mar 23 12:48:18 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 23 Mar 2012 12:48:18 +0200 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C4E51.7010603@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> Message-ID: <4F6C54F2.7020203@noa.gr> On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > how can i add "Trash:storage= " to have more place for deleting > messages like in > See, for example, my setup: http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html I use a single conf file (because it's small and it's more intuitive to me). Details: http://wiki2.dovecot.org/Quota/Configuration Regards, Nick From tss at iki.fi Fri Mar 23 12:50:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:50:04 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> On 21.3.2012, at 16.43, Luca Lesinigo wrote: > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. From tss at iki.fi Fri Mar 23 12:53:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:53:16 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <20120323104432.GB1353054@CIS.FU-Berlin.DE> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> Message-ID: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> On 23.3.2012, at 12.44, Heiko Schlichting wrote: > Timo wrote: >> So the only way I can think of how to change this is to add another >> option to optionally remove the dovecot/ suffix from the directory, but >> is this really worth the trouble? > > I would appreciate such option too. For large dedicated installations other > schemes than /etc/dovecot are common. > > See http://dovecot.org/list/dovecot/2009-January/036131.html Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). From alain.defrance at univ-evry.fr Fri Mar 23 12:58:09 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:58:09 +0100 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C54F2.7020203@noa.gr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> Message-ID: <4F6C5741.3000408@univ-evry.fr> thanks Nick so if i understand correctly i can mix the 2 quota_rule ? the one who came from ldap user_attrs (quota_rule=*:bytes=%$) and the other which from quota_rule2 = Trash:storage=+3%% in your case you add 3% quota more for Trash ? Am i write ? regards > On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > >> how can i add "Trash:storage= " to have more place for deleting >> messages like in >> > > See, for example, my setup: > http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html > > I use a single conf file (because it's small and it's more intuitive > to me). > > Details: http://wiki2.dovecot.org/Quota/Configuration > > Regards, > Nick > -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From jtam.home at gmail.com Fri Mar 23 12:58:36 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Fri, 23 Mar 2012 03:58:36 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 Message-ID: I ran into two issues trying to upgrade our dovecot installation (Solaris 10). 1) Does not compile with OpenSSL 0.9.7 Not a big deal, as I was able to successfully against OpenSSL 0.9.8, but does dovecot require OpenSSL >= 0.9.8 now? libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-test -std=gnu99 -O3 -fomit-frame-pointer -mcpu=ultrasparc -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -MT istream-openssl.lo -MD -MP -MF .deps/istream-openssl.Tpo -c istream-openssl.c -fPIC -DPIC -o .libs/istream-openssl.o iostream-openssl-context.c:9:28: openssl/engine.h: No such file or directory iostream-openssl-context.c: In function `ssl_iostream_deinit_global': iostream-openssl-context.c:431: warning: implicit declaration of function `ENGINE_finish' iostream-openssl-context.c:432: warning: implicit declaration of function `ENGINE_cleanup' ... 2) Dovecot's LDA does not work After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe exact same config file, local mail delivery tempfails: Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address Mar 23 02:51:51 server dovecot: auth: Error: userdb connection: Failed to get peer's credentials Mar 23 02:51:51 server dovecot: lda: Error: userdb lookup(j.tam): Disconnected unexpectedly Mar 23 02:51:51 server dovecot: lda: Fatal: Internal error occurred. Refer to server log for more information. # Sendmail reports stat=Deferred: local mailer (/var/dovecot/libexec/dovecot-lda) exited with EX_TEMPFAIL After seeing 2) in the logs, I had to revert back to 2.0.16. Any hints on what could be wrong? Joseph Tam # 2.0.16: /var/dovecot/etc/dovecot/dovecot.conf # OS: SunOS 5.10 sun4u nfs auth_cache_negative_ttl = 10 mins auth_cache_size = 64 k auth_cache_ttl = 1 days auth_failure_delay = 5 secs auth_master_user_separator = * auth_socket_path = /var/dovecot/run/auth-userdb auth_username_chars = abcdefghijklmnopqrstuvwxyz01234567890.-_ auth_worker_max_count = 1 base_dir = /var/dovecot/run default_vsz_limit = 64 M deliver_log_format = first_valid_gid = 10000 first_valid_uid = 10000 hostname = our.mail.domain last_valid_gid = 19999 last_valid_uid = 19999 lda_mailbox_autocreate = yes log_timestamp = login_greeting = Ready. mail_location = mbox:/nfs/home/%n/mail:INBOX=/nfs/mail/%n:INDEX=/data/dc-cache/%n mail_nfs_storage = yes mail_temp_dir = /var/tmp mbox_very_dirty_syncs = yes mbox_write_locks = dotlock_try fcntl namespace { inbox = yes location = prefix = separator = / } namespace { hidden = yes list = no location = prefix = / separator = / } namespace { hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace { hidden = yes list = no location = prefix = mail/ separator = / } passdb { args = /var/dovecot/etc/master-users driver = passwd-file master = yes pass = yes } passdb { args = /var/yp/etc/passwd driver = passwd-file } postmaster_address = MAILER-DAEMON at our.mail.domain protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth-worker { user = dovecot } service auth { idle_kill = 1 hours } service imap-login { process_limit = 2 service_count = 0 } service imap { process_limit = 512 } service pop3-login { process_limit = 1 service_count = 0 } service pop3 { process_limit = 64 } shutdown_clients = no ssl_cert = References: Message-ID: On Fri, 23 Mar 2012, dovecot-request at dovecot.org wrote: >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the > dovecot/ suffix from other directories as well. That might be > something worth doing (--without-package-suffix or something?). +1. I fake it now with symlinks (e.g. etc/dovecot -> .). Joseph Tam From rainer.frey at inxmail.de Fri Mar 23 13:19:45 2012 From: rainer.frey at inxmail.de (Rainer Frey) Date: Fri, 23 Mar 2012 12:19:45 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <69027C8C-A95B-41C2-B06B-824345F738DA@inxmail.de> On Mar 23, 2012, at 10:51 AM, Timo Sirainen wrote: >>> :2012-03-22T11:55:Noel Butler: >>> >>>> perhaps it should be renamed then, given it violates the known normal >>>> for SYSCONF dir, you've just created another form of --datadir >>> >>> Not really. The way I see it works as expected. >> >> The directory for installing read-only data files that pertain >> to a single machine?that is to say, files for configuring a >> host. Mailer and network configuration files, ?/etc/passwd?, and >> so forth belong here. All the files in this directory should be >> ordinary ASCII text files. This directory should normally be >> ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are >> using Autoconf, write it as ?@sysconfdir@?.) Well, I don't see that that prevents organizing the files in sysconfdir into a subdirectory. > ton of software installs into /etc// directory. [...] > So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? I really don't think so. What for? Nobody has shown a real-world problem with that subdirectory. From tss at iki.fi Fri Mar 23 13:26:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 13:26:54 +0200 Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: References: Message-ID: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> On 23.3.2012, at 12.58, Joseph Tam wrote: > I ran into two issues trying to upgrade our dovecot installation (Solaris 10). > > 1) Does not compile with OpenSSL 0.9.7 > > Not a big deal, as I was able to successfully against OpenSSL 0.9.8, > but does dovecot require OpenSSL >= 0.9.8 now? Hm. Maybe it's time by now? :) It could be fixed with some more #ifdefs but those make code more unreadable. > 2) Dovecot's LDA does not work > > After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe > exact same config file, local mail delivery tempfails: > > Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? From hsn at filez.com Fri Mar 23 13:41:24 2012 From: hsn at filez.com (Radim Kolar) Date: Fri, 23 Mar 2012 12:41:24 +0100 Subject: [Dovecot] delivering with maildrop Message-ID: <4F6C6164.2050506@filez.com> Can somebody provide maildrop syntax for using deliver-lda as final delivery program during sorting mail in user mailfilter? i mean replacement for "to" statement if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) { to $MAIL/.dovecot/ } From jtl+dovecot at uvm.edu Fri Mar 23 14:13:21 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 08:13:21 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C228E.5060902@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> Message-ID: <4F6C68E1.4030400@uvm.edu> On 3/23/12 3:13 AM, Stan Hoeppner wrote: >> Speaking as an admin who has run Dovecot on top of GFS both with and >> without the director, I would never go back to a cluster without the >> director. The cluster performs *so* much better when glocks can be >> cached on a single node, and this can't happen if a single user has IMAP >> processes on separate nodes. >> >> No, you don't strictly need the director if you have GFS, but if you can >> manage it, you'll be a lot happier. > Did/do you see the Director/glock benefit with both maildir and mdbox > Jim? Do you see any noteworthy performance differences between the two > formats on GFS, with and without Director? BTW, are you hitting FC or > iSCSI LUNs? > Actually, we're all mbox. This primarily has to do with how users do self-service mail recovery from backup: one folder = one file. I'd like to move to mdbox, but it would mean the recovery scripts will need to understand which files are associated with which folders, as well as restoring the associated index files. That's a to-do. We're using fibrechannel (IBM v7000) storage, but I would expect to see the same thing with iSCSI. It's mostly about different nodes contending over locks on the same files (although I'm sure cache locality helps a great deal, too.) If you end up with imap processes for the same folder on different nodes, or mail delivery happening on one node and imap on the other, you will feel the lag in your IMAP client. "Oh, my INBOX has been unresponsive for 10 seconds, I must be getting a lot of mail right now!" That's an exaggeration, but not by much. Jim From amateo at um.es Fri Mar 23 14:15:40 2012 From: amateo at um.es (Angel L. Mateo) Date: Fri, 23 Mar 2012 13:15:40 +0100 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> References: <4F6AF72E.9030206@um.es> <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> Message-ID: <4F6C696C.5030900@um.es> El 22/03/12 19:57, Timo Sirainen escribi?: > On 22.3.2012, at 11.55, Angel L. Mateo wrote: > >> The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. > > service auth { > idle_kill = 0 > } > In a test server I have, this have solved the problem. In my productions servers it is still being restarted. Could it be another parameter involve in this? service_count is set to 0. I have also seen that, whenever dovecot/auth is restarted, dovecot/config has also been restarted. Could be related? My config related with this service auth is: service auth { chroot = client_limit = 4096 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } -- Angel L. Mateo Mart?nez Secci?n de Telem?tica ?rea de Tecnolog?as de la Informaci?n _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 From list at airstreamcomm.net Fri Mar 23 15:39:07 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 08:39:07 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> On Wed, 21 Mar 2012 09:56:12 -0600, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=4220 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=5088 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: > fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: > fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index > > > This is my dovecot config currently: > > jdevine at test-gluster-client2:~> dovecot -n > # 2.0.13: /etc/dovecot/dovecot.conf > # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 > lock_method = dotlock > mail_fsync = always > mail_location = mdbox:~/mdbox > mail_nfs_index = yes > mail_nfs_storage = yes > mmap_disable = yes > passdb { > driver = pam > } > protocols = " imap" > ssl_cert = ssl_key = userdb { > driver = passwd > } I was able to get dovecot working across a gluster cluster a few weeks ago and it worked just fine. I would recommend using the native gluster mount option (need to install gluster software on clients), and using distributed replicated as your replication mechanism. If you're running two gluster servers you should have a replica count of two with distributed replicated. You should test first to make sure you can create a file in both mounts and see it from every mount point in the cluster, as well as interact with it. It's also very important to make sure your servers are running with synchronized clocks from an NTP server. Very bad things happen to a (dovecot or gluster) cluster out of sync with NTP. From eliezer at ngtech.co.il Fri Mar 23 15:57:30 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Fri, 23 Mar 2012 15:57:30 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <4F6C814A.2050803@ngtech.co.il> On 23/03/2012 12:53, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > >> Timo wrote: >>> So the only way I can think of how to change this is to add another >>> option to optionally remove the dovecot/ suffix from the directory, but >>> is this really worth the trouble? >> >> I would appreciate such option too. For large dedicated installations other >> schemes than /etc/dovecot are common. >> >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > well squid is using another way such as the directory you specify and without the /dovecot (squid) suffix. it's not that important. if you do change the config directory you know where you are putting it. i,m using the /opt/(service name) to install most of my self complied software so idont really care about it. but if the sysconfig directory as a directive it should be the default. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il From stan at hardwarefreak.com Fri Mar 23 16:02:41 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 09:02:41 -0500 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C6164.2050506@filez.com> References: <4F6C6164.2050506@filez.com> Message-ID: <4F6C8281.10906@hardwarefreak.com> On 3/23/2012 6:41 AM, Radim Kolar wrote: > Can somebody provide maildrop syntax for using deliver-lda as final > delivery program during sorting mail in user mailfilter? > > i mean replacement for "to" statement > > if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) > { > to $MAIL/.dovecot/ > } Dovecot's local delivery agent uses the Sieve language: http://wiki.dovecot.org/LDA/Sieve The syntax is quite different than maildrop or procmail. -- Stan From tss at iki.fi Fri Mar 23 16:06:25 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 16:06:25 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On 23.3.2012, at 15.39, wrote: > I was able to get dovecot working across a gluster cluster a few weeks ago > and it worked just fine. I would recommend using the native gluster mount > option (need to install gluster software on clients), and using distributed > replicated as your replication mechanism. Have you tried stress testing it with imaptest? Run in parallel for both servers: imaptest host=gluster1 user=testuser pass=testpass imaptest host=gluster2 user=testuser pass=testpass http://imapwiki.org/ImapTest And see if Dovecot logs any errors. From micah at riseup.net Fri Mar 23 17:52:02 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 11:52:02 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply References: <87bonon1j4.fsf@algae.riseup.net> Message-ID: <87ty1fl3h9.fsf@algae.riseup.net> Micah Anderson writes: > dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; > dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 > dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. I'm also getting similar strange results with my regular dsync backup: dsync-local(user at example.com): Error: Unexpected reply from server: 0 23bdce147b43674f8e2700002c449efa 1242 146 \Recent 1332335848 this is with 2.1.3. micah From micah at riseup.net Fri Mar 23 18:25:27 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 12:25:27 -0400 Subject: [Dovecot] doveadm user -f index Message-ID: <87pqc3l1xk.fsf@algae.riseup.net> I've configured my mail_location to have a different location for performance reasons so they aren't in the same location as the mail_location. The 'doveadm user -f home' is useful to find where a user's home directory is for various scripting purposes, but I can't seem to find a way to determine the location of the user's indexes. I can do something with the output of dovecot -a to find the mail_location and then look for a configured INDEX, but then I don't have a good way of translating the %d/%1n/%n type string formatters into their values for a user. thanks for any suggestions! micah -- From stan at hardwarefreak.com Fri Mar 23 19:11:49 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 12:11:49 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C68E1.4030400@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> Message-ID: <4F6CAED5.4000206@hardwarefreak.com> On 3/23/2012 7:13 AM, Jim Lawson wrote: > On 3/23/12 3:13 AM, Stan Hoeppner wrote: > >>> Speaking as an admin who has run Dovecot on top of GFS both with and >>> without the director, I would never go back to a cluster without the >>> director. The cluster performs *so* much better when glocks can be >>> cached on a single node, and this can't happen if a single user has IMAP >>> processes on separate nodes. >>> >>> No, you don't strictly need the director if you have GFS, but if you can >>> manage it, you'll be a lot happier. >> Did/do you see the Director/glock benefit with both maildir and mdbox >> Jim? Do you see any noteworthy performance differences between the two >> formats on GFS, with and without Director? BTW, are you hitting FC or >> iSCSI LUNs? >> > > Actually, we're all mbox. This primarily has to do with how users do > self-service mail recovery from backup: one folder = one file. Yeah, mbox isn't as dead as some people contend, but it just doesn't have legs for newer deployment architectures. > I'd like to move to mdbox, but it would mean the recovery scripts will > need to understand which files are associated with which folders, as > well as restoring the associated index files. That's a to-do. That's an easy weekend project. ;) > We're using fibrechannel (IBM v7000) storage, but I would expect to see > the same thing with iSCSI. It's mostly about different nodes contending > over locks on the same files (although I'm sure cache locality helps a > great deal, too.) If you end up with imap processes for the same folder > on different nodes, or mail delivery happening on one node and imap on > the other, you will feel the lag in your IMAP client. "Oh, my INBOX has > been unresponsive for 10 seconds, I must be getting a lot of mail right > now!" That's an exaggeration, but not by much. I was asking about your SAN storage unrelated to the locking issue. Just a curiosity thing. Note my email domain. ;) I'm an FC fan but iSCSI seems to be more popular in many circles, actually pretty much market wide these days. So when I come across another SAN user I'm naturally curious as to what hardware they use. Just so nobody gets the wrong idea, I wasn't advocating against Director earlier in the thread. I think it's fantastic and solves some critical scalability problems. As in your case, it allows one to use his mail storage format of choice with a cluster filesystem while mostly avoiding the locking headaches. In the past one pretty much had to use maildir with a cluster FS to avoid the locking performance killed. But one had to suffer the higher IOPS load on the storage. Not always a good tradeoff, especially for busy mail systems. I assume you do still have some minor locking/performance issues with the INBOX, even with Director, when LDA and the user MUA are both hitting the INBOX index and mbox files. You'll still see this with mdbox, but probably to a lesser degree if you use a smallish mdbox_rotate_size value. To mitigate this INBOX locking you could go with a dual namespaces, using maildir or sdbox for the INBOX and mdbox for the other user mail folders. -- Stan From tss at iki.fi Fri Mar 23 19:19:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 19:19:26 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <3845B569-3CE5-4C0C-BB60-B9CA91FF8B56@iki.fi> On 23.3.2012, at 19.11, Stan Hoeppner wrote: > I assume you do still have some minor locking/performance issues with > the INBOX, even with Director, when LDA and the user MUA are both > hitting the INBOX index and mbox files. You'll still see this with > mdbox, but probably to a lesser degree if you use a smallish > mdbox_rotate_size value. To mitigate this INBOX locking you could go > with a dual namespaces, using maildir or sdbox for the INBOX and mdbox > for the other user mail folders. The biggest difference is that mbox requires read locks, mdbox doesn't. mdbox lock waits are very similar to maildir's. Of course, I don't know about the cluster filesystems' internal locking, but I thought it was even worse with Maildir than with mbox because it had to get a read lock for each read file, but I guess this depends on the filesystem. From ruskie at codemages.net Fri Mar 23 19:22:11 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Fri, 23 Mar 2012 18:22:11 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: :2012-03-23T12:53:Timo Sirainen: > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). I would suggest to have a --layout=gnu|opt That would either do what it currently does(gnu) and opt to install everything into a single dir i.e.: /opt/dovecot/ With subdirs under there. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Be advised: causing a disturbance may result in fines, detainment, bodily harm, or death. Enjoy your stay. From jtl+dovecot at uvm.edu Fri Mar 23 19:33:42 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 13:33:42 -0400 Subject: [Dovecot] recovery of mdbox folders (was: Re: distributed mdbox) In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <4F6CB3F6.5010006@uvm.edu> On 3/23/12 1:11 PM, Stan Hoeppner wrote: > On 3/23/2012 7:13 AM, Jim Lawson wrote: > > >> I'd like to move to mdbox, but it would mean the recovery scripts will >> need to understand which files are associated with which folders, as >> well as restoring the associated index files. That's a to-do. > That's an easy weekend project. ;) > Out of curiosity, does anyone do self-service restoration of individual mdbox folders? If I'm going to write a script to do it, it'd be nice to avoid any pitfalls someone else has already run into. :-) We're already backing up from snapshots, so the synchronization issues are solved (at least at backup time...) Jim From list at airstreamcomm.net Fri Mar 23 19:43:34 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 12:43:34 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On Fri, 23 Mar 2012 16:06:25 +0200, Timo Sirainen wrote: > On 23.3.2012, at 15.39, > wrote: > >> I was able to get dovecot working across a gluster cluster a few weeks >> ago >> and it worked just fine. I would recommend using the native gluster >> mount >> option (need to install gluster software on clients), and using >> distributed >> replicated as your replication mechanism. > > Have you tried stress testing it with imaptest? Run in parallel for both > servers: > > imaptest host=gluster1 user=testuser pass=testpass > imaptest host=gluster2 user=testuser pass=testpass > > http://imapwiki.org/ImapTest > > And see if Dovecot logs any errors. I did stress test it, but we have developed a "mail bot net" tool for the purpose. I should mention this was tested using dovecot 1.2, as this is our current production version (hopefully will be upgrading soon). Its comprised of a control server that starts a bot network of client machines that creates pop/imap connections (smtp as well) on our test cluster of dovecot (and postfix) servers. In my test I distributed the load across a two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN storage attached to it. I actually didn't change my configuration from when I had a test NFS server connected to the test servers (mmap disabled, fcntl locking, etc), because glusterfs was an afterthought when we were stress testing our new netapp system using NFS. We have everything in VMware, including the glusterfs servers. Using five bot servers and connecting 7 times a second from each server (35 connections per second) for both pop and imap (70 total connections per second) split between two dovecot servers I was not seeing any big issues. The load average was low, and there were no errors to speak of in dovecot (or postfix). I was mounting the storage with the glusterfs native client, not using NFS (which I have not tested). I would like to do a more thorough test of glusterfs using Dovecot 2.0 on some dedicated hardware and see how much further I can push the system. From busseniu at in.tum.de Fri Mar 23 20:02:10 2012 From: busseniu at in.tum.de (=?UTF-8?B?Q2hyaXN0b3BoIEJ1w59lbml1cw==?=) Date: Fri, 23 Mar 2012 19:02:10 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: <4F6CBAA2.5020409@in.tum.de> Hi, maybe try "dsync -o mail_fsync=never". Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From luca at lm-net.it Fri Mar 23 20:12:56 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Fri, 23 Mar 2012 19:12:56 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> Message-ID: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: > Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... >> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) > Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... > I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. > However, I can confirm you that IMAP IDLE does work with imap proxy. That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. > You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. > You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). -- Luca Lesinigo From gedalya at gedalya.net Fri Mar 23 20:24:11 2012 From: gedalya at gedalya.net (Gedalya) Date: Fri, 23 Mar 2012 14:24:11 -0400 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Message-ID: <4F6CBFCB.60209@gedalya.net> On 03/23/2012 02:12 PM, Luca Lesinigo wrote: > Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. > I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. >>> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) >> Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. > Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) > > Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >>> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... >> I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. The same proxy_maybe (dummy proxy) setup works great for POP3 too. Very simple to set up, works like a charm. Nothing much to think about. > >> However, I can confirm you that IMAP IDLE does work with imap proxy. > That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. It will work as well as it was working with your existing courier server. But it will work great for accounts migrated to native dovecot. >> You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. > Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. It's a very good idea to have completely separate machines for outgoing mail. Once you have imap-only boxes, you can eliminate the need for an MTA by using the dovecot LMTP server. Your postfix transport map can send mail to either smtp:imap.yourdomain.com:25 or lmtp:imap.yourdomain.com:2525 on a per account basis, and you can get rid of the MTA in due time. >> You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. > On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Since you're sending everything back to the MX, you might as well have your MX use LMTP, looking up the correct protocol and host from the database, and spend the next couple of years telling your customers to change their mail client configuration to use a dedicated outgoing mail server. It's worth the trouble. > > Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). > > -- > Luca Lesinigo From ednitido at gmail.com Fri Mar 23 21:44:23 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 15:44:23 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are the same Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 From tss at iki.fi Fri Mar 23 21:46:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:46:53 +0200 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: On 23.3.2012, at 21.44, Ed Nitido wrote: > I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are > the same > > Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 Set auth_debug=yes. What does it log with v2.1.3? Also what's in your dovecot-ldap.conf.ext? From tss at iki.fi Fri Mar 23 21:49:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:49:38 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4F6CBFCB.60209@gedalya.net> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> <4F6CBFCB.60209@gedalya.net> Message-ID: On 23.3.2012, at 20.24, Gedalya wrote: > On 03/23/2012 02:12 PM, Luca Lesinigo wrote: >> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >>> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. >> I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. >> I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... > I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. Right, a lot of people have done migration from Courier -> Dovecot using the dummy proxying. Since v2.0 the proxying automatically handles any CAPABILITY string issues. From ednitido at gmail.com Fri Mar 23 22:26:46 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 16:26:46 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: <950E30E6-38A5-4F5F-B2D6-B12C810AB439@iki.fi> Message-ID: Ooops, didn't email the list... it working now thanks to Timo, solution below On Fri, Mar 23, 2012 at 4:14 PM, Timo Sirainen wrote: > >> On 23.3.2012, at 22.01, Ed Nitido wrote: >> >> > pass_attrs = >> uid=user,userPassword=password,=proxy,=master=doveadmin,=pass=xxxxxx >> >> I guess it doesn't like the "=proxy" part. I guess I should fix it. For >> now just set "=proxy=y". >> > > From ncjeffgus at zimage.com Fri Mar 23 22:42:23 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Fri, 23 Mar 2012 13:42:23 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6CBAA2.5020409@in.tum.de> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> Message-ID: <1332535343.5601.6.camel@sally> On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > Hi, > > maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync *should* be able to be just as fast, but it currently is spending way too much time doing something. What is it? ...Jeff From post at michael-neubert.de Fri Mar 23 22:57:28 2012 From: post at michael-neubert.de (Michael Neubert) Date: Fri, 23 Mar 2012 21:57:28 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 Message-ID: <4F6CE3B8.7020507@michael-neubert.de> Hello, I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". The config was not touched but now IMAP connections are not possible anymore (LMTP works fine). When I try to connect to a mailbox, the connect fails. Some log entries: ############################################################################################################### Mar 23 21:45:28 imap-login: Warning: SSL: where=0x10, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv2/v3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: auth client connected (pid=3431) Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server done A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write session ticket A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write change cipher spec A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x20, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: client in: AUTH 1 PLAIN service=imap secured lip=yyy.yyy.yyy.yyy rip=xxx.xxx.xxx.xxx lport=993 rport=51379 Mar 23 21:45:28 auth: Debug: client out: CONT 1 Mar 23 21:45:28 auth: Debug: client in: CONT 1 AG5lbWlAdmlzaXQtd29ybGQuZGUAUHJvNDUwLnN1 Mar 23 21:45:28 auth-worker(3433): Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 21:45:28 auth-worker(3433): Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_mysql.so Mar 23 21:45:28 auth-worker(3433): Info: mysql(zzz.zzz.zzz.zzz): Connected to database dovecot Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): query: SELECT password, 'directory' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid FROM users WHERE username = 'username' AND domain = 'domain' AND active = 'Y' Mar 23 21:45:28 auth: Debug: client out: OK 1 user=username Mar 23 21:45:28 auth: Debug: master in: REQUEST 2286813185 3394 1 4727968fd3514dd45f623ad9f944e305 Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): SELECT home, uid, gid FROM users WHERE username = 'username' AND domain = 'domain' Mar 23 21:45:28 auth: Debug: master out: USER 2286813185 username home=directory uid=8 gid=8 Mar 23 21:45:28 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=3434, TLS Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap(username): Info: Connection closed in=0 out=303 ############################################################################################################### The MySQL authentification seems to work fine, but after this the connection is closed with the SSL alert. In Dovecot 2.1.2 everything worked fine. The SSL certifcate is also correct. Any hints are welcome to identify the problem. Thanks in advance. Beste wishes Michael From tss at iki.fi Fri Mar 23 23:03:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:03:01 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> On 23.3.2012, at 19.43, wrote: >> Have you tried stress testing it with imaptest? Run in parallel for both >> servers: > I did stress test it, but we have developed a "mail bot net" tool for the > purpose. I should mention this was tested using dovecot 1.2, as this is > our current production version (hopefully will be upgrading soon). Its > comprised of a control server that starts a bot network of client machines > that creates pop/imap connections (smtp as well) on our test cluster of > dovecot (and postfix) servers. In my test I distributed the load across a > two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN > storage attached to it. I actually didn't change my configuration from > when I had a test NFS server connected to the test servers (mmap disabled, > fcntl locking, etc), because glusterfs was an afterthought when we were > stress testing our new netapp system using NFS. We have everything in > VMware, including the glusterfs servers. Using five bot servers and > connecting 7 times a second from each server (35 connections per second) > for both pop and imap (70 total connections per second) split between two > dovecot servers I was not seeing any big issues. The load average was low, > and there were no errors to speak of in dovecot (or postfix). I was > mounting the storage with the glusterfs native client, not using NFS (which > I have not tested). I would like to do a more thorough test of glusterfs > using Dovecot 2.0 on some dedicated hardware and see how much further I can > push the system. What did the bots do? Add messages and delete messages as fast as they could? I guess that's mostly enough to see if things work. imaptest anyway hammers the server as fast as it can with all kinds of commands. From tss at iki.fi Fri Mar 23 23:25:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:25:28 +0200 Subject: [Dovecot] dsync redesign Message-ID: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: http://dovecot.org/tmp/dsync-redesign-problem.txt It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. From list at airstreamcomm.net Sat Mar 24 01:39:11 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 18:39:11 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> Message-ID: <7e40b18742c5053948aeaaa51d41ceca@mail.airstreamcomm.net> On Fri, 23 Mar 2012 23:03:01 +0200, Timo Sirainen wrote: > On 23.3.2012, at 19.43, > wrote: > >>> Have you tried stress testing it with imaptest? Run in parallel for both >>> servers: >> I did stress test it, but we have developed a "mail bot net" tool for the >> purpose. I should mention this was tested using dovecot 1.2, as this is >> our current production version (hopefully will be upgrading soon). Its >> comprised of a control server that starts a bot network of client >> machines >> that creates pop/imap connections (smtp as well) on our test cluster of >> dovecot (and postfix) servers. In my test I distributed the load across >> a >> two node dovecot (/postfix) cluster back ended by glusterfs, which has >> SAN >> storage attached to it. I actually didn't change my configuration from >> when I had a test NFS server connected to the test servers (mmap >> disabled, >> fcntl locking, etc), because glusterfs was an afterthought when we were >> stress testing our new netapp system using NFS. We have everything in >> VMware, including the glusterfs servers. Using five bot servers and >> connecting 7 times a second from each server (35 connections per second) >> for both pop and imap (70 total connections per second) split between two >> dovecot servers I was not seeing any big issues. The load average was >> low, >> and there were no errors to speak of in dovecot (or postfix). I was >> mounting the storage with the glusterfs native client, not using NFS >> (which >> I have not tested). I would like to do a more thorough test of glusterfs >> using Dovecot 2.0 on some dedicated hardware and see how much further I >> can >> push the system. > > What did the bots do? Add messages and delete messages as fast as they > could? I guess that's mostly enough to see if things work. imaptest anyway > hammers the server as fast as it can with all kinds of commands. We created two python scripts on the bots that listed all the messages in the inbox then deleted all the messages in the inbox, one script doing pop and the other doing imap. The bots were also sending messages to the server simultaneously to repopulate inboxes. I didn't know about imaptest, thanks! From noel.butler at ausics.net Sat Mar 24 03:19:50 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sat, 24 Mar 2012 11:19:50 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <1332551990.11835.19.camel@tardis> On Fri, 2012-03-23 at 12:53 +0200, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > > > Timo wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't take much effort at all to change that to look for the long time default of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ No-one is suggesting putting all the individual conf files in /etc, only for existence of dovecot.conf itself. There are plenty of linux and unix systems that have been using /etc for as long as I can recall (even early redhat did), its only certain distros that build as /etc/foo/ the ones that use rpms or debs are obviously not running anything special (we all know no build config process will suite all operations) there are a large number i'm sure who use source (besides, with debian and redhat, who knows WHAT butchering they've done to upstreams code)... Which brings up another question, may I ask why some of the options to disable some passwd types were removed from build process? Systems that dont use system password files (amongst other formats) dont need to build them, that's not a criticism, 'just sayin'. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Sat Mar 24 03:50:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:50:07 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332551990.11835.19.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: On 24.3.2012, at 3.19, Noel Butler wrote: >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't > take much effort at all to change that to look for the long time default > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > No-one is suggesting putting all the individual conf files in /etc, only > for existence of dovecot.conf itself. So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Which brings up another question, may I ask why some of the options to > disable some passwd types were removed from build process? Systems that > dont use system password files (amongst other formats) dont need to > build them, that's not a criticism, 'just sayin'. There's also no harm in having that code included. They add no extra library dependencies. The only thing they do is to use a few kilobytes of more disk space, and possibly a few kilobytes of more memory (even that isn't certain). All options just increase the number of combinations that can cause things to go wrong. If I add some code to be compiled optionally, it just adds more combinations that should be tested together to see if the code still even compiles. Previously I've broken SSL code many times by not testing if after changes Dovecot builds without OpenSSL. So the less options there are, the more robust Dovecot is, and the less work I have to do to keep it working when adding new features. So I add an option only when there is a good use case for it and I expect more than one person to use it. From tss at iki.fi Sat Mar 24 03:52:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:52:56 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: On 23.3.2012, at 19.22, Andra? 'ruskie' Levstik wrote: > :2012-03-23T12:53:Timo Sirainen: > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > I would suggest to have a --layout=gnu|opt > > That would either do what it currently does(gnu) and opt to install > everything into a single dir i.e.: > /opt/dovecot/ > > With subdirs under there. Yes, --with-layout=gnu|opt could be useful. Anyone want to volunteer to implement it? :) From dovecot at tlinx.org Sat Mar 24 08:12:44 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:12:44 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6D65DC.7030304@tlinx.org> Jeff Gustafson wrote: > On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > >> Hi, >> >> maybe try "dsync -o mail_fsync=never". >> > > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? > ...Jeff > --- Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can use large buffers (~16MB gives best results on my local Gbit network), but it misses all those pesky acls and extended attrs, not to mention file perms...*sigh* Compare that to the I/O done 4k at a time by many older utils... If I'm writing to the LOCAL HD, instead of the network, then a 1GB-4GB buffer size gives best results (1GB/s raid5). Small buffers are such a PITA! From dovecot at tlinx.org Sat Mar 24 08:16:08 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:16:08 -0700 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? Message-ID: <4F6D66A8.3050208@tlinx.org> Is this redhat's version of the kernel only? Or does it apply to other linux kernels and other distros? Any idea what linux kernel versions might cause this? (from main dovecot webpage news) Thu Mar 22 14:38:53 EET 2012 Red Hat/CentOS users: A recent kernel update causes Dovecot to start failing after it has reached 1000 child processes. To fix this, downgrade your kernel until Red Hat releases a fixed kernel. From bra at fsn.hu Sat Mar 24 09:19:48 2012 From: bra at fsn.hu (Attila Nagy) Date: Sat, 24 Mar 2012 08:19:48 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <4F6D7594.10800@fsn.hu> On 03/23/12 22:25, Timo Sirainen wrote: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) It would bring Dovecot into the class of open source mail servers where there are currently no contenders. BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. For example, messages are immutable, so you won't face this issue in this area. And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). And of course there is always the problem of designing a good, consistent method which is also efficient. From jtam.home at gmail.com Sat Mar 24 11:36:33 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sat, 24 Mar 2012 02:36:33 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> References: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> Message-ID: On Fri, 23 Mar 2012, Timo Sirainen wrote: > On 23.3.2012, at 12.58, Joseph Tam wrote: > >> I ran into two issues trying to upgrade our dovecot installation (Solaris 10). >> >> 1) Does not compile with OpenSSL 0.9.7 >> >> Not a big deal, as I was able to successfully against OpenSSL 0.9.8, >> but does dovecot require OpenSSL >= 0.9.8 now? > > Hm. Maybe it's time by now? :) It could be fixed with some more > #ifdefs but those make code more unreadable. It might still compile with OpenSSL 0.9.7 if it is built with engine support (the default), but yeah, it's time to move to 0.9.8 or 1.0.0. >> 2) Dovecot's LDA does not work >> >> After stopping the the old dovecot, and starting dovecot 2.1.3 using the >> exact same config file, local mail delivery tempfails: >> >> Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address > > http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? Spot on, as usual. Thanks. Joseph Tam From janfrode at tanso.net Sat Mar 24 12:04:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 24 Mar 2012 11:04:07 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <20120324100407.GB31829@dibs.tanso.net> On Sat, Mar 24, 2012 at 08:19:48AM +0100, Attila Nagy wrote: > On 03/23/12 22:25, Timo Sirainen wrote: > > > Well, dsync is a very useful tool, but with continuous replication > it tries to solve a problem which should be handled -at least > partially- elsewhere. Storing stuff in plain file systems and > duplicating them to another one just doesn't scale. I don't see why this shouldn't scale. Mailboxes are after all changed relatively infrequently. One idea for making it more scalable might be to treat indexes/metadata and messages differently. Make index/metadata updates synchronous over the clusters/locations (with re-sync capability in case of lost synchronisation), while messages are store in one "altstorage" per cluster/location. For a two-location solution, message-data should be stored in: mail_location = mdbox:~/mdbox ALTcache=mdbox:~/mdbox-remoteip-cache ALT=dfetch://remoteip/ <-- new protocol If a message is in the index, look for it in that order: local mdbox ALTcache ALT if it finds the message in ALT, make a copy into ALTcache (or local mdbox?). Syncronizing messages could be a very low frequency job, and could be handled by simple rsync of ALT to ALTcache. No need for specialized tool for this job. Syncronizing ALTcache to local mdbox could be done with a reversed doveadm-altmove, but might not be necessary. Of course this is probably all very naive.. but you get the idea :-) -jf From stan at hardwarefreak.com Sat Mar 24 12:06:25 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 24 Mar 2012 05:06:25 -0500 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? In-Reply-To: <4F6D66A8.3050208@tlinx.org> References: <4F6D66A8.3050208@tlinx.org> Message-ID: <4F6D9CA1.9050008@hardwarefreak.com> On 3/24/2012 1:16 AM, Linda Walsh wrote: > Is this redhat's version of the kernel only? Or does it apply to other > linux kernels and other distros? > > Any idea what linux kernel versions might cause this? > > (from main dovecot webpage news) > > Thu Mar 22 14:38:53 EET 2012 > > Red Hat/CentOS users: A recent kernel update > causes Dovecot to > start failing after it has reached 1000 child processes. To fix this, > downgrade your kernel until Red Hat releases a fixed kernel. It appears to be a Red Hat centric regression. They added a patch to fix one thing and broke other things, Dovecot, in the process, because the Red Hat programmer made an incorrect assumption about what real world applications were doing, apparently without investigating such first. Note that one won't see this problem on their REHL/CentOS system if they never hit 1000 child processes. And as Timo states in the bug report it's *possible* Postfix could suffer the same problem as it uses the same pipe/epoll system. However nobody runs 1000 Postfix smtp[d]s. Few, if any, run over 200. The ones that do usually don't know how to properly tune Postfix, and they use a high smtp[d] daemon count to compensate for suboptimal configuration elsewhere in the system. A properly setup Postfix server can handle 200-300 msgs/second with the default 100 smtp[d] processes. 1000 smtp[d]s would suggest a message rate 10x that, or 2000-3000 msgs/second. The server plus disk subsystem required to queue that kind of message rate would be impressive, and expensive, for a mail server. This same message rate can typically be achieved by a much less expensive scale out farm. If anyone on the planet is running a properly tuned 1000 process Postfix server, I'd love to read about it. -- Stan From tss at iki.fi Sat Mar 24 13:49:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 13:49:36 +0200 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <0B23962D-A067-4A71-9A10-067FCA76B06D@iki.fi> On 24.3.2012, at 9.19, Attila Nagy wrote: > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. dsync solves several other problems besides replication. Even if Dovecot had a super efficient replicated storage, dsync would still exist for doing things like: - migrating between mailbox formats - migrating from other imap/pop3 servers - creating (incremental) backups - the redesign works great for super-high latency replication (USB sticks, cross-planet replication :) - and when you really just don't want any kind of a complex replicated database, just something simple So I'll need to get this working well in any case. And with the redesign the replication should be efficient enough to scale pretty well. > I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) Yes, I'm also planning on implementing that, but not yet. > It would bring Dovecot into the class of open source mail servers where there are currently no contenders. > > BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. > For example, messages are immutable, so you won't face this issue in this area. > And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. My plan was to create similar index files as currently exists in filesystem. It would work pretty much the same as you described: There's a "log" where changes are appended, and once in a while the changes are written into an "index" snapshot. When reading you first read the snapshot and then apply new changes from the log. The conflict resolution if DB supports it would work by reading the two logs in parallel and figure out a way to merge them consistently, similar to how dsync does pretty much the same thing. Hmm. Perhaps the metadata log could exist exactly as the dsync data format and have dsync code do the merging?.. > Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). This is already optionally done in v2.0+dbox. MIME attachments can be stored in plain binary form if they can be reconstructed back into their original form. It doesn't break any signed stuff. From CMarcus at Media-Brokers.com Sat Mar 24 14:01:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:01:07 -0400 Subject: [Dovecot] SIS and restoring from backups Message-ID: <4F6DB783.3050808@Media-Brokers.com> On 2012-03-24 7:49 AM, Timo Sirainen wrote: > This is already optionally done in v2.0+dbox. MIME attachments can be > stored in plain binary form if they can be reconstructed back into > their original form. It doesn't break any signed stuff. Hey Timo, Splitting this off into a separate thread... On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? Or maybe I misunderstood the problem? -- Best regards, Charles From tss at iki.fi Sat Mar 24 14:08:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 14:08:13 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DB783.3050808@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> Message-ID: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> On 24.3.2012, at 14.01, Charles Marcus wrote: > On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? > > Or maybe I misunderstood the problem? You can do full backups from a filesystem snapshot, which works "well enough" (might leave some unused attachments lying around in some rare cases, but that can also happen if Dovecot crashes/dies). The other possibility is to already use dsync (doveadm backup) to do full backups. With the redesigned dsync you would be able to do incremental backups also. In any case the solution involves de-SISing mails for backup. From CMarcus at Media-Brokers.com Sat Mar 24 14:54:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:54:14 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> Message-ID: <4F6DC3F6.70306@Media-Brokers.com> On 2012-03-24 8:08 AM, Timo Sirainen wrote: > You can do full backups from a filesystem snapshot, which works > "well enough" (might leave some unused attachments lying around in > some rare cases, but that can also happen if Dovecot crashes/dies). But the problem isn't with backups, but with restores, right? > The other possibility is to already use dsync (doveadm backup) to do > full backups. With the redesigned dsync you would be able to do > incremental backups also. In any case the solution involves > de-SISing mails for backup. So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Doesn't sound ideal... I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. Am I correct that enabling SIS as it is currently implemented would break this backup tool? I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... -- Best regards, Charles From post at michael-neubert.de Sat Mar 24 15:04:55 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 14:04:55 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6DC677.1000100@michael-neubert.de> The problem starts just after authorization: Console: ################################################################### openssl s_client -connect mailserver.com:993 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. a login "username" "password" closed ################################################################### Here are the logs of this moment: ################################################################### Mar 24 13:48:46 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=10662, TLS Mar 24 13:48:46 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 24 13:48:46 imap(username): Info: Connection closed in=0 out=303 ################################################################### So just after sucussful login with correct username / password the connection is closed. From tss at iki.fi Sat Mar 24 15:16:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:16:38 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DC3F6.70306@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: On 24.3.2012, at 14.54, Charles Marcus wrote: > On 2012-03-24 8:08 AM, Timo Sirainen wrote: >> You can do full backups from a filesystem snapshot, which works >> "well enough" (might leave some unused attachments lying around in >> some rare cases, but that can also happen if Dovecot crashes/dies). > > But the problem isn't with backups, but with restores, right? Ah, right. Then it gets tricky. >> The other possibility is to already use dsync (doveadm backup) to do >> full backups. With the redesigned dsync you would be able to do >> incremental backups also. In any case the solution involves >> de-SISing mails for backup. > > So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Some backup systems can do internal deduplication. > I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. > > Am I correct that enabling SIS as it is currently implemented would break this backup tool? I'm not sure. Are you running rsnapshot on live filesystem or on a snapshot? On live filesystem there would be race conditions. > I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... This should be possible, just point the namespace to such snapshot. You may need to point CONTROL dir to some temporary directory and index dir as well to either temp or to memory. From tss at iki.fi Sat Mar 24 15:17:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:17:28 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6DC677.1000100@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> Message-ID: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> On 24.3.2012, at 15.04, Michael Neubert wrote: > openssl s_client -connect mailserver.com:993 > * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. > a login "username" "password" > closed And what happens without SSL? e.g. telnet localhost 143 From mcbdovecot at robuust.nl Sat Mar 24 15:21:51 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Sat, 24 Mar 2012 14:21:51 +0100 (CET) Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: On Fri, 23 Mar 2012, Jeff Gustafson wrote: > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per second. Depending on the harddisks used, that would be quite a challenge. If you use rsync to only transfer the files that changed (based on file modification time) you may or may not miss files that have changed but still have the same time stamp. I assume you didn't use the --checksum parameter to rsync, right? dsync does so much more than simply copy some files... -- Maarten From post at michael-neubert.de Sat Mar 24 18:00:13 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:00:13 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> Message-ID: <4F6DEF8D.7090309@michael-neubert.de> > And what happens without SSL? e.g. telnet localhost 143 Without SSL it is no problem: ############################################## telnet imap-server 143 Trying xxx.xxx.xxx.xxx... Connected to imap-server. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. a login "username" "password" a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE QUOTA] Logged in b select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded \*)] Flags permitted [...] ############################################## From tss at iki.fi Sat Mar 24 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 18:02:59 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". So what exactly is this version? dovecot --version? From post at michael-neubert.de Sat Mar 24 18:09:43 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:09:43 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> Message-ID: <4F6DF1C7.1020306@michael-neubert.de> >> I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using >> Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > So what exactly is this version? dovecot --version? At the moment the version is "2.1.3-0~auto+6" from rename-it.nl. dovecot -n: # 2.1.3 (4ae85f573c93): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.4 ocfs2 From jackie.craig.sparks at live.com Sat Mar 24 22:34:49 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 16:34:49 -0400 Subject: [Dovecot] dovecot and cloudfile systems Message-ID: I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. I cant lock files, This is accessed using the fuse library. I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. From tss at iki.fi Sat Mar 24 22:43:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 22:43:34 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: Message-ID: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> On 24.3.2012, at 22.34, jackie sparks wrote: > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > I cant lock files, > > This is accessed using the fuse library. > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From jackie.craig.sparks at live.com Sat Mar 24 23:15:58 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 17:15:58 -0400 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. > From: tss at iki.fi > Date: Sat, 24 Mar 2012 22:43:34 +0200 > To: jackie.craig.sparks at live.com > CC: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot and cloudfile systems > > > On 24.3.2012, at 22.34, jackie sparks wrote: > > > > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > > I cant lock files, > > > > This is accessed using the fuse library. > > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. > > http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From tss at iki.fi Sat Mar 24 23:27:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 23:27:47 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: <00827B46-E6B3-4DD8-A035-421A522F489F@iki.fi> What you're trying to do is quite unlikely to work with any IMAP server / cloud filesystem combination. And if it does work, the performance will most likely be horrible. Of course, if it does work with any kind of a combination I'm interested in knowing about it. On 24.3.2012, at 23.15, jackie sparks wrote: > > This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. > It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= > This e-mail (including attachments) is covered by the Electronic > Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is > confidential, and is intended solely for the use of the individuals or > entities to whom it is addressed. If you are not the intended > recipient or the person responsible for delivering the e-mail to the > intended recipient, be advised that you have received this e-mail in > error and that any use, dissemination, forwarding, printing, or > copying of this e-mail and any file attachments is strictly > prohibited. If you have received this e-mail in error, please > immediately notify me by email at jackie.craig.sparks at live.com. You must destroy > the original transmission and its contents. > > >> From: tss at iki.fi >> Date: Sat, 24 Mar 2012 22:43:34 +0200 >> To: jackie.craig.sparks at live.com >> CC: dovecot at dovecot.org >> Subject: Re: [Dovecot] dovecot and cloudfile systems >> >> >> On 24.3.2012, at 22.34, jackie sparks wrote: >> >>> >>> I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. >>> I cant lock files, >>> >>> This is accessed using the fuse library. >>> I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. >>> I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. >> >> http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. > From post at michael-neubert.de Sun Mar 25 01:00:27 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sun, 25 Mar 2012 00:00:27 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6E520B.4060303@michael-neubert.de> I just did some more tests with different binaries. The problem occurs since: Dovecot 2.1.3-0~auto+5 dovecot --version 2.1.3 (f30437ed63dc) Dovecot 2.1.3-0~auto+4 works fine dovecot --version 2.1.3 (ff5c341f8838) So my title is wrong. The problem only affects people "who like to live on the edge" of 2.1.3 release ;) The stable Dovecot 2.1.3 release http://dovecot.org/list/dovecot-news/2012-March/000219.html is not affected. From noel.butler at ausics.net Sun Mar 25 05:48:36 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:48:36 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332643716.4515.23.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > > > No-one is suggesting putting all the individual conf files in /etc, only > > for existence of dovecot.conf itself. > > So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Not at all, I'm suggesting that in search for dovecot.conf file only, the search path be preferenced by @sysconfdir@/dovecot.conf such as --sysconfdir=/etc it looks for /etc/dovecot.conf, if not found, the config file location search continues on to look for /etc/dovecot/dovecot.conf I might be wrong, there might only be a handful of people annoyed by this change, but as more and more using custom builds test out moving from 1.2 to 2.x, well, more and more might be caught out, wouldn't it be better to, as you said previously, " avoid emails". As I'm sure you got better things to do than read any tripe I post :) Anyway I have made my suggestion, nothing more I think I can say on the matter that I haven't said already, so I'll leave it there, if it's implemented, great, if not, well, it's not... Cheers -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From noel.butler at ausics.net Sun Mar 25 05:56:06 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:56:06 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332644166.4515.30.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > grrr meant to comment on this too, umm since a default custom build doesnt install any config files, this would only become a confusion if one were using say an RPM package, and then decided to custom install, but IIRC, RPM renames the old config anyway, least it used to in some packages, dont know about .deb stuff though, it lacks a lot of intelligence so probably not (/me starts flamewars) Now I've said my bit.. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From stsiol at yahoo.co.uk Sun Mar 25 10:24:56 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 08:24:56 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot Message-ID: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Hello all, I want to ask about something I never did before. I have a dovecot/XMail/LAMP/Horde installation on a CentOS 5.5 32-bit system with two domains : domainA and domainB All the users used to have their mailboxes on domainA. However the personell dept decided that 90% of the users will have to have their mailboxes set to domainB and the other 10% will stay at domainA. So, I need to migrate those mailboxes from domainA to domainB. The only thing is I haven't done that ever. The directory structure is : "/var/MailRoot/domains/domainX/Username/Maildir/" and under there the usual suspects : cur (directory) dovecot.index.cache (file) dovecot-keywords (file) dovecot-uidvalidity (file) ????????? new (directory) tmp (directory) dovecot.index (file) dovecot.index.log (file) dovecot-uidlist (file) dovecot-uidvalidity.4cc055c6 (file) subscriptions (file) Using dovecot v1.2.15 Any help would be appreciated. Iam supposed to do this today !! :-) Thank you all, s. ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis? From jtam.home at gmail.com Sun Mar 25 10:46:25 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sun, 25 Mar 2012 00:46:25 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value Message-ID: Subject: Different user messages clustered around the same date.saved value After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." to iterate through all user trash folders and expunge old messages. However, I noticed a strange thing: querying what would have been deleted doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d showed many date.saved values are clustered around the same timestamp, even among different user's Trash mailbox. One user's trash mailbox having the same date.saved is explained by a user deleting a lot of message at one time, but I can't explain why many different users would have messages with the same (or closeby) date.saved value. For example, the output of the above query on my system showed the 10s window /2012-03-05 18:08:0[0-9]/ matched 7658 messages among 22 different user Trash mailboxes, which is statistically unlikely. I did't see anything special in the dovecot logs at this time to explain this. What would cause this? Joseph Tam From gedalya at gedalya.net Sun Mar 25 11:23:50 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 25 Mar 2012 04:23:50 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Message-ID: <4F6ED616.5090504@gedalya.net> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: > The directory structure is : > > > "/var/MailRoot/domains/domainX/Username/Maildir/" You can probably just: 1. Do something to prevent the user from logging in, and any deliveries from happening, e.g. delete the user. 2. Kick any existing connections. 3. Just move the Username directory from domainA to domainB. 4. Create the new user in the new domain Test the procedure first. Don't let a client log in to a mailbox and see something he's not supposed to see, like an empty mailbox. It can cause the client to drop its local cache and possibly other data. From lists at wildgooses.com Sun Mar 25 14:16:55 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:16:55 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6EFEA7.3030406@wildgooses.com> On 24/03/2012 13:21, Maarten Bezemer wrote: > > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > >> That didn't seem to make much of a difference. On a 3.1GB backup it >> shaved off 5 seconds. dsync's time was over 6 minutes with or without >> the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. >> It seems to me that dsync *should* be able to be just as fast, >> but it >> currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB > per second. Depending on the harddisks used, that would be quite a > challenge. rsync is only going to transfer files it believes has changed, so the transfer bandwidth will likely be lower > If you use rsync to only transfer the files that changed (based on > file modification time) you may or may not miss files that have > changed but still have the same time stamp. I assume you didn't use > the --checksum parameter to rsync, right? Dovecot is not very resiliant to files changing under it, but without the filename changing. I have no idea if it's supposed to work at all, but you might at least expect to see problems if you start doing this? > dsync does so much more than simply copy some files... Quite probably, but I don't think your expose above illustrates this? Regards Ed W From lists at wildgooses.com Sun Mar 25 14:41:35 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:41:35 +0100 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C8281.10906@hardwarefreak.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> Message-ID: <4F6F046F.1070003@wildgooses.com> On 23/03/2012 14:02, Stan Hoeppner wrote: > On 3/23/2012 6:41 AM, Radim Kolar wrote: >> Can somebody provide maildrop syntax for using deliver-lda as final >> delivery program during sorting mail in user mailfilter? >> >> i mean replacement for "to" statement >> >> if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) >> { >> to $MAIL/.dovecot/ >> } > Dovecot's local delivery agent uses the Sieve language: > http://wiki.dovecot.org/LDA/Sieve > > The syntax is quite different than maildrop or procmail. > I think that's why he asked the question? I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? In answer to the OP: read the maildropex man pages, but you have several options, eg: to "| someprogram" or: xfilter someprogram `someprogram` However, almost certainly I think you want the top option? Good luck Ed W From fumiyas at osstech.jp Sun Mar 25 15:17:36 2012 From: fumiyas at osstech.jp (SATOH Fumiyasu) Date: Sun, 25 Mar 2012 21:17:36 +0900 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <87zkb4j2n3.wl%fumiyas@osstech.jp> At Fri, 23 Mar 2012 12:53:16 +0200, Timo Sirainen wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > OpenLDAP's (and smbldap-tools's) configure script has --with-subdir option. This option defaults to "/openldap". (/smbldap-tools) $ grep subdir configure.in build/top.mk configure.in:dnl --with-subdir configure.in:ldap_subdir="/openldap" configure.in:AC_ARG_WITH(subdir, configure.in:[ --with-subdir=DIR change default subdirectory used for installs], configure.in: no) ldap_subdir="" configure.in: ldap_subdir="$withval" configure.in: ldap_subdir="/$withval" configure.in:AC_SUBST(ldap_subdir)dnl build/top.mk:ldap_subdir = @ldap_subdir@ build/top.mk:datadir = @datadir@$(ldap_subdir) build/top.mk:moduledir = @libexecdir@$(ldap_subdir) build/top.mk:sysconfdir = @sysconfdir@$(ldap_subdir) -- -- Name: SATOH Fumiyasu (fumiyas @ osstech co jp) -- Business Home: http://www.OSSTech.co.jp/ -- Personal Home: https://github.com/fumiyas/ From dovecot at vosslamber.nl Sun Mar 25 15:53:16 2012 From: dovecot at vosslamber.nl (Luuk@dovecot) Date: Sun, 25 Mar 2012 14:53:16 +0200 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6ED616.5090504@gedalya.net> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> Message-ID: <4F6F153C.2020005@vosslamber.nl> On 25-03-2012 10:23, Gedalya wrote: > On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >> The directory structure is : >> >> >> "/var/MailRoot/domains/domainX/Username/Maildir/" > > You can probably just: > 1. Do something to prevent the user from logging in, and any deliveries > from happening, e.g. delete the user. > 2. Kick any existing connections. > 3. Just move the Username directory from domainA to domainB. > 4. Create the new user in the new domain > > Test the procedure first. > > Don't let a client log in to a mailbox and see something he's not > supposed to see, like an empty mailbox. It can cause the client to drop > its local cache and possibly other data. > > i would also setup a forward from domainA to domainB for all the users that have moved, at least until most people who do send email know the 'old'address has changed. From stsiol at yahoo.co.uk Sun Mar 25 16:04:33 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 14:04:33 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <1332680673.52988.YahooMailNeo@web132203.mail.ird.yahoo.com> >On 25-03-2012 10:23, Gedalya wrote: >> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >>> The directory structure is : >>> >>> >>> "/var/MailRoot/domains/domainX/Username/Maildir/" >>? >> You can probably just: >> 1. Do something to prevent the user from logging in, and any deliveries >> from happening, e.g. delete the user. >> 2. Kick any existing connections. >> 3. Just move the Username directory from domainA to domainB. >> 4. Create the new user in the new domain >>? >> Test the procedure first. >>? >> Don't let a client log in to a mailbox and see something he's not >> supposed to see, like an empty mailbox. It can cause the client to drop >> its local cache and possibly other data. >>? >>? > >i would also setup a forward from domainA to domainB for all the users >that have moved, at least until most people who do send email know the >'old'address has changed. Hi chaps and thank you for your replies, Sorry forgot to mention. Most of the users use thunderbird. And yes, thank you for the forwarding issue of old e-mail accounts to the new ones. However, that is not my problem. Let me suggest something . . . : What if : 1. I setup the new accounts 2. Users log on normally from thunderbird 3. While they are using thunderbird, I get rid of their old e-mail address, create the new one (in thunderbird) 4. I make sure that the newly created address syncs with dovecot Wouldn't that be enough to migrate "on-the-fly" so to speak their existing directory tree structure and related messages to their new e-mail account ? Any ideas ? Cheers, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From CMarcus at Media-Brokers.com Sun Mar 25 18:01:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:01:39 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <4F6F3353.6000700@Media-Brokers.com> On 2012-03-25 8:53 AM, Luuk at dovecot wrote: > i would also setup a forward from domainA to domainB for all the users > that have moved, at least until most people who do send email know the > 'old'address has changed. I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 25 18:12:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:12:58 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: <4F6F35FA.6050207@Media-Brokers.com> On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: > >> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>> You can do full backups from a filesystem snapshot, which works >>> "well enough" (might leave some unused attachments lying around in >>> some rare cases, but that can also happen if Dovecot crashes/dies). >> >> But the problem isn't with backups, but with restores, right? > > Ah, right. Then it gets tricky. Yeah, I seem to remember it was a comment like that that scared me about enabling it... Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) >> So, this would make the backup storage requirements larger - maybe >> dramatically larger for sites that have a lot of large >> attachments? > Some backup systems can do internal deduplication. Hmmm... and actually, rsnapshot (which uses rsync) does just that, which is *why* each additional snapshot only requires a small fraction of additional disk space (compared to the first main/full snapshot). >> Am I correct that enabling SIS as it is currently implemented would >> break this backup tool? > I'm not sure. Are you running rsnapshot on live filesystem or on a > snapshot? On live filesystem there would be race conditions. I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] but the rsnapshot guys assured me this will and does not cause any real problems, other than those files don't get backed up. I am however looking forward to migrating this to a VM so I can do snapshot for backups to get consistent point-in-time backups. >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. This is great news! I'm looking forward to getting this all working. -- Best regards, Charles From stsiol at yahoo.co.uk Sun Mar 25 18:40:08 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 16:40:08 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F3353.6000700@Media-Brokers.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> <4F6F3353.6000700@Media-Brokers.com> Message-ID: <1332690008.99877.YahooMailNeo@web132204.mail.ird.yahoo.com> Thanks Charles, s. ? ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis >________________________________ > From: Charles Marcus >To: dovecot at dovecot.org >Sent: Sunday, 25 March 2012, 17:01 >Subject: Re: [Dovecot] migrating mailboxes on dovecot > >On 2012-03-25 8:53 AM, Luuk at dovecot wrote: >> i would also setup a forward from domainA to domainB for all the users >> that have moved, at least until most people who do send email know the >> 'old'address has changed. > >I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. > >What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. > >-- >Best regards, > >Charles > > > From hsn at filez.com Mon Mar 26 12:25:47 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 26 Mar 2012 11:25:47 +0200 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6F046F.1070003@wildgooses.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> <4F6F046F.1070003@wildgooses.com> Message-ID: <4F70361B.1070304@filez.com> I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? yes > In answer to the OP: read the maildropex man pages, but you have > several options, eg: > Yes found that dovecot-lda -m will do it nicely. echo "mail message Test " | /usr/local/libexec/dovecot/dovecot-lda -m dovecot just user agents are not able to display message with 0 headers. That confused me. From jeetuindian at gmail.com Mon Mar 26 12:51:00 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Mon, 26 Mar 2012 15:21:00 +0530 Subject: [Dovecot] dovecot.log warning Message-ID: Hi Guys, Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data i.e home directory and mail data from previous server which was on dovecot 1.2.8 to new one. Every thing is working fine. Mails are going and coming. But in dovecot.log file I an getting like imap(user at example.com):Warning: fscking index file /home/ example.com/user/mail/.imap/VISA/dovecot.index Can any one specify why its coming and how can I fix it ? -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From Attila.Sipos at netcall.com Mon Mar 26 16:28:31 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 14:28:31 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied Message-ID: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Hi, When I issue an IMAP copy command using the wrong UID, the server gives an "OK No messages copied" response. This seems like the wrong response to me. If the UID doesn't exist, then it should respond with a "No" response - maybe something like "NO - copy error: bad UID" I believe "OK No messages copied" would only be a suitable response if the email with the supplied UID had already been known to be copied successfully. I am using dovecot 1.2.9 - has this been fixed in newer versions of dovecot? Regards Attila Attila Sipos Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From bob at db.org Mon Mar 26 16:29:08 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Mon, 26 Mar 2012 15:29:08 +0200 Subject: [Dovecot] fts-solr not indexing body content Message-ID: This could easily be me missing something, but I can't seem to get the fts-solr plugin to index message bodies. Tcpdump shows me the following being sent from Dovecot to Solr as a messages is indexed: 6549fde08816e80d6b4f26650000b5f0b4b2user6549/fde08816e80d6b4f26650000b5f0b4b2/user Return-path: ... As you can see, the -node contains just an empty line. The above is just a snippet to illustrate the problem. A full dump can be found at the following URL: http://db.org/temp/solr.xml.txt Full text search is configures such: plugin { fts = solr fts_solr = break-imap-search url=http://127.0.0.1:8080/solr/ } So; am I missing something, or is this a Dovecot problem? dovecot.conf: http://db.org/temp/dovecot.conf Regards, B?rd Johannessen From tss at iki.fi Mon Mar 26 17:45:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:45:45 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332773145.26095.121.camel@innu> On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server gives > an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response - > maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. From tss at iki.fi Mon Mar 26 17:47:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:47:15 +0300 Subject: [Dovecot] dovecot.log warning In-Reply-To: References: Message-ID: <1332773235.26095.122.camel@innu> On Mon, 2012-03-26 at 15:21 +0530, Jitendra Bhaskar wrote: > Hi Guys, > > Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data > i.e home directory and mail data from previous server which was on dovecot > 1.2.8 to new one. Every thing is working fine. Mails are going and coming. > But in dovecot.log file I an getting like > > imap(user at example.com):Warning: fscking index file /home/ > example.com/user/mail/.imap/VISA/dovecot.index > > Can any one specify why its coming and how can I fix it ? This warning should not exist alone. Isn't there anything else logged? From tomislav.mihalicek at gmail.com Mon Mar 26 17:48:13 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 07:48:13 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap Message-ID: <33544743.post@talk.nabble.com> Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't create namespace 'Share/' for user miha-share at example.com: userdb didn't return a home directory, but location used it (%h): maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u Where is the problem, the same config worked with Dovecot 1.2.x -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544743.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Mon Mar 26 18:01:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:01:18 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <1332774078.26095.124.camel@innu> On Sun, 2012-03-25 at 00:46 -0700, Joseph Tam wrote: > Subject: Different user messages clustered around the same date.saved value > > After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." > to iterate through all user trash folders and expunge old messages. > > However, I noticed a strange thing: querying what would have been deleted > > doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d > > showed many date.saved values are clustered around the same > timestamp, even among different user's Trash mailbox. One user's trash > mailbox having the same date.saved is explained by a user deleting a > lot of message at one time, but I can't explain why many different users > would have messages with the same (or closeby) date.saved value. Which mailbox format? With Maildir the date.saved is taken from dovecot.index.cache file, and in some cases that might get dropped. If it does, then it fallbacks to using the file's ctime. From tss at iki.fi Mon Mar 26 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:02:59 +0300 Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <1332774179.26095.125.camel@innu> On Mon, 2012-03-26 at 07:48 -0700, Tomislav Mihalicek wrote: > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x I'm guessing it didn't work properly with v1.2. Anyway, these would help giving suggestions: 1. dovecot -n output 2. Logs with auth_debug=yes and mail_debug=yes enabled From tss at iki.fi Mon Mar 26 18:06:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:06:58 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: Message-ID: <1332774418.26095.126.camel@innu> On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > This could easily be me missing something, but I can't seem to get the > fts-solr plugin to index message bodies. What Dovecot version? From Attila.Sipos at netcall.com Mon Mar 26 18:13:29 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:13:29 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332773145.26095.121.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 15:46 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server > gives an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response > - maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:18:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:18:46 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332774418.26095.126.camel@innu> References: <1332774418.26095.126.camel@innu> Message-ID: <1332775126.26095.127.camel@innu> On Mon, 2012-03-26 at 18:06 +0300, Timo Sirainen wrote: > On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > > This could easily be me missing something, but I can't seem to get the > > fts-solr plugin to index message bodies. > > What Dovecot version? Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This should fix it: http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 From tss at iki.fi Mon Mar 26 18:22:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:22:59 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332775379.26095.130.camel@innu> It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response if > > the email with the supplied UID had already been known to be copied > > successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From lukas.mueller at newmedia.ch Mon Mar 26 18:25:54 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Mon, 26 Mar 2012 15:25:54 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: Thanks for the quick answer. I realised, that the error didn't occur since quite a while, opposed to what our client suggested. Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. Is it possible that those could have improved the situation? For now I don't have a way of reproducing the problem, so I will have to wait for an error to happen. Until then I will consider the problem as "Solved until happens again ;-) ", since the last error occured a while back (as mentioned above). >> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. >> My suspicion/speculation what happens is the following: >> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >> Somehow this leads to problems with Locks on NFS, which leads to the crash. >Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). I had a quick look a tour NFS (NetApp), but didn't find anything useful. In case the problem persists, I will check with the coworker responsible for NetApp. I will check what sort of locking is used by postfix, since I'm not sure if postfix and dovecot are configured to use the same mechanisms. If not, I think it could be part of the Problem. >> I have no idea how to solve this problem and any help is greatly appreciated. >The only way to fully fix this is: http://wiki2.dovecot.org/Director Unfortunately that is not an option right now, but I will keep it in mind. Thanks again. From Attila.Sipos at netcall.com Mon Mar 26 18:30:24 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:30:24 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332775379.26095.130.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> OK, clearly I am not experienced enough in IMAP to argue. I am sure the imap-protocol people will tell me to get lost! Thanks for your time. If you could possibly tell me how to know if an IMAP "UID COPY" is successful, I would appreciate it. Basically I'm moving a message from one folder to another. I thought I could issue a COPY command, check for success, then delete the email from the source folder. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:23 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" > > response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response > > if the email with the supplied UID had already been known to be > > copied successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : > 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 > 1BB > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:38:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:38:26 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332776306.26095.139.camel@innu> On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. From Attila.Sipos at netcall.com Mon Mar 26 18:51:19 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:51:19 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332776306.26095.139.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> thanks. I find it odd that the 2nd copy attempt returns OK. I would've thought the "expungeissued" reason would still stand. For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:38 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 19:08:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 19:08:44 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332778124.26095.141.camel@innu> Note how Dovecot sent the client EXPUNGE notification. Prior to that client knew that the message existed. After that client knows that the message no longer exists. It was only during the COPY command that client didn't know that the message had already been expunged. On Mon, 2012-03-26 at 16:51 +0100, Attila Sipos wrote: > thanks. > I find it odd that the 2nd copy attempt returns OK. > I would've thought the "expungeissued" reason would still stand. > > For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? > > Regards > Attila > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 16:38 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > > Thanks for your time. If you could possibly tell me how to know if an > > IMAP "UID COPY" is successful, I would appreciate it. > > Basically I'm moving a message from one folder to another. I thought > > I could issue a COPY command, check for success, then delete the email > > from the source folder. > > What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. > > If the client sees that some UID exists, but another session deletes it, the COPY will fail: > > a fetch 1 uid > * 1 FETCH (UID 820) > a OK Fetch completed. > b uid copy 820 Trash > * 1 EXPUNGE > b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. > c uid copy 820 Trash > c OK No messages copied. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From ncjeffgus at zimage.com Mon Mar 26 22:11:40 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:11:40 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6D65DC.7030304@tlinx.org> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> <4F6D65DC.7030304@tlinx.org> Message-ID: <1332789100.28702.7.camel@sally> On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: > Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of > the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can > use large buffers (~16MB gives best results on my local Gbit network), > but it > misses all those pesky acls and extended attrs, not to mention file > perms...*sigh* Compare that to the I/O done 4k at a time by many older > utils... cp -ax: real 0m3.088s user 0m0.034s sys 0m3.054s rsync -axHAX real 0m15.850s user 0m19.314s sys 0m8.816s dsync's time was over six minutes. Each time I cleared out the destination folder. dsync is doing something that is taking much, much, much longer to do. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:25:28 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:25:28 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <1332789928.28702.16.camel@sally> On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > > > That didn't seem to make much of a difference. On a 3.1GB backup it > > shaved off 5 seconds. dsync's time was over 6 minutes with or without > > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > > It seems to me that dsync *should* be able to be just as fast, but it > > currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per > second. Depending on the harddisks used, that would be quite a challenge. > If you use rsync to only transfer the files that changed (based on file > modification time) you may or may not miss files that have changed but > still have the same time stamp. I assume you didn't use the --checksum > parameter to rsync, right? The destination directory was empty. I was doing a full backup. > dsync does so much more than simply copy some files... I realize that. I am hoping that the extra data that dsync has available to it would improve the speed of syncing backups. My baseline testing of simply backing up a mailbox to an empty directory shows that dsync is takes way too long to backup a single mailbox. I have over a terabyte of data to backup. I'm currently using rsync and it must traverse tens of thousands of files and check the time information. It works, but I was hoping dsync would be a better solution. dsync should be able to sync faster, by gulping in the index information for each mailbox. I haven't even moved to the point of sync'ing since the baseline test of simply exporting a mailbox is so slow. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:34:50 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:34:50 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <1332790490.28702.23.camel@sally> On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: > > I personally think that Dovecot could gain much more if the amount of > work going into fixing or improving dsync would go into making Dovecot > to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences > compared to the "low latency" and consistency problem free local file > systems, but its fruits are also sweeter for the long term. :) Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. I think a solid, fast dsync implementation would be very useful for a large installation. ...Jeff From tomislav.mihalicek at gmail.com Tue Mar 27 00:08:36 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 14:08:36 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <33544762.post@talk.nabble.com> When i put service=lib-storage to users in ldap everything works. Is this a bug? cartman dovecot: auth: Debug: master in: USER 1 user at example.net service=lib-storage Tomislav Mihalicek wrote: > > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x > -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544762.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Tue Mar 27 00:14:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 27 Mar 2012 00:14:17 +0300 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > > Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Fixed: http://hg.dovecot.org/dovecot-2.1/rev/339b1337aab0 From andrei.michescu at miau.ca Tue Mar 27 01:14:22 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Mon, 26 Mar 2012 18:14:22 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Hello Timo, Thank you very much for planning a redesign of the dsycn and for opening this discussion. As I can see from the replies that came until now everybody misses the main point of IMAP: IMAP has been designed to work as a disconnected, high-latency data store. To make this more clear: once and IMAP client finishes the synchronization with the server, both have client and server have a consistent state of the mailbox. After this both the "client" and the "server" act like master for their own local copy (on the "server" new emails get created etc, on the "client" existing emails get changed (flags) and moved, and new emails appear (sent items)). So the protocol is designed, originally, to handle the master-master replication. And as this it make sense a deployment global-wide, where servers work independently and from time to time they "merge" the changes. This being said and acknowledged here are my 2 cents: I think that the current '1 brain / 2 workers' seems to be the correct model. The "the client" connects to the "server" and pushes the local changes and after retrieves the updated/new items from the "server". "The brain" considers first server as the "local storage" and the second server as "server storage". For the split design, "come to the same conclusion of the state" is very race-condition prone. As long as the algorithm is kept as you described it in the original document then the backups should really be incremental (because you only do the changes since last sync). As the most changes are "metadata-only" the sync can be pretty fast by merging indexes. Thank you, Andrei > In case anyone is interested in reading (and maybe helping!) with a dsync > redesign that's intended to fix all of its current problems, here are some > possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising > as an algorithm class problem :) If anyone has thoughts on how to solve > it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but > those should be much simpler. > > > !DSPAM:4f6cea4c260302917022693! > > From abruce at tumnus.co.nz Tue Mar 27 03:57:04 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Tue, 27 Mar 2012 13:57:04 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage Message-ID: Hi there, We're setting up a Dovecot virtual email setup - we've got everything working perfect with LDAP logins authenticating against AD and so forth, but we're having issues with retrieving the maxStorage value from AD (this is a pre-setup field in AD that we'd like to use to set per user quotas). In our LDAP lookup, we have the maxStorage entry listed under user_attrs for the quota (user_attrs = maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see it trying to get the entry, but it fails with: Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=maxStorage Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no fields returned by the server At this point, we then see the default quota applied. If we change the name of the field from maxStorage to instanceType we see the value show up in the logs and passed through to the quota system and applied successfully: Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=instanceType Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: instanceType(quota_rule=*:storage=%$M)=*:storage=4M Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 username at site quota_rule=*:storage=4M Which seems a bit weird. If we use ldapsearch and pass it the same search string and look for the field maxStorage, we clearly see the field and the value being returned. The result looks the same if we also lookup instanceType. We're using Dovecot 2.0.9. Does anyone have any idea as to why we can't use this field? Thanks, Andrew From jtam.home at gmail.com Tue Mar 27 04:16:24 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 26 Mar 2012 18:16:24 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >> However, I noticed a strange thing: querying what would have been >> deleted >> >> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >> >> showed many date.saved values are clustered around the same timestamp, >> even among different user's Trash mailbox. >> ... >> I can't explain why many different users would have messages with the >> same (or closeby) date.saved value. > > Which mailbox format? With Maildir the date.saved is taken from > dovecot.index.cache file, and in some cases that might get dropped. If > it does, then it fallbacks to using the file's ctime. mbox. A further look into this reveals that the clustered date.saved values are the earliest values for every mailbox in the system. This timestamp is close to the time I was testing "doveadm ... -A", so the likely explanation is that I accidentally deleted/updated these values using some variation of doveadm, even though I remember confining my testing to query/search/fetch. This appears to be a case of PEBKAC. These "wrong" values shouldn't cause problems with expunge queries since they err on the side of safety. Thanks for the insight though. Joseph Tam From koshikov at gmail.com Tue Mar 27 09:14:25 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Tue, 27 Mar 2012 09:14:25 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: Message-ID: <20120327091425.73963576@jimbo> On Tue, 27 Mar 2012 13:57:04 +1300 Bruce, Andrew wrote: > Hi there, > > We're setting up a Dovecot virtual email setup - we've got everything > working perfect with LDAP logins authenticating against AD and so > forth, but we're having issues with retrieving the maxStorage value > from AD (this is a pre-setup field in AD that we'd like to use to set > per user quotas). > > In our LDAP lookup, we have the maxStorage entry listed under > user_attrs for the quota (user_attrs = > maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > it trying to get the entry, but it fails with: > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=maxStorage > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > fields returned by the server > > At this point, we then see the default quota applied. > Try to change your quota rule to be like: maxStorage=quota_rule=*:bytes=%$ ^^^^^^^^^ And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > > If we change the name of the field from maxStorage to instanceType we > see the value show up in the logs and passed through to the quota > system and applied successfully: > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=instanceType > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > instanceType(quota_rule=*:storage=%$M)=*:storage=4M > Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 > username at site quota_rule=*:storage=4M > > > Which seems a bit weird. > > If we use ldapsearch and pass it the same search string and look for > the field maxStorage, we clearly see the field and the value being > returned. The result looks the same if we also lookup instanceType. > > We're using Dovecot 2.0.9. > > Does anyone have any idea as to why we can't use this field? > > Thanks, > > Andrew From luca.palazzo at unict.it Tue Mar 27 09:57:32 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 27 Mar 2012 08:57:32 +0200 Subject: [Dovecot] 2.1.2 Corrupted squat uidlist Message-ID: <4F7164DC.7010706@unict.it> Hi Timo and All, after upgrading to 2.1.2 i'm getting a lot of these messages: Error: Corrupted squat uidlist file XXXXXX wrong indexid I did not have them before. Ideas? Luca From bob at db.org Tue Mar 27 10:12:42 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Tue, 27 Mar 2012 09:12:42 +0200 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332775126.26095.127.camel@innu> References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: 2012/3/26 Timo Sirainen : > Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This > should fix it: > > http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 Nope; exactly same result; body field contains just the empty line. -- B?rd Johannessen From nmilas at noa.gr Tue Mar 27 11:13:35 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 27 Mar 2012 11:13:35 +0300 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C5741.3000408@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> <4F6C5741.3000408@univ-evry.fr> Message-ID: <4F7176AF.6000607@noa.gr> On 23/3/2012 12:58 ??, Alain DEFRANCE wrote: > so if i understand correctly i can mix the 2 quota_rule ? > the one who came from ldap user_attrs (quota_rule=*:bytes=%$) > and the other which from quota_rule2 = Trash:storage=+3%% Actually, in user_attrs you define the applicable ldap attributes and associated info. Rules are specified in the plugin {} section, but quota values may be overridden by ldap attribute values (but not for Trash). > in your case you add 3% quota more for Trash ? > Am i write ? Yes. Nick From janfrode at tanso.net Tue Mar 27 12:47:10 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 27 Mar 2012 11:47:10 +0200 Subject: [Dovecot] doveadm purge on clusterfs Message-ID: <20120327094710.GA10878@dibs.tanso.net> Since doveadm service proxying apparently doesn't work with dovecot v2.0, we need to find a way to safely run doveadm purge on the host the user is logged into. Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? -jf From pw at wk-serv.de Tue Mar 27 13:11:59 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Tue, 27 Mar 2012 12:11:59 +0200 Subject: [Dovecot] Merge mails from two mail_locations Message-ID: <4F71926F.30500@wk-serv.de> Hi guys, recently I had some trouble with my ocfs2 cluster and it unmounted itself from /var/mail. Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. Regards Patrick From jacek at hapay.pl Tue Mar 27 14:20:14 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:20:14 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time Message-ID: <4F71A26E.5030400@hapay.pl> Hi all, I want to start new server with Postfix (I still have qmail ) and I think I have a problem with authentication in dovecot - it takes 3 seconds. Is this normal time? My configuration: Usernames: from MySQL Passwords: from Active Directory dovecot -n # 1.1.20: /etc/dovecot.conf # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 log_path: /var/log/dovecot.log protocols: pop3 pop3s imap imaps listen: * ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem ssl_key_file: /etc/pki/tls/private/iRedMail.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_uid: 500 mail_gid: 500 mail_location: maildir:/xxx/%u:INDEX=/xxx/%u mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota zlib mail_plugins(imap): quota imap_quota zlib mail_plugins(pop3): quota zlib mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 lda: postmaster_address: postmaster at domain.com mail_plugins: cmusieve quota log_path: /var/log/sieve.log auth default: mechanisms: plain login default_realm: infor.pl username_format: %Lu debug: yes debug_passwords: yes passdb: driver: pam passdb: driver: sql args: /etc/dovecot-ldap.conf userdb: driver: sql args: /etc/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/dovecot-auth mode: 438 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 438 user: vmail group: vmail plugin: quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 quota: maildir quota_rule: *:storage=300M quota_rule2: *:messages=0 expire: Trash 7 Trash/* 7 Junk 30 expire_dict: proxy::expire auth_socket_path: /var/run/dovecot/auth-master sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve dict: expire: db:/xxx/expire.db grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf driver = mysql default_pass_scheme = CRYPT connect = host=localhost dbname=xxx user=xxx password=xxx #password_query = SELECT password FROM mailbox WHERE username='%u' AND active='1' user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' postfix: postfix-2.5.9-5.ired dovecot: dovecot-1.1.20-1_98.el5 mysql: mysql-server-5.0.77-4.el5_6.6 This is not a TCP connection problem, because i have results from tcpdump and wireshark. There is a information that Active directory is answering with password in 0,2 second. Regards Jacek From jacek at hapay.pl Tue Mar 27 14:39:32 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:39:32 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time [SOLVED] In-Reply-To: <4F71A26E.5030400@hapay.pl> References: <4F71A26E.5030400@hapay.pl> Message-ID: <4F71A6F4.7030203@hapay.pl> W dniu 27.03.2012 13:20, Jacek Kowalski pisze: > Hi all, > > I want to start new server with Postfix (I still have qmail ) and I > think I have a problem with authentication in dovecot - it takes 3 > seconds. Is this normal time? > > My configuration: > Usernames: from MySQL > Passwords: from Active Directory > > > dovecot -n > > # 1.1.20: /etc/dovecot.conf > # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 > log_path: /var/log/dovecot.log > protocols: pop3 pop3s imap imaps > listen: * > ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem > ssl_key_file: /etc/pki/tls/private/iRedMail.key > login_dir: /var/run/dovecot/login > login_executable(default): /usr/libexec/dovecot/imap-login > login_executable(imap): /usr/libexec/dovecot/imap-login > login_executable(pop3): /usr/libexec/dovecot/pop3-login > mail_uid: 500 > mail_gid: 500 > mail_location: maildir:/xxx/%u:INDEX=/xxx/%u > mail_executable(default): /usr/libexec/dovecot/imap > mail_executable(imap): /usr/libexec/dovecot/imap > mail_executable(pop3): /usr/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota zlib > mail_plugins(imap): quota imap_quota zlib > mail_plugins(pop3): quota zlib > mail_plugin_dir(default): /usr/lib/dovecot/imap > mail_plugin_dir(imap): /usr/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 > lda: > postmaster_address: postmaster at domain.com > mail_plugins: cmusieve quota > log_path: /var/log/sieve.log > auth default: > mechanisms: plain login > default_realm: infor.pl > username_format: %Lu > debug: yes > debug_passwords: yes > passdb: > driver: pam > passdb: > driver: sql > args: /etc/dovecot-ldap.conf > userdb: > driver: sql > args: /etc/dovecot-mysql.conf > socket: > type: listen > client: > path: /var/spool/postfix/dovecot-auth > mode: 438 > user: postfix > group: postfix > master: > path: /var/run/dovecot/auth-master > mode: 438 > user: vmail > group: vmail > plugin: > quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 > quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 > quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 > quota: maildir > quota_rule: *:storage=300M > quota_rule2: *:messages=0 > expire: Trash 7 Trash/* 7 Junk 30 > expire_dict: proxy::expire > auth_socket_path: /var/run/dovecot/auth-master > sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve > dict: > expire: db:/xxx/expire.db > > > > grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf > > driver = mysql > default_pass_scheme = CRYPT > connect = host=localhost dbname=xxx user=xxx password=xxx > #password_query = SELECT password FROM mailbox WHERE username='%u' AND > active='1' > user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, > '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule > FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' > > > postfix: postfix-2.5.9-5.ired > dovecot: dovecot-1.1.20-1_98.el5 > mysql: mysql-server-5.0.77-4.el5_6.6 > > > > This is not a TCP connection problem, because i have results from > tcpdump and wireshark. There is a information that Active directory is > answering with password in 0,2 second. > > > Regards > > Jacek Ok. Problem Solved. It was "passdb pam" problem. Regards Jacek From campbell at cnpapers.com Tue Mar 27 17:40:11 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 27 Mar 2012 10:40:11 -0400 Subject: [Dovecot] Namespace, prefix questions Message-ID: <4F71D14B.2010301@cnpapers.com> We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) disable_plaintext_auth = no listen = * mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mbox_write_locks = fcntl namespace { hidden = yes inbox = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = separator = / type = private } namespace { hidden = yes list = no location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = "#mbox/" separator = / type = private } namespace { hidden = yes list = no location = prefix = mail/ separator = / type = private } namespace { hidden = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = ~/mail/ separator = / type = private } namespace { hidden = yes list = no location = prefix = ~%u/mail/ separator = / type = private } passdb { driver = pam } protocols = pop3 imap ssl_cert = Message-ID: <87obrixcyp.fsf@algae.riseup.net> Timo Sirainen writes: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: thank you for opening this discussion about dsync! besides the problems I've encountered with dsync, there are a couple things that I think would be great to build into the new vision of the protocol. One would be the ability to perform *intelligent* incremental/rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it would be more intelligent if this were baked into the backup process. Secondly, being able to filter out mailboxes could result in much more efficient syncing. Now there is the capability to operate on only specific mailboxes, but this doesn't scale well when I am trying to backup thousands of users and I want to omit the Spam and Trash folders from the sync. I would have to get a mailbox list of each user, and then iterate over each mailbox for each user, skipping the Spam and Trash folders, forking a new 'dsync backup' for each of their mailboxes, for each user. Lastly, there isn't a good method for restoring backups. I can reverse the backup process, onto the user's "live" mailbox, but that brings the user into an undesirable state (eg. their mailbox state one day ago). Better would be if their backup could be restored in such a way that the user can resolve the missing pieces manually, as they know best. thanks again for your work on this, from my position dovecot is an amazing piece of software, the only part that seems to have some issues is dsync and I applaud the effort to redesign to fix things! micah From me at benschumacher.com Tue Mar 27 19:49:50 2012 From: me at benschumacher.com (Ben Schumacher) Date: Tue, 27 Mar 2012 10:49:50 -0600 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: > the zlib_save question reminds me of a wish: > I think it's not possible to set zlib_save parameter per namespace (or even > mailbox). Per namespace would be something for the wish list to get rid of > the cron job method to compress archival mailboxes. > And maybe an option to add a "Z" flag to compressed maildir message files > as recommended in the wiki regarding compress crob job. +1 on this request. I have a slightly different use case -- I have both an dbox and Maildir. Incoming email goes to Maildir, but I archive off to dbox (using Thunderbird). After I archive my emails, compression seems like a reasonable choice. Any idea if this feature will be available at some point? Thanks, Ben From lists at wiesinger.com Tue Mar 27 20:28:56 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:28:56 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F8D8.6040700@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From gerhard at wiesinger.com Tue Mar 27 20:27:59 2012 From: gerhard at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:27:59 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F89F.9060903@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From CMarcus at Media-Brokers.com Tue Mar 27 22:34:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 27 Mar 2012 15:34:40 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <87obrixcyp.fsf@algae.riseup.net> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <87obrixcyp.fsf@algae.riseup.net> Message-ID: <4F721650.4030901@Media-Brokers.com> On 2012-03-27 11:47 AM, Micah Anderson wrote: > One would be the ability to perform *intelligent* incremental / > rotated backups. I can do this now by running a dsync backup > operation and then doing manual hardlinking or moving of the backup > directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it > would be more intelligent if this were baked into the backup process. There are already numerous tools that do this flawlessly - I've been using rsnapshot (which uses rsync) for this for years. I don't know if Timo should be spending his time reinventing the wheel. I'm much more interested in dsync working flawlessly to keep one or more secondary servers in sync, and leave backups to backup software. > Lastly, there isn't a good method for restoring backups. I can reverse > the backup process, onto the user's "live" mailbox, but that brings the > user into an undesirable state (eg. their mailbox state one day > ago). Better would be if their backup could be restored in such a way > that the user can resolve the missing pieces manually, as they know > best. Again, best left to the backup software I think? Although, one interesting piece that I am hopeful I'll be able to implement soon (with Timo's professional help) is the ability to easily and automatically map my rsnapshot snapshots directory to a read-only 'Backups' namespace that automatically shows the snapshots by date and time as they are produced. This way users could 'go back in time' anytime they wanted without having to call me... :) > thanks again for your work on this, from my position dovecot is an > amazing piece of software, the only part that seems to have some issues > is dsync and I applaud the effort to redesign to fix things! Ditto all of that! :) -- Best regards, Charles From stan at hardwarefreak.com Tue Mar 27 23:09:44 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 27 Mar 2012 15:09:44 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332790490.28702.23.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> Message-ID: <4F721E88.8020309@hardwarefreak.com> On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > Do you have any suggestions for a distributed replicated filesystem > that works well with dovecot? I've looked into glusterfs, but the > latency is way too high for lots of small files. They claim this problem > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > see how any of the distributed filesystems would help me. I've also > tried out ZFS, but it appears to have issues with metadata look ups with > directories that have tens or hundreds of thousands of files in them. > For me, the best filesystem is straight up ext4 running on locally > attached storage. > I think a solid, fast dsync implementation would be very useful for a > large installation. It sounds like you're in need of a more robust and capable storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or incremental snapshot capability. Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands. It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput by a factor of 2-50+ depending on thread count, and decreases IOPS and MB/s hitting the storage by about the same factor, depending on thread count. Before this patch XFS sucked at the write portion of the maildir workload due to the extremely high IOPS and MB/s hitting just the log journal, not including the actual file writes. It's parallel maildir read performance was better than any other, but the write was so bad it bogged down the storage producing high latency for everything. With the delaylog patch, XFS now trounces every filesystem at medium to high parallelism levels. Delaylog was introduced in mid 2009, included in 2.6.35 as experimental, and is the default in 2.6.39 and later. If you're a Red Hat or CentOS user it's included in 6.2. This one patch, which was 5+ years in development, dramatically changed the character of XFS with this class of metadata intensive parallel workloads. Many people with such a workload who ran from XFS in the past, as if it were the Fukushima reactor, are now adopting it in droves. What a difference a few hundred lines of very creative code can make... -- Stan From abruce at tumnus.co.nz Tue Mar 27 23:39:37 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 09:39:37 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120327091425.73963576@jimbo> References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:36, Bruce, Andrew wrote: > On 27 March 2012 19:14, Nikita Koshikov wrote: >> On Tue, 27 Mar 2012 13:57:04 +1300 >> Bruce, Andrew wrote: >> >> Hi there, >> >> We're setting up a Dovecot virtual email setup - we've got everything >> working perfect with LDAP logins authenticating against AD and so >> forth, but we're having issues with retrieving the maxStorage value >> from AD (this is a pre-setup field in AD that we'd like to use to set >> per user quotas). >> >> In our LDAP lookup, we have the maxStorage entry listed under >> user_attrs for the quota (user_attrs = >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >> it trying to get the entry, but it fails with: >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=maxStorage >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >> fields returned by the server >> >> At this point, we then see the default quota applied. >> > Try to change your quota rule to be like: > maxStorage=quota_rule=*:bytes=%$ > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >> >> If we change the name of the field from maxStorage to instanceType we >> see the value show up in the logs and passed through to the quota >> system and applied successfully: >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=instanceType >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >> username at site ? ?quota_rule=*:storage=4M >> >> >> Which seems a bit weird. >> >> If we use ldapsearch and pass it the same search string and look for >> the field maxStorage, we clearly see the field and the value being >> returned. ?The result looks the same if we also lookup instanceType. >> >> We're using Dovecot 2.0.9. >> >> Does anyone have any idea as to why we can't use this field? >> >> Thanks, >> >> Andrew Tried your suggestion Nikita, no joy unfortunately. ?It still looks like the value never gets returned from the LDAP server to Dovecot. It definitely has something in the field (equivalent of 10GB, but in bytes as suggested) and I changed the user_attrs also, but still get the same "no fields returned by the server" error message. Modifying the user_attrs to lookup from a different field (instanceType) definitely works. What exact version are you using - perhaps it's a problem with our copy of 2.0.9. Thanks, Andrew From ncjeffgus at zimage.com Tue Mar 27 23:57:41 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Tue, 27 Mar 2012 13:57:41 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F721E88.8020309@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> Message-ID: <1332881861.29480.8.camel@sally> On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: > On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > > > Do you have any suggestions for a distributed replicated filesystem > > that works well with dovecot? I've looked into glusterfs, but the > > latency is way too high for lots of small files. They claim this problem > > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > > see how any of the distributed filesystems would help me. I've also > > tried out ZFS, but it appears to have issues with metadata look ups with > > directories that have tens or hundreds of thousands of files in them. > > For me, the best filesystem is straight up ext4 running on locally > > attached storage. > > It sounds like you're in need of a more robust and capable > storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or > incremental snapshot capability. We do have a FC system that another department is using. The company dropped quite a bit of cash on it for a specific purpose. Our department does not have access it to. People are somewhat afraid of iSCSI around here because they believe it will add too much latency to the overall IO performance. They're a big believer in locally attached disks. Less features, but very good performance. We thought ZFS would provide us with a nice snapshot and backup system (with zfs send). We never got that far once we discovered that ZFS doesn't work very well in this context. Running rsync on it gave us terrible performance. > Also, you speak of a very large maildir store, with hundreds of > thousands of directories, obviously many millions of files, of 1TB total > size. Thus I would assume you have many thousands of users, if not 10s > of thousands. > > It's a bit hard to believe you're not running XFS on your storage, given > your level of parallelism. You'd get much better performance using XFS > vs EXT4. Especially with kernel 2.6.39 or later which includes the > delayed logging patch. This patch increases metadata write throughput > by a factor of 2-50+ depending on thread count, and decreases IOPS and > MB/s hitting the storage by about the same factor, depending on thread > count. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. ...Jeff From abruce at tumnus.co.nz Wed Mar 28 00:06:55 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 10:06:55 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:39, Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: >> On 27 March 2012 19:14, Nikita Koshikov wrote: >>> On Tue, 27 Mar 2012 13:57:04 +1300 >>> Bruce, Andrew wrote: >>> >>> Hi there, >>> >>> We're setting up a Dovecot virtual email setup - we've got everything >>> working perfect with LDAP logins authenticating against AD and so >>> forth, but we're having issues with retrieving the maxStorage value >>> from AD (this is a pre-setup field in AD that we'd like to use to set >>> per user quotas). >>> >>> In our LDAP lookup, we have the maxStorage entry listed under >>> user_attrs for the quota (user_attrs = >>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>> it trying to get the entry, but it fails with: >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=maxStorage >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>> fields returned by the server >>> >>> At this point, we then see the default quota applied. >>> >> Try to change your quota rule to be like: >> maxStorage=quota_rule=*:bytes=%$ >> ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ >> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >> >> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>> >>> If we change the name of the field from maxStorage to instanceType we >>> see the value show up in the logs and passed through to the quota >>> system and applied successfully: >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=instanceType >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >>> username at site ? ?quota_rule=*:storage=4M >>> >>> >>> Which seems a bit weird. >>> >>> If we use ldapsearch and pass it the same search string and look for >>> the field maxStorage, we clearly see the field and the value being >>> returned. ?The result looks the same if we also lookup instanceType. >>> >>> We're using Dovecot 2.0.9. >>> >>> Does anyone have any idea as to why we can't use this field? >>> >>> Thanks, >>> >>> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Further investigation shows that there are a few other fields that we can't retrieve in Dovecot, but can using the same search string and lookup user with ldapsearch. maxStorage is obviously one, but I tried a couple of other fields of varying types: mobile - Octet String and logonCount - Integer. Doesn't seem to be the type that restricts the search, just some fields won't return. From sorr at rightnow.com Wed Mar 28 00:38:19 2012 From: sorr at rightnow.com (Orr, Steve) Date: Tue, 27 Mar 2012 21:38:19 +0000 Subject: [Dovecot] Using getmail with sieve Message-ID: <1AA32A754D17E9478500E421F4099F9D1D13A5E8@IS-BOZ-MB02.corp.rightnow.com> I'm trying to setup a personal "mailmover" where I use getmail to retrieve remote IMAP server mail and load it into my local Dovecot then filter all email on the Dovecot server side with sieve. (I'm using Dovecot v. 2.0.9 with IMAP/Maildir.) 1) From the docs I gather that Dovecot sieve will not work as mail is loaded by getmail and I need to refilter the email after it has been loaded with getmail, right? 2) In http://wiki2.dovecot.org/HowTo/RefilterMail it says, "This HOWTO helps you create a folder for mail that needs refiltering..." But contrary to the doc I don't see any actual instructions about creating said folder. The HOWTO assumes mail already exists in a folder called "REFILTER' but doesn't say how this folder was populated. "How to?" 3) As a test I manually created the "REFILTER" folder and put my inbox mail in it (with Thunderbird) then following the HOWTO I ran my getmail script and my sieve script performed as expected. 4) How do I automate this? I tried a new getmail script to run against the inbox and tag new email so a subsequent getmail script could move it into the REFILTER folder for later sieve processing. This seems like way too much work. The HOWTO doc seems incomplete or presumes other knowledge, especially for an IMAP server and email sieve nubie. TIA, D. B. ---------------------------------------------- $ dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-131.0.15.el6.i686 i686 Red Hat Enterprise Linux Server release 6.1 (Santiago) log_path = /var/log/dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = test at host1 protocols = imap lmtp service lmtp { user = test } ssl_cert = I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that is optimized to convert mailboxes without the fancy locking that dsync needs to do. Does have (or seen) a tool that could do this? We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process. ...Jeff From stonegate at stonegate.homeip.net Wed Mar 28 01:24:59 2012 From: stonegate at stonegate.homeip.net (stonegate) Date: Tue, 27 Mar 2012 15:24:59 -0700 (PDT) Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook Message-ID: <33544803.post@talk.nabble.com> Hi, i use dovecot 2.1.3 on a gentoo system. Before i installed my new imap server box, everything worked. Now with Dovecot it does not. Problem: When i receive a new email, it does not appear in my Outlook unless i have the IMAP Inbox Folder open (highlighted selection). Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize it unless i click on the inbox folder. Before that problem occured on my old system ( i think it was dovecot as well ) the inbox folder refreshed automatically and i instantly knew when i had new emails. It was kind of like with my exchange account. Does anyone have a clue what might be wrong ? Since i did not re-install outlook or something it cannot be a client thing since no settings changed on the client side. Maybe i have to do something with dovecots config file? I think it should keep the connection to the server open or something. I?d appreciate any kind of help. Thanks alot Stoney Dovecot is capable of the following things: telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. Thats my dovecot.conf: # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-gentoo-r2 x86_64 Gentoo Base System release 2.0.3 auth_mechanisms = plain login listen = * disable_plaintext_auth = no mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } #passdb { # args = * # driver = pam #} #passdb { # args = /etc/dovecot/dovecot-sql.conf.ext # driver = sql #} passdb { driver = shadow } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = References: <1332888019.29480.17.camel@sally> Message-ID: <4F727ECE.4050305@r.paypc.com> On 3/27/2012 3:40 PM, Jeff Gustafson wrote: > I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. Is it slower than doing an IMAP APPEND over an authenticated dovecot connection? I've used a simple PERL script based on Mail::IMAPClient and Mail::Box to import 180,000+ mailboxes into dovecot's mdbox at fairly high speed, and all it does is IMAP APPENDs. (I had to shard the mailboxes because these PERL based tools exhaust RAM when run with mailboxes larger than about 600MB). On my development VM test box (32 bit Slack 13.37, 2G/2G split kernel, no RAID, Q6600 with only two cores allocated to the VM) and 8GB of DDR2 RAM does Emails=180,044 real 237m28.485s (12.5 emails/second) user 94m50.425s sys 10m09.389s 21,984,824 /mail/home I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox "transcoding" tool, since none appear to exist. To keep it simple, I/O to/from "remote" mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's worth of RAM (if one of the directions involves a remote connection), and so far when processing MIX, Maildir, and Mbox files, it's extremely fast. Adding support for [sm]dbox wouldn't appear to be problematic. At the moment, it supports everything Panda's c-client supports plus Maildir/Maildir++ (including Panda's "MIX"). Write support for Maildir's extremely UNDER-tested so far, as I've mainly used it to import Maildir hives. I've experimented with Maildir as a format, and while the one email to a file model seems like a sensible idea, it seems to simply transfer stress from one part of the system to another, mainly filesystems, and not many of those are really up for handling that many files in one directory very efficiently. None of my users have mailboxes with fewer than 100K emails in them, some have more than a million. =R= From koshikov at gmail.com Wed Mar 28 09:25:34 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Wed, 28 Mar 2012 09:25:34 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: <20120328092534.5690fa40@jimbo> On Wed, 28 Mar 2012 09:39:37 +1300 Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: > > On 27 March 2012 19:14, Nikita Koshikov wrote: > >> On Tue, 27 Mar 2012 13:57:04 +1300 > >> Bruce, Andrew wrote: > >> > >> Hi there, > >> > >> We're setting up a Dovecot virtual email setup - we've got everything > >> working perfect with LDAP logins authenticating against AD and so > >> forth, but we're having issues with retrieving the maxStorage value > >> from AD (this is a pre-setup field in AD that we'd like to use to set > >> per user quotas). > >> > >> In our LDAP lookup, we have the maxStorage entry listed under > >> user_attrs for the quota (user_attrs = > >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > >> it trying to get the entry, but it fails with: > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=maxStorage > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > >> fields returned by the server > >> > >> At this point, we then see the default quota applied. > >> > > Try to change your quota rule to be like: > > maxStorage=quota_rule=*:bytes=%$ > > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > >> > >> If we change the name of the field from maxStorage to instanceType we > >> see the value show up in the logs and passed through to the quota > >> system and applied successfully: > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=instanceType > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M > >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 > >> username at site ? ?quota_rule=*:storage=4M > >> > >> > >> Which seems a bit weird. > >> > >> If we use ldapsearch and pass it the same search string and look for > >> the field maxStorage, we clearly see the field and the value being > >> returned. ?The result looks the same if we also lookup instanceType. > >> > >> We're using Dovecot 2.0.9. > >> > >> Does anyone have any idea as to why we can't use this field? > >> > >> Thanks, > >> > >> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Show your full dovecot-ldap.conf file, also what port do you using ? maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) And show exact result of ldapsearch tool, binding under user from dovecot-ldap.conf + debug for this user when it trying to login and 'doveadm -D quota get -u $user' for this one. Also ensure that your search query returns only 1 result. We are using dovecot 2.0.19 now, but all versions of dovecot 2.0 branch was there in the past. I'm updating server since version 2.0.1 - no problem found. From janfrode at tanso.net Wed Mar 28 10:24:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 28 Mar 2012 09:24:07 +0200 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <1332888019.29480.17.camel@sally> References: <1332888019.29480.17.camel@sally> Message-ID: On Wed, Mar 28, 2012 at 12:40 AM, Jeff Gustafson wrote: > ? ? ? ?I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. I wonder if time could be saved with a > program that is optimized to convert mailboxes without the fancy locking > that dsync needs to do. Does have (or seen) a tool that could do this? Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir->mdbox convertion yesterday, using the attached script. I only converted a little over 10000 easy accounts (accounts with simple folder names, as I expect to run into problems once we start hitting accounts with trailing dot or broken latin1/utf8 characters in the folder names). I might agree it wasn't quick, but that really doesn't matter as the only downtime for the user is that he's potentially kicked out during the userdb update. -jf > ? ? ? ?We're hoping that converting away from Maildir will help us speed up > the backup processes by reducing the number of files to process. > -------------- next part -------------- A non-text attachment was scrubbed... Name: migrer-til-mdbox.sh Type: application/x-sh Size: 2131 bytes Desc: not available URL: From tomislav.mihalicek at gmail.com Wed Mar 28 11:57:07 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Wed, 28 Mar 2012 01:57:07 -0700 (PDT) Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage Message-ID: <33544816.post@talk.nabble.com> Hi Could someone explain what this strings mean in dovecot 2.1.3 debug log? Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 test1 at example.net service=lib-storage Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 test2 at example.net service=lib-storage -- View this message in context: http://old.nabble.com/Shared-mailboxes-with-dovecot-problem-service%3Dlib-storage-tp33544816p33544816.html Sent from the Dovecot mailing list archive at Nabble.com. From mafonso at hangas.net Wed Mar 28 13:13:17 2012 From: mafonso at hangas.net (Hangas) Date: Wed, 28 Mar 2012 10:13:17 +0000 (UTC) Subject: [Dovecot] dbox vs. mdbox References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: Timo Sirainen iki.fi> writes: > > 4. Are there real-world benchmarks showing measurable differences between > >maildir, sdbox mdbox? > > Not that I'm aware of. So far everyone I've tried to ask have replaced their > whole mail system and their storage, so the before/after numbers can't be > compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. The environment will be virtually the same as this is running in a virtualized environment. I'm keeping the same storage and storage network, same host hardware and about the same VM specs. Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information. My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly From campbell at cnpapers.com Wed Mar 28 17:10:32 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 28 Mar 2012 10:10:32 -0400 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: <4F731BD8.8050307@cnpapers.com> On 3/27/2012 10:40 AM, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The > client allows me to specify the root folder, but not a prefix or > namespace. I'm still struggling with some users on our new server that > have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix > definitions on the imap server? > > On some clients, like Thunderbird, I have the option of specifying > namespace OR prefix. How do these differ? I thought that the prefix > was the "name" of the namespace. > > It appears that I have to delete and re-create the account on these OE > 6 clients to make the list of folders show properly. Does that sound > right? > > This all came about because one of these OE 6 users was not able to > use their imap folders (server errors). Turns out it was one of the > users that had their folders directly under ~. So I moved them to > ~/mail, created a .subscriptions file from their .mailboxlist file and > tried everything in the world to get the folders to list properly. > Only after specifying the root folder as ~/mail after recreating the > account and restarting OE did it show properly and the folders > remained listed. My default config has this setup as the > "mail_location" parm, but blanks as the root folder don't seem to work > in this situation. I'm also wondering where I specify the "list", > "hidden" and other parms that are usually set in namespace blocks. > > dovecot -n > # 2.0.9: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) > disable_plaintext_auth = no > listen = * > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > driver = pam > } > protocols = pop3 imap > ssl_cert = ssl_key = userdb { > driver = passwd > } > protocol pop3 { > pop3_uidl_format = %08Xu%08Xv > } > > > Thanks > > steve campbell > > > > Thanks > > steve campbell > > After googling a bit, it seems that all 3 can come into play in the same or different meanings. Seems that prefix and namespace mean the same thing. Root folder can mean the same as above, but can also stand alone as an individual pointer to a personal folder that differs from from what the imap server uses. It's still not clear to me, but at least I'm getting an idea of what may or may not work. Still not sure why the null or blank prefixed namespace doesn't take precedence when nothing is set in the client. steve From kiwi at oav.net Wed Mar 28 18:50:54 2012 From: kiwi at oav.net (Xavier Beaudouin) Date: Wed, 28 Mar 2012 17:50:54 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. Message-ID: <4F73335E.2070800@oav.net> Hi there, I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. One thing used by some of powerusers are archiving mail automatically with autocreated folders based on year + month. Is there any good way to make that with sieve... One example require "fileinto"; if address :is ["From", "To"] "dovecot at dovecot.org" { fileinto "INBOX.mls.%Y.%m.dovecot"; } This will fill any mails into INBOX.mls.2012.03.dovecot uppon receiving... I don't know if some sieve guru can tell me how to do that... ? Kind regards, Xavier From stephan at rename-it.nl Wed Mar 28 19:04:48 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:04:48 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F7336A0.3070202@rename-it.nl> Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? require ["variables","date","fileinto","mailbox"]; # Extract date info if currentdate :matches "year" "*" { set "year" "${1}"; } if currentdate :matches "month" "*" { set "month" "${1}"; } # Archive Dovecot mailing list items by year and month. # Create folder when it does not exist. if header :is "list-id" "dovecot.dovecot.org" { fileinto :create "INBOX.mls.${year}.${month}.dovecot"; } The above also uses a more reliable way to detect the Dovecot mailinglist. Regards, Stephan From stan at hardwarefreak.com Wed Mar 28 19:07:59 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 28 Mar 2012 11:07:59 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332881861.29480.8.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> Message-ID: <4F73375F.3070200@hardwarefreak.com> On 3/27/2012 3:57 PM, Jeff Gustafson wrote: > We do have a FC system that another department is using. The company > dropped quite a bit of cash on it for a specific purpose. Our department > does not have access it to. People are somewhat afraid of iSCSI around > here because they believe it will add too much latency to the overall IO > performance. They're a big believer in locally attached disks. Less > features, but very good performance. If you use a software iSCSI initiator with standard GbE ports, block IO latency can become a problem, but basically in only 3 scenarios: 1. Slow CPUs or not enough CPUs/cores. This is unlikely to be a problem in 2012, given the throughput of today's multi-core CPUs. Low CPU throughput hasn't generally been the cause of software iSCSI initiator latency problems since pre-2007/8 with most applications. I'm sure some science/sim apps that tax both CPU and IO may have still had issues. Those would be prime candidates for iSCSI HBAs. 2. An old OS kernel that doesn't thread IP stack, SCSI encapsulation, and/or hardware interrupt processing amongst all cores. Recent Linux kernels do this rather well, especially with MSI-X enabled, older ones not so well. I don't know about FreeBSD, Solaris, AIX, HP-UX, Windows, etc. 3. System under sufficiently high CPU load to slow IP stack and iSCSI encapsulation processing, and or interrupt handling. Again, with today's multi-core fast CPUs this probably isn't going to be an issue, especially given that POP/IMAP are IO latency bound, not CPU bound. Most people running Dovecot today are going to have plenty of idle CPU cycles to perform the additional iSCSI initiator and TCP stack processing without introducing undue block IO latency effects. As always, YMMV. The simply path is to acquire your iSCSI SAN array and use software initiators on client hosts. In the unlikely event you do run into block IO latency issues, you simply drop an iSCSI HBA into each host suffering the latency. They run ~$700-900 USD each for single port models, and they eliminate block IO latency completely, which is one reason they cost so much. They have an onboard RISC chip and memory doing the TCP and SCSI encapsulation processing. They also give you the ability to boot diskless servers from LUNs on the SAN array. This is very popular with blade server systems, and I've done this many times myself, albeit with fibre channel HBAs/SANs, not iSCSI. Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. > We thought ZFS would provide us with a nice snapshot and backup system > (with zfs send). We never got that far once we discovered that ZFS > doesn't work very well in this context. Running rsync on it gave us > terrible performance. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. >> Also, you speak of a very large maildir store, with hundreds of >> thousands of directories, obviously many millions of files, of 1TB total >> size. Thus I would assume you have many thousands of users, if not 10s >> of thousands. >> >> It's a bit hard to believe you're not running XFS on your storage, given >> your level of parallelism. You'd get much better performance using XFS >> vs EXT4. Especially with kernel 2.6.39 or later which includes the >> delayed logging patch. This patch increases metadata write throughput >> by a factor of 2-50+ depending on thread count, and decreases IOPS and >> MB/s hitting the storage by about the same factor, depending on thread >> count. > > I've relatively new here, but I'll ask around about XFS and see if > anyone had tested it in the development environment. If they'd tested it properly, and relatively recently, I would think they'd have already replaced EXT4 on your Dovecot server. Unless others factors prevented such a migration. Or unless I've misunderstood the size of your maildir workload. -- Stan From gfinch at ldmltd.ca Wed Mar 28 19:08:07 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:08:07 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F733767.60003@ldmltd.ca> On 2012-03-28 8:50 AM, Xavier Beaudouin wrote: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? > > Kind regards, > > Xavier I don't remember where I found out how to do this, but the following is what I use: require ["fileinto", "imap4flags", "date", "variables"]; if currentdate :matches "month" "*" { set "month" "${1}"; } if currentdate :matches "year" "*" { set "year" "${1}"; } fileinto :flags "\\seen" "${year}-${month}"; Deliver/lmtp is set to allow creation of folders. -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From gfinch at ldmltd.ca Wed Mar 28 19:26:25 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:26:25 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F7336A0.3070202@rename-it.nl> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> Message-ID: <4F733BB1.5060804@ldmltd.ca> On 2012-03-28 9:04 AM, Stephan Bosch wrote: > Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: >> Hi there, >> >> I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. >> >> One thing used by some of powerusers are archiving mail automatically >> with autocreated folders based on year + month. >> >> Is there any good way to make that with sieve... >> >> One example >> >> require "fileinto"; >> >> if address :is ["From", "To"] "dovecot at dovecot.org" { >> fileinto "INBOX.mls.%Y.%m.dovecot"; >> } >> >> This will fill any mails into INBOX.mls.2012.03.dovecot uppon >> receiving... >> >> I don't know if some sieve guru can tell me how to do that... ? > > require ["variables","date","fileinto","mailbox"]; > > # Extract date info > if currentdate :matches "year" "*" { set "year" "${1}"; } > if currentdate :matches "month" "*" { set "month" "${1}"; } > > # Archive Dovecot mailing list items by year and month. > # Create folder when it does not exist. > if header :is "list-id" "dovecot.dovecot.org" { > fileinto :create "INBOX.mls.${year}.${month}.dovecot"; > } > > > The above also uses a more reliable way to detect the Dovecot > mailinglist. > > > Regards, > > Stephan Stephan, Is the "mailbox" extension the one that lets "fileinto" use ":create"? I've had a hard time trying to find a useful sieve reference that I can understand. Thank you, -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From stephan at rename-it.nl Wed Mar 28 19:31:31 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:31:31 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F733BB1.5060804@ldmltd.ca> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> <4F733BB1.5060804@ldmltd.ca> Message-ID: <4F733CE3.4050101@rename-it.nl> Op 3/28/2012 6:26 PM, Gregory Finch schreef: > On 2012-03-28 9:04 AM, Stephan Bosch wrote: >> >> require ["variables","date","fileinto","mailbox"]; >> >> # Extract date info >> if currentdate :matches "year" "*" { set "year" "${1}"; } >> if currentdate :matches "month" "*" { set "month" "${1}"; } >> >> # Archive Dovecot mailing list items by year and month. >> # Create folder when it does not exist. >> if header :is "list-id" "dovecot.dovecot.org" { >> fileinto :create "INBOX.mls.${year}.${month}.dovecot"; >> } >> >> >> The above also uses a more reliable way to detect the Dovecot >> mailinglist. > Stephan, > > Is the "mailbox" extension the one that lets "fileinto" use ":create"? > I've had a hard time trying to find a useful sieve reference that I can > understand. > > Thank you, Yes: http://tools.ietf.org/html/rfc5490#section-3.2 You can find links to specifications of the various Sieve extensions implemented for Pigeonhole here: http://pigeonhole.dovecot.org Regards, Stephan. From ncjeffgus at zimage.com Wed Mar 28 23:54:01 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:54:01 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F73375F.3070200@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> Message-ID: <1332968041.26122.19.camel@sally> On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > Locally attached/internal/JBOD storage typically offers the best > application performance per dollar spent, until you get to things like > backup scenarios, where off node network throughput is very low, and > your backup software may suffer performance deficiencies, as is the > issue titling this thread. Shipping full or incremental file backups > across ethernet is extremely inefficient, especially with very large > filesystems. This is where SAN arrays with snapshot capability come in > really handy. I'm a new employee at the company. I was a bit surprised they were not using iSCSI. They claim they just can't risk the extra latency. I believe that you are right. It seems to me that offloading snapshots and backups to an iSCSI SAN would improve things. The problem is that this company has been burned on storage solutions more than once and they are a little skeptical that a product can scale to what they need. There are some SAN vendor names that are a four letter word here. So far, their newest FC SAN is performing well. I think having more, small, iSCSI boxes would be a good solution. One problem I've seen with smaller iSCSI products is that feature sets like snapshotting are not the best implementation. It works, but doing any sort of automation can be painful. > The snap takes place wholly within the array and is very fast, without > the problems you see with host based snapshots such as with Linux LVM, > where you must first freeze the filesystem, wait for the snapshot to > complete, which could be a very long time with a 1TB FS. While this > occurs your clients must wait or timeout while trying to access > mailboxes. With a SAN array snapshot system this isn't an issue as the > snap is transparent to hosts with little or no performance degradation > during the snap. Two relatively inexpensive units that have such > snapshot capability are: How does this work? I've always had Linux create a snapshot. Would the SAN doing a snapshot without any OS buy-in cause the filesystem to be saved in an inconsistent state? I know that ext4 is pretty good at logging, but still, wouldn't this be a problem? > > http://www.equallogic.com/products/default.aspx?id=10613 > > http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html > > The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be > had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or > more host/network connection ports when equipped with dual controllers. > There are many other vendors with similar models/capabilities. I > mention these simply because Dell/HP are very popular and many OPs are > already familiar with their servers and other products. I will take a look. I might have some convincing to do. > There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, > Linux FUSE. Which were you using? If the last, that would fully > explain the suck. There is one more that I had never used before coming on board here: ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My understanding is that they were using it on the backup machines with the front end dovecot machines using ext4. I'm told the metadata issue is a ZFS thing and they have the same problem on Solaris/Nexenta. > > I've relatively new here, but I'll ask around about XFS and see if > > anyone had tested it in the development environment. > > If they'd tested it properly, and relatively recently, I would think > they'd have already replaced EXT4 on your Dovecot server. Unless others > factors prevented such a migration. Or unless I've misunderstood the > size of your maildir workload. I don't know the entire history of things. I think they really wanted to use ZFS for everything and then fell back to ext4 because it performed well enough in the cluster. Performance becomes an issue with backups using rsync. Rsync is faster than Dovecot's native dsync by a very large margin. I know that dsync is doing more than rsync, but still, seconds compared to over five minutes? That is a significant difference. The problem is that rsync can't get a perfect backup. ...Jeff From ncjeffgus at zimage.com Wed Mar 28 23:58:38 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:58:38 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: References: <1332888019.29480.17.camel@sally> Message-ID: <1332968318.26122.22.camel@sally> On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: > Why is it a problem that dsync takes a long time, when it can be done > without downtime for the users? > > I just started our maildir->mdbox convertion yesterday, using the > attached script. I only converted a little over 10000 easy accounts > (accounts with simple folder names, as I expect to run into problems > once we start hitting accounts with trailing dot or broken latin1/utf8 > characters in the folder names). I might agree it wasn't quick, but > that really doesn't matter as the only downtime for the user is that > he's potentially kicked out during the userdb update. I looked over your script. I plan on doing some trial runs with it. I think the trick where you re-run the sync and then boot the user off the connection should work pretty well. I hadn't totally fleshed out the scripting on the conversion since there is a lot more I need to do with the database and configuration files first. It appears I can use your script as a starting point for our configuration. ...Jeff > > > -jf > > > We're hoping that converting away from Maildir will help us speed up > > the backup processes by reducing the number of files to process. > > From ncjeffgus at zimage.com Thu Mar 29 00:01:19 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 14:01:19 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <4F727ECE.4050305@r.paypc.com> References: <1332888019.29480.17.camel@sally> <4F727ECE.4050305@r.paypc.com> Message-ID: <1332968479.26122.24.camel@sally> On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: > I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox > "transcoding" tool, since none appear to exist. To keep it simple, I/O > to/from "remote" mailbox (connections) are not pipelined. It won't > require more than MAXEMAILSIZE's worth of RAM (if one of the directions > involves a remote connection), and so far when processing MIX, Maildir, > and Mbox files, it's extremely fast. This sounds interesting. If it could so [sm]dbox, it would be very, very useful to large installations. ...Jeff From tss at iki.fi Thu Mar 29 01:30:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:30:34 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Message-ID: On 27.3.2012, at 1.14, Michescu Andrei wrote: > This being said and acknowledged here are my 2 cents: > > I think that the current '1 brain / 2 workers' seems to be the correct > model. The "the client" connects to the "server" and pushes the local > changes and after retrieves the updated/new items from the "server". "The > brain" considers first server as the "local storage" and the second server > as "server storage". This design makes it too easy to design it in a way that adds extra roundtrips = extra latency. It also kind of hides other problems as well. For example now dsync can way too easily just fail if something unexpected happens during dsync (e.g. mailbox gets renamed/deleted). And there are of course some bugs that I don't really understand why some people are seeing them at all. > For the split design, "come to the same conclusion of the state" is very > race-condition prone. It's race-condition prone with the brain design as well. dsync can't just lock the mailbox during its sync, since the sync can take a long time. With a "brainless" design it's clear from the beginning that there are race conditions and they need to be dealt with. From tss at iki.fi Thu Mar 29 01:43:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:43:07 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: On 23.3.2012, at 23.25, Timo Sirainen wrote: > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. Step #3 was more difficult than I first realized. I spent last two days figuring out a way to make it work, and looks like I finally did. I didn't update the document yet, but I wrote a test program: http://dovecot.org/tmp/test-dsync.c Step #2 should be easy enough. Step #4 I think I'll forget about and just implement a per-mailbox dsync lock. The main reason I wanted to get rid of locks was because a per-user lock can't work with shared mailboxes. But a per-mailbox lock is okay enough. Note that #3 allows the two dsyncs to run in parallel and send duplicate changes, just not modifying the same mailbox at the same time (which would duplicate mails due to two transactions adding the same mails). From tss at iki.fi Thu Mar 29 01:52:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:52:38 +0300 Subject: [Dovecot] Merge mails from two mail_locations In-Reply-To: <4F71926F.30500@wk-serv.de> References: <4F71926F.30500@wk-serv.de> Message-ID: On 27.3.2012, at 13.11, Patrick Westenberg wrote: > recently I had some trouble with my ocfs2 cluster and it unmounted > itself from /var/mail. > > Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. > > Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. You can use "doveadm import" to copy mails from one location to another. From tss at iki.fi Thu Mar 29 01:59:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:59:15 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: <8C621D75-BC36-4C85-B80A-28473279DFAB@iki.fi> On 27.3.2012, at 10.12, B?rd Johannessen wrote: > 2012/3/26 Timo Sirainen : >> Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This >> should fix it: >> >> http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 > > Nope; exactly same result; body field contains just the empty line. Always? It worked in my tests, and I don't see anything else wrong in the code.. From tss at iki.fi Thu Mar 29 02:12:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:12:13 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: On 22.3.2012, at 23.25, Jeff Gustafson wrote: > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > mdbox:/home/bu/testuser > > real 1m9.519s > user 1m7.592s > sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox From tss at iki.fi Thu Mar 29 02:25:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:25:57 +0300 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: On 27.3.2012, at 17.40, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? Like you noticed, these are pretty much the same things, since most clients don't understand about namespaces. And Dovecot doesn't "select" a namespace for clients. They mostly affect mailbox listing.. Like when a client asks Dovecot to list mailboxes under foo/, then Dovecot checks if a foo/ namespace exists. > On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. I have no idea how Thunderbird handles them differently. > It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? Not really. > This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. .. > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } All of your namespaces have hidden=yes. There should be (at least) one with hidden=no (this prefix= namespace in your case). I guess I'll need to add a check to have Dovecot fail if there aren't any. Also it's not necessary to duplicate the "location" setting, since it defaults to the global mail_location. > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } This namespace is unlikely to be useful to you, unless you actually have some IMAP client configured to use #mbox/ prefix. > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } This shouldn't have list=yes .. I'm not sure what even happens with it. I guess if client lists all mailboxes from root you'll get a mailbox named "~", which has a "mail" child, which has all of your mailboxes duplicated. From tss at iki.fi Thu Mar 29 02:30:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:30:51 +0300 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On 28.3.2012, at 13.13, Hangas wrote: > Timo Sirainen iki.fi> writes: > >>> 4. Are there real-world benchmarks showing measurable differences between >>> maildir, sdbox mdbox? >> >> Not that I'm aware of. So far everyone I've tried to ask have replaced their >> whole mail system and their storage, so the before/after numbers can't be >> compared. I'm very interested in knowing myself too. > > I think I can give my contribution here. I'm planning to migrate from dovecot > 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at > the time, but now I'm trying to decide the mailbox format for a 2.x > fresh install. .. > My ideia is to install a fresh server and replicate the production maildir on it > to build a test "source disk" that I'll use then to experiment the conversions > to sdbox and mdbox. > I then plan to test the performance of the dbox formats, but I can include > Maildir measurements just for the record. > > I'm open to suggestions on how to test this properly The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much. From tss at iki.fi Thu Mar 29 02:32:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:32:01 +0300 Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage In-Reply-To: <33544816.post@talk.nabble.com> References: <33544816.post@talk.nabble.com> Message-ID: On 28.3.2012, at 11.57, Tomislav Mihalicek wrote: > Could someone explain what this strings mean in dovecot 2.1.3 debug log? > > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 > test1 at example.net service=lib-storage > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 > test2 at example.net service=lib-storage Dovecot is asking a user's home directory via userdb lookup. Looks like your userdb isn't returning a home directory. There should be an error message about it? From tss at iki.fi Thu Mar 29 02:34:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:34:54 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28.3.2012, at 0.06, Bruce, Andrew wrote: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server .. > Further investigation shows that there are a few other fields that we > can't retrieve in Dovecot, Looks to me like you can't retrieve any fields from LDAP, possibly because the dn user doesn't have access to the information or some other reason. From tss at iki.fi Thu Mar 29 02:37:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:37:03 +0300 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: <28C4EF51-12FB-4B11-A3FB-54949CAF0444@iki.fi> On 27.3.2012, at 19.49, Ben Schumacher wrote: > On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: >> the zlib_save question reminds me of a wish: >> I think it's not possible to set zlib_save parameter per namespace (or even >> mailbox). Per namespace would be something for the wish list to get rid of >> the cron job method to compress archival mailboxes. >> And maybe an option to add a "Z" flag to compressed maildir message files >> as recommended in the wiki regarding compress crob job. > > +1 on this request. I have a slightly different use case -- I have > both an dbox and Maildir. Incoming email goes to Maildir, but I > archive off to dbox (using Thunderbird). After I archive my emails, > compression seems like a reasonable choice. > > Any idea if this feature will be available at some point? For v2.2 I'm hoping to have per-namespace mail settings, although I'm not sure if it actually gets implemented - so many other things to do as well. Then (among other things) you could do: namespace foo { plugin { zlib_save = gz } } From tss at iki.fi Thu Mar 29 02:42:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:42:50 +0300 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <4F71F8D8.6040700@wiesinger.com> References: <4F71F8D8.6040700@wiesinger.com> Message-ID: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. doveconf -n output? > 2.0.x: with Prefix ~/Mail > A0003 LIST "" ~/Mail/sent > A0003 OK List completed.: Works with my v2.0.19 config: x list "" ~/Mail/sent * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" x OK List completed. From tss at iki.fi Thu Mar 29 02:46:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:46:17 +0300 Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook In-Reply-To: <33544803.post@talk.nabble.com> References: <33544803.post@talk.nabble.com> Message-ID: <009AB5A8-303D-45F9-B1CF-33E26D859B0C@iki.fi> On 28.3.2012, at 1.24, stonegate wrote: > Problem: When i receive a new email, it does not appear in my Outlook unless > i have the IMAP Inbox Folder open (highlighted selection). > > Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize > it unless i click on the inbox folder. > > Before that problem occured on my old system ( i think it was dovecot as > well ) the inbox folder refreshed automatically and i instantly knew when i > had new emails. It was kind of like with my exchange account. > > Does anyone have a clue what might be wrong ? Since i did not re-install > outlook or something it cannot be a client thing since no settings changed > on the client side. > > Maybe i have to do something with dovecots config file? I think it should > keep the connection to the server open or something. I?d appreciate any kind > of help. I'm not aware of any Dovecot change or any Dovecot setting that could change this behavior in Outlook. It's the client's choice how it monitors the mailboxes. From tss at iki.fi Thu Mar 29 02:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:48:53 +0300 Subject: [Dovecot] doveadm purge on clusterfs In-Reply-To: <20120327094710.GA10878@dibs.tanso.net> References: <20120327094710.GA10878@dibs.tanso.net> Message-ID: On 27.3.2012, at 12.47, Jan-Frode Myklebust wrote: > Would it be OK to run purge in the pop/imap postlogin scripts? We > already do a conditional: > > test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today > then > touch /var/log/activemailaccounts/imap/$USER > fi > > so adding a: > > doveadm purge -u $USER > > in this section would make it run once every day the users that log in. > Does that sound like an OK solution? Yeah, should work fine. Or you should make it run in background so user's login won't slow down because of the purging. I'm not sure if simply adding & at the end works (or if the process dies after login is finished, or what happens if it runs over 30 secs which is when normally post-login script is killed by SIGALRM). From tss at iki.fi Thu Mar 29 02:49:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:49:56 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <712362F5-EDFF-4BFA-B932-1C6E19855F7D@iki.fi> On 27.3.2012, at 4.16, Joseph Tam wrote: >>> However, I noticed a strange thing: querying what would have been >>> deleted >>> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >>> showed many date.saved values are clustered around the same timestamp, >>> even among different user's Trash mailbox. >>> ... >>> I can't explain why many different users would have messages with the >>> same (or closeby) date.saved value. >> Which mailbox format? With Maildir the date.saved is taken from >> dovecot.index.cache file, and in some cases that might get dropped. If >> it does, then it fallbacks to using the file's ctime. > > mbox. Ah, with mbox there isn't any usable fallback for date.saved. If it's not in dovecot.index.cache, the current time is used. > These "wrong" values shouldn't cause problems with expunge queries since > they err on the side of safety. Right. From ncjeffgus at zimage.com Thu Mar 29 02:51:49 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 16:51:49 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> Message-ID: <1332978709.26122.29.camel@sally> On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: > On 22.3.2012, at 23.25, Jeff Gustafson wrote: > > > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > > mdbox:/home/bu/testuser > > > > real 1m9.519s > > user 1m7.592s > > sys 0m1.126s > > Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: > > - mbox:/tmp/mbox > - mbox:/tmp/mbox:INDEX=MEMORY > - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. ...Jeff From tss at iki.fi Thu Mar 29 02:53:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:53:35 +0300 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: <2DF0AEB9-7410-473C-8404-1D270711A89B@iki.fi> On 26.3.2012, at 18.25, M?ller Lukas wrote: > Thanks for the quick answer. > > I realised, that the error didn't occur since quite a while, opposed to what our client suggested. > Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. > > Is it possible that those could have improved the situation? Unlikely. >>> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) > .. > >>> My suspicion/speculation what happens is the following: >>> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >>> Somehow this leads to problems with Locks on NFS, which leads to the crash. > >> Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > > I had a quick look a tour NFS (NetApp), but didn't find anything useful. > In case the problem persists, I will check with the coworker responsible for NetApp. I guess you just had very bad luck. NetApp has a good NFS implementation and normally works fine with Dovecot. From tss at iki.fi Thu Mar 29 02:57:22 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:57:22 +0300 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6F35FA.6050207@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> Message-ID: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> On 25.3.2012, at 18.12, Charles Marcus wrote: > On 2012-03-24 9:16 AM, Timo Sirainen wrote: >> On 24.3.2012, at 14.54, Charles Marcus wrote: >> >>> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>>> You can do full backups from a filesystem snapshot, which works >>>> "well enough" (might leave some unused attachments lying around in >>>> some rare cases, but that can also happen if Dovecot crashes/dies). >>> >>> But the problem isn't with backups, but with restores, right? >> >> Ah, right. Then it gets tricky. > > Yeah, I seem to remember it was a comment like that that scared me about enabling it... > > Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) It's easy to restore a full backup. And it's easy to restore specific users if you have the full backup easily accessible (just run doveadm import with proper settings pointing to backup). What's difficult is if you just want to restore a specific user from the backup and can't easily do random access to all files. Then you'll first need to restore the user's dbox files and then somehow figure out which attachments to restore from the SIS directory. >>> Am I correct that enabling SIS as it is currently implemented would >>> break this backup tool? > >> I'm not sure. Are you running rsnapshot on live filesystem or on a >> snapshot? On live filesystem there would be race conditions. > > I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: > > file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," > rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] I'd guess that with rsnapshot + Maildir you can get duplicate Maildir files if the rsnapshot is accessing a large maildir at the same time as user is changing a message flag. Dovecot usually notices these duplicates and logs a warning about them. From tss at iki.fi Thu Mar 29 03:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 03:06:53 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332978709.26122.29.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> Message-ID: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> On 29.3.2012, at 2.51, Jeff Gustafson wrote: >> Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: >> >> - mbox:/tmp/mbox >> - mbox:/tmp/mbox:INDEX=MEMORY >> - sdbox:/tmp/sdbox > > My tests show that maildir to mdbox or sdbox backup/conversions take > about the same length in time. I noticed maybe a second or two > difference between mdbox and sdbox). On a 3.1GB mailbox either one took > about 6 minutes. Rsync, on the other hand, took less than a minute. I > will re-run the tests with a maildir to maildir backup and see how long > it takes. Try also with INDEX=MEMORY, since the problem may be related to updating the indexes. Another way to test if the problem is dsync or Dovecot's generic mail saving code is to run: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all Or if it's the mail reading code: time doveadm fetch -u user at domain text all > /dev/null From ncjeffgus at zimage.com Thu Mar 29 03:48:06 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 17:48:06 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> Message-ID: <1332982086.26122.34.camel@sally> On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/user at domain.com/Maildir "" all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test > Or if it's the mail reading code: > > time doveadm fetch -u user at domain text all > /dev/null This ran quicker than a full dsync. Only 40s for 3.1GB. rsync still beat it clocking in at 16s. I ran the fetch command twice figuring the files would get cached by the OS. ...Jeff From tss at iki.fi Thu Mar 29 04:07:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 04:07:51 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332982086.26122.34.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> Message-ID: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> On 29.3.2012, at 3.48, Jeff Gustafson wrote: > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > This tried to write to /root for some reason and failed (dovecot > 2.1.3): > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > maildir:/home/users/user at domain.com/Maildir "" all > doveadm(root): Error: chdir(/root/) failed: Permission denied > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > in group 0(root), dir owned by 0:0 mode=0550) > doveadm(root): Error: chdir(/root) failed: Permission denied > doveadm(root): Error: Can't find namespace for mailbox Trash > doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. From ncjeffgus at zimage.com Thu Mar 29 05:07:13 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 19:07:13 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> Message-ID: <1332986833.26122.36.camel@sally> On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: > On 29.3.2012, at 3.48, Jeff Gustafson wrote: > > > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > > > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > > > This tried to write to /root for some reason and failed (dovecot > > 2.1.3): > > > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > > maildir:/home/users/user at domain.com/Maildir "" all > > doveadm(root): Error: chdir(/root/) failed: Permission denied > > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > > in group 0(root), dir owned by 0:0 mode=0550) > > doveadm(root): Error: chdir(/root) failed: Permission denied > > doveadm(root): Error: Can't find namespace for mailbox Trash > > doveadm(root): Error: Can't find namespace for mailbox test > > > Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash doveadm(user at domain.com): Error: Can't find namespace for mailbox test ...Jeff From jtam.home at gmail.com Thu Mar 29 05:41:16 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 28 Mar 2012 19:41:16 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >>> Which mailbox format? With Maildir the date.saved is taken from >>> dovecot.index.cache file, and in some cases that might get dropped. If >>> it does, then it fallbacks to using the file's ctime. >> >> mbox. > > Ah, with mbox there isn't any usable fallback for date.saved. If it's > not in dovecot.index.cache, the current time is used. I'm a little confused as to why it needed a fallback. In other words, why wasn't date.saved put into the index as soon as the IMAP operation copied it into "Trash"? If this data isn't set at that time, when does it get instantiated? When I actually ask for it? Joseph Tam From tss at iki.fi Thu Mar 29 07:04:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 07:04:26 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332986833.26122.36.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> Message-ID: <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> On 29.3.2012, at 5.07, Jeff Gustafson wrote: > That fixed some errors, but it still is having some sort of trouble > with that command: > > # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all > doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash > doveadm(user at domain.com): Error: Can't find namespace for mailbox test Oh, you don't have prefix="" namespace? If you have e.g. prefix="INBOX." namespace then use: time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all From lists at wiesinger.com Thu Mar 29 08:25:17 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Thu, 29 Mar 2012 07:25:17 +0200 (CEST) Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> References: <4F71F8D8.6040700@wiesinger.com> <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> Message-ID: On Thu, 29 Mar 2012, Timo Sirainen wrote: > On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > >> After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. > > doveconf -n output? > >> 2.0.x: with Prefix ~/Mail >> A0003 LIST "" ~/Mail/sent >> A0003 OK List completed.: > > Works with my v2.0.19 config: > > x list "" ~/Mail/sent > * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" > x OK List completed. # 2.0.19: /etc/dovecot/dovecot.conf # OS: cutted for security reasons listen = * mail_full_filesystem_access = yes mail_location = mbox:~:INBOX=/var/mail/%u mbox_lazy_writes = no mbox_write_locks = fcntl passdb { driver = pam } protocols = imap service auth { unix_listener /var/run/dovecot-auth-master { group = users mode = 0660 } user = root } service imap { executable = imap postlogin } service postlogin { executable = script-login -d rawlog } ssl_cert = References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> Message-ID: <405020e5dfb341332e535e905ff183c3@alpha.zimage.com> On Thu, 29 Mar 2012 07:04:26 +0300, Timo Sirainen wrote: > On 29.3.2012, at 5.07, Jeff Gustafson wrote: > >> That fixed some errors, but it still is having some sort of trouble >> with that command: >> >> # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u >> user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> Trash >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> test > > Oh, you don't have prefix="" namespace? If you have e.g. > prefix="INBOX." namespace then use: > > time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all Oh! I should have known that was the problem. This was very, very fast. This test is maildir to maildir: # time doveadm -o mail=maildir:/home/bu/test import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m0.412s user 0m0.036s sys 0m0.088s But it was just as slow to import into mdbox: # time doveadm -o mail=mdbox:/home/bu/test2 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 7m12.738s user 6m46.161s sys 0m7.046s mbox... still pretty fast: # time doveadm -o mail=mbox:/home/bu/test3 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m58.534s user 0m52.264s sys 0m5.762s sdbox seems a little on the slow side too: # time doveadm -o mail=sdbox:/home/bu/test4 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 6m11.616s user 6m6.924s sys 0m4.579s Does information help? It seems that [sm]dbox is on the slow side for the purpose of doing backups. ...Jeff From fabio.ferrari at unimore.it Thu Mar 29 11:15:32 2012 From: fabio.ferrari at unimore.it (FABIO FERRARI) Date: Thu, 29 Mar 2012 10:15:32 +0200 (CEST) Subject: [Dovecot] Problem about dovecot Panic Message-ID: Good morning, we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some mail services in it (postfix and dovecot 2). The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). >From last week we have this dovecot problem: suddenly dovecot doesn't accept any new connections, the dovecot.log file reports lines like these Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ libdovecot.so.0 [0x36ea4362e8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) [0x36ea440b7f] -> /usr/li b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ imap [0x408179] Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 killed with signal 6 (core dumps disabled) Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command startup failed, throttling Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument and the kern.log file reports Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 can you help us? thanks in advance Fabio Ferrari From javierdemiguel at us.es Thu Mar 29 11:18:36 2012 From: javierdemiguel at us.es (=?UTF-8?Q?Javier_Miguel_Rodr=C3=ADguez?=) Date: Thu, 29 Mar 2012 10:18:36 +0200 Subject: [Dovecot] Problem about dovecot Panic In-Reply-To: References: Message-ID: <5ae4435a57981464edec3590216c7b41@us.es> We had the same problem. Reboot with an older kernel (2.6.18-274.17.1.el5 works for us). It is known bug of RHEL, see this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=681578 Regards Javier On Thu, 29 Mar 2012 10:15:32 +0200 (CEST), FABIO FERRARI wrote: > Good morning, > we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some > mail services in it (postfix and dovecot 2). > > The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). > > From last week we have this dovecot problem: suddenly dovecot doesn't > accept any new connections, the dovecot.log file reports lines like these > > Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ > libdovecot.so.0 [0x36ea4362e8] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) > [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) > [0x36ea440b7f] -> /usr/li > b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) > [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> > /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ > imap [0x408179] > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 > killed with signal 6 (core dumps disabled) > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command > startup failed, throttling > Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > > and the kern.log file reports > > Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 > Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 > > can you help us? > > thanks in advance > > Fabio Ferrari From c at roessner-network-solutions.com Thu Mar 29 13:24:18 2012 From: c at roessner-network-solutions.com (=?iso-8859-1?Q?Christian_R=F6=DFner?=) Date: Thu, 29 Mar 2012 12:24:18 +0200 Subject: [Dovecot] File/folder permission issues in 2.1.3 Message-ID: <8B296F70-22B8-487B-AD7A-47BEB8C84F62@roessner-network-solutions.com> Hi, I figured out that Dovecot does not honer secondary groups with auth/auth-worker (??), if doing LDAP/TLS stuff. I had to use file system acls to add the user "vmail" to /etc/ssl/private and to the corresponding key file: doveconf -n # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-40-generic-pae i686 Ubuntu 10.04.4 LTS auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes hostname = mail.roessner-net.de lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_access_groups = vmail mail_gid = vmail mail_location = mdbox:~/mdbox mail_plugins = autocreate quota acl fts fts_solr zlib mail_log notify mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = mdbox:%%h/mdbox prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox junkmail { special_use = \Junk } prefix = separator = / type = private } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/var/mail/virtual/shared-mailboxes.db autocreate = Trash autocreate2 = Sent autocreate3 = Drafts autocreate4 = junkmail autosubscribe = Trash autosubscribe2 = Sent autosubscribe3 = Drafts autosubscribe4 = junkmail fts = solr fts_solr = break-imap-search url=http://localhost:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:User quota::file:%h/mdbox/dovecot-quota quota_rule = *:storage=300M:messages=20000 quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = gz zlib_save_level = 6 } protocols = imap pop3 lmtp sieve service auth-worker { unix_listener auth-worker { user = vmail } user = vmail } service auth { unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = ::1 port = 24 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_ca = From mafonso at hangas.net Thu Mar 29 14:16:35 2012 From: mafonso at hangas.net (Miguel Afonso) Date: Thu, 29 Mar 2012 12:16:35 +0100 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen wrote: > > The main problem is that it's difficult to do any "real world" tests with > IMAP, especially when users are using many different kinds of IMAP clients. > So I'm very interested in hearing some numbers (and disk IO graphs for a > few weeks would be great) before your migration and after your migration, > but the numbers for your tests might not mean all that much. I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days. I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards. From me at junc.org Thu Mar 29 14:57:39 2012 From: me at junc.org (Benny Pedersen) Date: Thu, 29 Mar 2012 13:57:39 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <47266fb4a9b1a50c72ab892ac67d9744@junc.org> Den 2012-03-28 17:50, Xavier Beaudouin skrev: > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > is this valid sieve ? > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... not all sieve have date support, and imho no one have macro supported > I don't know if some sieve guru can tell me how to do that... ? why not keep lmtp ? :) http://sieve.info From stan at hardwarefreak.com Thu Mar 29 15:24:05 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 29 Mar 2012 07:24:05 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332968041.26122.19.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> Message-ID: <4F745465.1030304@hardwarefreak.com> On 3/28/2012 3:54 PM, Jeff Gustafson wrote: > On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > >> Locally attached/internal/JBOD storage typically offers the best >> application performance per dollar spent, until you get to things like >> backup scenarios, where off node network throughput is very low, and >> your backup software may suffer performance deficiencies, as is the >> issue titling this thread. Shipping full or incremental file backups >> across ethernet is extremely inefficient, especially with very large >> filesystems. This is where SAN arrays with snapshot capability come in >> really handy. > > I'm a new employee at the company. I was a bit surprised they were not > using iSCSI. They claim they just can't risk the extra latency. I The tiny amount of extra latency using a software initiator is a non argument for a mail server workload, unless the server is undersized for the workload--high CPU load and low memory constantly. As I said, in that case you drop in an iSCSI HBA and eliminate any possibility of block latency. > believe that you are right. It seems to me that offloading snapshots and > backups to an iSCSI SAN would improve things. If you get the right unit you won't understand how you ever lived without it. The snaps complete transparently, and the data is on the snap LUN within a few minutes, depending on the priority you give to internal operations, snaps/rebuilds/etc, vs external IO requests. Depending on model > The problem is that this > company has been burned on storage solutions more than once and they are > a little skeptical that a product can scale to what they need. There are More than once? More than once?? Hmm... > some SAN vendor names that are a four letter word here. So far, their > newest FC SAN is performing well. Interesting. Care to name them (off list)? > I think having more, small, iSCSI boxes would be a good solution. One > problem I've seen with smaller iSCSI products is that feature sets like > snapshotting are not the best implementation. It works, but doing any > sort of automation can be painful. As is most often the case, you get what you pay for. >> The snap takes place wholly within the array and is very fast, without >> the problems you see with host based snapshots such as with Linux LVM, >> where you must first freeze the filesystem, wait for the snapshot to >> complete, which could be a very long time with a 1TB FS. While this >> occurs your clients must wait or timeout while trying to access >> mailboxes. With a SAN array snapshot system this isn't an issue as the >> snap is transparent to hosts with little or no performance degradation >> during the snap. Two relatively inexpensive units that have such >> snapshot capability are: > > How does this work? I've always had Linux create a snapshot. Would the > SAN doing a snapshot without any OS buy-in cause the filesystem to be > saved in an inconsistent state? I know that ext4 is pretty good at > logging, but still, wouldn't this be a problem? Instead of using "SAN" as a generic term for a "box", which it is not, please use the terms "SAN" for "storage area network", "SAN array" or "SAN controller" when talking about a box with or without disks that performs the block IO shipping and other storage functions, "SAN switch" for a fiber channel switch, or ethernet switch dedicated to the SAN infrastructure. The acronym "SAN" is an umbrella covering many different types of hardware and network topologies. It drives me nuts when people call a fiber channel or iSCSI disk array a "SAN". These can be part of a SAN, but are not themselves, a SAN. If they are direct connected to a single host they are simple disk arrays, and the word "SAN" isn't relevant. Only uneducated people, or those who simply don't care to be technically correct, call a single intelligent disk box a "SAN". Ok, end rant on "SAN". Read this primer from Dell: http://files.accord.com.au/EQL/Docs/CB109_Snapshot_Basic.pdf The snapshots occur entirely at the controller/disk level inside the box. This is true of all SAN units that offer snap ability. No host OS involvement at all in the snap. As I previously said, It's transparent. Snaps are filesystem independent, and are point-in-time, or PIT copies of one LUN to another. Read up on "LUN" if you're not familiar with the term. Everything in SAN storage is based on LUNs. Now, as the document above will tell you, array based snapshots may or may not be a total backup solution for your environment. You need to educate yourself and see if this technology is a feature that fits your file backup and disaster avoidance and recovery needs. >> http://www.equallogic.com/products/default.aspx?id=10613 >> >> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html >> >> The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be >> had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or >> more host/network connection ports when equipped with dual controllers. >> There are many other vendors with similar models/capabilities. I >> mention these simply because Dell/HP are very popular and many OPs are >> already familiar with their servers and other products. > > I will take a look. I might have some convincing to do. SAN array features/performance are an easy sell. Price not so much. Each fully loaded ~24 drive SAN array is going to run you between $15k-30k USD depending on the vendor and how many spindles you need for IOPS, disk size for total storage, snap/replication features you need, expandability, etc. >> There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, >> Linux FUSE. Which were you using? If the last, that would fully >> explain the suck. > > There is one more that I had never used before coming on board here: > ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My It's a "roll your own" patch set not in mainline and not supported by any Linux distro/vendor, AFAIK. Which is why I didn't include it. > understanding is that they were using it on the backup machines with the > front end dovecot machines using ext4. I'm told the metadata issue is a > ZFS thing and they have the same problem on Solaris/Nexenta. I've never used ZFS, and don't plan to, so I can't really comment on this. That and I have no technical details of the problem. >>> I've relatively new here, but I'll ask around about XFS and see if >>> anyone had tested it in the development environment. >> >> If they'd tested it properly, and relatively recently, I would think >> they'd have already replaced EXT4 on your Dovecot server. Unless others >> factors prevented such a migration. Or unless I've misunderstood the >> size of your maildir workload. > > I don't know the entire history of things. I think they really wanted > to use ZFS for everything and then fell back to ext4 because it > performed well enough in the cluster. Performance becomes an issue with > backups using rsync. Rsync is faster than Dovecot's native dsync by a > very large margin. I know that dsync is doing more than rsync, but > still, seconds compared to over five minutes? That is a significant > difference. The problem is that rsync can't get a perfect backup. This happens with a lot of "fan boys". There was so much hype surrounding ZFS that even many logically thinking people were frothing at the mouth waiting to get their hands on it. Then, as with many/most things in the tech world, the goods didn't live up to the hype. XFS has been around since 1994, has never had hype surrounding it, has simply been steadily, substantially improved over time. It has been since day 1 the highest performance filesystem with parallel workloads, and finally overcame its last barrier preventing it from being suitable for just about any workload: metadata write performance. Which makes it faster than any FS with the maildir workload when sufficient parallelism/concurrency is present. Meaning servers with a few thousand active users will benefit. Those with 7 users won't. -- Stan From CMarcus at Media-Brokers.com Thu Mar 29 16:56:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 29 Mar 2012 09:56:39 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> Message-ID: <4F746A17.8040500@Media-Brokers.com> On 2012-03-28 7:57 PM, Timo Sirainen wrote: > It's easy to restore a full backup. And it's easy to restore specific > users if you have the full backup easily accessible (just run doveadm > import with proper settings pointing to backup). What's difficult is > if you just want to restore a specific user from the backup and can't > easily do random access to all files. Then you'll first need to > restore the user's dbox files and then somehow figure out which > attachments to restore from the SIS directory. Well, I think I'm not going to worry about this, since you recently said: On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. If we really can get these snapshots to automatically show up under a 'Backups' namespace, with each users folders under each snapshot showing by date, so they can easily 'go back in time' and retrieve anything they want from them, that totally eliminates any need for me to do individual restores... :) > I'd guess that with rsnapshot + Maildir you can get duplicate Maildir > files if the rsnapshot is accessing a large maildir at the same time > as user is changing a message flag. Dovecot usually notices these > duplicates and logs a warning about them. This won't be a problem wither, because our new system will be performing filesystem snapshots for rsnapshot to use as a source. Thanks again! -- Best regards, Charles From dovecot at r.paypc.com Thu Mar 29 21:13:58 2012 From: dovecot at r.paypc.com (Robin) Date: Thu, 29 Mar 2012 11:13:58 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F745465.1030304@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> <4F745465.1030304@hardwarefreak.com> Message-ID: <4F74A666.4000705@r.paypc.com> On 3/29/2012 5:24 AM, Stan Hoeppner wrote: > This happens with a lot of "fan boys". There was so much hype > surrounding ZFS that even many logically thinking people were frothing > at the mouth waiting to get their hands on it. Then, as with many/most > things in the tech world, the goods didn't live up to the hype. The problem with zfs especially is that there are so many different implementations, with only the commercial Sun, er, Oracle paid Solaris having ALL of the promised features and the bug-fixes to make them safely usable. For those users, with very large RAM-backed Sun, er, Oracle, hardware, it probably works well. FreeBSD and even the last versions of OpenSolaris lack fixes for some wickedly nasty box-bricking bugs in de-dup, as well as many of the "sexy" features in zpool that had people flocking to it in the first place. The bug database that used to be on the OpenSolaris portal by Sun's gone dark, but you may have some luck through archive.org. I know when I tried it out for myself using the "Community Edition" of Solaris, I did feel annoyed by the bait-and-switch, and the RAM requirements to run de-dupe with merely adequate performance were staggering if I wanted to have plenty of spare block cache left over for improving performance overall. Sun left some of the FOSS operating systems a poison pill with its CDDL licence, which is the main reason why the implementations of zfs on Linux are immature and is being "re-implemented" with US DOE sponsorship, ostensibly in a GNU compatible licence. zfs reminds me a great deal of TIFF - lots of great ideas in the "White Paper", but an elusive (or very very costly) white elephant to acquire. "Rapidly changing", "bleeding edge", and "hot & new" are not descriptors for filesystems I want to trust more than a token amount of data to. =R= From abruce at tumnus.co.nz Thu Mar 29 22:05:24 2012 From: abruce at tumnus.co.nz (Andrew Bruce) Date: Fri, 30 Mar 2012 08:05:24 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120328092534.5690fa40@jimbo> References: <20120327091425.73963576@jimbo> <20120328092534.5690fa40@jimbo> Message-ID: <4F74B274.8000600@tumnus.co.nz> On 28/03/2012 19:25, Nikita Koshikov wrote: > On Wed, 28 Mar 2012 09:39:37 +1300 > Bruce, Andrew wrote: > >> On 28 March 2012 09:36, Bruce, Andrew wrote: >>> On 27 March 2012 19:14, Nikita Koshikov wrote: >>>> On Tue, 27 Mar 2012 13:57:04 +1300 >>>> Bruce, Andrew wrote: >>>> >>>> Hi there, >>>> >>>> We're setting up a Dovecot virtual email setup - we've got everything >>>> working perfect with LDAP logins authenticating against AD and so >>>> forth, but we're having issues with retrieving the maxStorage value >>>> from AD (this is a pre-setup field in AD that we'd like to use to set >>>> per user quotas). >>>> >>>> In our LDAP lookup, we have the maxStorage entry listed under >>>> user_attrs for the quota (user_attrs = >>>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>>> it trying to get the entry, but it fails with: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=maxStorage >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server >>>> >>>> At this point, we then see the default quota applied. >>>> >>> Try to change your quota rule to be like: >>> maxStorage=quota_rule=*:bytes=%$ >>> ^^^^^^^^^ >>> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >>> >>> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>>> If we change the name of the field from maxStorage to instanceType we >>>> see the value show up in the logs and passed through to the quota >>>> system and applied successfully: >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=instanceType >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>>> Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 >>>> username at site quota_rule=*:storage=4M >>>> >>>> >>>> Which seems a bit weird. >>>> >>>> If we use ldapsearch and pass it the same search string and look for >>>> the field maxStorage, we clearly see the field and the value being >>>> returned. The result looks the same if we also lookup instanceType. >>>> >>>> We're using Dovecot 2.0.9. >>>> >>>> Does anyone have any idea as to why we can't use this field? >>>> >>>> Thanks, >>>> >>>> Andrew >> Tried your suggestion Nikita, no joy unfortunately. It still looks >> like the value never gets returned from the LDAP server to Dovecot. >> It definitely has something in the field (equivalent of 10GB, but in >> bytes as suggested) and I changed the user_attrs also, but still get >> the same "no fields returned by the server" error message. >> >> Modifying the user_attrs to lookup from a different field >> (instanceType) definitely works. >> >> What exact version are you using - perhaps it's a problem with our >> copy of 2.0.9. >> >> Thanks, >> >> Andrew > maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) > Dead on - it was a restriction of ldap port 3268 - as soon as we pointed ldapsearch at the same port, we got the same result - some of the fields were missing. It all makes perfect sense and I wish I noticed that earlier. Now need to work out why Dovecot can get the fields and username back from ldap on port 389, but it can't do the auth through it like it could with 3268. Thanks Nikita for your help. Andrew From tss at iki.fi Fri Mar 30 03:07:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 03:07:40 +0300 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6B5CB9.9080204@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: On 22.3.2012, at 19.09, Martin Schitter wrote: > Am 2012-03-22 03:46, schrieb Gedalya: >>> >>> doveadm sync/backup via impac puts the same message all over the place... >> >> Thanks Martin, I've set up a test platform to investigate this further >> but I've been short on time... > > after some debugging a few more remarks about this problem: > > the bug only appears on recursive folder hierarchies. > if you specity option "-m INBOX" everything works fine. > > for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( > > has anyone a clue how to fix this problem in the source code? http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. From tss at iki.fi Fri Mar 30 05:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 05:27:09 +0300 Subject: [Dovecot] Dovecot migration from any IMAP/POP3 server Message-ID: With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: http://wiki2.dovecot.org/Migration/Dsync The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. From neuronetv at gmail.com Fri Mar 30 00:33:57 2012 From: neuronetv at gmail.com (neuronetv) Date: Thu, 29 Mar 2012 14:33:57 -0700 (PDT) Subject: [Dovecot] newbie: keep getting same emails in mail client Message-ID: <33544893.post@talk.nabble.com> dovecot-2.0.9-2.el6_1.1.i686 I've just set up dovecot in centos 6.2 (server install) and finally got it working (kind of). I set up a unix user (not a virtual user) sent a test email to this user but in my mail client I keep getting this test email over and over again. I don't think the fault is with the email client because other emails work fine and never duplicate and I've tweaked the account settings too, so it must be something I've done wrong in the dovecot setup. Here is my dovecot.conf file: # Dovecot configuration file protocols = pop3 imap disable_plaintext_auth = no mail_location = mbox:~/mail:INBOX=/var/spool/mail/unix-username ssl_cert = References: Message-ID: <4F7521D7.2080808@gedalya.net> On 3/29/2012 10:27 PM, Timo Sirainen wrote: > With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: > > http://wiki2.dovecot.org/Migration/Dsync > > The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. > Bravo!! From joe at tao.org.uk Fri Mar 30 14:29:39 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 12:29:39 +0100 Subject: [Dovecot] How do I test if the anti spam plugin is working? Message-ID: <3779AD95-CA9E-484B-8B63-039F50B0426E@tao.org.uk> I've configured the dspam anti spam plugin, but it doesn't appear to be doing anything when I move mail between mailboxes. Can anyone help me determine what's going on? The plugin appears to be loading; at least if I don't define all the required configuration options I get a complaint in the log file. But beyond that I don't see any activity. Here's the config: babel# dovecot --version 2.1.3 The dovecot anti spam plugin config is: protocol imap { mail_plugins = $mail_plugins antispam } and plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = dspam antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_trash_pattern_ignorecase = trash;Deleted * antispam_spam_pattern = SPAM antispam_dspam_binary = /usr/local/bin/dspam antispam_dspam_args = --deliver=;--user;%n at _%d } I don't even appear to be seeing any log entries from the plugin. I've moving an email from my main mailbox into a mailbox called 'SPAM', which is how I thought that it was supposed to be triggered. Cheers, Joe From pw at wk-serv.de Fri Mar 30 14:31:58 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 13:31:58 +0200 Subject: [Dovecot] Hints for a NFS-Setup Message-ID: <4F7599AE.9080300@wk-serv.de> Hi everyone, as I have often trouble with OCFS2 I want to switch to NFS but I'm not sure how to rebuild my cluster with regard to locking and indexing problems. By now my I have a four server configuration (there are another 2 servers for outgoing mail but they can be ignored): MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 As far as I understood I will get poor performance if I'd just switch from OCFS2 to NFS (while keeping this configuration) with 4 hosts accessing the NFS-share and the index files on it and it is recommended to assign users to a specific host (http://wiki2.dovecot.org/NFS). I'm uncertain what's the meaning of "user" in this context. Is it an IMAP-User or every incoming mail? An IMAP-User assigned to a specific IMAP-Server is ok for me and I could store and profit from local index files. However, I want my incoming mailservers to be equally receiving mails. Both should accept mails for every mailbox but in this case I won't have local indexes. I would appreciate any hints. Patrick From busseniu at in.tum.de Fri Mar 30 14:37:44 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Fri, 30 Mar 2012 13:37:44 +0200 Subject: [Dovecot] Dovecot allows creation of folders outside of a user's directory Message-ID: <4F759B08.1060603@in.tum.de> Hi, in our dovecot 2.0 setup with shared folders, users can make dovecot create directories outside their mail directory. Which is a bit scary imho. The following command: . create inbox.shared.abc123 or even . create "inbox.shared.strange &ANY- characters" -- even though it will fail with a "permission denied" error -- will create a directory like "/mail/users/strange &ANY- characters". That directory will only contain a subdirectory "Maildir" and therein dovecot-acl-list. I think basically the reason for this behaviour is that Dovecot checks whether the directory has enough ACLs for the user to access it, and auto-creates the directory in the process. Is there way to avoid this auto-creation - or maybe a way to make Dovecot check whether the directory name is an existing username? Here's a config to reproduce this: # 2.0.19: /usr/local/dovecot/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-35-server x86_64 Ubuntu 10.04.4 LTS auth_username_format = %Ln disable_plaintext_auth = no mail_gid = vmail mail_home = /mail/users/%u mail_location = maildir:~/Maildir mail_plugins = " acl" mail_uid = vmail maildir_very_dirty_syncs = yes namespace default { inbox = yes location = prefix = INBOX. separator = . type = private } namespace sharedns { inbox = no list = children location = maildir:/mail/users/%%u/Maildir prefix = INBOX.shared.%%u. separator = . subscriptions = no type = shared } passdb { args = /usr/local/dovecot/etc/dovecot/users driver = passwd-file } plugin { acl = vfile:/usr/local/dovecot/etc/dovecot/global-acls:cache_secs=300 acl_shared_dict = file:/mail/vmail/shared-mailboxes.db } service auth { unix_listener auth-userdb { group = vmail mode = 0660 } } ssl_cert = Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From nick at mobilia.it Fri Mar 30 15:38:01 2012 From: nick at mobilia.it (Nick Warr) Date: Fri, 30 Mar 2012 14:38:01 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F7599AE.9080300@wk-serv.de> References: <4F7599AE.9080300@wk-serv.de> Message-ID: <4F75A929.9090902@mobilia.it> Il 30/03/2012 13.31, Patrick Westenberg ha scritto: > Hi everyone, > > as I have often trouble with OCFS2 I want to switch to NFS but > I'm not sure how to rebuild my cluster with regard to locking > and indexing problems. > > By now my I have a four server configuration (there are another 2 > servers for outgoing mail but they can be ignored): > > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 > IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 > > As far as I understood I will get poor performance if I'd just switch > from OCFS2 to NFS (while keeping this configuration) with 4 hosts > accessing the NFS-share and the index files on it and it is recommended > to assign users to a specific host (http://wiki2.dovecot.org/NFS). > > I'm uncertain what's the meaning of "user" in this context. Is it an > IMAP-User or every incoming mail? > > An IMAP-User assigned to a specific IMAP-Server is ok for me and I > could store and profit from local index files. However, I want my > incoming mailservers to be equally receiving mails. Both should accept > mails for every mailbox but in this case I won't have local indexes. > > I would appreciate any hints. > > Patrick If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). SMTP wouldn't have to be balanced in the same way, you could just use round robin in that case.. I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. From andy at xecu.net Fri Mar 30 16:03:19 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:03:19 -0400 (EDT) Subject: [Dovecot] Multiple instances In-Reply-To: References: Message-ID: <20120330083210.L21999@shell.xecu.net> Sorry to respond to an old post, but I've just recently begun implementing multiple instances to facilitate our director proxies running along with our normal dovecot config on the same servers in the cluster. This is a VERY useful feature Timo, it may need just a little refinement. On Mon, 6 Feb 2012, Timo Sirainen wrote: > # doveadm instance remove proxy Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but when I do this, while it dissapears from doveadm, it still responds to pop/imap requests, and the process continues to run. Is remove supposed to be different than say, "stop"? > It would be possible to add commands to start/shutdown some/all > instances using doveadm, but is it all that useful? I'd guess people > would have their own init.d scripts anyway doing that. Eh, in a FreeBSD port-build environment, I have to hack something in place in the rc script that gets installed, and then make sure to duplicate it every time I upgrade dovecot...not ideal. So, if dovecot had some sort of mechanism in the main config file to alert it of the additional instances to start and their config files, that would be nice. Or, if you were to add a "instance_enable" switch in the config files and then have dovecot scan the /usr/local/etc/dovecot directory for appropriate config files to automatically parse. I dunno, it doesn't feel right to push the startup of the additional instances outside of dovecot. For example, consider postfix's master.cf file. BTW somebody needs to poke the dovecot port maintaner, he still has 2.0.18, I had to manually update the port to get 2.1.3. > Anything else that could be useful related to this? Yes...we should probably be able to start instances back up as well. Other then that, looks good. Definitely a great feature. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From andy at xecu.net Fri Mar 30 16:25:11 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:25:11 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides Message-ID: <20120330091204.B22325@shell.xecu.net> I've recently set up a director proxy environment on my test servers, with the intention of deploying on our cluster soon. One thing I found confusing in the proxying documentation [1] was the first bit about their being two ways to do the authentication...either you have the proxy forward the auth to the real server for authentication, or you have the proxy authenticate it and then login to the real server with a master password. Well, we use /bin/checkpassword authentication which hooks into a variety of subsytems for various specific customer needs, and sometimes we need to know the username AND password of the user in order to determine their home directory information. So, using a master password (which requires the back-end server not getting the user password) is out. However, when we have the front-end server do a static director proxy, the problem is that authentication failures are logged on the back-end server with a source IP of the proxy, and no authentication failure with the client IP address is logged on the proxy. So, fail2ban (which is a MUST these days, at least for us) will not be able to properly filter out the brute force attackers. My solution was an alternative: I authenticate with our /bin/checkpassword on the proxy, which authenticates the user and only at that point returns the proxy=y nopassword=y switch to proxy the connection and forward the authentication. As a result, we get logs on the proxy for failed attempts, and the full username and password is supplied to the back-end server for proper processing. Food for thought in case anybody else is implementing this. Thanks, Andy [1] http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From ms at mur.at Fri Mar 30 17:15:28 2012 From: ms at mur.at (Martin Schitter) Date: Fri, 30 Mar 2012 16:15:28 +0200 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: <4F75C000.2010201@mur.at> Am 2012-03-30 02:07, schrieb Timo Sirainen: >> has anyone a clue how to fix this problem in the source code? > > http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. thanks! -- now it works! :) From tss at iki.fi Fri Mar 30 17:30:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:30:12 +0300 Subject: [Dovecot] Multiple instances In-Reply-To: <20120330083210.L21999@shell.xecu.net> References: <20120330083210.L21999@shell.xecu.net> Message-ID: <1E4998FE-26F5-4F2C-A1DB-D4EF244A15A5@iki.fi> On 30.3.2012, at 16.03, Andy Dills wrote: > On Mon, 6 Feb 2012, Timo Sirainen wrote: > >> # doveadm instance remove proxy > > Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but > when I do this, while it dissapears from doveadm, it still responds to > pop/imap requests, and the process continues to run. > > Is remove supposed to be different than say, "stop"? Yes, the "remove" is meant to simply remove already stopped instances, e.g. some test instances. You can stop instances with "doveadm -i proxy stop". Dunno if there should be another "doveadm instance stop proxy" alias for that?.. >> It would be possible to add commands to start/shutdown some/all >> instances using doveadm, but is it all that useful? I'd guess people >> would have their own init.d scripts anyway doing that. > > Eh, in a FreeBSD port-build environment, I have to hack something in place > in the rc script that gets installed, and then make sure to duplicate it > every time I upgrade dovecot...not ideal. > > So, if dovecot had some sort of mechanism in the main config file to alert > it of the additional instances to start and their config files, that would > be nice. > > Or, if you were to add a "instance_enable" switch in the config files and > then have dovecot scan the /usr/local/etc/dovecot directory for > appropriate config files to automatically parse. Hmm. Perhaps a "doveadm instance auto yes|no" command to set which instances are started up automatically when Dovecot starts up. From tss at iki.fi Fri Mar 30 17:35:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:35:33 +0300 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: On 30.3.2012, at 15.38, Nick Warr wrote: >> As far as I understood I will get poor performance if I'd just switch >> from OCFS2 to NFS (while keeping this configuration) with 4 hosts >> accessing the NFS-share and the index files on it and it is recommended >> to assign users to a specific host (http://wiki2.dovecot.org/NFS). >> >> I'm uncertain what's the meaning of "user" in this context. Is it an >> IMAP-User or every incoming mail? IMAP, POP3, LDA so everything. > If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). IP stickiness isn't enough if user uses more than one IMAP client, which is pretty common nowadays. And doesn't help at all with LDA. > I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. Anything else except Dovecot director will cause corruption with NFS. Several really large sites already use director in production. From tss at iki.fi Fri Mar 30 17:39:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:39:34 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330091204.B22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> Message-ID: On 30.3.2012, at 16.25, Andy Dills wrote: > However, when we have the front-end server do a static director proxy, the > problem is that authentication failures are logged on the back-end server > with a source IP of the proxy, and no authentication failure with the > client IP address is logged on the proxy. So, fail2ban (which is a MUST > these days, at least for us) will not be able to properly filter out the > brute force attackers. This is a simple fix (and something you should do anyway): Add the proxy's IP/netmask to login_trusted_networks setting in the remote server. For this to work with POP3 you need v2.1.2+. > My solution was an alternative: I authenticate with our /bin/checkpassword > on the proxy, which authenticates the user and only at that point returns > the proxy=y nopassword=y switch to proxy the connection and forward the > authentication. Hm. Doesn't it do that even without nopassword=y? From andy at xecu.net Fri Mar 30 17:51:00 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 10:51:00 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: References: <20120330091204.B22325@shell.xecu.net> Message-ID: <20120330104543.N22325@shell.xecu.net> On Fri, 30 Mar 2012, Timo Sirainen wrote: > On 30.3.2012, at 16.25, Andy Dills wrote: > > > However, when we have the front-end server do a static director proxy, the > > problem is that authentication failures are logged on the back-end server > > with a source IP of the proxy, and no authentication failure with the > > client IP address is logged on the proxy. So, fail2ban (which is a MUST > > these days, at least for us) will not be able to properly filter out the > > brute force attackers. > > This is a simple fix (and something you should do anyway): Add the > proxy's IP/netmask to login_trusted_networks setting in the remote > server. For this to work with POP3 you need v2.1.2+. Well, the problem isn't that my proxies would be banned; the problem is I have no way of seeing the remote IP of the failed authentication so I can ban the people who should be banned. > > My solution was an alternative: I authenticate with our /bin/checkpassword > > on the proxy, which authenticates the user and only at that point returns > > the proxy=y nopassword=y switch to proxy the connection and forward the > > authentication. > > Hm. Doesn't it do that even without nopassword=y? Perhaps...I was going by the docs which seemed to suggest that nopassword=y was how you get the proxy to forward the users authentication credentials to the back-end server. I had been trying a lot of different things, and it was only when I realized I needed to not do a static passdb on the proxy, but instead do a full authentication so that the auth failure would be logged on the proxy for fail2ban, that things began to work the way I needed. It seems obvious in retrospect, but for whatever reason the way the docs were written made me feel like having the full authentication happen on both the proxy and the backend wasn't possible. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From tss at iki.fi Fri Mar 30 18:03:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 18:03:00 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330104543.N22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> <20120330104543.N22325@shell.xecu.net> Message-ID: <0E6CE6EC-8682-4DF3-B983-DA0906B41BF2@iki.fi> On 30.3.2012, at 17.51, Andy Dills wrote: > On Fri, 30 Mar 2012, Timo Sirainen wrote: > >> On 30.3.2012, at 16.25, Andy Dills wrote: >> >>> However, when we have the front-end server do a static director proxy, the >>> problem is that authentication failures are logged on the back-end server >>> with a source IP of the proxy, and no authentication failure with the >>> client IP address is logged on the proxy. So, fail2ban (which is a MUST >>> these days, at least for us) will not be able to properly filter out the >>> brute force attackers. >> >> This is a simple fix (and something you should do anyway): Add the >> proxy's IP/netmask to login_trusted_networks setting in the remote >> server. For this to work with POP3 you need v2.1.2+. > > Well, the problem isn't that my proxies would be banned; the problem is I > have no way of seeing the remote IP of the failed authentication so I can > ban the people who should be banned. This is what the setting changes. The remote IP will be seen by the backends. > It seems obvious in retrospect, but for whatever reason the way the docs > were written made me feel like having the full authentication happen on > both the proxy and the backend wasn't possible. Oh. This is a pretty common configuration. I guess the docs could be clarified. From pw at wk-serv.de Fri Mar 30 22:37:05 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 21:37:05 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: <4F760B61.3090209@wk-serv.de> Nick Warr schrieb: > I think some of the new Dovecot (director?) software is user aware, but > I don't know if it's quite ready for production. Yes, with director it should be something like that: MTA --(lmtp)--\ /--(lmtp)--> backend1 --\ -- director -- -- NFS MTA --(lmtp)--/ \--(lmtp)--> backend2 --/ IMAP-User --> frontend1 --\ /--(imap)--> backend1 --\ -- director -- -- NFS IMAP-User --> frontend2 --/ \--(imap)--> backend2 --/ So what happens if user1 at example.tld receives a mail? - The director decides to connect to backend1 which in turn stores the mail on the NFS share and the index file locally? - Then, user1 at example.tld connects to one of the frontends. Does the director know that, earlier, this user received a mail and proxies him to backend1 too? From joe at tao.org.uk Fri Mar 30 22:44:08 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:44:08 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? Message-ID: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> I'm excited to hear that dovecot now supports imap as a mailbox type. I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. But, I can't find much documentation on imapc, other than it exists. Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? Thanks :), Joe From tss at iki.fi Fri Mar 30 22:52:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:52:18 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > I'm excited to hear that dovecot now supports imap as a mailbox type. > > I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. > > But, I can't find much documentation on imapc, other than it exists. > > Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. From joe at tao.org.uk Fri Mar 30 22:55:27 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:55:27 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> On 30 Mar 2012, at 20:52, Timo Sirainen wrote: > On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > >> I'm excited to hear that dovecot now supports imap as a mailbox type. >> >> I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. >> >> But, I can't find much documentation on imapc, other than it exists. >> >> Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? > > One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. > > http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? Joe From tss at iki.fi Fri Mar 30 22:59:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:59:47 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> Message-ID: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: >> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >> >> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >> > > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. From joe at tao.org.uk Fri Mar 30 23:02:46 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:02:46 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: On 30 Mar 2012, at 20:59, Timo Sirainen wrote: > On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: > >>> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >>> >>> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >>> >> >> But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? > > It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? Joe From tss at iki.fi Fri Mar 30 23:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:08:07 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: >> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. > > I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. From joe at tao.org.uk Fri Mar 30 23:26:24 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:26:24 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> Message-ID: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> On 30 Mar 2012, at 21:08, Timo Sirainen wrote: > On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: > >>> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. >> >> I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? > > I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. Do you think that this will reliably do the trick? Joe -------------- next part -------------- A non-text attachment was scrubbed... Name: imapc.patch Type: application/octet-stream Size: 2299 bytes Desc: not available URL: From tss at iki.fi Fri Mar 30 23:28:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:28:53 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> Message-ID: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: >> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. > > Do you think that this will reliably do the trick? > > If it works at all, I guess it should work reliably. :) From joe at tao.org.uk Fri Mar 30 23:42:13 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:42:13 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> Message-ID: <695D3A65-CFBF-4DC3-9DAC-E0C299ED0E6D@tao.org.uk> On 30 Mar 2012, at 21:28, Timo Sirainen wrote: > On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: > >>> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. >> >> Do you think that this will reliably do the trick? >> >> > > If it works at all, I guess it should work reliably. :) Hmm. Didn't appear to: % dsync -Dv -u joe at local.com -o imapc_host=mail.remoteimap.com -o imapc_port=143 -o imapc_username=joe@ remoteimap.com -o imapc_password='somepass' mirror imapc: > & /tmp/output produced the following output. What do you think? Joe. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output2.txt URL: From ka at pacific.net Sat Mar 31 00:30:05 2012 From: ka at pacific.net (Ken A) Date: Fri, 30 Mar 2012 16:30:05 -0500 Subject: [Dovecot] Problem managing mbox In-Reply-To: <1332165600.26095.75.camel@innu> References: <4F64E923.5060401@gmail.com> <1332165600.26095.75.camel@innu> Message-ID: <4F7625DD.1050109@pacific.net> I have seen this error msg too. Deleting the index files 'fixes' the error messages in the log. The indexes seem to get corrupted sometimes, when the client is logged doing: copy dest=Trash, delete, expunge Ken On 3/19/2012 9:00 AM, Timo Sirainen wrote: > On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: >> Hello, >> >> I have a problem with dovecot. seems that do not erase mail that mail >> client request to be erased. > > Are you sure the clients have actually issued the EXPUNGE command, > rather than simply marked the mail with \Deleted flag? > >> And I have this errors: >>> Error: Next message unexpectedly corrupted in mbox file >> Info: > > mbox code isn't perfect, but if this doesn't happen often it shouldn't > matter much. doveconf -n output might have been helpful in giving more > suggestions. > > > -- Ken Anderson Pacific Internet - http://www.pacific.net Latest Pacific.Net Status - http://twitter.com/pacnetstatus From dm-list-email-dovecot at scs.stanford.edu Sat Mar 31 18:38:48 2012 From: dm-list-email-dovecot at scs.stanford.edu (dm-list-email-dovecot at scs.stanford.edu) Date: Sat, 31 Mar 2012 08:38:48 -0700 Subject: [Dovecot] dovecot 2.1 breaks FTS + pre-auth? Message-ID: <878vigok53.wl@ta.scs.stanford.edu> Hi. I use dovecot in the simplest possible way, as an IMAP server in pre-auth mode over ssh or just locally over a unix-domain socket (e.g., with offlineimap, which runs much faster using dovecot for the local message store). Ideally I would like to avoid running any extra daemons or setting up anything as root. Until recently, this has worked fine by just setting the CONFIG_FILE environment variable to something in my home directory. Here is my configuration: $ export CONFIG_FILE=$HOME/etc/dovecot.conf $ dovecot -n # 2.1.3: /home/dm/etc/dovecot.conf # OS: Linux 3.2.13-1-ARCH x86_64 mail_location = maildir:~/Mail/inbox mail_plugins = " fts fts_squat" plugin { fts = squat fts_squat = partial=4 full=10 } doveconf: Error: ssl enabled, but ssl_cert not set doveconf: Fatal: Error in configuration file /home/dm/etc/dovecot.conf: ssl enabled, but ssl_cert not set Full text search used to work just fine with this configuration, and still does on a machine I have running dovecot 2.0.13. However, on the machine with 2.1, I get errors about /var/run/dovecot/index not existing. $ printf "a select INBOX\nb search text xyzzy\nc logout\n" \ | /usr/lib/dovecot/imap * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE] Logged in as dm imap(dm): Error: net_connect_unix(/var/run/dovecot/indexer) failed: No such file or directory ... Needless to say, no dovecot.index.search or dovecot.index.search.uids file is created after this error. While I can't write /var/run/dovecot, this is not a permission issue. For example, adding base_dir=/home/dm (my home directory) to the configuration file yields the same error for /home/dm/indexer. I'm guessing something has changed where imap requires an indexer daemon and doesn't launch it in pre-auth mode any more, but I can't find anything about this in the documentation. In short, if anyone can tell me how to use FTS in conjunction with pre-auth mode or point me to a working example, I would appreciate it. From ghilt at shadowprojects.org Sat Mar 31 19:55:47 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 18:55:47 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 Message-ID: <4F773713.8030904@shadowprojects.org> Hello, Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. dovecot-antispam package is compiled for dovecot 2.0.15. So, i'm trying to compile a new dovecot antispam plugin. I followed dovecot wiki but i'm running into this error : Successfully compiled dspam.c (plugin). mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying_via_save' mailbox.c: In function 'antispam_save_finish': mailbox.c:174:12: error: 'struct mail_save_context' has no member named 'copying_via_save' Failed to compile mailbox.c (plugin)! Any hint please ? Regards, -- Guillaume Hilt From e-frog at gmx.de Sat Mar 31 20:01:53 2012 From: e-frog at gmx.de (e-frog) Date: Sat, 31 Mar 2012 19:01:53 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773713.8030904@shadowprojects.org> References: <4F773713.8030904@shadowprojects.org> Message-ID: <4F773881.10907@gmx.de> On 31.03.2012 18:55, wrote Guillaume Hilt: > Hello, > > Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. > dovecot-antispam package is compiled for dovecot 2.0.15. > > So, i'm trying to compile a new dovecot antispam plugin. > I followed dovecot wiki but i'm running into this error : > Successfully compiled dspam.c (plugin). > mailbox.c: In function 'antispam_save_begin': > mailbox.c:138:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > mailbox.c: In function 'antispam_save_finish': > mailbox.c:174:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > Failed to compile mailbox.c (plugin)! > > Any hint please ? > > Regards, > Revert this patch and it should work: http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 From ghilt at shadowprojects.org Sat Mar 31 20:13:55 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 19:13:55 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773881.10907@gmx.de> References: <4F773713.8030904@shadowprojects.org> <4F773881.10907@gmx.de> Message-ID: <4F773B53.5070004@shadowprojects.org> Thanks, compilation works fine now. Guillaume Hilt Le 31/03/2012 19:01, e-frog a ?crit : > On 31.03.2012 18:55, wrote Guillaume Hilt: >> Hello, >> >> Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. >> dovecot-antispam package is compiled for dovecot 2.0.15. >> >> So, i'm trying to compile a new dovecot antispam plugin. >> I followed dovecot wiki but i'm running into this error : >> Successfully compiled dspam.c (plugin). >> mailbox.c: In function 'antispam_save_begin': >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> mailbox.c: In function 'antispam_save_finish': >> mailbox.c:174:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> Failed to compile mailbox.c (plugin)! >> >> Any hint please ? >> >> Regards, >> > > Revert this patch and it should work: > > http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 > > > > From mcbdovecot at robuust.nl Thu Mar 1 00:37:24 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Wed, 29 Feb 2012 23:37:24 +0100 (CET) Subject: [Dovecot] remove messages once downloaded In-Reply-To: References: <4F4C8DB2.9060302@mobilia.it> <6477DAB9-6840-410D-875D-5885A7E9962D@leuxner.net> Message-ID: On Tue, 28 Feb 2012, Thomas Leuxner wrote: > Am 28.02.2012 um 09:46 schrieb Thomas Leuxner: > >> Assuming this is the task you could also use: >> >> $ doveadm expunge -u jane.doe at example.org before 2012-02-01 unseen >> >> Play around with the scope a bit before you actually run it: >> >> $ doveadm search -u jane.doe at example.org before 2012-02-01 unseen | wc -l > > Of course this was supposed to read 'seen' :/ Maybe so, maybe not. I've seen people with over 2GB of unread mail in their inbox alone. Subscribed to dozens of mailing lists, and just not reading 95% of all incoming email. Don't know how pop handles 'leave on server' with regard to mail storage. Will messages loose their 'unseen' state when doing something like a TOP msgnumber maxint or something similar? Will messages loose their 'unseen' state when an RSET is issued after downloading? If you don't want people to keep a lot of mail in popboxes, quota and sending out warning messages would be the better way to handle this, I think. Applying 'magic' to customer's email might not be seen as a very polite action. Just my 2 cents.. -- Maarten From dbenfell at gmail.com Thu Mar 1 02:54:40 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 16:54:40 -0800 Subject: [Dovecot] need simpler directions on user authentication Message-ID: <20120301005440.GA5290@-e> Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From dbenfell at gmail.com Thu Mar 1 03:40:56 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 17:40:56 -0800 Subject: [Dovecot] need simpler instructions for user authentication Message-ID: Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com From stan at hardwarefreak.com Thu Mar 1 04:15:27 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 20:15:27 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4BB559.6050405@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> Message-ID: <4F4EDBBF.40004@hardwarefreak.com> On 2/27/2012 10:54 AM, Charles Marcus wrote: > These two locations will be connected via a private Gb ethernet > connection, and each location will have its own internet connection (I > think - still waiting on some numbers to present to the owner to see > what he wants to do in that regard, but that will be my recommendation), > so bandwidth for replication won't be an issue. Say you're a boutique mail services provider or some such. In your own datacenter you have a Dovecot server w/64 processors, 512GB RAM, and 4 dual port 8Gb fiber channel cards. It's connected via 8 redundant fiber channel links to 4 SAN array units, each housing 120 x15k SAS drives, 480 drives total, ~140,000 random IOPs. This gear eats 36U of a 40U rack, and about $400,000 USD out of your wallet. In the remaining 4U at the top of the rack you have a router, with two GbE links connected to the server, and an OC-12 SONET fiber link (~$15k-20k USD/month) to a national ISP backbone. Not many years ago OC-12s comprised the backbone links of the net. OC-48s handle that today. Today OC-12s are most often used to link midsized ISPs to national ISPs, act as the internal backbone of midsized ISPs, and link large ISPs' remote facilities to the backbone. Q: How many concurrent IMAP clients could you serve with this setup before hitting a bottleneck at any point in the architecture? What is the first bottleneck you'd run into? The correct answer to this question, and the subsequent discussion that will surely take place, may open your eyes a bit, and prompt you to rethink some of your assumptions that went into the architectural decisions you've presented here. -- Stan From david at davidrenstrom.com Thu Mar 1 04:53:31 2012 From: david at davidrenstrom.com (=?us-ascii?Q?David_Renstrom?=) Date: Thu, 1 Mar 2012 03:53:31 +0100 Subject: [Dovecot] Permission problems Message-ID: <34D750D1EC41408BA8AAA495077048CD@daviddator> Hi, I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and Postfixadmin on Fedora. Almost everything works correctly now. The only thing bugging me right now is that I always have to change the access privileges of the directories in /var/run/dovecot/ manually to 777 to make it work. As you understand, I'm not particularly happy about this as it doesn't feel secure at all. And I also have to do this every time I reboot since Dovecot resets the old access priviliges (600). As you see I've got a vmail user who belongs to the mail group. Almost all directories under /var/run/dovecot belong to user root and group root. They all have the permissions rw--------. Below is some output from the log file dovecot.log: Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Mar 01 03:31:17 master: Error: service(auth): command startup failed, throttling Mar 01 03:31:46 imap-login: Info: Disconnected (auth failed, 1 attempts): method=PLAIN, rip=94.254.99.6, lip=91.123.204.174 What is wrong here and how can I correct it? Thanks in advance! /David R. From stan at hardwarefreak.com Thu Mar 1 06:16:47 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 22:16:47 -0600 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: <4F4EF82F.6090306@hardwarefreak.com> On 2/29/2012 8:53 PM, David Renstrom wrote: > Hi, > > I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and > Postfixadmin on Fedora. Almost everything works correctly now. The only > thing bugging me right now is that I always have to change the access > privileges of the directories in /var/run/dovecot/ manually to 777 to make > it work. As you understand, I'm not particularly happy about this as it > doesn't feel secure at all. And I also have to do this every time I reboot > since Dovecot resets the old access priviliges (600). > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. Is this a Fedora Dovecot RPM, or did you use source? Do you have selinux/apparmor enabled? Do you like shellfish? -- Stan From tss at iki.fi Thu Mar 1 06:30:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:30:19 +0200 Subject: [Dovecot] need simpler instructions for user authentication In-Reply-To: References: Message-ID: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> On 1.3.2012, at 3.40, David Benfell wrote: > My situation is this: I have postfix running on a Linode under Arch Linux. > A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." You're missing userdb. > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: userdb { driver = passwd } From tss at iki.fi Thu Mar 1 06:34:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:34:19 +0200 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: On 1.3.2012, at 4.53, David Renstrom wrote: > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. That's how they should be mostly. > Below is some output from the log file dovecot.log: > > Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory > /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) > missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Why is your auth process running as vmail? Show your dovecot -n output From dbenfell at gmail.com Thu Mar 1 07:53:37 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 21:53:37 -0800 Subject: [Dovecot] SOLVED - Re: need simpler instructions for user authentication In-Reply-To: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> References: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> Message-ID: <20120301055337.GC25866@-e> On Thu, Mar 01, 2012 at 06:30:19AM +0200, Timo Sirainen wrote: > > You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: Sorry. The Arch Linux package is dovecot-2.1.1-3. > > userdb { > driver = passwd > } It *looks* like this worked. Thank you *very* much! David Benfell benfell at parts-unknown.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From joshua at hybrid.pl Thu Mar 1 10:21:50 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Thu, 1 Mar 2012 09:21:50 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? Message-ID: Hi all, While reading the "Multiple locations, 2 servers - planning questions..." thread, it just hit me that I was planning to do something that seems to be a bit hazardous... I am setting up an high-availability server set, which consists of two (or more) servers with common disk space, all set behind redundant hardware load balancers. At first, there will be two servers and disk space will be kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what creates kind of networked RAID1 storage space). This will be done mostly to keep WWW service available in case when one of servers fails. However, if we have everything redundant, why not have the same with SMTP and POP3/IMAP? But - won't anything fail if two (or more) dovecots are accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If it is possible, that would be a good solution for the mentioned thread. If not - where are possible dangers, and are there any plans to make it possible in the future? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From jtam.home at gmail.com Thu Mar 1 10:44:58 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Thu, 1 Mar 2012 00:44:58 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid I would like to run various doveadm commands that involves all (mail) users like doveadm expunge -A mailbox Trash savedbefore 30d but any doveadm command that uses "-A" to iterate through all users will stop processing at the first account with UIDname mapping for utilities like ls, chown, etc. There are various workaround like iterating manually, or setting first_valid_uid to zero, or even reordering users to put all system accounts at the end, but is there a better way to do this? Joseph Tam From CMarcus at Media-Brokers.com Thu Mar 1 13:43:47 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 01 Mar 2012 06:43:47 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4EDBBF.40004@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> Message-ID: <4F4F60F3.5050508@Media-Brokers.com> On 2012-02-29 9:15 PM, Stan Hoeppner wrote: > Q: How many concurrent IMAP clients could you serve with this setup > before hitting a bottleneck at any point in the architecture? No idea how to calculate it... > What is the first bottleneck you'd run into? Unless this is a trick question, the OC-12 link (since it is only 644Mb), and the next bottleneck would be the 2 GbE server connections to the router (are these bonded? if so, what mode?... > The correct answer to this question, and the subsequent discussion that > will surely take place, may open your eyes a bit, and prompt you to > rethink some of your assumptions that went into the architectural > decisions you've presented here. Since the vast majority of our connections will be *local*, I'm unconcerned about the internet connect speeds (one office has a 100/10Mb Cable (Comcast Business Class) connection, the other will have a 100/100Mb fiber/ethernet connection). My main priority is that the user experience at each physical location be optimal, which is why I'm more focused on making sure each offices users are connected to only the local server for all services (file/print/mail). I also neglected to mention how each server would be physically connected to the network, which I guess I should have done, since I'm fairly sure that will be the bottleneck I should mostly be concerned about... My choices are, as I see it, single GbE connections, or add some multiport GbE cards (these Dells support up to 3 PCIe cards) and bond some ports together for each VM. 10GbE is simply not in our price range (and I don't think we need it anyway), although I did stumble on these while googling and am waiting on pricing, since they claim to be 'much cheaper': http://www.mellanox.com/ethernet/ Since neither the multi-port GbE cards or decent switches that have reliable support for bonding/teaming are really not that expensive (especially when comparing to 10GbE solutions), I don't really see any reason *not* to do this (at a minimum I'd get redundancy if one of the ports on the server failed), but I'm also not sure which mode would be best - round-robin or IEEE 802.3ad dynamic link aggregation? Obviously, I don't have the experience or expertise to answer these questions myself (never analyzed IMAP traffic to have an idea of the bandwidth each user uses, and probably wouldn't trust my efforts if I made the attempt). Hopefully, there are some people here who have a rough idea, which is why I brought this question up here. Oh - and I am/will be working with a local I.T. services company to help with the design and implementation (since obviously I don't have the experience to do this myself), and will be asking them these same questions, I just like to usually know the general answers to questions like this ahead of time, so that I know if the guys I'm hiring know what they are doing and are giving me the best options for my budget. Thanks for your thoughts... -- Best regards, Charles From mark at ecs.vuw.ac.nz Thu Mar 1 13:49:11 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:49:11 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> Message-ID: <4F4F6237.1060100@ecs.vuw.ac.nz> On 02/29/12 23:41, Timo Sirainen wrote: >> Oh, yes, this is a bug in Dovecot.. > > Hmm. Or it looked like a bug, since it replied only with "+", so I thought all auth mechanisms would have such a bug, but no.. So I'm not really sure why it's not sending more data. I don't have a Kerberos setup to test this with. v2.1's GSSAPI code is anyway identical to v2.0's. With auth debugging on a successful connection gives: Mar 2 00:33:34 bats dovecot: auth: Debug: auth client connected (pid=1584) Mar 2 00:33:34 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49116 Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 YIGaBgkqhkiG9xIBAgICAG+BijCBh6ADAgEFoQMCAQ+iezB5oAMCARKicgRwXldfEmBHqH3DiVbw7aXtx54iBNjo1Rv/KxBSK5G3TmYFm3YskYN/23EiaOQ0Tdyi4bc4jhv5cFWMpH/xM89wAFJVW8Ue27/fmCasfDWXE+i4TKA3UCm78Wy8YyiNVae8X341LspBk86R1Zl5MNRMvA== Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): Negotiated security layer Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 BQQF/wAMAAwAAAAA47846FHFUOykdXinGYvMKwH///8= Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: client out: OK 1 user=mark and the failing kmail gives Mar 2 00:38:08 bats dovecot: auth: Debug: auth client connected (pid=2720) Mar 2 00:38:08 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49118 resp= Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:38:08 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:38:08 bats dovecot: auth: Debug: client in: CONT Mar 2 00:38:08 bats dovecot: auth: GSSAPI(mark at ECS.VUW.AC.NZ,130.195.5.88): Invalid base64 data in continued response Mar 2 00:38:08 bats dovecot: auth: Debug: client out: FAIL 1 reason=Invalid base64 data in continued response so what bit of the code should I be looking at to see what happens between the "security context state completed" and the "client out"? cheers mark From tss at iki.fi Thu Mar 1 13:52:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 13:52:34 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6237.1060100@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> Message-ID: <1330602754.2081.34.camel@innu> On Fri, 2012-03-02 at 00:49 +1300, Mark Davies wrote: > so what bit of the code should I be looking at to see what happens > between the "security context state completed" and the "client out"? All of the code is in mech-gssapi.c Are these working vs. non-working Dovecots in same or different servers? From joseba.torre at ehu.es Thu Mar 1 13:52:55 2012 From: joseba.torre at ehu.es (Joseba Torre) Date: Thu, 01 Mar 2012 12:52:55 +0100 Subject: [Dovecot] need simpler directions on user authentication In-Reply-To: <20120301005440.GA5290@-e> References: <20120301005440.GA5290@-e> Message-ID: <4F4F6317.2090605@ehu.es> El 01/03/12 01:54, David Benfell escribi?: > Hi all, > > My situation is this: I have postfix running on a Linode under Arch > Linux. A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." > > I do not want virtual users. I do not particularly want SQL or LDAP. I > do have multiple domains. I do use Maildir. I already have SSL > certificates for the server which I did manage to get dovecot to quit > complaining about. > > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? > > Thank you! > David Benfell > dbenfell at gmail.com If you give us some information maybe we could help you. At least, we need the output of doveadm config -n and the relevant log lines. From mark at ecs.vuw.ac.nz Thu Mar 1 13:57:56 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:57:56 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330602754.2081.34.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> Message-ID: <4F4F6444.1050107@ecs.vuw.ac.nz> On 03/02/12 00:52, Timo Sirainen wrote: >> so what bit of the code should I be looking at to see what happens >> between the "security context state completed" and the "client out"? > > All of the code is in mech-gssapi.c Yes, I'm just trying to work out the flow of the calls in and out of there. > Are these working vs. non-working Dovecots in same or different servers? All the working and non working connections are against a single dovecot instance, just using different clients. cheers mark From tss at iki.fi Thu Mar 1 14:04:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 14:04:30 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6444.1050107@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> Message-ID: <1330603470.2081.37.camel@innu> On Fri, 2012-03-02 at 00:57 +1300, Mark Davies wrote: > > On 03/02/12 00:52, Timo Sirainen wrote: > >> so what bit of the code should I be looking at to see what happens > >> between the "security context state completed" and the "client out"? > > > > All of the code is in mech-gssapi.c > > Yes, I'm just trying to work out the flow of the calls in and out of there. The problem is that that mech_gssapi_sec_context() calls gss_accept_sec_context(), which is supposed to return some output in output_token, but it doesn't. So I don't think there's anything in Dovecot code that is helpful in debugging this. You'd have to look into the GSSAPI/Kerbereros libraries. > > Are these working vs. non-working Dovecots in same or different servers? > > All the working and non working connections are against a single dovecot > instance, just using different clients. Oh. So GSSAPI in general is working, just not with kmail. I think if you downgraded to Dovecot v2.0 in your current system it would fail as well. The difference between your previously working system and currently working system is the GSSAPI/Kerberos libraries. From campbell at cnpapers.com Thu Mar 1 16:16:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 01 Mar 2012 09:16:39 -0500 Subject: [Dovecot] Multiple namespaces seems to be used at the same time Message-ID: <4F4F84C7.1060502@cnpapers.com> I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Thanks for any help. The learning curve is getting less steep, but it's still a ways off to the top of the hill. steve campbell From xchris89x at googlemail.com Thu Mar 1 17:08:28 2012 From: xchris89x at googlemail.com (Chris) Date: Thu, 1 Mar 2012 16:08:28 +0100 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: 2012/2/28 Ralf Hildebrandt : > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. Is that fixed? -- Chris From tss at iki.fi Thu Mar 1 18:15:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 18:15:05 +0200 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: On 28.2.2012, at 20.46, Ralf Hildebrandt wrote: > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. I can't reproduce this. From slusarz at curecanti.org Thu Mar 1 20:03:46 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 11:03:46 -0700 Subject: [Dovecot] LIST-STATUS issue Message-ID: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: >> Timestamp: Thu, 01 Mar 2012 10:55:47 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. 3 seconds later... >> Timestamp: Thu, 01 Mar 2012 10:55:50 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: * OK [XPROXYREUSE] IMAP connection reused by squirrelmail-imap_proxy S: 1 OK User logged in C: 2 EXAMINE IN.dovecot (QRESYNC (1254166849 25800 18484,20260,20393,20705,20860:20868)) S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft impflag1 impflag2 impflag0 Junk seen NonJunk) S: * OK [PERMANENTFLAGS ()] Read-only mailbox. S: * 13 EXISTS S: * 0 RECENT S: * OK [UNSEEN 5] First unseen. S: * OK [UIDVALIDITY 1254166849] UIDs valid S: * OK [UIDNEXT 20869] Predicted next UID S: * OK [HIGHESTMODSEQ 25800] Highest S: 2 OK [READ-ONLY] Select completed. [...] C: 7 SEARCH RETURN (COUNT) UNSEEN S: * ESEARCH (TAG "7") COUNT 8 S: 7 OK Search completed (0.000 secs). I can confirm that I did not receive 8 dovecot list messages in 3 seconds. :) Once the mailbox is accessed, I am seeing correct counts in LIST-STATUS return again: >> Timestamp: Thu, 01 Mar 2012 10:57:28 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 8) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. I am using imapproxy for these connections, so all access are happening in the same Dovecot access. michael From tss at iki.fi Thu Mar 1 20:22:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 20:22:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: On 1.3.2012, at 20.03, Michael M Slusarz wrote: > Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... > > UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: Only with LIST-STATUS, or also STATUS itself? And with what kind of config (doveconf -n)? Can you reproduce this? LIST-STATUS should use the exact same code as STATUS. STATUS should use pretty much the same code as SELECT. mailbox_list_index=yes makes a difference, but other than that I can't really think of a reason. From slusarz at curecanti.org Thu Mar 1 21:17:41 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 12:17:41 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 1.3.2012, at 20.03, Michael M Slusarz wrote: > >> Looked at the hg commits since 2.1.1, but didn't see anything that >> was exactly on-point to this... >> >> UNSEEN counts seem to be broken when done in a LIST-STATUS call. I >> see this, for example: > > Only with LIST-STATUS, or also STATUS itself? And with what kind of > config (doveconf -n)? Can you reproduce this? I've been seeing this intermittently the last few days. When I notice it, I try to reproduce and never can. Then it eventually comes back, as in the first time I access the next morning. Maybe its some kind of cache/timeout issue. (e.g. - I can't reproduce right now) > LIST-STATUS should use the exact same code as STATUS. STATUS should > use pretty much the same code as SELECT. mailbox_list_index=yes > makes a difference, but other than that I can't really think of a > reason. # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-1-ARCH x86_64 auth_mechanisms = plain login namespace { inbox = yes location = maildir:~/Maildir prefix = separator = . type = private } namespace { location = maildir:~/mailtest/mboxtesting2 prefix = "#shared." separator = . type = shared } namespace { location = mbox:~/mailtest/mail prefix = "#public2." separator = . type = public } passdb { args = dovecot driver = pam } plugin { acl = vfile } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_cert = I'm running imapproxy as shown at http://wiki2.dovecot.org/HowTo/ImapcProxy. In fact, that's my config in the wiki. 8-) It's been working, but has performance issues when the Exchange server that's hard-coded as imapc_host=xxx.xxx.xxx.xxx doesn't happen to be the user's home exchange server. I'd like to point dovecot at the correct Exchange server based on an LDAP query, and in fact, have an LDAP search that works: DC=example,DC=com (&(objectCategory=person)(objectClass=user)(!(userAccountcontrol:1.2.840.113556.1.4.803:=2))(sAMAccountName=username)) With the exchange server being returned in the msExchHomeServerName property as: /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Can anybody toss me a clue? Once I get it working, I'll update the wiki. Thanks! Terry From dbenfell at gmail.com Fri Mar 2 03:23:41 2012 From: dbenfell at gmail.com (David Benfell) Date: Thu, 1 Mar 2012 17:23:41 -0800 Subject: [Dovecot] Thanks, was Re: need simpler instructions for user authentication Message-ID: Thanks to everyone who responded to my request. And I apologize that this message isn't properly threaded. I'm in trouble whenever I have to send mail from my gmail account because I automatically pull it down into the same place from which I send my regular mail (from benfell at parts-unknown.org ) and I haven't configured that to deal with the gmail account. That makes it difficult to reply properly to mail on this list which I have subscribed to with the gmail account so that I'd actually receive it even when I'm having problems with my main mail system. Timo Sirainen responded that I needed a userdb and offered that I should probably add the following to dovecot.conf : userdb { driver = passwd } As near as I can tell--and it's been over a day now--that solved the problem. Thanks! From stan at hardwarefreak.com Fri Mar 2 03:38:13 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 01 Mar 2012 19:38:13 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F502485.9070503@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > On 2012-02-29 9:15 PM, Stan Hoeppner wrote: >> Q: How many concurrent IMAP clients could you serve with this setup >> before hitting a bottleneck at any point in the architecture? > > No idea how to calculate it... The correct answer is approximately 120,000 concurrent users, based on an assumed average of ~3MB-5MB of ram consumed in all processes for each user. >> What is the first bottleneck you'd run into? > > Unless this is a trick question, the OC-12 link (since it is only > 644Mb), and the next bottleneck would be the 2 GbE server connections to > the router (are these bonded? if so, what mode?... It was a bit of a trick question, with a somewhat elaborate setup, designed to shift your focus/thinking. Apparently I failed in my effort here. The correct answer is that RAM will be the first bottleneck. Then disk IOPS, finally followed by the OC-12 assuming we beef up the others. > Since the vast majority of our connections will be *local*, I'm > unconcerned about the internet connect speeds (one office has a 100/10Mb > Cable (Comcast Business Class) connection, the other will have a > 100/100Mb fiber/ethernet connection). You didn't grasp why I used the OC-12 in my example. It had nothing to do with LAN/WAN, local or remote, but the total users/traffic a 600Mb/s link can carry. > My main priority is that the user experience at each physical location > be optimal, which is why I'm more focused on making sure each offices > users are connected to only the local server for all services > (file/print/mail). A single MAN (Metropolitan Area Network) 1000BASE-LX link, good for 5km, likely what you will have, is more than sufficient to carry the 2nd office site traffic while keeping all of your servers/etc where they are now. > My choices are, as I see it, single GbE connections, or add some > multiport GbE cards (these Dells support up to 3 PCIe cards) and bond > some ports together for each VM. 10GbE is simply not in our price range > (and I don't think we need it anyway), although I did stumble on these > while googling and am waiting on pricing, since they claim to be 'much > cheaper': With specs like that you must be supporting 100,000 users. ;) > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Your company/employer has less than 250 users IIRC. Is this right? You're a media company that works with files much larger than the average company. Is that correct? Let's cut to the chase shall we? Your 1000BASE-LX MAN link has an after link overhead bandwidth of just over 100MB/s full duplex. To put this into real world perspective, you can copy a single 4.7GB DVD in 47 seconds, or 1 in each direction in the same time, 2 total, 9.4GB total. You can copy 20 full DVDs over this link, 10 in each direction, in less than 8 minutes. Add heavy IMAP traffic for 500 concurrent users and it's still less than 10 minutes and the IMAP users won't have a clue if the switch VLAN QOS is setup correctly. You see GbE as mundane, slow, because it has been ubiquitous for some time, being a freebie on both servers and desktops. This is why I used the OC-12 example at $15K/month, hoping you'd start to grasp that cost has little direct relationship to performance. GbE is "free" now because the cost of the silicon to drive a 1000MHz signal over 300 meters of copper wire is no longer higher than for 100BASE-T. Here's another comparison. All internet backbone links are OC-48 at 2.5Gb/s. It takes only 2.5 GbE links to equal a backbone link. Backbone links carry the traffic of *millions* of users, all applications, all data stream types. And that's *only* 250MB/s. So, the point is, a single 1000BASE-LX MAN link is far more than plenty to carry all of the traffic you'll throw at it, and quite a bit more, with some minor QOS configuration. Consider how much money, time, and duplication of services and servers you are going to save now that you realize you need nothing other than the 1000BASE-LX MAN link, and closet switches at the second office site? Get yourself a qualified network architect. Pay for a full network traffic analysis. He'll attach sniffers at multiple points in your network to gather traffic/error/etc data. Then you'll discuss the new office, which employees/types with move there, and you'll be able to know almost precisely the average and peak bandwidth needs over the MAN link. He'll very likely tell you the same thing I have, that a single gigabit MAN link is plenty. If you hire him to do the work, he'll program the proper QOS setup to match the traffic patterns gleaned from the sniffers. -- Stan From sdavies at sdc.com.au Fri Mar 2 04:14:04 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Fri, 2 Mar 2012 12:44:04 +1030 Subject: [Dovecot] Log sybnch error Message-ID: <201203021244.05034.sdavies@sdc.com.au> My mail log has many entries like: Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: Extension header update points outside header size Version 2.1.1 dovecot -n gives: # 2.1.1: /usr/etc/dovecot/dovecot.conf # OS: Linux 2.6.33.5-desktop-2mnb i686 Mandriva Linux 2010.2 mail_access_groups = mail mail_privileged_group = mail passdb { driver = pam } protocols = pop3 imap service anvil { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = mode = 0600 user = } unix_listener anvil { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = root vsz_limit = 18446744073709551615 B } service config { chroot = client_limit = 0 drop_priv_before_exec = no executable = config extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = config unix_listener config { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service dict { chroot = client_limit = 1 drop_priv_before_exec = no executable = dict extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dict { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service director { chroot = . client_limit = 0 drop_priv_before_exec = no executable = director extra_groups = fifo_listener login/proxy-notify { group = mode = 00 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener director-admin { group = mode = 0600 user = } unix_listener login/director { group = mode = 00 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service dns_client { chroot = client_limit = 1 drop_priv_before_exec = no executable = dns-client extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dns-client { group = mode = 0666 user = } unix_listener login/dns-client { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = port = 143 ssl = no } inet_listener imaps { address = port = 993 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service indexer-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = indexer-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 10 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service indexer { chroot = client_limit = 0 drop_priv_before_exec = no executable = indexer extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service ipc { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = ipc extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener ipc { group = mode = 0600 user = } unix_listener login/ipc-proxy { group = mode = 0600 user = $default_login_user } user = $default_internal_user vsz_limit = 18446744073709551615 B } service lmtp { chroot = client_limit = 1 drop_priv_before_exec = no executable = lmtp extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = lmtp service_count = 0 type = unix_listener lmtp { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service pop3-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = pop3-login extra_groups = group = idle_kill = 0 inet_listener pop3 { address = port = 110 ssl = no } inet_listener pop3s { address = port = 995 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = pop3 service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service pop3 { chroot = client_limit = 1 drop_priv_before_exec = no executable = pop3 extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = pop3 service_count = 1 type = unix_listener login/pop3 { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service ssl-params { chroot = client_limit = 0 drop_priv_before_exec = no executable = ssl-params extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = startup unix_listener login/ssl-params { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service stats { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = stats extra_groups = fifo_listener stats-mail { group = mode = 0600 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener stats { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } ssl_cert = References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> Message-ID: <4F503FFD.5040504@r.paypc.com> > My initial tests for CLucene were that it would take 30% of mailbox size > (compared to 50% for Xapian). But this was before I actually implemented > it to Dovecot.. I haven't really looked at how large the indexes > actually are. Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. =R= From tss at iki.fi Fri Mar 2 09:31:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:31:01 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F503FFD.5040504@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: On 2.3.2012, at 5.35, Robin wrote: > >> My initial tests for CLucene were that it would take 30% of mailbox size >> (compared to 50% for Xapian). But this was before I actually implemented >> it to Dovecot.. I haven't really looked at how large the indexes >> actually are. > > Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. From dchenusa at yahoo.com Fri Mar 2 09:33:14 2012 From: dchenusa at yahoo.com (D Chen) Date: Thu, 1 Mar 2012 23:33:14 -0800 (PST) Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! Message-ID: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! Thx. From tss at iki.fi Fri Mar 2 09:36:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:36:40 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203021244.05034.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: On 2.3.2012, at 4.14, Stephen Davies wrote: > My mail log has many entries like: > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error > at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: > Extension header update points outside header size What filesystem is this? Are you using NFS or some other remote/shared filesystem? Do these messages repeat more than once for the same mailbox? Have you recently upgraded from Dovecot v1.x? From slusarz at curecanti.org Fri Mar 2 09:53:27 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 00:53:27 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Message-ID: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Quoting Michael M Slusarz : > Quoting Timo Sirainen : > >> On 1.3.2012, at 20.03, Michael M Slusarz wrote: >> >>> Looked at the hg commits since 2.1.1, but didn't see anything that >>> was exactly on-point to this... >>> >>> UNSEEN counts seem to be broken when done in a LIST-STATUS call. >>> I see this, for example: >> >> Only with LIST-STATUS, or also STATUS itself? And with what kind of >> config (doveconf -n)? Can you reproduce this? > > I've been seeing this intermittently the last few days. When I > notice it, I try to reproduce and never can. Then it eventually > comes back, as in the first time I access the next morning. Maybe > its some kind of cache/timeout issue. (e.g. - I can't reproduce > right now) It happened again. Once it got around midnight local time I stopped seeing correct STATUS updates with the UNSEEN count. Running two MUAs: I opened a mailbox with new messages in one, and the next time the other MUA (webmail) polled it magically started seeing unseen messages again. And now if I change the seen flag in one MUA, it will continue to be updated (as normal) in the other MUA. michael From tss at iki.fi Fri Mar 2 10:27:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:27:16 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Message-ID: <1330676836.2081.46.camel@innu> On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > It happened again. Once it got around midnight local time I stopped > seeing correct STATUS updates with the UNSEEN count. Running two > MUAs: I opened a mailbox with new messages in one, and the next time > the other MUA (webmail) polled it magically started seeing unseen > messages again. And now if I change the seen flag in one MUA, it will > continue to be updated (as normal) in the other MUA. Next time try talking IMAP protocol directly? First giving the same LIST STATUS command, and if it doesn't show the updated counts, try STATUS directly and finally EXAMINE. I don't see anything in the code that could explain why this could be happening. What filesystem are you using? From tss at iki.fi Fri Mar 2 10:45:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:45:51 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: Message-ID: <1330677951.2081.49.camel@innu> On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: > I am setting up an high-availability server set, which consists of two (or > more) servers with common disk space, all set behind redundant hardware > load balancers. At first, there will be two servers and disk space will be > kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what > creates kind of networked RAID1 storage space). This will be done mostly > to keep WWW service available in case when one of servers fails. > > However, if we have everything redundant, why not have the same with SMTP > and POP3/IMAP? But - won't anything fail if two (or more) dovecots are > accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If both servers randomly access users' mails, with NFS you'll have some trouble, with OCFS2 probably less trouble. But in both cases you'll have better performance and no problems if you use Dovecot director in both servers (install both director and backend to both servers). http://wiki2.dovecot.org/Director From stan at hardwarefreak.com Fri Mar 2 11:17:21 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 02 Mar 2012 03:17:21 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F509021.2050202@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Expanding on my previous statements, and hopefully answering some questions here, or at least getting in the ballpark, lets see what a single GbE link is capable of. Let's assume an average transfer size of SMTP/IMAP email including headers is roughly 4096 bytes, or 32768 bits. TCP over GbE after all framing and protocol overhead = 992,697,000 bits/sec maximum bandwidth with jumbo frames = 941,482,000 bits/sec max without jumbo frames We'll go without jumbo frames in our example. Every GbE interface on one router segment must support jumbo or you can't enable it. If you do, interfaces that don't do jumbo will have bad to horrible performance, or maybe not work at all. Many workstation NICs don't do jumbo frames as well as many commercial routers. Typical IMAP command payload is absolutely tiny, so we'll concentrate on response traffic. Theoretical steady state IMAP server to client 4KB message transfer rates: = 28,731 msgs/sec = 1,723,905 msgs/minute = 103,434,301 msgs/hour = 2,482,423,242 msgs/day General file transfer bandwidth, 5MB JPG: = 22 files/sec = 1,346 files/minute = 80,808 files/hour = 1,939,393 files/day General file transfer bandwidth, 100MB TIFF: = 1 files/sec = 67 files/minute = 4,040 files/hour = 96,969 files/day General file transfer bandwidth, 500MB video file: = 1 files 4.5 seconds = 10 files 44.6 seconds = 100 files 7.4 minutes As you can see, a single GbE interface has serious capacity and will probably easily carry your inter-site traffic without needing duplicate servers at the second site. You mentioned putting multiple GbE interfaces on your servers. Very, very few servers *need* 900+ Mb/s of bandwidth, however having two links is good for redundancy. So I'd not worry about the aggregation performance, only the proper and seamless failover functionality. I obviously haven't seen your workflows Charles, but I recall you do a lot of media work. By 'you' I mean Media Brokers. So obviously your users will be hitting the network harder than average office workers. I'm taking that into account. My gut instinct, based on experience and the match, is that a single GbE inter site MAN link will be plenty, without the need to duplicate server infrastructure. Again, have a qualified network architect sniff your current network traffic patterns, and discuss with you the anticipated user traffic at the 2nd site to determine your average and peak inter-site b/w needs. The average will absolutely be much less than 1Gb/s, but the peak may be well above 1Gb/s. You can still avoid the myriad problems/costs of server duplication without incurring significant additional link costs. There are a couple of options that should be available to you: 1. A second fiber pair and GbE link You might negotiate a burst contract. You pay a flat monthly rate for a base bit rate of X and pay extra for bursts. Burst contract availability will depend on the provider's network topology. If at any point they're aggregating multiple customer's traffic on a single trunk fiber pair a burst contract should be available. Burst contract allow them to oversubscribe their trunks, just as ISPs and broadband providers do. Your network architect should be able to assist you in figuring out what you'd want for your base and peak bit rates for such a contract. Why pay for 1000Mb/s from 8pm to 6am if you're only using 20Kb/s? 2. Add a second GbE link on a different transceiver wavelength using a prism on each end to transmit both links on one fiber pair. This is typically cheaper when the provider has limited fiber runs in a given area or to a given building. You may or may not be able to save money with a burst contract in this scenario. Talk to your provider and find out what your options are. Wait until your architect has finished your network analysis before speaking to the provider. Treat this link as a traditional WAN link. Do NOT treat it as simply another switch segment. Put an IP router on each side of the GbE MAN link and create a separate IP subnet for hosts and devices in the new office. By doing this you keep broadcast traffic from traversing the link. This includes things like ARP discovery, DHCP, NTP broadcast, and most importantly: broadcast traffic from disk imaging software. If you don't make this an IP routed link, network disk imaging traffic will traverse the MAN link just as it traverses your entire switched LAN. This could be anywhere from 25-80MB/s (200-640Mb/s) of broadcast traffic. You obviously don't want this clogging the link. You *might* be able to eliminate broadcast traffic using special VLAN configurations on sufficiently advanced layer2-7 "switch routers", but it's cheaper and fool proof when done with standard IP routers. Again, chat with your architect. With this being a routed connection, and broadcast traffic being eliminated, any services that rely on broadcast traffic will need to be duplicated or tweaked accordingly. You will need a DHCP server in the new office. The router should be able to serve DHCP, unless you're currently serving some custom scope it can't handle. If you rely on broadcast for WINS, or have any other Microsoft services that rely on broadcast, you will need to address those. If you currently use NTP broadcast for time updates you'll need another NTP server in the new office. Again, the router should be able to broadcast NTP updates. The solutions to these things have been around forever, so I'm not going to go into all of them, but you need to be aware. You'll need to discuss these things with your network architect or a qualified Microsoft consultant. If you run no MS servers and don't use broadcast, then no need to worry about. And hooray for you, no MS! :) This may be of interest given the topic. At a previous $dayjob a few years back, we ran the traffic of about 580 desktops/wireless laptops through a single GbE uplink into an 11 blade server farm backed by a small fiber channel SAN. Blade-blade IP traffic was through a dedicated 14x6 port GbE switch module, so things like vmotion, backups, etc worked at full boogie. But the uplink from the switch module in the BladeCenter to the Cisco 5000 core switch was a single copper GbE uplink. All user traffic flowed over this link. We never had performance issues. We'd configured QOS to keep the IP phones happy but that's about it for traffic shaping. Before I left I jacked in a 2nd GbE uplink for redundancy and configured Cisco's link aggregation protocol. We didn't notice a performance difference. I could have aggregated 6 GbE uplinks. One did the job, two gave resiliency, more would have just wasted ports on the core switch. Hope you find this educational/informational/useful Charles, and maybe others. -- Stan From dovecot at r.paypc.com Fri Mar 2 11:51:53 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 01:51:53 -0800 Subject: [Dovecot] fts size In-Reply-To: References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: <4F509839.8010801@r.paypc.com> > No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. That sounds like a great deal to me! I'm glad you're still interested enough in it. =R= From Ralf.Hildebrandt at charite.de Fri Mar 2 12:25:01 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:25:01 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from Message-ID: <20120302102501.GZ11180@charite.de> Hi! Starting with 2.1.1 we suddely encounter quite a lot of these messages: Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) While this has (assumedly) been working with 2.0.18. Another issue with this: This fixes ONE file, and throws an error. Repeatedly accessing this folder fixes more files, until at some point all files were fixed. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From mailing at securitylabs.it Fri Mar 2 12:30:04 2012 From: mailing at securitylabs.it (mailing at securitylabs.it) Date: Fri, 02 Mar 2012 11:30:04 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <4F50A12C.4080304@securitylabs.it> Il 02/03/2012 11:25, Ralf Hildebrandt ha scritto: > Hi! > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) > > While this has (assumedly) been working with 2.0.18. Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. > > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, apparently it happens only on servers with qmail, not on servers with exim or dovecot as lda: Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: Broken physical size for mail UID 40669 Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) From tss at iki.fi Fri Mar 2 12:32:08 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:32:08 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F509839.8010801@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> <4F509839.8010801@r.paypc.com> Message-ID: On 2.3.2012, at 11.51, Robin wrote: >> No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. > > That sounds like a great deal to me! I'm glad you're still interested enough in it. Having more choices is always good. :) From Ralf.Hildebrandt at charite.de Fri Mar 2 12:34:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:34:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <4F50A12C.4080304@securitylabs.it> References: <20120302102501.GZ11180@charite.de> <4F50A12C.4080304@securitylabs.it> Message-ID: <20120302103445.GC11180@charite.de> > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, > apparently it happens only on servers with qmail, not on servers with > exim or dovecot as lda: I'm using the dovecot LDA, but then it's not clear if the messages affected are REALLY old and thus might predate the use of the dovecot LDA... > > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename > has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, > to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index > cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: > Broken physical size for mail UID 40669 > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Fri Mar 2 12:37:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:37:06 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S .. > While this has (assumedly) been working with 2.0.18. Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. > Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. From Ralf.Hildebrandt at charite.de Fri Mar 2 12:43:33 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:43:33 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> Message-ID: <20120302104333.GD11180@charite.de> * Timo Sirainen : > On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > .. > > While this has (assumedly) been working with 2.0.18. > > Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. COuld be! > > Another issue > > with this: This fixes ONE file, and throws an error. Repeatedly > > accessing this folder fixes more files, until at some point all files > > were fixed. > > Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Ah yes, good idea Mar 2 11:39:39 postamt dovecot: imap-login: Login: user=, method=PLAIN, rip=141.42.206.38, lip=141.42.206.36, mpid=28959, secured Mar 2 11:39:41 postamt dovecot: imap(user): Error: Cached message size smaller than expected (168202 < 170440) Mar 2 11:39:41 postamt dovecot: imap(user): Error: Maildir filename has wrong S value, renamed the file from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ Mar 2 11:39:41 postamt dovecot: imap(user): Error: Corrupted index cache file /home/g/z/user/Maildir/.Partys/dovecot.index.cache: Broken physical size for mail UID 81 Mar 2 11:39:41 postamt dovecot: imap(user): Error: read(/home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ) failed: Input/output error (uid=81) Mar 2 11:39:41 postamt dovecot: imap(user): Disconnected: Internal error occurred. Refer to server log for more information. [2012-03-02 11:39:41] in=735 out=5258 Look at that renaming operation: It simply reused the same name: from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ > Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. In the case above that mail was gzipped twice :( -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From CMarcus at Media-Brokers.com Fri Mar 2 14:40:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 02 Mar 2012 07:40:54 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F4D009F.7000107@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> Message-ID: <4F50BFD6.5010808@Media-Brokers.com> On 2012-02-28 11:28 AM, Charles Marcus wrote: > On 2012-02-28 11:05 AM, kfx wrote: >> Ok I feel ashame... it was a third party init scrip who was the >> problem :( >> >> Sorry for the noise and thank you for dovecot > > So... you're saying that Thunderbird now correctly uses server side search? Please respond... I need to know whether or not I need to pursue this, since we use Thunderbird in house and will be switching soon to dovecot... Thanks, -- Best regards, Charles From slusarz at curecanti.org Fri Mar 2 20:27:42 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 11:27:42 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <1330676836.2081.46.camel@innu> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> Message-ID: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Quoting Timo Sirainen : > On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > >> It happened again. Once it got around midnight local time I stopped >> seeing correct STATUS updates with the UNSEEN count. Running two >> MUAs: I opened a mailbox with new messages in one, and the next time >> the other MUA (webmail) polled it magically started seeing unseen >> messages again. And now if I change the seen flag in one MUA, it will >> continue to be updated (as normal) in the other MUA. > > Next time try talking IMAP protocol directly? First giving the same LIST > STATUS command, and if it doesn't show the updated counts, try STATUS > directly and finally EXAMINE. Weird. In the MUA I was seeing this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. Then I did this: slusarz at bigworm % dovecot * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.dovecot" * STATUS "IN.dovecot" (UNSEEN 7) * LIST () "." "INBOX" * STATUS "INBOX" (UNSEEN 1) 1 OK List completed. Sure enough, I went back to the MUA and now see this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 7) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. The only difference... in the MUA I am enabling QRESYNC. This is what the previous commands look like (before reaching the LIST-STATUS command): >> Timestamp: Fri, 02 Mar 2012 11:17:56 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: 1 OK User logged in C: 2 ENABLE QRESYNC S: * ENABLED QRESYNC S: 2 OK Enabled. C: 3 STATUS IN.horde.cvs (MESSAGES UIDNEXT UIDVALIDITY HIGHESTMODSEQ) S: * STATUS "IN.horde.cvs" (MESSAGES 11 UIDNEXT 16767 UIDVALIDITY 1239077891 HIGHESTMODSEQ 31409) S: 3 OK Status completed. Next time I see this issue, I will try to go back and issue ENABLE QRESYNC from the command line. > I don't see anything in the code that could explain why this could be > happening. What filesystem are you using? Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. michael From tss at iki.fi Fri Mar 2 20:36:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 20:36:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: On 2.3.2012, at 20.27, Michael M Slusarz wrote: >> I don't see anything in the code that could explain why this could be >> happening. What filesystem are you using? > > Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. jfs used to have a problem with not updating directory's mtime when link()ing files to it, which caused Dovecot not to notice new mails. This got fixed a few years ago though. But it is a bit suspicious that this is happening with jfs.. From dovecot at r.paypc.com Sat Mar 3 00:13:18 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 14:13:18 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F50BFD6.5010808@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> Message-ID: <4F5145FE.3070301@r.paypc.com> On 3/2/2012 4:40 AM, Charles Marcus wrote: > Please respond... I need to know whether or not I need to pursue this, > since we use Thunderbird in house and will be switching soon to dovecot... This mailing list is for dovecot, not Thunderbird support. The lack of replies to Thunderbird usage questions no doubt reflects this. I would look at the GUI interface and/or "manual" for Thunderbird to find the answer to that question. I suspect there is a check-box or configuration item that's been right in front of you all along that you've not thought twice about. =R= From mcguire at neurotica.com Sat Mar 3 00:18:12 2012 From: mcguire at neurotica.com (Dave McGuire) Date: Fri, 02 Mar 2012 17:18:12 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F514724.7070001@neurotica.com> On 03/02/2012 05:13 PM, Robin wrote: > On 3/2/2012 4:40 AM, Charles Marcus wrote: >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. Please forgive me for jumping in, but I believe this is very much on-topic. It isn't a matter of "Thunderbird support", it's a matter of Dovecot interoperability. Please DO keep stuff like this on-list. -Dave -- Dave McGuire, AK4HZ New Kensington, PA From sdavies at sdc.com.au Sat Mar 3 01:45:02 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Sat, 3 Mar 2012 10:15:02 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: <201203031015.02716.sdavies@sdc.com.au> No NFS. The file system is local. Yes. There are multiple copies of the message for multiple mailboxes for each of at least two users. Yes. Did recently upgrade from 1.2.15. Cheers and thanks, Stephen On Fri, 2 Mar 2012 06:06:40 PM Timo Sirainen wrote: > On 2.3.2012, at 4.14, Stephen Davies wrote: > > My mail log has many entries like: > > > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization > > error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS > > Emails/dovecot.index: Extension header update points outside header size > > What filesystem is this? Are you using NFS or some other remote/shared > filesystem? > > Do these messages repeat more than once for the same mailbox? Have you > recently upgraded from Dovecot v1.x? -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From slusarz at curecanti.org Sat Mar 3 02:48:09 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 17:48:09 -0700 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data Message-ID: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> I'm seeing this: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) It should be: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) since the RFC 2822 quoting characters must be removed. michael From slusarz at curecanti.org Sat Mar 3 03:48:23 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 18:48:23 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 2.3.2012, at 20.27, Michael M Slusarz wrote: > >>> I don't see anything in the code that could explain why this could be >>> happening. What filesystem are you using? >> >> Nothing fancy: jfs on Archlinux. I've been using it for years - >> it's not something I have recently changed. > > jfs used to have a problem with not updating directory's mtime when > link()ing files to it, which caused Dovecot not to notice new mails. > This got fixed a few years ago though. But it is a bit suspicious > that this is happening with jfs.. I can now verify that QRESYNC is triggering this behavior. * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 ENABLE QRESYNC * ENABLED QRESYNC 1 OK Enabled. 2 LIST "" (IN.horde.dev) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.horde.dev" * STATUS "IN.horde.dev" (UNSEEN 0) 2 OK List completed. 3 EXAMINE IN.horde.dev * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded NonJunk impflag0 impflag1) * OK [PERMANENTFLAGS ()] Read-only mailbox. * 3 EXISTS * 1 RECENT * OK [UNSEEN 3] First unseen. * OK [UIDVALIDITY 1255685339] UIDs valid * OK [UIDNEXT 2805] Predicted next UID * OK [HIGHESTMODSEQ 8266] Highest 3 OK [READ-ONLY] Select completed. (The unseen message arrived about 7 minutes before I issued these commands, so it didn't sneak in between command #2 and #3). michael From busseniu at in.tum.de Sat Mar 3 14:33:27 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Sat, 03 Mar 2012 13:33:27 +0100 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F4DF07A.7020408@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> Message-ID: <4F520F97.5030002@in.tum.de> On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: > when the private namespace has "prefix = INBOX." and you use doveadm > fetch to search for "mailbox INBOX", then it prints every message twice: Apparently the bug has been introduced with this changeset: changeset: 14112:f5353573d3a0 user: Timo Sirainen date: Sun Feb 12 02:50:49 2012 +0200 summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. http://hg.dovecot.org/dovecot-2.1/rev/f5353573d3a0 Cheers, Christoph From netwiz at crc.id.au Sat Mar 3 14:07:44 2012 From: netwiz at crc.id.au (Steven Haigh) Date: Sat, 03 Mar 2012 23:07:44 +1100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <20110311215739.GD13492@state-of-mind.de> References: <20110311215739.GD13492@state-of-mind.de> Message-ID: <4F520990.2000903@crc.id.au> Hi all, I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... -- Steven Haigh Email: netwiz at crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 >Timo, > >would you consider adding support for "IMAP LIST Extension for >Special-Use >Mailboxes" any time near >in the >future? > >I would really love to get rid of all those folders created by all >those >different mail clients just because they can't agree to use the same >folder >for special purpose. > >Obviously clients need to support it too. Having Dovecot support it >certainly >would make them adopt the standard sooner. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4952 bytes Desc: S/MIME Cryptographic Signature URL: From CMarcus at Media-Brokers.com Sat Mar 3 14:52:10 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 07:52:10 -0500 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5213FA.30700@Media-Brokers.com> On 2012-03-02 2:33 AM, D Chen wrote: > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start > successfully with lots of errors i.e. "dovecot: doveconf: Warning: > ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on > Ubuntu 11.10, by following the Postfix installation and configuration > instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section > on page 190, after run "sudo apt-get install dovecot-common", it > requires to edit the section of "auth default" and the "socket > listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my > /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth > default" "socket listen" ! Distro specific questions are usually much better asked on the distro support lists... -- Best regards, Charles From eliezer at ec.hadorhabaac.com Sat Mar 3 15:03:40 2012 From: eliezer at ec.hadorhabaac.com (Eliezer Croitoru) Date: Sat, 03 Mar 2012 15:03:40 +0200 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5216AC.3030602@ec.hadorhabaac.com> On 02/03/2012 09:33, D Chen wrote: use the command dovecot -n to get dovecot settings output and we can try to help you a bit. Regards, Eliezer > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. > "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! > > I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a > dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? > > BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! > > Thx. From arnaud.abelard at univ-nantes.fr Sat Mar 3 18:06:17 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Sat, 03 Mar 2012 17:06:17 +0100 Subject: [Dovecot] keywords/flags questions Message-ID: <4F524179.2040407@univ-nantes.fr> Hello, I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? For example: . STORE 1:1 flags &AOA-_refaire * 1 FETCH (FLAGS (&aoa-_refaire)) This makes retrieving the keyword properly impossible. What did I miss? My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? Thanks in advance, Arnaud -- Arnaud Ab?lard jabber: arnaud.abelard at univ-nantes.fr / twitter: ArnY Administrateur Syst?me DSI Universit? de Nantes - From public-mail at alekciy.ru Sat Mar 3 18:51:28 2012 From: public-mail at alekciy.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KHRg9C90LTRg9C60L7Qsg==?=) Date: Sat, 3 Mar 2012 20:51:28 +0400 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: 3 ????? 2012??. 20:06 ???????????? Arnaud Ab?lard ???????: > But docevot save keywords in lowercase Yes. http://www.dovecot.org/list/dovecot/2011-April/058493.html From tss at iki.fi Sat Mar 3 19:05:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 3 Mar 2012 19:05:13 +0200 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: <5CA4B56F-D26E-492B-9B4D-9BD8E6EAD018@iki.fi> On 3.3.2012, at 18.06, Arnaud Ab?lard wrote: > I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. > > First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? > > For example: > . STORE 1:1 flags &AOA-_refaire > * 1 FETCH (FLAGS (&aoa-_refaire)) > > This makes retrieving the keyword properly impossible. What did I miss? Sorry, doesn't work like that. This was recently discussed in imap-protocol mailing list though, but nothing has come of it. > My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? For now yes. I've been thinking about doing something about this for last 8 years, but it has never become a real problem so I haven't bothered. From trashcan at odo.in-berlin.de Sat Mar 3 20:03:42 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:03:42 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <1330346709.11500.324.camel@innu> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: Hi -- On 27.02.2012, at 13:45, Timo Sirainen wrote: > On Thu, 2012-02-23 at 20:55 +0100, Michael Grimm wrote: >> My working 2.0.18 syntax threw the following error: >> >> vmail> dsync -v -f -u test ssh vmail at remote-host.tld dsync -v -f -u test >> doveadm(vmail): Fatal: Unknown print formatter: -u >> dsync-local(test): Error: read() from worker server failed: EOF > > You left out "mirror" from that command, but after adding it the latest > hg version works. I did use 'mirror', I just forgot to paste it. JFTR: vmail> dovecot --version 20120303 (1002733ca266+) vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test dsync-local(test): Error: remote: dsync: illegal option -- f doveadm dsync-server [-u |-A] [-S ] dsync-local(test): Error: read() from worker server failed: EOF If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. >> Now I switched to the recommended new syntax as stated in http://wiki2.dovecot.org/Upgrading/2.1: >> >> vmail> doveadm sync -v -f -u test ssh vmail at remote-host.tld doveadm sync -v -f -u test >> doveadm: illegal option -- v >> doveadm sync [-u |-A] [-S ] [-fR] [-m ] > > The -v parameter is in wrong place now, needs to be "doveadm -v sync". Ah, yes. That was my mistake, sorry. >> After some trial by error I finally found a working syntax: >> >> vmail> doveadm sync -u test -f ssh vmail at remote-host.tld doveadm dsync-server -u test > > Oh, hmm. I hadn't thought about this problem, it shouldn't have been > necessary to give the dsync-server parameter. But I guess there's not a > whole lot of other possibilities to do this. Hmm. > > BTW. I think you can do this simply: > > doveadm sync -f -u test test at remote-host.tld vmail> doveadm sync -f -u test ssh vmail at remote-host.tld dsync-local(test): Error: remote: dsync-server: Command not found. dsync-local(test): Error: read() from worker server failed: EOF Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that I'm the only one reporting that. How could I help to debug this issue? Regards, Michael From tlx at leuxner.net Sat Mar 3 20:10:35 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Sat, 3 Mar 2012 19:10:35 +0100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <4F520990.2000903@crc.id.au> References: <20110311215739.GD13492@state-of-mind.de> <4F520990.2000903@crc.id.au> Message-ID: <83D77B81-EC49-4755-A866-E30B41E8B246@leuxner.net> Am 03.03.2012 um 13:07 schrieb Steven Haigh: > I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... Yes it was. It has been discussed extensively: http://www.dovecot.org/list/dovecot-news/2012-February/000213.html http://www.dovecot.org/list/dovecot/2011-December/062327.html Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From trashcan at odo.in-berlin.de Sat Mar 3 20:12:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:12:21 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <8D5B4989-571D-4F5D-927E-65198CECFADD@odo.in-berlin.de> Hi -- On 03.03.2012, at 19:03, Michael Grimm wrote: > Thus I can't test user suggestion. s/user/your/ Sorry, Michael From anyaddress at gmx.net Sat Mar 3 22:08:48 2012 From: anyaddress at gmx.net (Tom Fernandes) Date: Sat, 3 Mar 2012 21:08:48 +0100 Subject: [Dovecot] directly addressable public folders issues Message-ID: <201203032108.49489.anyaddress@gmx.net> Hi, I would like to have an address info at example.com whose mails are stored in a public folder. I also want certain users to be able to create sieve-filter-rules and subfolders. From what I understand it's a good idea to have a separate location for home and for mail_location. For my normal accounts I have: home = /var/vmail/ mail_location = ~/Maildir Is there a way to have the same for public folders? This are my current settings: namespace public { separator = / prefix = public/ location = maildir:/var/vmail/public subscriptions = no } user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The LDAP-homeDirectory-attribute for info at example.com is "public/.info" Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which is good. The MUA seems to read to read from /var/vmail/public/.info though. How can I make the MUA read from /var/vmail/public/.info/Maildir instead? And in case this works - is it possible to have only certain users modify the sieve-rule for this public folder or do I definitely need a passdb-entry for info at example.com? Modifying scripts via commandline is not an option... I'm using dovecot 1.2. regards, Tom Fernandes From CMarcus at Media-Brokers.com Sat Mar 3 22:14:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:14:55 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F527BBF.3060607@Media-Brokers.com> On 2012-03-02 5:13 PM, Robin wrote: > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? -- Best regards, Charles From CMarcus at Media-Brokers.com Sat Mar 3 22:20:24 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:20:24 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F509021.2050202@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> Message-ID: <4F527D08.6070508@Media-Brokers.com> Thanks very much for taking the time for your detailed reply, Stan, but I'll need more time to study it... On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > My gut instinct, based on experience and the match, is that a single GbE > inter site MAN link will be plenty, without the need to duplicate server > infrastructure. I just wanted to point out one thing - I have two primary goals - yes, one is to maximize performance, but the other is accomplish a level of *redundancy*... Also - I already have the servers (I have 3 Poweredge 2970's available to me, only one of which is currently being used)... So, the only extra expenses involved will be relatively minor hardware expenses (multi-port Gb NICs), and some consulting services for making sure I implement the VM environment (including the routing) correctly. So, honestly, we'd be incurring most of these expenses anyway, even if we didn't set up redundant servers, so I figure why not get redundancy too (now is the time to get the boss to pay for it)... -- Best regards, Charles From trashcan at odo.in-berlin.de Sat Mar 3 22:27:11 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 21:27:11 +0100 Subject: [Dovecot] Dovecot clustering with dsync-based replication In-Reply-To: <1330437834.2081.2.camel@innu> References: <1330437834.2081.2.camel@innu> Message-ID: Hi -- On 28.02.2012, at 15:03, Timo Sirainen wrote: > This document describes a design for a dsync-replicated Dovecot cluster. Whow! That's more than interesting, that's a real bummer ;-) At least for my setup of redundant mail servers. Looking forward to test it, Michael From piotr-l at netexpert.pl Sat Mar 3 22:41:24 2012 From: piotr-l at netexpert.pl (Piotr NetExpert) Date: Sat, 03 Mar 2012 21:41:24 +0100 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F5281F4.1070503@netexpert.pl> > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Take into account costs of administering a more complex environment too. -- pozdrawiam Piotr Szafarczyk http://www.netexpert.pl From bradley.giesbrecht at gmail.com Sat Mar 3 23:16:31 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sat, 3 Mar 2012 13:16:31 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F527BBF.3060607@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> Message-ID: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > On 2012-03-02 5:13 PM, Robin wrote: >> This mailing list is for dovecot, not Thunderbird support. The lack of >> replies to Thunderbird usage questions no doubt reflects this. > > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? Show dovecot misbehaving. On Feb 28, 2012, at 6:57 AM, Timo Sirainen wrote: > On Tue, 2012-02-28 at 15:47 +0100, kfx wrote: >>> Did you enable the 'Run search on server' option in the Advanced Search >>> window? Doing this *should* result in Thunderbird using dovecots indexes >>> server side. >>> >> >> Yes I did. >> >> Some more info: >> >> by telnet'ing directly and issuing: >> c search text pattern >> * SEARCH 1208 >> c OK Search completed (0.003 secs). > > So, Solr in Dovecot works perfectly. > >> But the same search in thunderbird return "No matches found" :( > > Thunderbird problem, nothing you can do about it from Dovecot's side. Regards, Bradley Giesbrecht From stan at hardwarefreak.com Sun Mar 4 02:51:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 03 Mar 2012 18:51:39 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F52BC9B.9000005@hardwarefreak.com> On 3/3/2012 2:20 PM, Charles Marcus wrote: > Thanks very much for taking the time for your detailed reply, Stan, but > I'll need more time to study it... > > On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > >> My gut instinct, based on experience and the match, is that a single GbE >> inter site MAN link will be plenty, without the need to duplicate server >> infrastructure. > > I just wanted to point out one thing - I have two primary goals - yes, > one is to maximize performance, but the other is accomplish a level of > *redundancy*... What type of redundancy are you looking for? I.e. is one reason for duplicating servers at site #2 to avoid disruption in the event the MAN link fails? Do you currently have redundant GbE links to each closet switch stack in site #1, and also redundant switches in the datacenter? I.e. do you skip a beat if a core or closet switch fails? If you do not currently have, nor plan to create such network redundancy internally at site #1, then why build application redundancy with the single goal of mitigating failure of a single network link? Do you have reason to believe there is a higher probability of failure of the MAN link than any other single link in the current network? > Also - I already have the servers (I have 3 Poweredge 2970's available > to me, only one of which is currently being used)... > > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Again, you don't need multi-port GbE NICs or bonding for performance--a single GbE link is all each server needs. Your switches should be able to demonstrate that, without even needing a sniffer, assuming they're decent managed units. If you're after link redundancy, use two single port NICs per server, or one mobo mounted port and once single port NIC. Most dual port NICs duplicate the PHYs but not the ethernet chip nor power circuits, etc. Thus, when a dual port NIC fails you usually loose both ports. > So, honestly, we'd be incurring most of these expenses anyway, even if > we didn't set up redundant servers, so I figure why not get redundancy > too (now is the time to get the boss to pay for it)... Don't forget power backup at site #2. Probably not a huge cost in the overall scheme of things, but it's still another $5000 or so. In summary, my advice is: 1. One 1000Mb MAN link is plenty of bandwidth for all users at site #2 including running internet traffic through site #1, saving the cost of an internet pipe at site #2 2. If truly concerned about link failure, get a backup 100Mb/s link, or get two GbE links with a burst contract, depending on price 3. Keep your servers in one place. If you actually desire application level redundancy (IMAP, SMB/CIFS, etc) unrelated to a network link failure, then do your clustering etc "within the rack". It will be much easier to manage and troubleshoot this than two datacenters w/ all kinds of replication etc between them 4. If site #1 is not already link redundant, it makes little sense to make a big redundancy push to cover a possible single network link failure, regardless of which link 5. Building a 2nd datacenter and using the MAN link for data replication gives you no performance advantage, and may actually increase overall utilization, vs using the link as a regular trunk 6. *Setup QOS appropriately to maintain low latency of IMAP and other priority data, giving a back seat to SMB/CIFS/FTP/HTTP and other bulk transfer protocols* With proper QOS the single GbE MAN link will simply scream for everyone, regardless of saturation level -- Stan From dchenusa at yahoo.com Sun Mar 4 08:25:12 2012 From: dchenusa at yahoo.com (D Chen) Date: Sat, 3 Mar 2012 22:25:12 -0800 (PST) Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... Message-ID: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! ?? admin at server:/etc/dovecot$ doveconf -n ? ? # 2.0.13: /etc/dovecot/dovecot.conf ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? # OS: Linux 3.0.0-16-server x86_64 Ubuntu 11.10? ? ? log_timestamp = "%Y-%m-%d %H:%M:%S " ? ? mail_location = maildir:~/Maildir ? ? mail_privileged_group = mail ? ? managesieve_notify_capability = mailto ? ? managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ? ? passdb { ? ? ? driver = pam - Ignored: ? ? } ? ? passdb { ? ? ? driver = pam ? ? } ? ? plugin { ? ? ? sieve = ~/.dovecot.sieve ? ? ? sieve_dir = ~/sieve ? ? } ? ? protocols = imap pop3 sieve ? ? service auth { ? ? ? unix_listener /var/spool/postfix/private/auth-client { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? unix_listener /var/spool/postfix/private/dovecot-auth { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? user = root ? ? } ? ? ssl_cert = was automatically rejected:%n%r ? ? } From tss at iki.fi Sun Mar 4 12:44:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 04 Mar 2012 12:44:46 +0200 Subject: [Dovecot] dsync replication available for testing Message-ID: <4F53479E.40703@iki.fi> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Below is a configuration for virtual user setup. System user configuration works pretty much the same, except doveadm/ssh is run as root. Try first that dsync works successfully with ssh in host1: doveadm sync -u user at domain remote:vmail at host2.example.com and also in host2: doveadm sync -u user at domain remote:vmail at host1.example.com ------ mail_plugins = $mail_plugins notify replication service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } From tss at iki.fi Sun Mar 4 13:05:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:05:26 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: On 3.3.2012, at 20.03, Michael Grimm wrote: > vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test > dsync-local(test): Error: remote: dsync: illegal option -- f > doveadm dsync-server [-u |-A] [-S ] > dsync-local(test): Error: read() from worker server failed: EOF > > If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 >> doveadm sync -f -u test test at remote-host.tld > > vmail> doveadm sync -f -u test ssh vmail at remote-host.tld > dsync-local(test): Error: remote: dsync-server: Command not found. > dsync-local(test): Error: read() from worker server failed: EOF Remove the "ssh" parameter from the middle and change vmail@ to test@ > Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I > can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's a good idea. In the latest hg version the preferred way is: doveadm sync -f -u test remote:vmail at host It automatically adds the remote -u test. > Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). > The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that > I'm the only one reporting that. How could I help to debug this issue? Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or other scripts that "rm" mails? From tss at iki.fi Sun Mar 4 13:13:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:13:18 +0200 Subject: [Dovecot] directly addressable public folders issues In-Reply-To: <201203032108.49489.anyaddress@gmx.net> References: <201203032108.49489.anyaddress@gmx.net> Message-ID: <7E27E7D3-DCE8-4FF9-9689-24815D2895CB@iki.fi> On 3.3.2012, at 22.08, Tom Fernandes wrote: > I would like to have an address info at example.com whose mails are stored in a > public folder. .. > From what I understand it's a good idea to have a separate location for home and > for mail_location. > > For my normal accounts I have: > home = /var/vmail/ > mail_location = ~/Maildir mail_location = maildir:~/Maildir to unnecessary avoid autodetection. > Is there a way to have the same for public folders? This are my current > settings: > > namespace public { > separator = / > prefix = public/ > location = maildir:/var/vmail/public > subscriptions = no > } That's ok. > user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The "mail" isn't necessary here, since it's already globally set. > The LDAP-homeDirectory-attribute for info at example.com is "public/.info" That's not going to work too well. > Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which > is good. The MUA seems to read to read from /var/vmail/public/.info though. > > How can I make the MUA read from /var/vmail/public/.info/Maildir instead? You can't. > And in case this works - is it possible to have only certain users modify the > sieve-rule for this public folder or do I definitely need a passdb-entry for > info at example.com? > Modifying scripts via commandline is not an option... The way I did it was to make info@, sales@ and others aliases to "company" user, which is a rather regular user (except can't actually log in). For this "company" user I've a Sieve script that puts the mails into the proper mailbox, e.g. fileinto "public/info". From tss at iki.fi Sun Mar 4 13:21:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:21:07 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Message-ID: <75EBCBDD-B30A-401F-A6D9-517C03B1873B@iki.fi> On 3.3.2012, at 3.48, Michael M Slusarz wrote: > I can now verify that QRESYNC is triggering this behavior. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/8cbc130c2b72 http://hg.dovecot.org/dovecot-2.1/rev/31ae11fe18b2 From trashcan at odo.in-berlin.de Sun Mar 4 13:31:45 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:31:45 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:05, Timo Sirainen wrote: > On 3.3.2012, at 20.03, Michael Grimm wrote: >> vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test >> dsync-local(test): Error: remote: dsync: illegal option -- f >> doveadm dsync-server [-u |-A] [-S ] >> dsync-local(test): Error: read() from worker server failed: EOF >> >> If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. > > Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: > http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 Ok, that means it has always been ignored in 2.0.x ;-) >>> doveadm sync -f -u test test at remote-host.tld >> >> vmail> doveadm sync -f -u test ssh vmail at remote-host.tld >> dsync-local(test): Error: remote: dsync-server: Command not found. >> dsync-local(test): Error: read() from worker server failed: EOF > > Remove the "ssh" parameter from the middle and change vmail@ to test@ That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: ssh -p 1234 vmail at remote-host.tld Sorry, I should have mentioned that before. In your other mail about 'dsync replication' you refer to a config option: #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} Would that allow for ssh options to be set? >> Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. >> Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. > > In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's > a good idea. In the latest hg version the preferred way is: > > doveadm sync -f -u test remote:vmail at host See above regarding ssh options. >> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >> I'm the only one reporting that. How could I help to debug this issue? > > Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or > other scripts that "rm" mails? No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. But no scripts or such are used to remove mail. I'm running mdbox only. Thanks and regards, Michael From trashcan at odo.in-berlin.de Sun Mar 4 13:34:03 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:34:03 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Hi -- On 04.03.2012, at 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Great news. I would love to test it, if I will be able to run this on a test account, only. All other users should become synced the "old way" for the time being. Would that be possible with the current implementation? Regards, Michael From tss at iki.fi Sun Mar 4 13:35:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:35:04 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Message-ID: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> On 4.3.2012, at 13.31, Michael Grimm wrote: > That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: > > ssh -p 1234 vmail at remote-host.tld > > Sorry, I should have mentioned that before. > > In your other mail about 'dsync replication' you refer to a config option: > #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > Would that allow for ssh options to be set? Yes. >> doveadm sync -f -u test remote:vmail at host > > See above regarding ssh options. So this works by changing the dsync_remote_cmd. >>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>> I'm the only one reporting that. How could I help to debug this issue? >> >> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >> other scripts that "rm" mails? > > No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. > But no scripts or such are used to remove mail. I'm running mdbox only. By "undeletable" do you mean you have mails that always come back after expunging them? I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they don't contain any sensitive information. From tss at iki.fi Sun Mar 4 13:38:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:38:14 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Message-ID: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> On 4.3.2012, at 13.34, Michael Grimm wrote: > On 04.03.2012, at 11:44, Timo Sirainen wrote: > >> In dovecot-2.1 hg you can now test dsync-based replication. > > Great news. I would love to test it, if I will be able to run this on a test > account, only. All other users should become synced the "old way" for the time > being. > > Would that be possible with the current implementation? 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) 2) You can enable replication plugin only for one user by changing mail_plugins setting via userdb extra fields. Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So you could even ignore 1) and just let it sync everyone at startup. From trashcan at odo.in-berlin.de Sun Mar 4 13:41:47 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:41:47 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> Message-ID: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:35, Timo Sirainen wrote: > On 4.3.2012, at 13.31, Michael Grimm wrote: >> That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: >> >> ssh -p 1234 vmail at remote-host.tld >> >> Sorry, I should have mentioned that before. >> >> In your other mail about 'dsync replication' you refer to a config option: >> #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} >> >> Would that allow for ssh options to be set? > > Yes. Good news. >>>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>>> I'm the only one reporting that. How could I help to debug this issue? >>> >>> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >>> other scripts that "rm" mails? >> >> No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. >> But no scripts or such are used to remove mail. I'm running mdbox only. > > By "undeletable" do you mean you have mails that always come back after expunging them? Yes. Deleting by the client will return them after the next dsync run. > I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they > don't contain any sensitive information. From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Thanks and regards, Michael From tss at iki.fi Sun Mar 4 13:54:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:54:34 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Message-ID: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> On 4.3.2012, at 13.41, Michael Grimm wrote: >> By "undeletable" do you mean you have mails that always come back after expunging them? > > Yes. Deleting by the client will return them after the next dsync run. > >> I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they >> don't contain any sensitive information. > > From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Just one mailbox where that consistently happens is enough: 1. Expunge the mail 2. Get a copy of the dbox-Mails/dovecot.index, dbox-Mails/dovecot.index.log and dbox-Mails/dovecot.index.log.2 from both servers 3. Run dsync, and verify that the message is undeleted 4. Get another copy of the indexes from both servers From tss at iki.fi Sun Mar 4 14:14:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:14:23 +0200 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data In-Reply-To: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> References: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> Message-ID: <90B35FA4-651C-40CA-8149-8FE7E3E09E50@iki.fi> On 3.3.2012, at 2.48, Michael M Slusarz wrote: > I'm seeing this: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > It should be: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > since the RFC 2822 quoting characters must be removed. Oops. This has been buggy forever. Added the fix to all Dovecot hg trees. From tss at iki.fi Sun Mar 4 14:33:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:33:35 +0200 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F520F97.5030002@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> <4F520F97.5030002@in.tum.de> Message-ID: <8AAD13E5-CE14-44BF-9CD9-DDB984B0BF31@iki.fi> On 3.3.2012, at 14.33, Christoph Bu?enius wrote: > On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: >> when the private namespace has "prefix = INBOX." and you use doveadm >> fetch to search for "mailbox INBOX", then it prints every message twice: > > Apparently the bug has been introduced with this changeset: > > changeset: 14112:f5353573d3a0 > user: Timo Sirainen > date: Sun Feb 12 02:50:49 2012 +0200 > summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/bbe6b6c2ee99 From tss at iki.fi Sun Mar 4 14:35:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:35:18 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203031015.02716.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> Message-ID: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> On 3.3.2012, at 1.45, Stephen Davies wrote: > No NFS. The file system is local. > > Yes. There are multiple copies of the message for multiple mailboxes for each > of at least two users. But does the error keep repeating for the same mailbox? It's supposed to fix itself automatically after logging the error once. > Yes. Did recently upgrade from 1.2.15. I think in earlier versions mbox used somewhat different index file structures and now Dovecot logs some errors about them. Anyway, one sure way to fix this is to just delete all the .imap/ directories. From tss at iki.fi Sun Mar 4 14:36:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:36:23 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Message-ID: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> On 4.3.2012, at 8.25, D Chen wrote: > Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf So, doveconf -n > dovecot-new.conf mv dovecot-new.conf /etc/dovecot/dovecot.conf That should do it. From tss at iki.fi Sun Mar 4 14:41:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:41:20 +0200 Subject: [Dovecot] Multiple namespaces seems to be used at the same time In-Reply-To: <4F4F84C7.1060502@cnpapers.com> References: <4F4F84C7.1060502@cnpapers.com> Message-ID: <42CE6C31-D246-4AE1-9E79-3DD457E20E39@iki.fi> On 1.3.2012, at 16.16, Steve Campbell wrote: > I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. > > On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. > > The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Difficult to say without knowing 1) doveconf -n output and 2) .subscriptions file contents. > Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Yes, assuming you have: mail_location = mbox:~/mail (with maybe the :INBOX=/var/mail/%u) Also the .subscriptions needs to be in ~/mail/ then. From tss at iki.fi Sun Mar 4 14:45:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:45:48 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> Message-ID: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> On 2.3.2012, at 0.35, Terry Carmen wrote: > With the exchange server being returned in the msExchHomeServerName property as: > > /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername > > I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. .. > The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Do all of the values have the same prefix? Then I guess you can do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%49.100$.example.com If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%-10$.example.com There's no otherwise nice way to parse this string. From tss at iki.fi Sun Mar 4 14:47:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:47:34 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <28E8C0DA-B388-42F0-B39E-B08CA7960D09@iki.fi> On 1.3.2012, at 10.44, Joseph Tam wrote: > I would like to run various doveadm commands that involves all (mail) users like > > doveadm expunge -A mailbox Trash savedbefore 30d > > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users > > However, these accounts are system accounts (locked password, no shell) > and are in userdb to provide UID<->name mapping for utilities like ls, > chown, etc. What userdb are you using? userdb passwd should already skip users that aren't in the valid range. And what Dovecot version are you using? From tss at iki.fi Sun Mar 4 14:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:48:53 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f From tss at iki.fi Sun Mar 4 14:51:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:51:03 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users And one more thing: Does it really even stop there? Looking at the code it's supposed to log an error and continue to next user. Note that it says "Failed to iterate through SOME users". From tss at iki.fi Sun Mar 4 15:32:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:32:15 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: <20120224012247.GA6512@krell.zikzak.de> <1330342560.11500.308.camel@innu.invalid> Message-ID: On 29.2.2012, at 3.03, Andreas M. Kirchwitz wrote: > Timo Sirainen wrote: > >>> There seems to be a new dependency in some modules (eg, lib-storage, >>> libdovecot-lda, libdovecot-ssl) on OpenSSL. In Dovecot 2.0, those >>> modules didn't require OpenSSL, but 2.1 does. >>> >>> For the linking process the path to the OpenSSL library isn't >>> specified properly (SSL_LIBS). Dovecot fails to build if OpenSSL >>> is in a non-standard path. (Haven't checked if SSL_CFLAGS isn't >>> properly used as well.) >> >> Maybe http://hg.dovecot.org/dovecot-2.1/rev/c07415305d9e fixes >> everything? > > That's the way to go. Makes things better, but I've found three more > dependencies. This patch is against the daily snapshot 20120228. > (Sorry for the changes to Makefile.in which you won't need. But this way > I don't have to rebuild Makefile.in from Makefile.am when compiling. ;-) The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 From tss at iki.fi Sun Mar 4 15:36:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:36:59 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> On 28.2.2012, at 19.45, Steve Platt wrote: > Most of this is working but I'm stuck on how to convert users' mail folders > from the existing setup to the new one. I'm using the convert plugin but of > course the problem is that the plugin executes as the "vmail" user and cannot > access the existing mail folders that belong to the users: and I'd be worried > if it could, of course! Convert plugin also has some other problems. > I have the idea that I should be able to run some command (as a privileged > user) on the mail server and have it do the conversion for me, changing the > ownership/permissions on the way. > > Can convert-tool do this? Possibly, but I remember it had some problems. The best solution would be to use Dovecot v2.0's dsync. Also you can use one of the scripts in http://wiki2.dovecot.org/Migration/MailFormat such as mb2md. > I'd prefer to go with the automatic (plugin) conversion if I can bodge the > ownership issues somehow. Failing that, some tool or script may be the next > best answer. You could set mail_drop_priv_before_exec=yes, mail_access_groups=vmail and chgrp vmail, chmod g+rw the old mailboxes. From tss at iki.fi Sun Mar 4 16:10:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:10:28 +0200 Subject: [Dovecot] [PATCH] Pop3 order in courier migration script In-Reply-To: <4F4B2F62.1020204@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> Message-ID: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> On 27.2.2012, at 9.23, Christoph Bu?enius wrote: > I found a problem in the courier conversion script (courier-dovecot-migrate.pl). In some cases, it does not correctly preserve the order of POP3 UIDLs. Thanks, updated. BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. From tss at iki.fi Sun Mar 4 16:23:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:23:39 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302104333.GD11180@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: >> Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. > > In the case above that mail was gzipped twice :( Yes, looks like Dovecot can't correctly fix the wrong S size for gzipped mails. I don't know if I should bother fixing it, especially since in your case the doubly-gzipped mails will look corrupted to user.. From terry at cnysupport.com Sun Mar 4 16:48:17 2012 From: terry at cnysupport.com (Terry Carmen) Date: Sun, 04 Mar 2012 09:48:17 -0500 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> Message-ID: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> ----- Message from Timo Sirainen ---------    Date: Sun, 4 Mar 2012 14:45:48 +0200    From: Timo Sirainen Subject: Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location      To: Terry Carmen      Cc: dovecot at dovecot.org > On 2.3.2012, at 0.35, Terry Carmen wrote: >> With the exchange server being returned in the msExchHomeServerName >> property as: >> >> /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername >> >> I believe this should somehow end up in the userdb section, >> which currently contains "driver = prefetch", but can't seem to >> figure out specifically what should be there. > .. > The only important part is "cn=exchangeservername", which is > the machine name and would need to be prepended to example.com to > get the fqdn. > Do all of the values have the same prefix? Then I guess you can do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%49.100$.example.com > > If the prefix differs, but all of the exchange server names have > the same length, for example 10, you can also do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%-10$.example.com > There's no otherwise nice way to parse this string. If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). homeservers.txt: user1:exch1.example.com user2:exch1.example.com user3:exch1.example.com user4:exch2.example.com Is it possible to do a lookup in a text file to get this? Terry From tss at iki.fi Sun Mar 4 16:58:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:58:59 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> Message-ID: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> On 4.3.2012, at 16.48, Terry Carmen wrote: >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >> >> If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: >> >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >> There's no otherwise nice way to parse this string. > > > If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. OK, so if the prefix or suffix isn't always the same length you can't do the above. > I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). > > homeservers.txt: > user1:exch1.example.com > user2:exch1.example.com > user3:exch1.example.com > user4:exch2.example.com > > Is it possible to do a lookup in a text file to get this? If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile Example line: user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com Note that you can't then return any userdb fields from passdb ldap lookup. From CMarcus at Media-Brokers.com Sun Mar 4 17:27:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:27:40 -0500 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <4F5389EC.1040503@Media-Brokers.com> On 2012-03-04 7:36 AM, Timo Sirainen wrote: > So, doveconf -n> dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. One suggestion... since 'doveconf -n' is sort of a clone of 'postconf -n', maybe it would be a good idea to clone the postfix way for upgrading the configuration file as well? Postfix does it as: postfix upgrade-configuration Man page details for options are here: http://www.postfix.org/postfix.1.html I hate to keep suggesting that you 'copy' anyone or anything, but if you *are*, I think postfix is one of the ones you'd want to emulate... ;) Just a thought... it would be more intuitive for those of us who use postfix with dovecot (and I think there are a lot)... -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 4 17:57:45 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:57:45 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> Message-ID: <4F5390F9.4000301@Media-Brokers.com> On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: > On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >> On 2012-03-02 5:13 PM, Robin wrote: >>> This mailing list is for dovecot, not Thunderbird support. The lack of >>> replies to Thunderbird usage questions no doubt reflects this. >> What precisely about a possible bug with *any* IMAP client when >> using dovecot+fts makes you think that this is not on topic for the >> dovecot list? > Show dovecot misbehaving. The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:21:13 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:21:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? > > The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? well as I wrote in the mini-tutorial, if you use roundcube search, you will see that it uses solr. So from my point of view it would be a Thunderbird thing. -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From CMarcus at Media-Brokers.com Sun Mar 4 18:42:04 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 11:42:04 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> Message-ID: <4F539B5C.3030003@Media-Brokers.com> On 2012-03-04 11:21 AM, Christian Roessner wrote: >> The OP showed where *something* was misbehaving - maybe you should >> read an entire thread before jumping in? >> >> The bottom line, though, until it can be determined that it *is* a >> Thunderbird bug, we won't know if it is a dovecot bug or not, will >> we? > well as I wrote in the mini-tutorial, if you use roundcube search, ? First post from you in this thread, much less a reference to some mini-tutorial you wrote > you will see that it uses solr. So from my point of view it would be > a Thunderbird thing. As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... Looks like he isn't interested in replying (or has unsubbed from the list), so looks like there is no point in pursuing this at this point. I'll just have to wait until we get switched over to dovecot, and see if we have any problems with fts... -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:59:42 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:59:42 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F539B5C.3030003@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> Message-ID: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> >>> > >> well as I wrote in the mini-tutorial, if you use roundcube search, > > ? First post from you in this thread, much less a reference to some mini-tutorial you wrote This: http://www.roessner-network-solutions.com/2012/02/19/full-text-search-with-solr-and-dovecot-on-ubuntu-10-04/ is from my blog ;) > >> you will see that it uses solr. So from my point of view it would be >> a Thunderbird thing. > > As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... > ok -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From bradley.giesbrecht at gmail.com Sun Mar 4 19:39:37 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sun, 4 Mar 2012 09:39:37 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: On Mar 4, 2012, at 7:57 AM, Charles Marcus wrote: > On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: >> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >>> On 2012-03-02 5:13 PM, Robin wrote: >>>> This mailing list is for dovecot, not Thunderbird support. The lack of >>>> replies to Thunderbird usage questions no doubt reflects this. > >>> What precisely about a possible bug with *any* IMAP client when >>> using dovecot+fts makes you think that this is not on topic for the >>> dovecot list? > >> Show dovecot misbehaving. > > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? I have been reading this thread from the beginning. You asked the question: On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. At what point should this issue be taken to a Thunderbird support venue? I will butt out now. From CMarcus at Media-Brokers.com Sun Mar 4 20:03:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 13:03:14 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> Message-ID: <4F53AE62.3000005@Media-Brokers.com> On 2012-03-04 11:59 AM, Christian Roessner wrote: >>> you will see that it uses solr. So from my point of view it would be >>> a Thunderbird thing. >> As I said, I would like confirmation *from the OP* about his last >> comment that his problem with Thunderbird was actually fixed by fixing >> whatever 'third party init script who was the problem'... > ok One other thing I neglected to mention - I'll probably use fts+lucene, since it appears to be simpler (is a plugin and only requires clucene as a dependency)... Next is to push for full support in Thunderbird for a per account config option to simply run all searches on the server Should only be enabled for an account the server for which support fts search indexes): https://bugzilla.mozilla.org/show_bug.cgi?id=564168 -- Best regards, Charles From trashcan at odo.in-berlin.de Sun Mar 4 23:39:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 22:39:22 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:38, Timo Sirainen wrote: > On 4.3.2012, at 13.34, Michael Grimm wrote: >> On 04.03.2012, at 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. >> >> Great news. I would love to test it, if I will be able to run this on a test >> account, only. All other users should become synced the "old way" for the time >> being. >> >> Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration > to return only one test user for replicator that avoids it. (You may be able to > do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > 2) You can enable replication plugin only for one user by changing mail_plugins > setting via userdb extra fields. > > Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So > you could even ignore 1) and just let it sync everyone at startup. Does that mean that the new functionality (queue) does only run dsync replication the usual way whenever new mail arrives? That's at least what I read in your code committed today (but I'm not that good in reading code I do have to confess). If you could approve my assumption, I'm willing to give it a try to all users. Regards, Michael From stan at hardwarefreak.com Mon Mar 5 00:29:32 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 16:29:32 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <4F53ECCC.7060302@hardwarefreak.com> On 3/4/2012 11:39 AM, Bradley Giesbrecht wrote: > It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. Most software contains workarounds to bugs/misfeatures in other vendors' programs. Dovecot already has many: NFS: mmap_disable = no mail_nfs_index = no Workarounds for various client bugs: delay-newmail: netscape-eoh: tb-extra-mailbox-sep: To state that a problem in other software that interacts with Dovecot is not worth discussing seems a bit naive, or arrogant, or both. Given how long it takes, never in some cases, for Mozilla to fix IMAP related problems in TBird, you can't blame the OP for looking in other directions for a solution. Note the bug I filed 2+ years on broken IMAP custom header search: https://bugzilla.mozilla.org/show_bug.cgi?id=546925 2 years later and it's not even been assigned to a dev... -- Stan From p at state-of-mind.de Mon Mar 5 00:47:13 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Sun, 04 Mar 2012 23:47:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53ECCC.7060302@hardwarefreak.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> Message-ID: <4F53F0F1.9010002@state-of-mind.de> On 04.03.2012 23:29, Stan Hoeppner wrote: > not worth discussing seems a bit naive, or arrogant, or both. Given how > long it takes, never in some cases, for Mozilla to fix IMAP related > problems in TBird, you can't blame the OP for looking in other > directions for a solution. Note the bug I filed 2+ years on broken IMAP > custom header search: > > https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > > 2 years later and it's not even been assigned to a dev... We started buying features/fixes. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5372 bytes Desc: S/MIME Cryptographic Signature URL: From stan at hardwarefreak.com Mon Mar 5 01:07:36 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 17:07:36 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F0F1.9010002@state-of-mind.de> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> Message-ID: <4F53F5B8.8070105@hardwarefreak.com> On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how >> long it takes, never in some cases, for Mozilla to fix IMAP related >> problems in TBird, you can't blame the OP for looking in other >> directions for a solution. Note the bug I filed 2+ years on broken IMAP >> custom header search: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 >> >> 2 years later and it's not even been assigned to a dev... > > We started buying features/fixes. Does Mozilla have a page listing such services and prices, err, required/expected donation amounts? -- Stan From sdavies at sdc.com.au Mon Mar 5 01:18:40 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Mon, 5 Mar 2012 09:48:40 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> Message-ID: <201203050948.40819.sdavies@sdc.com.au> Sorry. I wasn't clear. The message did repeat for the same mailbox. eg Mar 5 09:41:40 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42304 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:41:44 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42392 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42480 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Drafts/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Templates/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=12964 for /home/scldad/Mail/Mail/.imap/Sent/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42568 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Junk/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Outbox/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=60240 for /home/scldad/Mail/Mail/.imap/storage1/dovecot.index: Extension header update points outside header size I have deleted the .imap directories and the message seems to have disappeared. Cheers and thanks, Stephen On Sun, 4 Mar 2012 11:05:18 PM Timo Sirainen wrote: > On 3.3.2012, at 1.45, Stephen Davies wrote: > > No NFS. The file system is local. > > > > Yes. There are multiple copies of the message for multiple mailboxes for > > each of at least two users. > > But does the error keep repeating for the same mailbox? It's supposed to > fix itself automatically after logging the error once. > > > Yes. Did recently upgrade from 1.2.15. > > I think in earlier versions mbox used somewhat different index file > structures and now Dovecot logs some errors about them. > > Anyway, one sure way to fix this is to just delete all the .imap/ > directories. -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From amk at spamfence.net Mon Mar 5 02:32:17 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Mon, 5 Mar 2012 00:32:17 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: Message-ID: Hello Timo! Timo Sirainen wrote: > The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: > http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 Thanks for this patch. I've applied it to the dovecot-20120303 nightly snapshot. The good news is, compilation works fine. The bad news is, the libraries and binaries don't work because they don't find the custom SSL libraries. Greetings, Andreas =============================================================================== $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install =============================================================================== $ ldd src/*/.libs/*.so src/auth/.libs/libauthdb_imap.so: linux-gate.so.1 => (0x0013a000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0091d000) librt.so.1 => /lib/librt.so.1 (0x003c7000) libc.so.6 => /lib/libc.so.6 (0x00c6a000) libdl.so.2 => /lib/libdl.so.2 (0x009a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00491000) /lib/ld-linux.so.2 (0x007a4000) src/lib-dovecot/.libs/libdovecot.so: linux-gate.so.1 => (0x0053c000) libdl.so.2 => /lib/libdl.so.2 (0x0056d000) librt.so.1 => /lib/librt.so.1 (0x00925000) libc.so.6 => /lib/libc.so.6 (0x00626000) /lib/ld-linux.so.2 (0x00a61000) libpthread.so.0 => /lib/libpthread.so.0 (0x003ec000) src/lib-lda/.libs/libdovecot-lda.so: linux-gate.so.1 => (0x00b75000) libdovecot-storage.so.0 => /usr/local/src/dovecot-20120303/src/lib-storage/.libs/libdovecot-storage.so.0 (0x00c9a000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0062d000) librt.so.1 => /lib/librt.so.1 (0x00b3d000) libc.so.6 => /lib/libc.so.6 (0x00110000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x002f1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ab1000) /lib/ld-linux.so.2 (0x00f23000) src/lib-sql/.libs/libdovecot-sql.so: linux-gate.so.1 => (0x006d3000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0096c000) libdl.so.2 => /lib/libdl.so.2 (0x0078c000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x00119000) /lib/ld-linux.so.2 (0x00731000) libpthread.so.0 => /lib/libpthread.so.0 (0x00569000) src/lib-ssl-iostream/.libs/libdovecot-ssl.so: linux-gate.so.1 => (0x00ea1000) libdl.so.2 => /lib/libdl.so.2 (0x00b31000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00110000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00dcf000) librt.so.1 => /lib/librt.so.1 (0x00fa5000) libc.so.6 => /lib/libc.so.6 (0x002d3000) /lib/ld-linux.so.2 (0x002b4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d3c000) src/lib-storage/.libs/libdovecot-storage.so: linux-gate.so.1 => (0x002ee000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00395000) libdl.so.2 => /lib/libdl.so.2 (0x00958000) librt.so.1 => /lib/librt.so.1 (0x00333000) libc.so.6 => /lib/libc.so.6 (0x00d45000) /lib/ld-linux.so.2 (0x008e9000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f06000) src/login-common/.libs/libdovecot-login.so: linux-gate.so.1 => (0x00d66000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00c82000) librt.so.1 => /lib/librt.so.1 (0x00f64000) libc.so.6 => /lib/libc.so.6 (0x00110000) libdl.so.2 => /lib/libdl.so.2 (0x00b26000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /lib/ld-linux.so.2 (0x00520000) =============================================================================== $ ldd /usr/local/Dovecot-20120303/lib/dovecot/*.so /usr/local/Dovecot-20120303/*bin/* /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so: linux-gate.so.1 => (0x00230000) librt.so.1 => /lib/librt.so.1 (0x00b69000) libc.so.6 => /lib/libc.so.6 (0x00231000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a82000) /lib/ld-linux.so.2 (0x007eb000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_imap_acl_plugin.so: linux-gate.so.1 => (0x004fb000) lib01_acl_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so (0x00c6f000) librt.so.1 => /lib/librt.so.1 (0x0061d000) libc.so.6 => /lib/libc.so.6 (0x001dd000) libpthread.so.0 => /lib/libpthread.so.0 (0x00fae000) /lib/ld-linux.so.2 (0x00b89000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_lazy_expunge_plugin.so: linux-gate.so.1 => (0x00e5b000) librt.so.1 => /lib/librt.so.1 (0x00847000) libc.so.6 => /lib/libc.so.6 (0x00110000) libpthread.so.0 => /lib/libpthread.so.0 (0x0039d000) /lib/ld-linux.so.2 (0x00a4e000) /usr/local/Dovecot-20120303/lib/dovecot/lib05_snarf_plugin.so: linux-gate.so.1 => (0x001e2000) librt.so.1 => /lib/librt.so.1 (0x00441000) libc.so.6 => /lib/libc.so.6 (0x00220000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cfa000) /lib/ld-linux.so.2 (0x00acd000) /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so: linux-gate.so.1 => (0x00fa9000) librt.so.1 => /lib/librt.so.1 (0x00bb4000) libc.so.6 => /lib/libc.so.6 (0x00d5e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00be7000) /lib/ld-linux.so.2 (0x0055c000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_imap_quota_plugin.so: linux-gate.so.1 => (0x00426000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x004c9000) librt.so.1 => /lib/librt.so.1 (0x00e35000) libc.so.6 => /lib/libc.so.6 (0x009b5000) libpthread.so.0 => /lib/libpthread.so.0 (0x00146000) /lib/ld-linux.so.2 (0x00507000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_trash_plugin.so: linux-gate.so.1 => (0x00baf000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x00d8b000) librt.so.1 => /lib/librt.so.1 (0x0041e000) libc.so.6 => /lib/libc.so.6 (0x00bb6000) libpthread.so.0 => /lib/libpthread.so.0 (0x0088a000) /lib/ld-linux.so.2 (0x00a6d000) /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00f77000) libc.so.6 => /lib/libc.so.6 (0x00146000) libpthread.so.0 => /lib/libpthread.so.0 (0x00df9000) /lib/ld-linux.so.2 (0x004dd000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_autocreate_plugin.so: linux-gate.so.1 => (0x005d2000) librt.so.1 => /lib/librt.so.1 (0x007ed000) libc.so.6 => /lib/libc.so.6 (0x00262000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b48000) /lib/ld-linux.so.2 (0x00243000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_expire_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x0099e000) libc.so.6 => /lib/libc.so.6 (0x0013c000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a93000) /lib/ld-linux.so.2 (0x00460000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so: linux-gate.so.1 => (0x002c1000) librt.so.1 => /lib/librt.so.1 (0x00ef2000) libc.so.6 => /lib/libc.so.6 (0x00c3a000) libpthread.so.0 => /lib/libpthread.so.0 (0x0028d000) /lib/ld-linux.so.2 (0x003f5000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_listescape_plugin.so: linux-gate.so.1 => (0x0027c000) librt.so.1 => /lib/librt.so.1 (0x007cf000) libc.so.6 => /lib/libc.so.6 (0x00427000) libpthread.so.0 => /lib/libpthread.so.0 (0x00110000) /lib/ld-linux.so.2 (0x00d25000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_mail_log_plugin.so: linux-gate.so.1 => (0x00cff000) lib15_notify_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so (0x00a2b000) librt.so.1 => /lib/librt.so.1 (0x00152000) libc.so.6 => /lib/libc.so.6 (0x00355000) libpthread.so.0 => /lib/libpthread.so.0 (0x00680000) /lib/ld-linux.so.2 (0x00c16000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_virtual_plugin.so: linux-gate.so.1 => (0x0074d000) librt.so.1 => /lib/librt.so.1 (0x008d2000) libc.so.6 => /lib/libc.so.6 (0x004c2000) libpthread.so.0 => /lib/libpthread.so.0 (0x009c5000) /lib/ld-linux.so.2 (0x0038e000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so: linux-gate.so.1 => (0x004ab000) libz.so.1 => /lib/libz.so.1 (0x0091c000) libbz2.so.1 => /lib/libbz2.so.1 (0x00c4d000) librt.so.1 => /lib/librt.so.1 (0x00b30000) libc.so.6 => /lib/libc.so.6 (0x0075b000) libpthread.so.0 => /lib/libpthread.so.0 (0x00611000) /lib/ld-linux.so.2 (0x00bfd000) /usr/local/Dovecot-20120303/lib/dovecot/lib21_fts_squat_plugin.so: linux-gate.so.1 => (0x00df4000) lib20_fts_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so (0x00ea8000) librt.so.1 => /lib/librt.so.1 (0x0021d000) libc.so.6 => /lib/libc.so.6 (0x00bd1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d63000) /lib/ld-linux.so.2 (0x00b83000) /usr/local/Dovecot-20120303/lib/dovecot/lib30_imap_zlib_plugin.so: linux-gate.so.1 => (0x0059f000) lib20_zlib_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so (0x00652000) librt.so.1 => /lib/librt.so.1 (0x00f2d000) libc.so.6 => /lib/libc.so.6 (0x00735000) libz.so.1 => /lib/libz.so.1 (0x00110000) libbz2.so.1 => /lib/libbz2.so.1 (0x00125000) libpthread.so.0 => /lib/libpthread.so.0 (0x00136000) /lib/ld-linux.so.2 (0x001f9000) /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so: linux-gate.so.1 => (0x004c0000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x006b2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00992000) /lib/ld-linux.so.2 (0x00d92000) /usr/local/Dovecot-20120303/lib/dovecot/lib95_imap_stats_plugin.so: linux-gate.so.1 => (0x0076b000) lib90_stats_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so (0x00b90000) librt.so.1 => /lib/librt.so.1 (0x00215000) libc.so.6 => /lib/libc.so.6 (0x0021e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /lib/ld-linux.so.2 (0x00e48000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-lda.so: linux-gate.so.1 => (0x00d82000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x009bb000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00542000) libc.so.6 => /lib/libc.so.6 (0x002f6000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x006a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00189000) /lib/ld-linux.so.2 (0x002d7000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-login.so: linux-gate.so.1 => (0x00f96000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00e56000) librt.so.1 => /lib/librt.so.1 (0x00371000) libc.so.6 => /lib/libc.so.6 (0x00168000) libdl.so.2 => /lib/libdl.so.2 (0x00623000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ccd000) /lib/ld-linux.so.2 (0x0074f000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so: linux-gate.so.1 => (0x00bf3000) libdl.so.2 => /lib/libdl.so.2 (0x00a54000) librt.so.1 => /lib/librt.so.1 (0x00ad8000) libc.so.6 => /lib/libc.so.6 (0x00e63000) /lib/ld-linux.so.2 (0x00b1c000) libpthread.so.0 => /lib/libpthread.so.0 (0x005ac000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-sql.so: linux-gate.so.1 => (0x008b7000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00909000) libdl.so.2 => /lib/libdl.so.2 (0x005f5000) librt.so.1 => /lib/librt.so.1 (0x008bc000) libc.so.6 => /lib/libc.so.6 (0x00675000) /lib/ld-linux.so.2 (0x004bc000) libpthread.so.0 => /lib/libpthread.so.0 (0x00184000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-ssl.so: linux-gate.so.1 => (0x00ef2000) libdl.so.2 => /lib/libdl.so.2 (0x0033f000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00a3d000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x0034e000) librt.so.1 => /lib/librt.so.1 (0x002ea000) libc.so.6 => /lib/libc.so.6 (0x00110000) /lib/ld-linux.so.2 (0x007d4000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so: linux-gate.so.1 => (0x0089e000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00be9000) libdl.so.2 => /lib/libdl.so.2 (0x00852000) librt.so.1 => /lib/librt.so.1 (0x001aa000) libc.so.6 => /lib/libc.so.6 (0x00442000) /lib/ld-linux.so.2 (0x00b8a000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b0e000) /usr/local/Dovecot-20120303/bin/doveadm: linux-gate.so.1 => (0x00c6a000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00110000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00acf000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x0066e000) libc.so.6 => /lib/libc.so.6 (0x00247000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x001f4000) librt.so.1 => /lib/librt.so.1 (0x00ab5000) libfreebl3.so => /lib/libfreebl3.so (0x003d1000) /lib/ld-linux.so.2 (0x00228000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cc3000) /usr/local/Dovecot-20120303/bin/doveconf: linux-gate.so.1 => (0x00830000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x0032a000) libc.so.6 => /lib/libc.so.6 (0x00876000) libdl.so.2 => /lib/libdl.so.2 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x002b6000) /lib/ld-linux.so.2 (0x007d2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /usr/local/Dovecot-20120303/bin/dsync: linux-gate.so.1 => (0x00c8b000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00257000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00662000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00d15000) libc.so.6 => /lib/libc.so.6 (0x00dd1000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x009b3000) librt.so.1 => /lib/librt.so.1 (0x0037f000) libfreebl3.so => /lib/libfreebl3.so (0x00110000) /lib/ld-linux.so.2 (0x00a8e000) libpthread.so.0 => /lib/libpthread.so.0 (0x001e9000) /usr/local/Dovecot-20120303/sbin/dovecot: linux-gate.so.1 => (0x00f00000) libcap.so.2 => /lib/libcap.so.2 (0x0037c000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00555000) libc.so.6 => /lib/libc.so.6 (0x0080b000) libattr.so.1 => /lib/libattr.so.1 (0x00ece000) libdl.so.2 => /lib/libdl.so.2 (0x00ea2000) librt.so.1 => /lib/librt.so.1 (0x00dce000) /lib/ld-linux.so.2 (0x00d49000) libpthread.so.0 => /lib/libpthread.so.0 (0x00bb2000) =============================================================================== From isolecki at gmail.com Mon Mar 5 03:42:12 2012 From: isolecki at gmail.com (Ian Solecki) Date: Sun, 4 Mar 2012 20:42:12 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine Message-ID: Hello, hoping someone here might be able to help me or at least point me in the right direction. My company recently (last week) moved to a new dedicated server for website and email hosting. It is a fairly run-of-the-mill Linux machine running cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail server. When we first set up the server, I was able to create a mailbox, access it via any desktop mail client, webmail, AND by setting it up as a basic POP3 account on my BlackBerry. Mail was running fine to and from the BlackBerry, no problems. I deleted that account from my BlackBerry as it was a test account, and went to add my actual account but was unable to do so. Received the "Cannot log in. Verify your email address, user name and password. If the error persists, contact (my domain name)" message. I tried the test account that had been running successfully and sending/receiving emails not minutes earlier, and it would not set up either. I have since tried multiple accounts on multiple BlackBerry devices (different models) on multiple carriers on several of my different domains (all of which point to the same server, of course), to no avail. Yet, any of these accounts still work flawlessly in any desktop mail client (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these BlackBerrys work flawlessly with any other mail server. The username to log in to these mailboxes is not standard, it's mailboxname+ example.com if the email address is mailboxname at example.com and the mail server is mail.example.com. So, I know I have to access the "Advanced Settings" in BlackBerry email setup in order to put this username in. Still, no effect. So, there is something wrong with how mail SETUP works (not sending/receiving, though that may also not work, I have no way of knowing now) between RIM and my server, and it's something that has changed since the server was set up a week ago. My carrier(s) are clueless, my dedicated server provider (Lunarpages) is clueless. Can anyone help? From gedalya at gedalya.net Mon Mar 5 04:52:06 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 04 Mar 2012 21:52:06 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F542A56.6070104@gedalya.net> Do you have the dovecot logs? What do they say about connections coming from RIM? On 03/04/2012 08:42 PM, Ian Solecki wrote: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > From dchenusa at yahoo.com Mon Mar 5 05:57:16 2012 From: dchenusa at yahoo.com (Dennis Chen) Date: Sun, 4 Mar 2012 19:57:16 -0800 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Thanks Timo, I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? I also posted the similar question to ubuntuforums, but not much response. Hope you understand my point. It's very frustrated! Sent from my iPhone On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > On 4.3.2012, at 8.25, D Chen wrote: > >> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! > > v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > >> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf > > So, doveconf -n > dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. > From tss at iki.fi Mon Mar 5 08:41:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:41:36 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Message-ID: Dovecot v2.0 changed the settings a lot compared to v1.x. There are no longer auth default or socket listen sections. The doveconf -n generated dovecot.conf should contain all of the settings that you had in v1.x, converted for v2.0. So you should be able to use it directly without problems. If you want, you could look for Dovecot's example-config that probably comes with Ubuntu (in /usr/share/doc/dovecot*/ maybe?), copy those to /etc/dovecot/ and change the settings in there based on the generated dovecot.conf. In any case you shouldn't try to add those v1.x-specific things back there anymore, since they'll add back the "obsolete settings" warnings. On 5.3.2012, at 5.57, Dennis Chen wrote: > Thanks Timo, > > I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) > > Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? > > I also posted the similar question to ubuntuforums, but not much response. > > Hope you understand my point. > > It's very frustrated! > > > Sent from my iPhone > > On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > >> On 4.3.2012, at 8.25, D Chen wrote: >> >>> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! >> >> v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: >> >>> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf >> >> So, doveconf -n > dovecot-new.conf >> mv dovecot-new.conf /etc/dovecot/dovecot.conf >> >> That should do it. >> > From tss at iki.fi Mon Mar 5 08:43:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:43:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Message-ID: <86F8DB00-F1B0-4666-B3EC-B3EA25F87C0B@iki.fi> On 4.3.2012, at 23.39, Michael Grimm wrote: >> Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So >> you could even ignore 1) and just let it sync everyone at startup. > > Does that mean that the new functionality (queue) does only run dsync replication > the usual way whenever new mail arrives? That's at least what I read in your code > committed today (but I'm not that good in reading code I do have to confess). > > If you could approve my assumption, I'm willing to give it a try to all users. Yes, the replicator simply runs "doveadm sync -u user at domain -d" (and sometimes with -f). The -d gets the default location from mail_replica setting. From robert at schetterer.org Mon Mar 5 08:48:01 2012 From: robert at schetterer.org (Robert Schetterer) Date: Mon, 05 Mar 2012 07:48:01 +0100 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F5461A1.7000100@schetterer.org> Am 05.03.2012 02:42, schrieb Ian Solecki: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > this is the dovecot mail list, not support for blackberry, we cant help unless you have dovecot configs and dediacted logs to your problem -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From p at state-of-mind.de Mon Mar 5 08:53:39 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Mon, 5 Mar 2012 07:53:39 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F5B8.8070105@hardwarefreak.com> References: <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> <4F53F5B8.8070105@hardwarefreak.com> Message-ID: <20120305065339.GC5094@state-of-mind.de> Stan, * Stan Hoeppner : > On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > > > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > > > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how > >> long it takes, never in some cases, for Mozilla to fix IMAP related > >> problems in TBird, you can't blame the OP for looking in other > >> directions for a solution. Note the bug I filed 2+ years on broken IMAP > >> custom header search: > >> > >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > >> > >> 2 years later and it's not even been assigned to a dev... > > > > We started buying features/fixes. > > Does Mozilla have a page listing such services and prices, err, > required/expected donation amounts? to my knowledge they don't have a page listing services and prices. Recently they discussed pros and cons of crowd sourcing, but without much progress. I can get you in contact with one of the TB programmers, who implemented features for us, if you want to. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3603 bytes Desc: not available URL: From bra at fsn.hu Mon Mar 5 09:25:34 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 08:25:34 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <4F546A6E.6020400@fsn.hu> Hi, On 03/04/12 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Everything > isn't finished yet, but it appears to work and I've enabled it for my > @dovecot.fi mails. Some issues: > > - public namespace isn't replicated at all > - shared namespace is replicated, but not private mail flags > - I've only tested SSH replication setup now, not director > replication setup (and director setup is still missing many things) > - SSH replication setup uses aggregator process, which isn't really > necessary and can probably be avoided in future Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. (if I understand things correctly) Thanks for working on this. From janfrode at tanso.net Mon Mar 5 10:35:25 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 09:35:25 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <20120305083525.GA20889@dibs.tanso.net> On Sun, Mar 04, 2012 at 01:38:14PM +0200, Timo Sirainen wrote: > > > > Great news. I would love to test it, if I will be able to run this on a test > > account, only. All other users should become synced the "old way" for the time > > being. > > > > Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) IMHO it would be great if it didn't sync all users. We probably av have hundreds of thousands of inactive users that we would like to sync at a later point. Also when we provision users that's just an entry in a LDAP-directory without any files or directories. So dovecot shouldn't create any directories for these before they've received mail or logged in. So, ideally (for us), dovecot should keep a log over which accounts are active (has received or checked mail), and only sync users that has been active for the last $timeperiode on startup. -jf From ccourvoisier70 at yahoo.com Mon Mar 5 11:51:19 2012 From: ccourvoisier70 at yahoo.com (Charles C) Date: Mon, 5 Mar 2012 09:51:19 +0000 (GMT) Subject: [Dovecot] 1.0beta to latest and greatest? Message-ID: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> dear dovecot users, This is my first post and with a rather embarassing question. To soften the question up I just want to say I've been very pleased with Dovecot thus far and its low maintenance requirements - perhaps to pleased! I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Thanks for your input. Charles From tss at iki.fi Mon Mar 5 11:56:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 11:56:30 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> Message-ID: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> On 4.3.2012, at 13.54, Timo Sirainen wrote: > On 4.3.2012, at 13.41, Michael Grimm wrote: > >>> By "undeletable" do you mean you have mails that always come back after expunging them? >> >> Yes. Deleting by the client will return them after the next dsync run. Luckily this just started happening to me as well. After some debugging I found and fixed the problem: http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 From tss at iki.fi Mon Mar 5 12:08:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:08:35 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F546A6E.6020400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: On 5.3.2012, at 9.25, Attila Nagy wrote: > On 03/04/12 11:44, Timo Sirainen wrote: >> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >> > Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. Sure the idea is to improve the performance :) There are two ways: 1) Use longer running SSH sessions which dsync more than one user at a time. 2) Use TCP connections instead of SSH. > It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. From tss at iki.fi Mon Mar 5 12:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:41:30 +0200 Subject: [Dovecot] 1.0beta to latest and greatest? In-Reply-To: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> References: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> Message-ID: On 5.3.2012, at 11.51, Charles C wrote: > I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Just delete the indexes and you don't have to worry about problems related to them. The config file is different though and you'll probably have to spend some time converting it. (I'm assuming you're using mbox/maildir, not the broken dbox implementation.) From tss at iki.fi Mon Mar 5 12:45:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:45:26 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305083525.GA20889@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> Message-ID: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> On 5.3.2012, at 10.35, Jan-Frode Myklebust wrote: >> 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > IMHO it would be great if it didn't sync all users. We probably av have > hundreds of thousands of inactive users that we would like to sync at a > later point. Also when we provision users that's just an entry in a > LDAP-directory without any files or directories. So dovecot shouldn't > create any directories for these before they've received mail or logged in. > > So, ideally (for us), dovecot should keep a log over which accounts are > active (has received or checked mail), and only sync users that has been > active for the last $timeperiode on startup. Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. From CMarcus at Media-Brokers.com Mon Mar 5 13:13:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:13:07 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F547CAB.2030005@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F547CAB.2030005@gmail.com> Message-ID: <4F549FC3.5030602@Media-Brokers.com> On 2012-03-05 3:43 AM, kadafax at gmail.com wrote: > Le 02/03/12 13:40, Charles Marcus a ?crit : >> On 2012-02-28 11:28 AM, Charles Marcus wrote: >>> On 2012-02-28 11:05 AM, kfx wrote: >>>> Ok I feel ashame... it was a third party init scrip who was the >>>> problem :( >>> So... you're saying that Thunderbird now correctly uses server side >>> search? >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > Yes, for me thunderbird correctly use server side search. Just got confirmation from the OP that his problem with Thunderbird indeed is now gone, so it was a 3rd party init script causing his problems... One less thing to worry about... -- Best regards, Charles From CMarcus at Media-Brokers.com Mon Mar 5 13:24:28 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:24:28 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <1330441042.2081.24.camel@innu> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> Message-ID: <4F54A26C.4040005@Media-Brokers.com> On 2012-02-28 9:57 AM, Timo Sirainen wrote: > So, Solr in Dovecot works perfectly. Timo, a follow-up on this... Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? I don't have a dovecot test server set up yet, but even if I did, I wouldn't really know what to look for or how to test this myself... Thanks, -- Best regards, Charles From mark at ecs.vuw.ac.nz Mon Mar 5 13:26:22 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Tue, 06 Mar 2012 00:26:22 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330603470.2081.37.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> <1330603470.2081.37.camel@innu> Message-ID: <4F54A2DE.5000500@ecs.vuw.ac.nz> On 03/02/12 01:04, Timo Sirainen wrote: > The difference between your previously working system and currently > working system is the GSSAPI/Kerberos libraries. Just to close this thread off, seems that the bug was in the cyrus-sasl libraries that kmail uses. Reverting from the 2.1.25 version that the latest was trying to use to 2.1.23 that was on the older systems got it working again. When I get a moment I'll try and work out what specifically changed. cheers mark From bra at fsn.hu Mon Mar 5 14:15:39 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 13:15:39 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: <4F54AE6B.4060400@fsn.hu> On 03/05/12 11:08, Timo Sirainen wrote: > On 5.3.2012, at 9.25, Attila Nagy wrote: > >> On 03/04/12 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>> >> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. > Sure the idea is to improve the performance :) There are two ways: > > 1) Use longer running SSH sessions which dsync more than one user at a time. > > 2) Use TCP connections instead of SSH. Don't forget about connection pooling to get concurrency. :) BTW, despite being somewhat harder to implement, I personally like native connections better. > >> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. > The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. > It depends. For a moderately loaded server I get this: # time ssh root at be02 "echo 1" 1 0.000u 0.009s 0:00.30 0.0% 0+0k 0+0io 0pf+0w ICMP echo RTT is 0.878 ms. So the ssh connection adds ~29 ms overhead to each sync request. Yes, dsync seems to need some optimizations too. :) I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. The method was similar to yours: - an external library wrote modified user ids to a file - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) The runs were longer and longer... BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... From janfrode at tanso.net Mon Mar 5 14:35:08 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 13:35:08 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> Message-ID: <20120305123508.GA22845@dibs.tanso.net> On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: > > > > So, ideally (for us), dovecot should keep a log over which accounts are > > active (has received or checked mail), and only sync users that has been > > active for the last $timeperiode on startup. > > Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. > .. or we could keep touching /activemailaccounts/$address in post-login scripts, and run "doveadm sync" for any user updated the last $timeperiode and avoid the need for SQL-userdatabase. But we still don't have a last-login update on lmtp delivery... or has this changed? -jf From tss at iki.fi Mon Mar 5 14:48:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 14:48:40 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54AE6B.4060400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: On 5.3.2012, at 14.15, Attila Nagy wrote: >>> On 03/04/12 11:44, Timo Sirainen wrote: >>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>> >>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >> Sure the idea is to improve the performance :) There are two ways: >> >> 1) Use longer running SSH sessions which dsync more than one user at a time. >> >> 2) Use TCP connections instead of SSH. > Don't forget about connection pooling to get concurrency. :) There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. > BTW, despite being somewhat harder to implement, I personally like native connections better. Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. >>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >> > It depends. For a moderately loaded server I get this: > # time ssh root at be02 "echo 1" I meant doveadm/dsync costs, ssh startup is rather slow. > Yes, dsync seems to need some optimizations too. :) > I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. > The method was similar to yours: > - an external library wrote modified user ids to a file > - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) > > The runs were longer and longer... dsync doesn't currently take enough advantage of modseqs and send only the changed data. > BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... Definitely doesn't help. From apm at one.com Mon Mar 5 15:01:54 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 14:01:54 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST Message-ID: <4F54B942.9070005@one.com> Hi, I noticed a difference between courier and dovecot, and I'm not sure which of them is wrong wrt. RFC3501 - if any. I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: INBOX INBOX.Trash INBOX.INBOX.folder INBOX.INBOX.folder.a INBOX.INBOX.folder.b The INBOX.INBOX folder does not exist on disk and is not subscribed. Courier responds to: . list "" "*" with * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" But dovecot does not list that folder using "*". However, if you issue: . list "" "INBOX.%" Dovecot answers: * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". I know the recommended client way is to use "%", but I'm still curious about which is the correct behaviour. /Peter From bra at fsn.hu Mon Mar 5 15:11:06 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 14:11:06 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: <4F54BB6A.20702@fsn.hu> On 03/05/12 13:48, Timo Sirainen wrote: > On 5.3.2012, at 14.15, Attila Nagy wrote: > >>>> On 03/04/12 11:44, Timo Sirainen wrote: >>>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>>> >>>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >>> Sure the idea is to improve the performance :) There are two ways: >>> >>> 1) Use longer running SSH sessions which dsync more than one user at a time. >>> >>> 2) Use TCP connections instead of SSH. >> Don't forget about connection pooling to get concurrency. :) > There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. Good to hear. > >> BTW, despite being somewhat harder to implement, I personally like native connections better. > Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. Yes. For large installations there may be some backend channel already (SSL tunnels, IPSec etc), so it seems to be OK. > >>>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >>> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >>> >> It depends. For a moderately loaded server I get this: >> # time ssh root at be02 "echo 1" > I meant doveadm/dsync costs, ssh startup is rather slow. I see. Running from network makes this worse slightly. Long running processes with long running connections rule. :) > >> Yes, dsync seems to need some optimizations too. :) >> I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. >> The method was similar to yours: >> - an external library wrote modified user ids to a file >> - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) >> >> The runs were longer and longer... > dsync doesn't currently take enough advantage of modseqs and send only the changed data. Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. > >> BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... > Definitely doesn't help. I know, we are working on this. :) From tss at iki.fi Mon Mar 5 16:37:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:37:44 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54BB6A.20702@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> <4F54BB6A.20702@fsn.hu> Message-ID: <8E6FD156-E7F5-49BD-9C3A-1F012E600DD8@iki.fi> On 5.3.2012, at 15.11, Attila Nagy wrote: >> dsync doesn't currently take enough advantage of modseqs and send only the changed data. > Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? > I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? The plan is to get it working with at least a few thousand users to several tens of thousands. > BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. Yes, that's also in my future plans, but it's a larger change. Also I don't think Cassandra (or any nosql?) still supports application-level merging of data after split brain. From tss at iki.fi Mon Mar 5 16:45:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:45:55 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54B942.9070005@one.com> References: <4F54B942.9070005@one.com> Message-ID: On 5.3.2012, at 15.01, Peter Mogensen wrote: > I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: > > INBOX > INBOX.Trash > INBOX.INBOX.folder > INBOX.INBOX.folder.a > INBOX.INBOX.folder.b > > The INBOX.INBOX folder does not exist on disk and is not subscribed. > > Courier responds to: > . list "" "*" > with > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" I'm surprised Courier would return this. > But dovecot does not list that folder using "*". But it returns all of the mailboxes under INBOX.INBOX, right? > However, if you issue: > . list "" "INBOX.%" > > Dovecot answers: > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. > This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Also if you used LAYOUT=fs in Dovecot, it would always show the \Noselect mailboxes because they happen to exist physically. From tss at iki.fi Mon Mar 5 16:48:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:48:11 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54A26C.4040005@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> Message-ID: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> On 5.3.2012, at 13.24, Charles Marcus wrote: > On 2012-02-28 9:57 AM, Timo Sirainen wrote: >> So, Solr in Dovecot works perfectly. > > Timo, a follow-up on this... > > Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... > > Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? The regular IMAP protocol supports searching only from the selected folder. But I guess this quickfilter search also searches from only the selected folder. So I don't see a problem. From tss at iki.fi Mon Mar 5 16:56:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:56:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305123508.GA22845@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> <20120305123508.GA22845@dibs.tanso.net> Message-ID: On 5.3.2012, at 14.35, Jan-Frode Myklebust wrote: > On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: >>> >>> So, ideally (for us), dovecot should keep a log over which accounts are >>> active (has received or checked mail), and only sync users that has been >>> active for the last $timeperiode on startup. >> >> Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. >> > > .. or we could keep touching /activemailaccounts/$address in post-login > scripts, and run "doveadm sync" for any user updated the last $timeperiode > and avoid the need for SQL-userdatabase. But we still don't have a > last-login update on lmtp delivery... or has this changed? It would be pretty simple to write such a plugin that globally does it for all imap/pop3/lmtp. Here, works for v2.0 and v2.1: http://dovecot.org/patches/2.1/lastaccess-plugin.c From apm at one.com Mon Mar 5 16:56:52 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 15:56:52 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> Message-ID: <4F54D434.6090300@one.com> On 2012-03-05 15:45, Timo Sirainen wrote: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > I'm surprised Courier would return this. > >> But dovecot does not list that folder using "*". > > But it returns all of the mailboxes under INBOX.INBOX, right? Yes. And they exists on disk and are subscribed to. >> However, if you issue: >> . list "" "INBOX.%" >> >> Dovecot answers: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. Seems reasonable. >> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". > > What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. /Peter From tss at iki.fi Mon Mar 5 17:08:02 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 17:08:02 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54D434.6090300@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> Message-ID: <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> On 5.3.2012, at 16.56, Peter Mogensen wrote: >>> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". >> >> What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. > > Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. Couldn't the scripts be simply fixed to figure out that if foo.bar.baz is returned without foo or foo.bar, then just internally assume them being there as \noselect? From CMarcus at Media-Brokers.com Mon Mar 5 18:12:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 11:12:19 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> Message-ID: <4F54E5E3.2010509@Media-Brokers.com> On 2012-03-05 9:48 AM, Timo Sirainen wrote: > On 5.3.2012, at 13.24, Charles Marcus wrote: >> Thunderbird has a 'Quickfilter Toolbar' with a little searchbox >> that applies a filter of the current folder message view pane (to >> show you only messages in the pane that meet the criteria >> specified). The default criteria that are selected are just >> 'Sender', 'Recipients' and 'Subject', but you can also select >> 'Body' (and a few others like 'Unread', 'Contain attachments', >> etc), but the 'Body' criteria is the one that would want/need to >> use the fts indexes, so... >> >> Can dovecots fts indexes be used in a case like this? In other >> words, will it 'just work'? Or will it *not* work? Or, would the >> client need to specify the folder in the commands it sends to >> limit the search and/or results to just the currently selected >> folder? Or is this even possible for such a simple/limited use >> filtering mechanism? > The regular IMAP protocol supports searching only from the selected > folder. Interesting, thanks... so, just guessing, most likely Thunderbird simply iterates over each folder in an account when searching an entire account and 'Run search on server' is checked (only available in the Advanced Search window)... One last question then (couldn't find an answer on the wiki)... I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: https://bugzilla.mozilla.org/show_bug.cgi?id=541400 This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? This would pretty much neutralize/work around the above bug. > But I guess this quickfilter search also searches from only the > selected folder. So I don't see a problem. Correct, and thanks for the confirmation... -- Best regards, Charles From tss at iki.fi Mon Mar 5 19:01:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:01:45 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54E5E3.2010509@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> Message-ID: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> On 5.3.2012, at 18.12, Charles Marcus wrote: > One last question then (couldn't find an answer on the wiki)... > > I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... > > What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? Lucene doesn't really work that way. It only searches full words. But some words are "stop words" that are ignored, such as "a" or "the" in the English language. So fts-lucene won't find anything when you search for "a", but other character searches will return all messages that contain it as a full word. (Or: That's how I think it works, too lazy to test it now.) > The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: > > https://bugzilla.mozilla.org/show_bug.cgi?id=541400 > > This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... I doubt this is going to be a problem with Dovecot+fts. The search results will be pretty much instantaneous, even if the search matches all of the messages. > Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? Nope. From CMarcus at Media-Brokers.com Mon Mar 5 19:11:50 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 12:11:50 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> Message-ID: <4F54F3D6.9040508@Media-Brokers.com> On 2012-03-05 12:01 PM, Timo Sirainen wrote: > On 5.3.2012, at 18.12, Charles Marcus wrote: >> Thunderbird has an annoying behavior where it sends a new/separate >> query each time a character is typed, beginning with the very first >> character: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=541400 >> >> This causes massive performance degradation on my clients system >> that currently uses Courier-IMAP (this is the one we'll soon be >> converting to dovecot) when the users (usually accidentally >> (select the 'Body' criteria in the Quickfilter searchbox... > I doubt this is going to be a problem with Dovecot+fts. The search > results will be pretty much instantaneous, even if the search matches > all of the messages. Cool... I guess I'll just wait and see then, and revisit this if we run into problems... Thanks Timo... -- Best regards, Charles From joshua at hybrid.pl Mon Mar 5 19:25:43 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Mon, 5 Mar 2012 18:25:43 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: <1330677951.2081.49.camel@innu> References: <1330677951.2081.49.camel@innu> Message-ID: On Fri, 2 Mar 2012, Timo Sirainen wrote: > On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: >> However, if we have everything redundant, why not have the same with SMTP >> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > If both servers randomly access users' mails, with NFS you'll have some > trouble, with OCFS2 probably less trouble. But in both cases you'll have > better performance and no problems if you use Dovecot director in both > servers (install both director and backend to both servers). > http://wiki2.dovecot.org/Director Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From tss at iki.fi Mon Mar 5 19:53:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:53:12 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: <1330677951.2081.49.camel@innu> Message-ID: <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> On 5.3.2012, at 19.25, Jacek Osiecki wrote: >>> However, if we have everything redundant, why not have the same with SMTP >>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > >> If both servers randomly access users' mails, with NFS you'll have some >> trouble, with OCFS2 probably less trouble. But in both cases you'll have >> better performance and no problems if you use Dovecot director in both >> servers (install both director and backend to both servers). >> http://wiki2.dovecot.org/Director > > Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... You can safely do that with director. Also the problem with NFS isn't locks, but caching. > Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). From sam at robots.org.uk Mon Mar 5 20:45:36 2012 From: sam at robots.org.uk (Sam Morris) Date: Mon, 05 Mar 2012 18:45:36 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users Message-ID: <1330973136.70967.33.camel@leela.office.red-redemption.com> The attached patch makes it possible for Kerberos principals to be associated with a password database entry by adding a new "k5principals" passdb setting. A client that successfully authenticates using GSSAPI will be able to log in as any user who has been associated with the client's Kerberos principal. This means that users can now use their Kerberos identities to access virtual mail accounts. The patch definitely needs review by someone familiar with Dovecot. It works for me on a small test installation using the passwd-file backend. Things that should probably be improved: 1. The list of authorized principals is stored in struct auth_request. I would prefer to store it in struct gssapi_auth_request, but auth-request.c does not know about structs that are specific to the different authentication plugins. This could be fixed in a more general way by adding a new function to struct mech_module to allow authplugins to read fields during passdb lookups. 2. The gssapi authplugin now does a credential lookup in order to trigger parsing of the k5principals setting. In order for this to work, auth_request_set_username is now called before mech_gssapi_userok. AFAICT the only impact of this is that messages logged by this function (and the functions it calls) will now use the name of the virtual account. 3. The credentials lookup triggers an info log message saying that credentials for GSSAPI were requested, "but we have only (e.g.) MD5-CRYPT". The authplugin doesn't actually want the credential, but I think that the only way the authplugin can trigger a passdb lookup is by requesting it. 4. The final part of the code in mech_gssapi_unwrap was moved to the callback that's triggered when the credentials lookup is complete. The code still needs access to the GSSAPI data, so the buffer pointer & length are now stored in struct gssapi_auth_request, making the inbuf parameter to the mech_gssapi_{sec_context,wrap,unwrap} functions superfluous. The parameters should be removed. 5. The k5principals list won't be processed on Solaris. The code added to the end of mech_gssapi_krb5_userok would have to be moved to a separate function and then be called from the Solaris code. 6. GCC tells me about assignment to incompatible pointer types in the code that iterates through gssapi_k5principals. I must be missing something. The patch is licensed under the MIT license. Please let me know what you think. -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_1.patch Type: text/x-patch Size: 6953 bytes Desc: not available URL: From tss at iki.fi Mon Mar 5 20:52:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 20:52:19 +0200 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: <1330973136.70967.33.camel@leela.office.red-redemption.com> References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: On 5.3.2012, at 20.45, Sam Morris wrote: > 3. The credentials lookup triggers an info log message saying that > credentials for GSSAPI were requested, "but we have only (e.g.) > MD5-CRYPT". The authplugin doesn't actually want the credential, > but I think that the only way the authplugin can trigger a > passdb lookup is by requesting it. I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". From campbell at cnpapers.com Mon Mar 5 21:30:30 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Mon, 05 Mar 2012 14:30:30 -0500 Subject: [Dovecot] Shared mboxes Message-ID: <4F551456.102@cnpapers.com> I've been looking at some documentation on shared mail accounts. But I'm getting mixed thoughts on how this can or should be done. I use mbox for all my pop and imap folders since I've converted from a uw-imap server. The first thing that makes me wonder about setup is that I've been told to not use maildir and mbox on the same machine, although I'm not really sure why since it seems this would work OK, but anyway, I'm guessing I should stick with mbox for the shared accounts. Secondly, I'm sure I'd need a namespace to use which ever format, so there's private, public, and shared types. Most of the stuff I'm reading seems to suggest "public" as a type instead of "shared". So what's shared for anyway? I want to use this shared account so that email can be sent to this account, and be shared by only a few people, but I'm reading where locks and such don't work with mbox, so in my mind, how do you avoid corruption and why not just make a normal account and let people hack away at the data? I've not even got to the questions in my mind about how to set up the account, but figured if I could get the above straight, I might be able to fuddle my way through it. Help would be truly appreciated. steve campbell From tom at talpey.com Tue Mar 6 00:06:20 2012 From: tom at talpey.com (Tom Talpey) Date: Mon, 05 Mar 2012 17:06:20 -0500 Subject: [Dovecot] POP3C storage backend Message-ID: <4F5538DC.4060802@talpey.com> I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I don't see anything in the 2.1 doc directory or in the wiki. Can this be used to synchronize dovecot with external pop servers? Doing away with my current fetchmail and lmtp solution for this would be quite interesting. Thanks for any pointers to configuring and using this, if so... From stan at hardwarefreak.com Tue Mar 6 01:16:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Mon, 05 Mar 2012 17:16:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F551456.102@cnpapers.com> References: <4F551456.102@cnpapers.com> Message-ID: <4F55495B.10609@hardwarefreak.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. But I'm > getting mixed thoughts on how this can or should be done. > > I use mbox for all my pop and imap folders since I've converted from a > uw-imap server. The first thing that makes me wonder about setup is that > I've been told to not use maildir and mbox on the same machine, although > I'm not really sure why since it seems this would work OK, but anyway, > I'm guessing I should stick with mbox for the shared accounts. > > Secondly, I'm sure I'd need a namespace to use which ever format, so > there's private, public, and shared types. Most of the stuff I'm reading > seems to suggest "public" as a type instead of "shared". So what's > shared for anyway? > > I want to use this shared account so that email can be sent to this > account, and be shared by only a few people, but I'm reading where locks > and such don't work with mbox, so in my mind, how do you avoid > corruption and why not just make a normal account and let people hack > away at the data? > > I've not even got to the questions in my mind about how to set up the > account, but figured if I could get the above straight, I might be able > to fuddle my way through it. > > Help would be truly appreciated. Start here: http://wiki.dovecot.org/SharedMailboxes -- Stan From kgc at corp.sonic.net Tue Mar 6 03:33:32 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Mon, 5 Mar 2012 17:33:32 -0800 Subject: [Dovecot] Master Users Message-ID: <20120306013332.GE16881@corp.sonic.net> I have a setup where I need to use a Master User account to login on behalf of users normally authed via PAM. Is there any existing mechanism that will allow master users to be wired down to specific ip address rather than having these very magic user/pass combos be valid from any random host? It would be totally acceptable to be able to say that master logins were only valid from a specific list of hosts rather than wiring specific master users to specific hosts. -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtam.home at gmail.com Tue Mar 6 04:01:43 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 5 Mar 2012 18:01:43 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On Sun, 4 Mar 2012, Timo Sirainen writes: > > I would like to run various doveadm commands that involves all (mail) users like > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > but any doveadm command that uses "-A" to iterate through all users will > > stop processing at the first account with UID > What userdb are you using? userdb passwd should already skip users that > aren't in the valid range. And what Dovecot version are you using? passwd-file under dovecot 2.0.16. > And one more thing: Does it really even stop there? Looking at the code > it's supposed to log an error and continue to next user. Note that it says > "Failed to iterate through SOME users". The wording did not escape my notice, which is why I suspect it's not doing what it was designed to do. This is my test: # Command doveadm mailbox list -A # Start of password file sysdaemon:*:500:500:System daemon:/:/dev/null ... and the rest ... # dovecot.conf ... first_valid_uid = 10000 first_valid_gid = 10000 ... In this situation, doveadm will exit immediately with an UID error message. If I change the UID>10000, it will produce the analogous GID error message. If I satify both UID and GID constraints, it will fail on the next daemon entry. If I move the sysdaemon entry all the way to the bottom of the pasword file, I get user1 saved-messages user1 sent-mail user1 postponed-msgs user1 temp user1 temp/temp user1 INBOX user2 sent-mail user2 101 user2 345 user2 ckf ... all user's mailbox with UID>10000, then ... doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop privileges: Mail access for users with UID 500 not permitted (see first_valid_uid in config file, uid from userdb lookup). doveadm(sysdaemon): Error: User init failed doveadm: Error: Failed to iterate through some users > Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? Sorry for this error mismatch -- I cut&pasted the wrong test output; however, the problem I witnessed applies to both UID and GID (if either constraint is not met, user iteration terminates). > I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f It looks like I'll be upgrading. Oh, I just spotted this in the ChangeLog -- maybe you are undoing this? (2010-10-21) * src/auth/auth-settings.c, src/auth/auth-settings.h, src/auth/userdb- passwd.c: auth: userdb passwd iteration now lists only users within first_valid_uid..last_valid_uid range. [745ef289b0ea] Joseph Tam From apm at one.com Tue Mar 6 09:17:32 2012 From: apm at one.com (Peter Mogensen) Date: Tue, 06 Mar 2012 08:17:32 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> Message-ID: <4F55BA0C.5090606@one.com> On 2012-03-05 16:36, Timo Sirainen wrote: >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > Neither behavior is wrong, just different. :) Ok... I were in doubt if I had missed something from the RFC. However... for testing, I tried to create "INBOX.INBOX" on dovecot. But then dovecot answers NO and complains that the folder already exists. Though it's still not on disk and dovecot still doesn't list it with "*". /Peter From frank at moltke28.B.Shuttle.DE Tue Mar 6 09:37:45 2012 From: frank at moltke28.B.Shuttle.DE (Frank Elsner) Date: Tue, 6 Mar 2012 08:37:45 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve Message-ID: (auto-added) Hello all, I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 installed and working. But I've problems to get the avelsieve plugin for squirrelmail working with dovecot. The "Message Filters" show up in "Options" of squirrelmail, but "Could not log on to timsieved daemon on your IMAP server ........." dovecot log shows: Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured Where to look for configuration error(s)? | root at seymour: /usr/local/dovecot/src<156> dovecot -n | # 2.1.1: /usr/local/dovecot/etc/dovecot/dovecot.conf | # OS: Linux 2.6.35.14-106.fc14.i686.PAE i686 Fedora release 14 (Laughlin) ext3 | default_vsz_limit = 512 M | disable_plaintext_auth = no | first_valid_uid = 200 | last_valid_uid = 65534 | listen = * | lmtp_save_to_detail_mailbox = yes | login_greeting = c64.shuttle.de - IMAPs Service (dovecot) ready. | login_log_format_elements = %u %r %c | mail_location = maildir:/var/spool/mail/%u | mail_log_prefix = "%Us(%u,%r): " | mail_plugin_dir = /usr/dovecot/lib/dovecot/ | mail_plugins = " notify quota" | managesieve_notify_capability = mailto | managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave | passdb { | args = dovecot | driver = pam | } | plugin { | mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append | mail_log_fields = uid box from subject msgid size flags | mail_log_group_events = yes | quota = maildir:User quota | quota_rule = *:storage=2G | quota_rule2 = Trash:storage=+100M | sieve = ~/.sieve | sieve_dir = ~/sieve | } | postmaster_address = postmaster at moltke28.b.shuttle.de | protocols = imap sieve | service anvil { | client_limit = 1027 | } | service auth { | unix_listener auth-client { | group = exim | mode = 0660 | user = exim | } | } | service imap-login { | inet_listener imap { | port = 143 | } | inet_listener imaps { | port = 993 | ssl = yes | } | process_limit = 512 | process_min_avail = 10 | } | service imap-postlogin { | executable = script-login /usr/dovecot/bin/imap-post-login | } | service imap { | executable = imap imap-postlogin | } | service lmtp { | inet_listener lmtp { | address = 0.0.0.0 | port = 24 | } | } | service managesieve-login { | inet_listener sieve { | port = 4190 | } | } | service pop3-login { | inet_listener pop3 { | port = 110 | } | inet_listener pop3s { | port = 995 | ssl = yes | } | } | service pop3 { | process_limit = 1024 | } | ssl_cert = References: <1330677951.2081.49.camel@innu> <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> Message-ID: <4F55C9DB.2070809@mobilia.it> Il 05/03/2012 18.53, Timo Sirainen ha scritto: > On 5.3.2012, at 19.25, Jacek Osiecki wrote: > >>>> However, if we have everything redundant, why not have the same with SMTP >>>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? >>> If both servers randomly access users' mails, with NFS you'll have some >>> trouble, with OCFS2 probably less trouble. But in both cases you'll have >>> better performance and no problems if you use Dovecot director in both >>> servers (install both director and backend to both servers). >>> http://wiki2.dovecot.org/Director >> Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... > You can safely do that with director. > > Also the problem with NFS isn't locks, but caching. After reading a little bit, it seems that Director does the job of a decent load balancer, but in the middle instead of in front of your servers, I've limited problems with NFS by using "sticky" connections with long timeouts in my load balancer, unless they're disconnected for days, they'll always end up going through the same server for POP3/IMAP conections. Doesn't work great for the SMTP/LDA part though. > >> Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? > Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). > In the rare case it does happen, NFS locking and concurrent_connections set to one has seemed to reduce my problems to a minimum.. I like the Director idea though, since it's content aware it isn't organizing connections based on port/IP, but on the the actual users, especially if it does so with the LDA, it seems like an excellent solution to collisions (I guess they're called this) .. I wish it had been a reality when I was building my servers. From stephan at rename-it.nl Tue Mar 6 10:40:44 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 06 Mar 2012 09:40:44 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve In-Reply-To: (auto-added) References: (auto-added) Message-ID: <4F55CD8C.20108@rename-it.nl> On 3/6/2012 8:37 AM, Frank Elsner wrote: > Hello all, > > I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 > installed and working. But I've problems to get the avelsieve plugin for > squirrelmail working with dovecot. > > The "Message Filters" show up in "Options" of squirrelmail, but > "Could not log on to timsieved daemon on your IMAP server ........." > > dovecot log shows: > > Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured You should try to capture traffic between client and server with ngrep, e.g. sudo ngrep -d lo port 4190 However, I've noticed that avelsieve uses STARTTLS even on localhost, so if you want to see anything intelligible, you will have to turn that off temporarily. As far as I know, there is also a means to instruct managesieve-login to write its traffic somewhere (a login 'rawlog'), but I can't find where it is documented right now. > | protocol lmtp { > | mail_plugins = " notify quota quota" > | } > | protocol lda { > | mail_plugins = " notify quota quota" > | } > | protocol imap { > | imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags > | imap_logout_format = [%i/%o] > | mail_max_userip_connections = 0 > | mail_plugins = " notify quota mail_log quota imap_quota listescape" > | } Why do you have duplicate "quota" entries here? Also, "sieve" plugin is missing from lmtp and lda. Still, ManageSieve should accept connections with this config. Regards, Stephan. From dovecot at arvoreen.net Tue Mar 6 13:29:13 2012 From: dovecot at arvoreen.net (Pol Bettinger) Date: Tue, 06 Mar 2012 12:29:13 +0100 Subject: [Dovecot] LDAP auth_bind fails Message-ID: <4F55F509.4000507@arvoreen.net> Hello, I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. Dovecot version 2.1.1 (I started with 2.1.0 and hoped 2.1.1 would fix it) I tried to play around with the base, pass_attrs,pass_filter to no avail but didn't succeed. Looking at a wireshark trace i only saw 7 packets and it seemed to me dovecot did only an anonymous bind. any help would appreciated Sincerely Pol Bettinger output of mail.log: Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5#011service=imap#011secured#011lip=192.168.16.27#011rip=192.168.16.20#011lport=993#011rport=51838 Mar 6 12:16:34 Dell dovecot: auth: Debug: client out: CONT#0112#011PDQ1NjgyMjE3NjYyMDk3NjkuMTMzMTAzMjU5NEBEZWxsPg== Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: CONT Mar 6 12:16:34 Dell dovecot: auth: Debug: password(arvi at arvoreen.net,192.168.16.20): passdb doesn't support credential lookups Mar 6 12:16:36 Dell dovecot: auth: Debug: client out: FAIL#0112#011user=arvi at arvoreen.net output of dovecot -n: # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic i686 Ubuntu 11.10 ext4 auth_debug = yes auth_default_realm = arvoreen.net auth_mechanisms = plain digest-md5 cram-md5 auth_verbose = yes base_dir = /var/run/dovecot/ mail_location = maildir:/var/mail/%d/%1n/%n:INDEX=/var/indexes/%d/%1n/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Drafts { auto = create special_use = \Drafts } mailbox Junk { auto = create special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = create special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap_pass.conf.ext driver = ldap } plugin { sieve = /var/sieve/%d/%1n/%n sieve_dir = /var/sieve/%d/%1n/%n } protocols = imap lmtp sieve service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = From tss at iki.fi Tue Mar 6 13:43:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 6 Mar 2012 13:43:26 +0200 Subject: [Dovecot] LDAP auth_bind fails In-Reply-To: <4F55F509.4000507@arvoreen.net> References: <4F55F509.4000507@arvoreen.net> Message-ID: On 6.3.2012, at 13.29, Pol Bettinger wrote: > I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. .. > Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5 CRAM-MD5 can't work with auth_bind. http://wiki2.dovecot.org/Authentication/Mechanisms#Non-plaintext_authentication From jernej.porenta at arnes.si Tue Mar 6 15:28:50 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 6 Mar 2012 14:28:50 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) Message-ID: Heya, We are expiriencing issues with dovecot 2.1.1 on Linux with weird filenames in home directory of username. We are using mbox IMAP folders, with no special changes (mail_location = mbox:~/:INBOX=%h/.mailbox). Mar 6 13:37:17 machine dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 6 13:37:17 machine dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb79450] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb794a6] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb78963] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c87ebd5] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c88c12c] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_next+0x1b4) [0x2ba41c88c494] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c885342] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_next+0x234) [0x2ba41c885604] -> dovecot/imap [0x40b2d1] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410427] -> dovecot/imap [0x40f4cd] -> dovecot/imap [0x40f582] -> dovecot/imap(client_handle_input+0x3f) [0x40f6cf] -> dovecot/imap(client_input+0x62) [0x410052] -> /opt/dovecot Mar 6 13:37:17 machine dovecot: imap(username): Fatal: master: service(imap): child 20873 killed with signal 6 (core dumps disabled) The bug is reproducible by using home folder structure available from: http://bit.ly/x8pTXS AFAIK, the problem lies in processing the file list of home folder, which can contain filenames that do not have proper UTF-8 encoding of filenames, which causes dovecot to crash. On the other hand, UTF-8 filenames created on the system by hand (using touch), are not displayed in IMAP LIST command (sample is included in the folder structure; single letter file). Cheers, Jernej From campbell at cnpapers.com Tue Mar 6 16:28:55 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 09:28:55 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F55495B.10609@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> Message-ID: <4F561F27.5000102@cnpapers.com> On 3/5/2012 6:16 PM, Stan Hoeppner wrote: > On 3/5/2012 1:30 PM, Steve Campbell wrote: >> I've been looking at some documentation on shared mail accounts. But I'm >> getting mixed thoughts on how this can or should be done. >> >> I use mbox for all my pop and imap folders since I've converted from a >> uw-imap server. The first thing that makes me wonder about setup is that >> I've been told to not use maildir and mbox on the same machine, although >> I'm not really sure why since it seems this would work OK, but anyway, >> I'm guessing I should stick with mbox for the shared accounts. >> >> Secondly, I'm sure I'd need a namespace to use which ever format, so >> there's private, public, and shared types. Most of the stuff I'm reading >> seems to suggest "public" as a type instead of "shared". So what's >> shared for anyway? >> >> I want to use this shared account so that email can be sent to this >> account, and be shared by only a few people, but I'm reading where locks >> and such don't work with mbox, so in my mind, how do you avoid >> corruption and why not just make a normal account and let people hack >> away at the data? >> >> I've not even got to the questions in my mind about how to set up the >> account, but figured if I could get the above straight, I might be able >> to fuddle my way through it. >> >> Help would be truly appreciated. > Start here: > http://wiki.dovecot.org/SharedMailboxes That's where most of my questions originated, but thanks for the reply. (Sorry for the first response - I sent it to the poster, not the list). Maybe I'm misunderstanding concepts here and I'm trying to use something I don't need to use. I'm really new to dovecot, and as I learn all the ins and outs, I'm finding a lot of this doesn't seem to be "turning on any light bulbs" until after I've played with it a while. What I've done in the past with the old imap server is to create an account (unix account), so the smtp server puts the mbox (what is referred to as the INbox) in /var/spool/mail. Users who needed to "share" this mailbox would be give the account user name and the password for this account and would add an Imap account to their mail client. This would sometimes cause locking problems or client corruption due to email removals mostly. This is basically a normal, non-shared account. Now that I've moved to dovecot on a new, updated server, I'd like to use the facilities of dovecot for the truly shared accounts. I'm not sure if I need to create the account like before, but seems like I'd have to in order to get the smtp server to deliver new email to /var/spool/mail/%u. As I see it, I've got to create a namespace for shared accounts and configure this on the multiple-user's clients so that when they access the Inbox and imap files under /home/%u/mail, they don't butt heads, so they're some locking involved. I could use acls for this, but don't have to according to the documentation. I can grant permissions to each user that is included in the acl, and I can create dovecot "groups" to use as a basis for this permission. I'm hoping this is pretty much the way it's done, and I want to keep with mbox format for all files and folders. I'm also hoping that this is the way it's supposed to be used, but I get conflicting ideas about what the documentation is really telling me. Anyway, I'll play with this and see where I get. I've still not found out where to create these dovecot "groups" other than it seems to use a userdb file somewhere. Thanks for the help so far steve > From khoroshyy at gmail.com Tue Mar 6 18:58:55 2012 From: khoroshyy at gmail.com (Khoroshyy Petro) Date: Tue, 6 Mar 2012 17:58:55 +0100 Subject: [Dovecot] Dovecot saves mails in "wrong" folder. Message-ID: Hi all I have installed dovecot 1.2.15 and try to use it together with offlineimap and gnus. my problem is that it saves emails into /var/mail/petro instead of ~/Maildir Thanks. Petro. This is my .dovecot.conf default_mail_env = maildir:%h/Maildir And this is my .offlineimaprc [general] accounts = Gmail maxsyncaccounts = 1 [Account Gmail] localrepository = Local remoterepository = Remote [Repository Local] type = IMAP remotehost = localhost port = 143 remoteuser = petro [Repository Remote] type = IMAP remotehost = imap.gmail.com remoteuser = myname at gmail.com ssl = yes maxconnections = 1 realdelete = no folderfilter = lambda foldername: foldername in ['INBOX'] -- From kgc at corp.sonic.net Tue Mar 6 19:33:08 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Tue, 06 Mar 2012 09:33:08 -0800 Subject: [Dovecot] Master Users In-Reply-To: <20120306013332.GE16881@corp.sonic.net> References: <20120306013332.GE16881@corp.sonic.net> Message-ID: <4F564A54.9050400@corp.sonic.net> On 03/05/12 17:33, Kelsey Cummings wrote: > I have a setup where I need to use a Master User account to login on > behalf of users normally authed via PAM. Is there any existing mechanism > that will allow master users to be wired down to specific ip address rather Ah, found it. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/AllowNets -K From sam at robots.org.uk Tue Mar 6 20:12:01 2012 From: sam at robots.org.uk (Sam Morris) Date: Tue, 06 Mar 2012 18:12:01 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: <1331057521.84875.2.camel@leela.office.red-redemption.com> On Mon, 2012-03-05 at 20:52 +0200, Timo Sirainen wrote: > On 5.3.2012, at 20.45, Sam Morris wrote: > > > 3. The credentials lookup triggers an info log message saying that > > credentials for GSSAPI were requested, "but we have only (e.g.) > > MD5-CRYPT". The authplugin doesn't actually want the credential, > > but I think that the only way the authplugin can trigger a > > passdb lookup is by requesting it. > > I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". Thanks for pointing that out. Here's a newer version of the patch with that change. I also realised that the gss_buffer is not required in the code that runs once the passdb lookup is complete, so I removed the code that stashes it in struct gssapi_auth_request. Regards, -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_2.patch Type: text/x-patch Size: 6020 bytes Desc: not available URL: From stan at hardwarefreak.com Tue Mar 6 22:17:22 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 06 Mar 2012 14:17:22 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F561F27.5000102@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> Message-ID: <4F5670D2.9090004@hardwarefreak.com> On 3/6/2012 8:28 AM, Steve Campbell wrote: >> http://wiki.dovecot.org/SharedMailboxes > That's where most of my questions originated, but thanks for the reply. Steve, all the information you need is behind that link. > Maybe I'm misunderstanding concepts here Very possibly. > What I've done in the past with the old imap server is to create an > account (unix account), so the smtp server puts the mbox (what is > referred to as the INbox) in /var/spool/mail. Users who needed to > "share" this mailbox would be give the account user name and the > password for this account and would add an Imap account to their mail > client. This would sometimes cause locking problems or client corruption > due to email removals mostly. This is basically a normal, non-shared > account. Locking problems with multiple users hitting mbox files is unavoidable. The same is true when a single user hits an mbox from multiple client devices simultaneously--PC, smart phone, tablet, etc. Which is why you do not want to use mbox file format for shared mailboxes, but maildir instead, because each email is a separate file. Please note, from the link I provided: ********************************************************************** Maildir: Per-user \Seen flag With Maildir a dovecot-shared file controls if the \Seen flags are shared or private. The file must be created separately inside each Maildir, although if the file already exists in the Maildir root it's automatically copied for newly created mailboxes. If dovecot-shared file doesn't exist in Maildir, the \Seen flags are shared. If it exists, the \Seen flag state is stored only in the user's index files. By making each user have their own private index files, you can make the \Seen flag private for the users. ********************************************************************** Simple concept above: each user of the shared mailbox sees "new" mail. One user accessing new mail and marking it as read doesn't mark that message as read for other shared users. You can not do this with mbox file format, only maildir. ********************************************************************** Maildir: Keyword sharing Make sure you don't try to use per-user CONTROL directory. Otherwise dovecot-keywords file doesn't get shared and keyword mapping breaks. Other mailbox formats Currently you can't have any per-user flags with other mailbox formats than Maildir. ********************************************************************** -- Stan From campbell at cnpapers.com Tue Mar 6 23:01:08 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 16:01:08 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F5670D2.9090004@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> Message-ID: <4F567B14.3030908@cnpapers.com> On 3/6/2012 3:17 PM, Stan Hoeppner wrote: > On 3/6/2012 8:28 AM, Steve Campbell wrote: > >>> http://wiki.dovecot.org/SharedMailboxes >> That's where most of my questions originated, but thanks for the reply. > Steve, all the information you need is behind that link. I've gone over that set of links on that page a dozen times. Perhaps I'm trying to put a square peg in a round hole by using mbox, but they keep providing information on it, so I guess I was just pounding away. But then there's that "don't use maildir and mbox together". All of the accounts on this server are carry-overs from the UW-IMAP server, so perhaps I should have converted those to maildir. Seems as though it's OK when they don't apply to the same type namespace. > >> Maybe I'm misunderstanding concepts here > Very possibly. > >> What I've done in the past with the old imap server is to create an >> account (unix account), so the smtp server puts the mbox (what is >> referred to as the INbox) in /var/spool/mail. Users who needed to >> "share" this mailbox would be give the account user name and the >> password for this account and would add an Imap account to their mail >> client. This would sometimes cause locking problems or client corruption >> due to email removals mostly. This is basically a normal, non-shared >> account. > Locking problems with multiple users hitting mbox files is unavoidable. > The same is true when a single user hits an mbox from multiple client > devices simultaneously--PC, smart phone, tablet, etc. Which is why you > do not want to use mbox file format for shared mailboxes, but maildir > instead, because each email is a separate file. Please note, from the > link I provided: I've experienced that type of locked mailbox before on the old server. Users insist on accessing their email account as a pop account on their desktop with the "check for new mail every so many minutes" turned on and still keep their smartphones on while accessing it as an imap account so they can still download the files to their desktop when they return. > > ********************************************************************** > Maildir: Per-user \Seen flag > > With Maildir a dovecot-shared file controls if the \Seen flags are > shared or private. The file must be created separately inside each > Maildir, although if the file already exists in the Maildir root it's > automatically copied for newly created mailboxes. If dovecot-shared file > doesn't exist in Maildir, the \Seen flags are shared. If it exists, the > \Seen flag state is stored only in the user's index files. By making > each user have their own private index files, you can make the \Seen > flag private for the users. > ********************************************************************** > > > Simple concept above: each user of the shared mailbox sees "new" mail. > One user accessing new mail and marking it as read doesn't mark that > message as read for other shared users. You can not do this with mbox > file format, only maildir. > > > ********************************************************************** > Maildir: Keyword sharing > > Make sure you don't try to use per-user CONTROL directory. Otherwise > dovecot-keywords file doesn't get shared and keyword mapping breaks. > > Other mailbox formats > > Currently you can't have any per-user flags with other mailbox formats > than Maildir. > ********************************************************************** So just to clarify, is it OK to have a maildir account setup on this server for these shared/imap access only accounts along with the mbox accounts already on there? Thanks for the patience and help steve From sdavies at sdc.com.au Wed Mar 7 01:00:50 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:30:50 +1030 Subject: [Dovecot] Log sync errors (again) Message-ID: <201203070930.50847.sdavies@sdc.com.au> As suggested earlier, I deleted all .imap directories and the log sync errors stopped - for a while. They have now returned. It seems to happen for every mailbox that gets accessed. Dovecot version 2.1.1 with pidgeonhole 3.0.0 on Mandriva Linux. Could this interfere with sieve filters? Several users have filters but none of them seem to do anything. Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=38708 for /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index: Extension header update points outside header size Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=41576 for /home/john/Mail/INBOX/.imap/Zerna/dovecot.index: Extension header update points outside header size Cheers and thanks, Stephen -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From sdavies at sdc.com.au Wed Mar 7 01:07:06 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:37:06 +1030 Subject: [Dovecot] Fscking warnings Message-ID: <201203070937.06545.sdavies@sdc.com.au> Google tells me that these "should go away" but they don't. Seems to happen continuously while a user is viewing email. Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Archive/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Davies/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/FieldNET/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Invoices Out/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Lawrence and Hanson/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Call/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Reset/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/River Murray/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Soil Moisture Alert/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Management Alarm/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Usage/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Zerna/dovecot.index -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From jk at jkart.de Wed Mar 7 01:19:10 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:19:10 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? Message-ID: <4F569B6E.1080905@jkart.de> Hello, you knows, that http://xi.rename-it.nl is down? -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Die Oper ist eine h?bsche Unterhaltung, die noch besser w?re, wenn nicht dabei gesungen w?rde. (Claude Debussy) From stephan at rename-it.nl Wed Mar 7 01:33:09 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 07 Mar 2012 00:33:09 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569B6E.1080905@jkart.de> References: <4F569B6E.1080905@jkart.de> Message-ID: <4F569EB5.7030204@rename-it.nl> On 3/7/2012 12:19 AM, Jim Knuth wrote: > Hello, > > you knows, that http://xi.rename-it.nl is down? > Yep, and back. Regards, Stephan. From jk at jkart.de Wed Mar 7 01:36:42 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:36:42 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569EB5.7030204@rename-it.nl> References: <4F569B6E.1080905@jkart.de> <4F569EB5.7030204@rename-it.nl> Message-ID: <4F569F8A.1090200@jkart.de> am 07.03.12 00:33 schrieb Stephan Bosch : > On 3/7/2012 12:19 AM, Jim Knuth wrote: >> Hello, >> >> you knows, that http://xi.rename-it.nl is down? >> > > Yep, and back. > > Regards, > > Stephan. wow. Thank you -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Dilettanten erkennt man an der Plumpheit ihrer Komplimente. Der routinierte Verf?hrer riskiert Kritik. (Cath?rine Deneuve) From 24x7server at 24x7server.net Wed Mar 7 04:22:23 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Wed, 7 Mar 2012 07:52:23 +0530 (Asi) Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file Message-ID: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> hi i am using qmailtoaster with dovecot version 2 mailbox format is maildir i have a domain with around 5000 users which are distributed over 2 servers webmail (squirrelmail) runs using dovecot v2 is being used from server number one server number 2 had all the data stored in it and also has pop and smtp running from it. i am not using dovecot for pop as yet on the server with dovecot i get such errors in the log file access to data on server number 2 is via nfs on server number 1 i get errors as such Error: fcntl(read-lock) locking failed for file Input/output error squirrelmail gives error imap connection closed and i am not able to login so i set the parameters as such in the dovecot conf file and the error stopped mmap_disable=yes dotlock_use_excl = yes lock_method = dotlock can somebody please advise me if the above is correct ? or is it preferred to use fcntl with lockd (note that my mailbox is maildir format) thanks very much for your help rajesh From jd.beaubien at gmail.com Wed Mar 7 05:19:05 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Tue, 6 Mar 2012 22:19:05 -0500 Subject: [Dovecot] mdbox + gzip and rsync Message-ID: Hi, After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip): "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation." How will the mailbox files (m.X) files be modified when I move or delete emails using mdbox+gzip. Will the resulting gzipped mdbox files be rsync-able or will they need a full re-upload? If I plan on using rsync for backups, am I better off not using the gzip feature (if i can spare the extra storage)??? Thanks, -JD From CMarcus at Media-Brokers.com Wed Mar 7 15:32:32 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 08:32:32 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? Message-ID: <4F576370.8040706@Media-Brokers.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. > But I'm getting mixed thoughts on how this can or should be done. This brings up a question I have been meaning to ask. One thing I want to do on my new converted system is to implement shared mail. There will be two different scenarios - users sharing 'folders', which looks to be fairly simple using virtual ACL files - but for the other scenario, I'm not sure about a specific requirement we will have... I want to give multiple people shared access to some actual accounts with all of the special use folders, with the following requirements: 1. They can all read/reply to new messages as they come in, 2. They use shared \seen, \replied and \forwarded flags, so once someone else has read/dealt with a message, the others see that, 3. When they reply to/forward a message, the Sent message gets saved to that accounts 'Sent' folder, 4. They can *move* messages to other folders in that account (ie, 'file' them), and last (this is the tricky part) 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. These emails deal with financial transactions (AP and AR issues) and Faxes, thus the requirement to not be able to delete them. Can this be accomplished with the current state of things? Or would this require some coding? If the latter, could it be done as a plug-in, or would it require changes to the core code? Thanks, -- Best regards, Charles From lists at wildgooses.com Wed Mar 7 18:39:13 2012 From: lists at wildgooses.com (Ed W) Date: Wed, 07 Mar 2012 16:39:13 +0000 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F578F31.3000303@wildgooses.com> > I want to give multiple people shared access to some actual accounts > with all of the special use folders, with the following requirements: I have done this (unsatisfactorarily) by making it a normal mail account with normal login credentials. Add it like any other mail account. It then satisfies all your requirements, although: behind a nat, on thunderbird and with condstore, I sometimes see read/unread get out of sync... Believed to be a thunderbird bug, but unsure. Easy to resync > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) Interested to hear proper answers... Ed W From wgillespie at es2eng.com Wed Mar 7 20:04:44 2012 From: wgillespie at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 11:04:44 -0700 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F57A33C.3050808@es2eng.com> On 3/7/2012 6:32 AM, Charles Marcus wrote: > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. If you are using ACLs, just don't give them the delete permission? But I guess now that I am thinking about it as I write, you did want them to be able to move the messages (which is really a copy + delete). So... maybe not. From e-frog at gmx.de Wed Mar 7 20:17:36 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 07 Mar 2012 19:17:36 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command Message-ID: <4F57A640.1030202@gmx.de> Hello Timo, There seems to be something broken in v2.1 latest hg version: # 2.1.1 (94de7605f50f) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL * OK Namespace completed. Please note that the "OK Namespace completed." is send untagged. It worked on below version: # 2.1.1 (315f0d8cc2b2) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL 1 OK Namespace completed. Thanks, e-frog From CMarcus at Media-Brokers.com Wed Mar 7 21:03:30 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 14:03:30 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F57A33C.3050808@es2eng.com> References: <4F576370.8040706@Media-Brokers.com> <4F57A33C.3050808@es2eng.com> Message-ID: <4F57B102.80400@Media-Brokers.com> On 2012-03-07 1:04 PM, Willie Gillespie wrote: > On 3/7/2012 6:32 AM, Charles Marcus wrote: >> 5. No one other than a designated user or users (Master User(s)? Users >> in a specified Group?) can delete any messages in this account, in any >> of the folders. > If you are using ACLs, just don't give them the delete permission? But I > guess now that I am thinking about it as I write, you did want them to > be able to move the messages (which is really a copy + delete). > > So... maybe not. Right... although my understanding is that dovecot does indeed use mv (at least on linux) to do moves when using maildir, so maybe there is a way... I'll wait and see what Timo says about this... no hurry, as I'm still in the design stage, this is just how I'd *like* it to work, but if it won't/can't, I'll figure something else out. Thanks for the replies so far... -- Best regards, Charles From micah at riseup.net Wed Mar 7 21:43:49 2012 From: micah at riseup.net (Micah Anderson) Date: Wed, 07 Mar 2012 14:43:49 -0500 Subject: [Dovecot] dot named folders Message-ID: <87aa3s2o3u.fsf@algae.riseup.net> When a user makes a folder called 'x.y' it actually creates a folder called 'x' with a folder called 'y' inside, rather than a folder called 'x.y'. I'm guessing this has to do with an internal folder separator namespace configuration, but I'm a bit confused by how this works. I'm using 2.0.15 with mdbox and this is what I have configured for my namespaces: namespace { separator = . prefix = inbox = yes } namespace { separator = . prefix = INBOX. inbox = no hidden = yes list = no } I migrated from courier maildirs, so perhaps I no longer need some of these now that the conversion is finished? thanks for any suggestions, I've got my head mixed up on this issue, micah -- From wgillespie+dovecot at es2eng.com Wed Mar 7 22:41:25 2012 From: wgillespie+dovecot at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 13:41:25 -0700 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <4F57C7F5.4030803@es2eng.com> On 03/07/2012 12:43 PM, Micah Anderson wrote: > > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. Correct. Similar to how in Linux, I could create a folder mkdir test1/test2 It will create test2 inside of test1. The difference being that IMAP doesn't necessarily need the parent mailbox to exist, where Linux would throw an error if test1/ didn't exist first. So basically, as far as I know, you can't have a folder with a "." in the name with the namespaces you have set up. From stan at hardwarefreak.com Wed Mar 7 22:47:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 14:47:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F567B14.3030908@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> Message-ID: <4F57C96F.7090602@hardwarefreak.com> On 3/6/2012 3:01 PM, Steve Campbell wrote: > I've experienced that type of locked mailbox before on the old server. > Users insist on accessing their email account as a pop account on their > desktop with the "check for new mail every so many minutes" turned on > and still keep their smartphones on while accessing it as an imap > account so they can still download the files to their desktop when they > return. Using IMAP on the phone and POP on the PC doesn't make any sense. Is there a (valid) reason why these people insist on this phone/IMAP and PC/POP setup? This seems seriously counter intuitive/productive. > So just to clarify, is it OK to have a maildir account setup on this > server for these shared/imap access only accounts along with the mbox > accounts already on there? Yes. With Dovecot it is possible to specify mail_location on a per user basis: http://wiki.dovecot.org/MailLocation You can even do a split mailbox type setup per user using multiple namespaces, for example specifying that INBOX use mbox with all other mail being stored in maildir format: http://wiki.dovecot.org/Namespaces > Thanks for the patience and help Sure thing. -- Stan From stan at hardwarefreak.com Wed Mar 7 23:03:35 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 15:03:35 -0600 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: <4F57CD27.3000207@hardwarefreak.com> On 3/6/2012 5:07 PM, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. Is this thread what "Google tells you"? http://dovecot.org/list/dovecot/2010-October/053909.html Timo is the creator of Dovecot, if you didn't know. So you can take his words for gospel. Also note his last statement in that thread: "The next time you could do it with dsync to avoid these kind of problems." It would seem you omitted a very important detail from your problem report, which is that you recently performed a migration. Please don't omit such critical details in future requests for help. Provide as much relevant detail as possible. This speeds the process up for everyone, and avoids guesswork on our part. -- Stan From M.Roos at roosit.eu Thu Mar 8 01:26:55 2012 From: M.Roos at roosit.eu (Marc) Date: Thu, 8 Mar 2012 00:26:55 +0100 Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap Message-ID: Anybody also experiencing that imap processes are kept running/open by mac osx mail.app, so eventually users are getting to the mail_max_userip_connections limit? Outlook / other clients seem to run fine. Thanks, Marc From jd.beaubien at gmail.com Thu Mar 8 03:30:26 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Wed, 7 Mar 2012 20:30:26 -0500 Subject: [Dovecot] Single instance storage Message-ID: I have read most of the doc on the dovecot website, and couldn't find any info on the single instance storage feature, so I'm posting my questions here. - Are these 3 parameters the only one necessary for single instance storage? I cannot find any doc on this feature on the website; is there anything specific I need to know about them? (the last one isn't exactly self-explanatory). - mail_attachment_dir = /srv/vmail/attachments - mail_attachment_hash = %{sha256} - mail_cache_min_mail_count = 2 - Is this feature ready for production? Thanks, -JD From schut at sarvision.nl Thu Mar 8 11:56:35 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 10:56:35 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. Message-ID: Hi, I'm currently migrating our old (colocated) mail server (running a [terribly outdated, I know] dovecot 1.1.11) to a new VPS (virtual private server). The old server was running gentoo linux (which is mainly the culprit of the old dovecot version: gentoo was too much trouble to keep updating); the new server will run debian (stable: 6). Debian currently has dovecot 1.2.15 in its repositories; not that much newer... I read in the docs about the auto-generated-from-hg debian dovecot packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what version to use... OK, 2.2 is development, which leaves the choice to: 1.2.15; 2.0.x, or 2.1.x. I would appreciate any consideration or thoughts on what version to choose. On a related note, there is the possibility to switch from maildir to dbox. I did not really find much pros or cons, except from performance and standards-compliance (ability to use e.g. mutt on the server itself). Any thoughts? About the server: we're just a small company. Think about 15 accounts, normal mail traffic, sometimes relatively large attachments (20mb+). Some accounts have many folders; some accounts are very large (5Gb+). Storage is on ext3, raid10. Performance has never been an issue; reliability and ease of maintenance is more important. Thanks, Vincent Schut. From trashcan at odo.in-berlin.de Thu Mar 8 11:59:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 10:59:37 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> Message-ID: HI -- On 05.03.2012 10:56, Timo Sirainen wrote: > On 4.3.2012, at 13.54, Timo Sirainen wrote: >> On 4.3.2012, at 13.41, Michael Grimm wrote: >>>> By "undeletable" do you mean you have mails that always come back >>>> after expunging them? >>> >>> Yes. Deleting by the client will return them after the next dsync >>> run. > > Luckily this just started happening to me as well. After some > debugging I found and fixed the problem: > > http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 I can confirm, that you fixed that issue successfully. Thanks and regards, Michael From trashcan at odo.in-berlin.de Thu Mar 8 12:26:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 11:26:56 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Hi -- On 04.03.2012 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. > Everything isn't finished yet, but it appears to work and I've > enabled > it for my @dovecot.fi mails. I did give it a try starting some days ago, and I can confirm that you are right, dsync replication can be used, but there are some issues, see below. Let me start with replicator's configuration ... > Below is a configuration for virtual user setup. [...] > service doveadm { > # if you're using a single virtual user, set this to > # start ssh as vmail (not root) > user = vmail > } ... that led to the following complaints at start-up: | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up | dovecot: doveadm: Error: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied | [...] | (repeatedly, presumably for the number of users in userdb?) Therefore, I modified dsync_remote_cmd ... > dsync_remote_cmd = ssh -p 1234 -l vmail %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} ... and used an empty 'service doveadm { }' instead. That worked, but I would love to run doveadm as vmail user (security), though. How should I do that without running into the error messages above? Now some observations regarding replicator: 1) I see a lot of error messages whenever replicator is in action like (although everything is being synced correctly): | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: save: box=INBOX, uid=27, msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld (admin), flags=() | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: flag_change: box=TEST, uid=27568, msgid=<20120307144810.6360A74F013 at example.tld>, size=435, from=test at example.tld, flags=(\Seen) JFTR: I do have mail_log plugin activated. Some testing results: 1) I ran a test by sending locally produced mails every other minute on both servers simultaneously. That test ran for ~5 hours. All mails became synced correctly, and no losses were observable, but some duplicates. 2) I did send 100 small test mails from a distant server to my mailservers (mx1 and mx2): a) replicator and dsync deactivated: received 100 distinct mails (57 at mx1, 43 at mx2). b) now, replicator active: 172 mails (100 distinct, a lot of duplicates (up to 8 incarnations of the very same mail). Ok, 2b) is a rather 'mailbomb-like' scenario, but it worries me a bit: One of my users is receiving mails from a mailing list that sends individual mails batch-wise ... 3) replicator active: 1000 mails sent ended in 4523 mails at every server. Well, that was a mailbomb :-) 4) replicator active: 100 (and even 1000) locally produced mails at one server only: all 100 (and 1000 mails) became synced, prefectly well, without duplicates. 5) replicator active: 100 locally produced mails at both servers simultaneously: 341 mails, thus a lot of multiple incarnations. (This test differed from 1) because all mails were sent in one batch.) Final note to these tests: It doesn't matter whether sieve with redirecting, or sieve with redirecting and copying, or no sieve at all has been involved. It seems to me, that whenever a larger number of mails arrive on both servers simultaneously, the replicator gets into trouble [1]. I am unsure if one can expect that a replicator should deal with such stress, though. Or? R?sum?: The overall performance of replicator is very good from my point of view for my conditions (handful users, average workload of roughly 1000 mails a day). Thank you for replicator and regards, Michael [1] JFTR: I did similar tests in the past with dsync running from cron every other minute with similar results. From tss at iki.fi Thu Mar 8 13:35:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 08 Mar 2012 13:35:34 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> References: <4F53479E.40703@iki.fi> <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Message-ID: <1331206534.2081.101.camel@innu> On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > Let me start with replicator's configuration ... > > > Below is a configuration for virtual user setup. > [...] > > service doveadm { > > # if you're using a single virtual user, set this to > > # start ssh as vmail (not root) > > user = vmail > > } > > ... that led to the following complaints at start-up: > > | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up > | dovecot: doveadm: Error: Error reading configuration: > net_connect_unix(/var/run/dovecot/config) failed: Permission denied > | [...] > | (repeatedly, presumably for the number of users in userdb?) You can do for example: service config { unix_listener config { user = vmail } } > Now some observations regarding replicator: > > 1) I see a lot of error messages whenever replicator is in action > like (although everything is being synced correctly): > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: save: box=INBOX, uid=27, > msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld > (admin), flags=() > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: flag_change: box=TEST, uid=27568, > msgid=<20120307144810.6360A74F013 at example.tld>, size=435, > from=test at example.tld, flags=(\Seen) > > JFTR: I do have mail_log plugin activated. Hmm. Right. I guess all the logging should go to the log files instead of via the ssh pipe. Of course that would also require that dsync has write access to your log files. > It seems to me, that whenever a larger number of mails arrive on both > servers simultaneously, > the replicator gets into trouble [1]. I am unsure if one can expect > that a replicator should > deal with such stress, though. Or? Were these mails delivered via LMTP or dovecot-lda? The locks should prevent duplicates I think, so there's something still going wrong. From Leo.Baltus at omroep.nl Thu Mar 8 13:56:41 2012 From: Leo.Baltus at omroep.nl (Leo Baltus) Date: Thu, 8 Mar 2012 12:56:41 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <4F459344.5020407@rename-it.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> Message-ID: <20120308115641.GB5700@omroep.nl> Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: > On 2/22/2012 12:15 AM, Adam Szpakowski wrote: > >On 22.02.2012 00:09, Timo Sirainen wrote: > >>Well, it would be possible to build a doveadm script that > >>deletes the duplicates after delivery, but currently there's no > >>implementation to avoid delivering duplicate Message-IDs in the > >>first place. > >> > >>I don't really like such a Message-ID-based deduplication > >>feature enabled by default, but something like this could be > >>nice: > >> > >>fileinto :copy :x-deduplicate "boss"; > >> > >>Anyway, probably not going to be implemented anytime soon. > >Maybe there is a way to use a procmail with something like this: > > > >:0 Wh: msgid.lock > >| formail -D 8192 .msgid.cache > > > >But is there a safe way to use it together with sieve? Using > >Pigeonhole Sieve Pipe Plugin? > > > > There are a few options: > > * You can use Procmail as primary delivery agent and invoke > dovecot-lda/sieve from within Procmail once Procmail has determined > that it is not a duplicate. > > * Invoke procmail from Sieve using the pipe extension (i.e. the > other way around). This has the disadvantage that Procmail will > have to take care of final delivery, meaning the Dovecot indexes are > not updated. > > * For Pigeonhole v0.3 there is the possibility to "filter" the > message through Procmail using the sieve_extprograms plugin, but I > haven't actually tested something like that. > > * I've just created an alternative that implements something similar > to the Procmail code you posted above, but from within Sieve itself. > It is a custom language extension called vnd.dovecot.duplicate and > it adds the "duplicate" test. This test keeps track of which > Message-IDs it has seen before in earlier deliveries and yields a > true result if the message was seen before, e.g.: > > require "vnd.dovecot.duplicate"; > > if duplicate { > discard; > } > > Read the specification for details ("name" argument is not yet implemented): > > http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate/raw-file/4b1dbda4d3fc/doc/rfc/spec-bosch-sieve-duplicate.txt > > The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate > > This plugin is only a few hours old, experimental, and largely > untested, so test it thoroughly before considering to use this. Read > the INSTALL file for compile and installation instructions. > > Comments are welcome. > I did some very basic testing and it seems to work fine. The example in spec-bosch-sieve-duplicate.txt however says: if duplicate { fileinto :create "Trash/Duplicate"; } This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' So this leads to: failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name I am not sure if this a bug or not, I suppose you know the rfc's better than I do, is the sieve language supposed to be agnostic of the internals of the storage-engine (dovecot)? -- Leo Baltus, internetbeheerder /\ NPO ICT Internet Services /NPO/\ Sumatralaan 45, 1217 GP Hilversum, Filmcentrum, west \ /\/ beheer at omroep.nl, 035-6773555 \/ From CMarcus at Media-Brokers.com Thu Mar 8 14:03:05 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 07:03:05 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F589FF9.7080608@Media-Brokers.com> On 2012-03-08 4:56 AM, Vincent Schut wrote: > The old server was running gentoo linux (which is mainly the culprit of > the old dovecot version: gentoo was too much trouble to keep updating); Please stop with the FUD... I've been running gentoo for 8+ years, and it is a *breeze* to keep updated, *especially* long term (since it is a 'rolling release' type of distro)... Yes, it actually does require some minimum amount of attention from the admin, like, say, once per week or once per month updates - buy so should *any* system... and yes, it does require a little more willingness to learn and 'get your hands dirty' (especially for the installation), but it is well worth it. Oh - and Portage rocks... :) -- Best regards, Charles From stephan at rename-it.nl Thu Mar 8 14:05:37 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 08 Mar 2012 13:05:37 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <20120308115641.GB5700@omroep.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> <20120308115641.GB5700@omroep.nl> Message-ID: <4F58A091.7090704@rename-it.nl> On 3/8/2012 12:56 PM, Leo Baltus wrote: > Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: >> The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate >> >> This plugin is only a few hours old, experimental, and largely >> untested, so test it thoroughly before considering to use this. Read >> the INSTALL file for compile and installation instructions. >> >> Comments are welcome. > I did some very basic testing and it seems to work fine. > > The example in spec-bosch-sieve-duplicate.txt however says: > > if duplicate { > fileinto :create "Trash/Duplicate"; > } > > This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' > > So this leads to: > failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name > > I am not sure if this a bug or not, I suppose you know the rfc's better > than I do, is the sieve language supposed to be agnostic of the > internals of the storage-engine (dovecot)? For Sieve, the mailbox name is pretty much opaque. Usually, it matches what is used through IMAP. http://tools.ietf.org/html/rfc5228#section-4.1 So, in your case, just use "Trash.Duplicate" instead. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 8 14:19:28 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 13:19:28 +0100 Subject: [Dovecot] dsync replication available for testing Message-ID: Hi -- On 08.03.2012 12:35, Timo Sirainen wrote: > On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > You can do for example: > > service config { > unix_listener config { > user = vmail > } > } I will try that later. >> It seems to me, that whenever a larger number of mails arrive on >> both >> servers simultaneously, the replicator gets into trouble [1]. I am >> unsure if one can expect that a replicator should deal with such >> stress, >> though. Or? > > Were these mails delivered via LMTP or dovecot-lda? LMTP > The locks should prevent duplicates I think, so there's something > still > going wrong. Just to be sure that I didn't misunderstand your proposed configuration: @mx1: plugin { mail_replica = remote:vmail at mx2.example.tld } @mx2: plugin { mail_replica = remote:vmail at mx1.example.tld } I do need to define one mail_replica plugin at each server pointing to the other one, correct? Regards, Michael From as at 3a.pl Thu Mar 8 14:24:13 2012 From: as at 3a.pl (Adam Szpakowski) Date: Thu, 08 Mar 2012 13:24:13 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F58A4ED.8070704@3a.pl> On 08.03.2012 10:56, Vincent Schut wrote: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... > I read in the docs about the auto-generated-from-hg debian dovecot > packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what > version to use... OK, 2.2 is development, which leaves the choice to: > 1.2.15; 2.0.x, or 2.1.x. > > I would appreciate any consideration or thoughts on what version to > choose. On several production machines we are using dovecot from debian testing repos, so 2.0.x. It's working stable for us and is quite easy to maintain. Please be careful and very selectively install packages from testing. If possible, the package dependences should be installed from stable/security. -- Adam Szpakowski From schut at sarvision.nl Thu Mar 8 15:53:54 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 14:53:54 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F589FF9.7080608@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: On 03/08/2012 01:03 PM, Charles Marcus wrote: > On 2012-03-08 4:56 AM, Vincent Schut wrote: >> The old server was running gentoo linux (which is mainly the culprit of >> the old dovecot version: gentoo was too much trouble to keep updating); > > Please stop with the FUD... > > I've been running gentoo for 8+ years, and it is a *breeze* to keep > updated, *especially* long term (since it is a 'rolling release' type of > distro)... Right. I should've known I shouln't mention anyone's favourite distro... :-) Hey, listen, sorry I offended you... its really nothing I have against gentoo, I'm sorry it might have sounded like that. It's just that I appeared not to have the time and energy to do regular updates, and when I tried to update something some months later, I had problems which I had no time and energy to start solving. Thus I decided a rolling distro was no good combination for my server and me. Which is why I will switch to a less rolling distro. That's really all there is to say about. I do still have a rolling distro which-will-not-be-named on my desktop, which I can and do update often and easy. > > Yes, it actually does require some minimum amount of attention from the > admin, like, say, once per week or once per month updates - buy so > should *any* system... and yes, it does require a little more > willingness to learn and 'get your hands dirty' (especially for the > installation), but it is well worth it. Yes, I have learned lots from some years with gentoo. No bad feelings. Just bad combo this time. > > Oh - and Portage rocks... :) > Well, yes, so does granite. Or iron maiden. Or whatever. As long as you like it :-) But maybe you also have something useful to say on the questions I *did* ask? About dovecot versions, and/or maildir vs. dbox for example? As the subject said, I was seeking advice, not rant nor war... Best, Vincent. From campbell at cnpapers.com Thu Mar 8 16:38:36 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 08 Mar 2012 09:38:36 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F57C96F.7090602@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> <4F57C96F.7090602@hardwarefreak.com> Message-ID: <4F58C46C.8000202@cnpapers.com> On 3/7/2012 3:47 PM, Stan Hoeppner wrote: > On 3/6/2012 3:01 PM, Steve Campbell wrote: > >> I've experienced that type of locked mailbox before on the old server. >> Users insist on accessing their email account as a pop account on their >> desktop with the "check for new mail every so many minutes" turned on >> and still keep their smartphones on while accessing it as an imap >> account so they can still download the files to their desktop when they >> return. > Using IMAP on the phone and POP on the PC doesn't make any sense. Is > there a (valid) reason why these people insist on this phone/IMAP and > PC/POP setup? This seems seriously counter intuitive/productive. The bulk of these type users are sales staff. They use their desktop when their in the office. For years, the only type of email account we used was pop just because that was the way it was. We used horde for webmail, which read these type of accounts just fine. Once they needed email in the field, it was necessary to either set up their phones to use pop and keep email on the server so that they could download the email to their desktop, or use imap on the phones. They typically don't use any folders they've created on the imap account when accessing mail on the desktop. It would be a nightmare going to each desktop, finding a time when each and every user would have the time to allow us to change things, and switching all of the accounts. It may not seem to be a good way of doing things, but it's just the way our system here has evolved. Now that we're down to skeleton-type staffing, it's not easy to find the time and manpower to accomplish change when it "ain't broke". The occasional locked mailbox was easier to resolve that the massive change to all user's accounts. This all came about because I installed a new server to replace the old, and dovecot became the pop/imap server. > >> So just to clarify, is it OK to have a maildir account setup on this >> server for these shared/imap access only accounts along with the mbox >> accounts already on there? > Yes. With Dovecot it is possible to specify mail_location on a per user > basis: > > http://wiki.dovecot.org/MailLocation > > You can even do a split mailbox type setup per user using multiple > namespaces, for example specifying that INBOX use mbox with all other > mail being stored in maildir format: > > http://wiki.dovecot.org/Namespaces > >> Thanks for the patience and help > Sure thing. Again, thanks for the help. From CMarcus at Media-Brokers.com Thu Mar 8 18:04:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 11:04:14 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: <4F58D87E.3040704@Media-Brokers.com> On 2012-03-08 8:53 AM, Vincent Schut wrote: > But maybe you also have something useful to say on the questions I *did* > ask? About dovecot versions, and/or maildir vs. dbox for example? As the > subject said, I was seeking advice, not rant nor war... Yeah, sorry, and I wasn't offended, I just dislike it when someone says something like that without clarification... As for version, it is generally recommended for obvious reasons to stay within the confines of your distros package manager unless you are comfortable installing from source. I've never used Debian, so can't speak to which repos you can safely use or the implications if you do... As for what mailbox format, there is no more 'dbox', it is either sdbox (like mbox one file per folder) or mdbox (multiple files per folder) - that said, mdbox seems to be the best general purpose, but my understanding is it can complicate things if something goes wrong, but it seems to be very solid. -- Best regards, Charles From micah at riseup.net Thu Mar 8 18:27:43 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:27:43 -0500 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> Message-ID: <87ty1zys5c.fsf@algae.riseup.net> Willie Gillespie writes: > On 03/07/2012 12:43 PM, Micah Anderson wrote: >> >> When a user makes a folder called 'x.y' it actually creates a folder >> called 'x' with a folder called 'y' inside, rather than a folder called >> 'x.y'. I'm guessing this has to do with an internal folder separator >> namespace configuration, but I'm a bit confused by how this works. > > Correct. > Similar to how in Linux, I could create a folder > mkdir test1/test2 > It will create test2 inside of test1. > > The difference being that IMAP doesn't necessarily need the parent mailbox to > exist, where Linux would throw an error if test1/ didn't exist first. > > So basically, as far as I know, you can't have a folder with a "." in the name > with the namespaces you have set up. That makes sense, however I'm not sure that I need these namespaces any longer if I no longer am using the maildir format (mdbox). In either case, it seems like the internal folder separator should not be exposed to the user like this. What is happening now is the user gets something other than they expect (a folder within a folder, instead of a folder with a dot in the name) because of some unknown internal configuration. If moving to mdbox is not enough to remove these namespace configurations that cause this, then it would be good if the user was unable to create such a folder, because it was prohibited, rather than creating something other than they expect. micah From micah at riseup.net Thu Mar 8 18:29:46 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:29:46 -0500 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. References: Message-ID: <87pqcnys1x.fsf@algae.riseup.net> Vincent Schut writes: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... No, Debian has 1.2.15 in its /stable (squeeze)/ repositories, there are newer versions available in other Debian repositories. micah From robert at schetterer.org Thu Mar 8 18:32:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Thu, 08 Mar 2012 17:32:49 +0100 Subject: [Dovecot] dot named folders In-Reply-To: <87ty1zys5c.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> Message-ID: <4F58DF31.3040203@schetterer.org> Am 08.03.2012 17:27, schrieb Micah Anderson: > Willie Gillespie writes: > >> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>> >>> When a user makes a folder called 'x.y' it actually creates a folder >>> called 'x' with a folder called 'y' inside, rather than a folder called >>> 'x.y'. I'm guessing this has to do with an internal folder separator >>> namespace configuration, but I'm a bit confused by how this works. >> >> Correct. >> Similar to how in Linux, I could create a folder >> mkdir test1/test2 >> It will create test2 inside of test1. >> >> The difference being that IMAP doesn't necessarily need the parent mailbox to >> exist, where Linux would throw an error if test1/ didn't exist first. >> >> So basically, as far as I know, you can't have a folder with a "." in the name >> with the namespaces you have set up. > > That makes sense, however I'm not sure that I need these namespaces any > longer if I no longer am using the maildir format (mdbox). > > In either case, it seems like the internal folder separator should not > be exposed to the user like this. What is happening now is the user gets > something other than they expect (a folder within a folder, instead of a > folder with a dot in the name) because of some unknown internal > configuration. > > If moving to mdbox is not enough to remove these namespace > configurations that cause this, then it would be good if the user was > unable to create such a folder, because it was prohibited, rather than > creating something other than they expect. > > micah > http://wiki.dovecot.org/Plugins/Listescape may help -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From e-frog at gmx.de Thu Mar 8 19:43:25 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 08 Mar 2012 18:43:25 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command In-Reply-To: <4F57A640.1030202@gmx.de> References: <4F57A640.1030202@gmx.de> Message-ID: <4F58EFBD.9080205@gmx.de> On 07.03.2012 19:17, wrote e-frog: > # 2.1.1 (94de7605f50f) > 1 namespace > * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL > * OK Namespace completed. > > Please note that the "OK Namespace completed." is send untagged. Ok, it's working again today with 2.1.1 (7a26c427fc78). From busseniu at in.tum.de Thu Mar 8 19:56:03 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Thu, 08 Mar 2012 18:56:03 +0100 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> Message-ID: <4F58F2B3.9070407@in.tum.de> On 03/04/2012 03:10 PM, Timo Sirainen wrote: > BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. Oh, I was not aware that this feature exists. I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Our migration process involves: 1) Converting the maildir from Courier using the Perl script 2) Converting to mdbox using dsync -R backup The POP3 ordering seems to get lost during the second step. I.e., if Dovecot is set up to server POP3 mails from a maildir having "O" flags, the POP3 ordering is as intended. After changing the configuration to mdbox format and converting the mails using dsync, the POP3 ordering is different. Is this known or am I missing something? (I tried Dovecot 2.1.1.) Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 20:46:50 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 18:46:50 +0000 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: Your message of "Sun, 04 Mar 2012 15:36:59 +0200." <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> Message-ID: Thank you for your help, Timo. > use Dovecot v2.0's dsync I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) on a dovecot (v1) installation; presumably with its own configuration file(s). > You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... Yes, I think we could do that; I should have thought of it myself, thanks again. I think there was one other problem with the automatic conversion which I've now remembered: I note that the first time a user connects to th eimap service dovecot creates their (virtual) home directory for them with all the right permissions. That's great and I use the existence of that directory as an indication to our MTA that the user wants delivery into the dovecot store rather than their old system mailbox. However once I tried using the convert plugin the process fails because (it seems) the conversion tries to take place before the home directory has been created. Is there any configuration change that might change this order? Can I configure the convert plugin on LDA delivery, for example, instead of as part of the "protocol imap" section? Many thanks, Steve Platt From tss at iki.fi Thu Mar 8 20:51:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 20:51:26 +0200 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <4F58F2B3.9070407@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> <4F58F2B3.9070407@in.tum.de> Message-ID: <0FEB6932-0FE5-42C6-B72F-3FD914B3A7BB@iki.fi> On 8.3.2012, at 19.56, Christoph Bu?enius wrote: > On 03/04/2012 03:10 PM, Timo Sirainen wrote: >> BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. > > Oh, I was not aware that this feature exists. > > I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Yeah, for now it's only for Maildir. Probably wouldn't be difficult to implement for dbox by adding it as dbox metadata (although how to add it there? dsync can't copy that). From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 21:04:47 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 19:04:47 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 Message-ID: I've set up a list of ciphers that excludes SSLv2 ciphers (and other weak ones) in the hope of preventing SSLv2 connections: ssl_cipher_list = TLSv1+HIGH : !SSLv2 : RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH However, this doesn't prevent the SSLv2 connection being allowed as our Nessus scans show and I'm tasked with trying to plug that "hole". I see Dovecot2 had the following change a year or so ago, in file src/login-common/ssl-proxy-openssl.c: - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); I tried making the same change to dovecot1's src tree on our test system and it seems to have the desired effect; however I am very hesitant about putting this into our production system without seeking advice here first :-) Have I missed anything that's obviously bad about doing this please? Thanks again, Steve Platt From markus at mpetri.org Thu Mar 8 21:18:12 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 8 Mar 2012 20:18:12 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 Message-ID: <20120308201812.2932e90c@legolas.home.ceotex.de> Hi, after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use shared folders with mutt anymore. 2.1 lists the shared namespace prefix once per user sharing an folder in LIST "" "%". I also noticed, that with 2.1 the user folder (Shared/) is no longer tagged as \NoSelect. Is this the intended behaviour and mutt simply cannot cope with it or is it a dovecot problem? Here an example with three users sharing a folder to the logged in user with Dovecot 2.1.1: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\HasChildren) "/" "Shared/test" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasChildren) "/" "Shared/test2" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. The same three users and config with Dovecot 2.0.18: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared/test" * LIST (\Noselect \HasChildren) "/" "Shared/test2" * LIST (\Noselect \HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. Markus # 2.1.1: /opt/dovecot-2.1/etc/dovecot/dovecot.conf # OS: Linux 3.2.0-1-amd64 x86_64 Debian wheezy/sid auth_mechanisms = plain login disable_plaintext_auth = no listen = 192.168.56.11 mail_location = maildir:~/Maildir mail_plugins = acl managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { inbox = yes location = prefix = separator = / subscriptions = yes type = private } namespace { inbox = no list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/index/shared/%%u prefix = Shared/%%u/ separator = / subscriptions = no type = shared } passdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } plugin { acl = vfile acl_anyone = allow acl_shared_dict = file:/var/lib/vdovecot/shared-mailboxes.db } protocols = imap service auth { unix_listener auth-userdb { mode = 0600 user = vdovecot } } ssl = no userdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } verbose_proctitle = yes protocol imap { mail_plugins = acl imap_acl } From tss at iki.fi Thu Mar 8 21:36:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 21:36:09 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <20120308201812.2932e90c@legolas.home.ceotex.de> References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: On 8.3.2012, at 21.18, Markus Petri wrote: > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > once per user sharing an folder in LIST "" "%". > > I also noticed, that with 2.1 the user folder (Shared/) is no > longer tagged as \NoSelect. > > Is this the intended behaviour and mutt simply cannot cope with it or > is it a dovecot problem? Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even if it did. Also Dovecot probably should add \Noselect, especially if the mailbox isn't really selectable (there's some weirdness between shared/user being equal to shared/user/INBOX, but I'm not sure what to do about it). From Bennett.Tony at con-way.com Fri Mar 9 01:23:00 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Thu, 8 Mar 2012 15:23:00 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? Message-ID: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. (The output of "dovecot -n" is at the bottom of this email.) I'm trying "baby steps" to get it up, before I give it the final configuration. (My apologies: I was pointed to RFC3501 and told to get an IMAP server, build it, configure it, and bring it up) What is currently occurring when I start dovecot is: Error: service(pop3-login): listen(::, 110) failed: Address already in use Error: service(pop3-login): listen(::, 995) failed: Address already in use Error: service(imap-login): listen(::, 143) failed: Address already in use Error: service(imap-login): listen(::, 993) failed: Address already in use Fatal: Failed to start listeners Using TRUSS and recompiling with log messages I've determined that dovecot is successfully creating and binding to AF_INET sockets... but is failing when trying to do the "bind" the same port to an AF_INET6 socket. The failure is "EADDRINUSE". The logic in the dovecot sources seems driven off of the define of HAVE_IPV6 (defined in config.h by configure) So, the questions I have are: - Is this the correct behavior - If this is the correct behavior, has this been tested against AIX 6.1, and if so, does anyone have an idea of what I did wrong...??? If it has not been tested against AIX 6.1 and is NOT the correct behavior, should I just change "config.h", and undefined HAVE_IPV6 ... or is there a better way to move beyond this issue... (like a change to "configure")??? Thanks, -tony Here is the output of "dovecot -n": # 2.1.1: /attic/usr/local/etc/dovecot/dovecot.conf # OS: AIX 1 00C30F654C00 default_login_user = dovecot disable_plaintext_auth = no namespace { inbox = yes location = mailbox { special_use = \Drafts name = Drafts } mailbox { special_use = \Junk name = Junk } mailbox { special_use = \Sent name = Sent } mailbox { special_use = \Sent name = Sent Messages } mailbox { special_use = \Trash name = Trash } prefix = name = inbox } passdb { args = scheme=CRYPT username_format=%u /attic/usr/local/etc/dovecot/users driver = passwd-file } service anvil-auth-penalty { name = anvil } service auth-worker { name = auth-worker } service auth-client { name = auth } service config { name = config } service dict { name = dict } service login/proxy-notify { name = director } service dns-client { name = dns_client } service doveadm-server { name = doveadm } service imap { name = imap-login } service login/imap { name = imap } service indexer-worker { name = indexer-worker } service indexer { name = indexer } service ipc { name = ipc } service lmtp { name = lmtp } service log-errors { name = log } service pop3 { name = pop3-login } service login/pop3 { name = pop3 } service login/ssl-params { name = ssl-params } service stats-mail { name = stats } ssl_cert = References: Message-ID: <201203091030.20828.sdavies@sdc.com.au> Yes that is the google thread that I saw. I don't see the relevance of your reference to dsync. As I read the man pages for dsync it is used to sync separate servers, to make backups or to convert mailbox formats. When I upgraded from 1.2.15 to 2.1.1 I saw nothing in the doco to suggest that dsync was relevant to my scenario. In a previous thread here (Log sync errors), Timo suggested that the migration fix was to delete all .imap directories. My understanding was that this should fix any differences between 1.2.15 files and 2.1.1. If that were the case, mentioning the migration again would seem irrelevant. However, it seems that deleting the .imap files did not fix the log sync errors or the fscking warnings. Both are still happening continuously. Cheers, Stephen On Thu, 8 Mar 2012 08:26:55 PM dovecot-request at dovecot.org wrote: > Date: Wed, 07 Mar 2012 15:03:35 -0600 > From: Stan Hoeppner > To: dovecot at dovecot.org > Subject: Re: [Dovecot] Fscking warnings > Message-ID: <4F57CD27.3000207 at hardwarefreak.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On 3/6/2012 5:07 PM, Stephen Davies wrote: > > Google tells me that these "should go away" but they don't. > > > > > > > > Seems to happen continuously while a user is viewing email. > > Is this thread what "Google tells you"? > > http://dovecot.org/list/dovecot/2010-October/053909.html > > Timo is the creator of Dovecot, if you didn't know. So you can take his > words for gospel. Also note his last statement in that thread: > > "The next time you could do it with dsync to avoid these kind of > problems." > > It would seem you omitted a very important detail from your problem > report, which is that you recently performed a migration. Please don't > omit such critical details in future requests for help. Provide as much > relevant detail as possible. This speeds the process up for everyone, > and avoids guesswork on our part. > > -- > Stan > > > ------------------------------ > > Message: 10 > Date: Thu, 8 Mar 2012 00:26:55 +0100 > From: "Marc" > To: > Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From mmielke at sapphire.gi Fri Mar 9 02:50:47 2012 From: mmielke at sapphire.gi (Martin Mielke) Date: Fri, 9 Mar 2012 00:50:47 +0000 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. Message-ID: Hi all, I have inherited an old Dovecot installation which is causing headaches almost every day. I know that one of the rules says "Don't bother asking questions about v0.99.x versions. They're no longer supported."...but please bear with me, this will be quick as I only need some advise from experienced Dovecot gurus out there. I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... Thanks a lot in advance! Regards, Martin From ybhu at hk1.ibm.com Fri Mar 9 10:09:37 2012 From: ybhu at hk1.ibm.com (Andy YB Hu) Date: Fri, 9 Mar 2012 16:09:37 +0800 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? Message-ID: In the real world, the mails which belongs to one thread could be dispersed in different mailbox, at least Inbox and SENT, so whether the command can search different mailboxes and grap them in one talk with dovecot? If Not, any other approach to do that? BTW, what the THREAD=REFS stands for? In the RFC, http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. From janfrode at tanso.net Fri Mar 9 10:31:34 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 9 Mar 2012 09:31:34 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <20120309083134.GA8248@dibs.tanso.net> On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: > > As for what mailbox format, there is no more 'dbox', it is either > sdbox (like mbox one file per folder) or mdbox (multiple files per > folder) - Sdbox is like maildir, one message per file, while mdbox is more like mbox: http://wiki2.dovecot.org/MailboxFormat/dbox > that said, mdbox seems to be the best general purpose, but > my understanding is it can complicate things if something goes > wrong, but it seems to be very solid. It's a leap of faith to go with dovecot's own format, and no longer be able to use grep and mutt to poke in mail folders directly, but as a serverside storage format it seems like the right way to go. -jf From varia at e-healthexpert.org Fri Mar 9 11:35:17 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Fri, 9 Mar 2012 09:35:17 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: References: Message-ID: <20120309093517.30979c04@e-healthexpert.org> On Thu, 08 Mar 2012 19:04:47 +0000, Steve Platt wrote: > I've set up a list of ciphers that excludes SSLv2 ciphers (and other > weak ones) in the hope of preventing SSLv2 connections: > > ssl_cipher_list = TLSv1+HIGH : !SSLv2 : > RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH > > I tried making the same change to dovecot1's src tree on our test > system and it seems to have the desired effect; No need to change sources. Try this and see if it serves your purpose: ssl = required ssl_cipher_list = HIGH:!SSLv2:!aNULL:!MD5!DES:!3DES M. From CMarcus at Media-Brokers.com Fri Mar 9 16:45:35 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 09 Mar 2012 09:45:35 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <4F5A178F.1060404@Media-Brokers.com> On Thu, Mar 09, 2012 at 12:30AM -0500, Jan-Frode Myklebust wrote: > On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: >> As for what mailbox format, there is no more 'dbox', it is either >> sdbox (like mbox one file per folder) or mdbox (multiple files per >> folder) - > > Sdbox is like maildir, one message per file, while mdbox is more > like mbox: > > http://wiki2.dovecot.org/MailboxFormat/dbox Wow, I've no idea how that bit of incorrect data got lodged inside my head. Thanks Jan-Frode for the correction! -- Best regards, Charles From steve.platt at mrc-bsu.cam.ac.uk Fri Mar 9 17:05:26 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Fri, 09 Mar 2012 15:05:26 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: Message from Mark Alan of "Fri, 09 Mar 2012 09:35:17 GMT." <20120309093517.30979c04@e-healthexpert.org> Message-ID: Hi Mark, I think I may not have been clear enough in my query, sorry! What I'm trying to do is to prevent SSLv2 connections being made to our IMAP server while allowing SSLv3 and TLSv1 connections. I think I've prevented the use of SSLv2 ciphers but this does not prevent SSLv2 protocol connections (as far as I can tell). (Once connected, the SSLv2 client finds it has no ciphers so the session fails at that point but this is not enough to satisfy our security audit. I want to disable the use of the SSLv2 protocol itself, not just the SSLv2 ciphers) steve.platt at mrc-bsu.cam.ac.uk said: > I see Dovecot2 had the following change a year or so ago, in file src/ > login-common/ssl-proxy-openssl.c: > > - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); > + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > > I tried making the same change to dovecot1's src tree on our test system and > it seems to have the desired effect ... I'm testing this by using: openssl s_client -ssl2 -connect mailhost:993 This should fail immediately with "ssl handshake failure" (for a happy audit!). Thanks again, Steve From user+dovecot at localhost.localdomain.org Fri Mar 9 22:19:15 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Fri, 09 Mar 2012 21:19:15 +0100 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> Message-ID: <4F5A65C3.4080404@localhost.localdomain.org> On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From Bennett.Tony at con-way.com Fri Mar 9 22:34:56 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Fri, 9 Mar 2012 12:34:56 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <4F5A65C3.4080404@localhost.localdomain.org> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> <4F5A65C3.4080404@localhost.localdomain.org> Message-ID: <9E085D377965634187A85638358AE61101A291DA92@DCXPRCL017.cnf.prod.cnf.com> Bingo... It worked... Thanks, Pascal -tony -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Pascal Volk Sent: Friday, March 09, 2012 12:19 PM To: Dovecot Mailing List Subject: Re: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From sca at andreasschulze.de Fri Mar 9 23:40:16 2012 From: sca at andreasschulze.de (Andreas Schulze) Date: Fri, 9 Mar 2012 22:40:16 +0100 Subject: [Dovecot] sieve and utf-7 foldernames Message-ID: <20120309214016.GA5584@doran.andreasschulze.de> Hi all, since many dovecot/pigeonhole versions I have an error: Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) setup: dovecot-2.1.1 / pigeonhole-0.3.0 postfix deliver to dovecot-lda .dovecot.sieve contains this: require ["fileinto","reject","vacation","relational","comparator-i;ascii-numeric","regex"]; if header :contains "To" "green at example.org" { fileinto "INBOX.gr&APw-n"; stop; } a mail to the mentioned address produces this logging: Mar 9 22:23:10 test dovecot: lda(foo): save: box=INBOX.gr&APw-n, uid=2, msgid=<4711 at example.org>, size=4642 also a mail_debug=yes log no more lines related to the foldername but at the end I found the mail not in the expected folder. It places in "INBOX.gr&-APw-n" notice the extra minus after & I think it's pigeonholes fault but I may be wrong ... # doveconf -n # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686 i686 Debian 5.0.9 ext3 ... lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = 127.0.0.1, ::1 protocols = " imap sieve" protocol lda { mail_plugins = quota notify mail_log sieve } ... Andreas From user+dovecot at localhost.localdomain.org Sat Mar 10 02:13:43 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Sat, 10 Mar 2012 01:13:43 +0100 Subject: [Dovecot] sieve and utf-7 foldernames In-Reply-To: <20120309214016.GA5584@doran.andreasschulze.de> References: <20120309214016.GA5584@doran.andreasschulze.de> Message-ID: <4F5A9CB7.1040804@localhost.localdomain.org> On 03/09/2012 10:40 PM Andreas Schulze wrote: > Hi all, > > since many dovecot/pigeonhole versions I have an error: > Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) > ? Behaves as documented - since Dovecot 1.2.0: Wiki > Upgrading > v1.1 to v1.2 > Sieve: * You should consider migrating from CMU Sieve to Dovecot Sieve (see the link for instructions) http://wiki.dovecot.org/LDA/Sieve/Dovecot#Migration_from_CMUSieve: * Be sure to use UTF8 for the mailbox argument of the fileinto command. Older CMUSieve installations used modified UTF7 (as IMAP does) for the mailbox parameter. If not adjusted, the new Sieve plugin will use the wrong folder name for storing the message. Regards, Pascal -- The trapper recommends today: cafefeed.1207001 at localdomain.org From tss at iki.fi Sat Mar 10 18:01:27 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:01:27 +0200 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. In-Reply-To: References: Message-ID: On 9.3.2012, at 2.50, Martin Mielke wrote: > I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... http://wiki2.dovecot.org/Upgrading/1.0 points out a few things: - rename .subscriptions -> subscriptions - rename .customflags -> dovecot-keywords - default pop3 UIDL format changed From tss at iki.fi Sat Mar 10 18:02:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:02:40 +0200 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? In-Reply-To: References: Message-ID: <4956B88C-3B55-460C-AAD2-E8253DA8627B@iki.fi> On 9.3.2012, at 10.09, Andy YB Hu wrote: > > In the real world, the mails which belongs to one thread could be dispersed > in different mailbox, at least Inbox and SENT, so whether the command can > search different mailboxes and grap them in one talk with dovecot? > > If Not, any other approach to do that? Create a virtual mailbox containing all mails. Then you see all the mails within a thread. http://wiki2.dovecot.org/Plugins/Virtual > BTW, what the THREAD=REFS stands for? In the RFC, > http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. It's from http://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05 which never made it to an actual RFC. From tss at iki.fi Sat Mar 10 18:04:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:04:44 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <79C53550-348C-4812-AAA1-2C3D6D9F59B4@iki.fi> On 8.3.2012, at 20.46, Steve Platt wrote: >> use Dovecot v2.0's dsync > > I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) > on a dovecot (v1) installation; presumably with its own configuration file(s). Yes, although in some situations it might write stuff to index files that v1.x complains about. But deleting index files afterwards fixes that. >> You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... > > Yes, I think we could do that; I should have thought of it myself, thanks > again. > > I think there was one other problem with the automatic conversion which I've > now remembered: I note that the first time a user connects to th eimap service > dovecot creates their (virtual) home directory for them with all the right > permissions. That's great and I use the existence of that directory as an > indication to our MTA that the user wants delivery into the dovecot store > rather than their old system mailbox. However once I tried using the convert > plugin the process fails because (it seems) the conversion tries to take place > before the home directory has been created. > > Is there any configuration change that might change this order? No. > Can I configure the convert plugin on LDA delivery, for example, instead of as > part of the "protocol imap" section? Yes. From tss at iki.fi Sat Mar 10 18:06:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:06:20 +0200 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <729DCBA4-E353-41EA-903D-0DDF897E5208@iki.fi> On 7.3.2012, at 21.43, Micah Anderson wrote: > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. > > I'm using 2.0.15 with mdbox and this is what I have configured for my > namespaces: > > namespace { > separator = . > prefix = > inbox = yes > } Keep this. > namespace { > separator = . > prefix = INBOX. > inbox = no > hidden = yes > list = no > } > > I migrated from courier maildirs, so perhaps I no longer need some of > these now that the conversion is finished? It depends on if you have any users whose clients are using INBOX. namespace. If there are, and you remove it, the users won't see anything except INBOX anymore. From tss at iki.fi Sat Mar 10 18:11:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:11:55 +0200 Subject: [Dovecot] Single instance storage In-Reply-To: References: Message-ID: <508532C3-B8D0-47E8-9566-6A570A3233F3@iki.fi> On 8.3.2012, at 3.30, Jean-Daniel Beaubien wrote: > I have read most of the doc on the dovecot website, and couldn't find any > info on the single instance storage feature, so I'm posting my questions > here. > > - Are these 3 parameters the only one necessary for single instance > storage? I cannot find any doc on this feature on the website; is there > anything specific I need to know about them? (the last one isn't exactly > self-explanatory). > - mail_attachment_dir = /srv/vmail/attachments > - mail_attachment_hash = %{sha256} > - mail_cache_min_mail_count = 2 > > - Is this feature ready for production? mail_cache_min_mail_count isn't related to single instance storage at all. I didn't really even remember that such a setting existed. I'm not sure if it's actually useful in any setups.. Maybe you were thinking about mail_attachment_min_size? Other than that, yeah, the mail_attachment_dir is really the only thing you need to set to enable SIS. From tss at iki.fi Sat Mar 10 18:13:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:13:39 +0200 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: On 7.3.2012, at 1.07, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. > > Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file > /home/john/Mail/INBOX/.imap/Archive/dovecot.index What Dovecot version? Anyway, something wrong in the mbox I guess. Just do rm -rf /home/john/Mail/INBOX/.imap/ From tss at iki.fi Sat Mar 10 18:20:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:20:33 +0200 Subject: [Dovecot] POP3C storage backend In-Reply-To: <4F5538DC.4060802@talpey.com> References: <4F5538DC.4060802@talpey.com> Message-ID: <7B049F23-AE14-4BCE-857C-91D70E02A7E8@iki.fi> On 6.3.2012, at 0.06, Tom Talpey wrote: > I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I > don't see anything in the 2.1 doc directory or in the wiki. Can this > be used to synchronize dovecot with external pop servers? Doing away > with my current fetchmail and lmtp solution for this would be quite > interesting. > > Thanks for any pointers to configuring and using this, if so... It could possibly be used to do that with http://wiki2.dovecot.org/Plugins/Snarf Although that would probably connect to POP3 server quite often. And when IDLEing it wouldn't see new POP3 mails (that would need a small change to snarf plugin). From tss at iki.fi Sat Mar 10 18:44:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:44:21 +0200 Subject: [Dovecot] dovecot Digest, Vol 107, Issue 20 Fscking warnings In-Reply-To: <201203091030.20828.sdavies@sdc.com.au> References: <201203091030.20828.sdavies@sdc.com.au> Message-ID: <7F823A5F-36DC-4444-A13E-3182FA243EE1@iki.fi> On 9.3.2012, at 2.00, Stephen Davies wrote: > However, it seems that deleting the .imap files did not fix the log sync errors > or the fscking warnings. > > Both are still happening continuously. If you're talking about errors like these: Mar 10 18:21:38 imap(tss): Error: Log synchronization error at seq=1,offset=26896 for /home/tss/mail/.imap/INBOX/dovecot.index: Extension header update points outside header size and the following fsck error, then deleting all of the .imap directories should get rid of them (maybe you didn't delete all of them? note that each subdirectory has its own, so there's more than just ~/mail/.imap/). Anyway, this is now also fixed: http://hg.dovecot.org/dovecot-2.0/rev/dc88712581c9 http://hg.dovecot.org/dovecot-2.1/rev/1289b79241bb From tss at iki.fi Sat Mar 10 18:50:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:50:15 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <7F351B29-16BF-4098-8C3A-3FC84D85ADD4@iki.fi> On 7.3.2012, at 15.32, Charles Marcus wrote: > 4. They can *move* messages to other folders in that account (ie, 'file' them), and last > > (this is the tricky part) > > 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. There is unfortunately no "default ACL" feature currently. Although you could somewhat easily add an ugly hack to the code for that. And I guess it wouldn't be difficult to implement it, maybe by reading it from $mail_root/dovecot-acl-default file or something.. So without code changes you could: - create all of the necessary folders - set such ACLs that user can't create any more folders - disallow expunging in all folders From tss at iki.fi Sat Mar 10 18:51:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:51:16 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F578F31.3000303@wildgooses.com> References: <4F576370.8040706@Media-Brokers.com> <4F578F31.3000303@wildgooses.com> Message-ID: <75B1D406-190B-4824-B575-E1AF1F76B207@iki.fi> On 7.3.2012, at 18.39, Ed W wrote: >> 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. > > Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) The maildir file's read permission doesn't matter, the parent cur/ or new/ directory's write permission matters. And removing those prevents moving mails from new/ to cur/ and from keeping the flag states in the filename.. Not very good. From tss at iki.fi Sat Mar 10 18:53:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:53:38 +0200 Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file In-Reply-To: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> References: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> Message-ID: <35158279-4A11-4806-A481-548314E27282@iki.fi> On 7.3.2012, at 4.22, Rajesh M wrote: > on server number 1 i get errors as such > Error: fcntl(read-lock) locking failed for file Input/output error > squirrelmail gives error imap connection closed and i am not able to login > > > so i set the parameters as such in the dovecot conf file and the error > stopped > > mmap_disable=yes > dotlock_use_excl = yes > lock_method = dotlock > > can somebody please advise me if the above is correct ? That should work. > or is it preferred to use fcntl with lockd That would probably be more efficient. > (note that my mailbox is maildir format) The fcntl locking is used for Dovecot index files, not for maildir files. From tss at iki.fi Sat Mar 10 18:56:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:56:50 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: References: Message-ID: On 7.3.2012, at 5.19, Jean-Daniel Beaubien wrote: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." > > How will the mailbox files (m.X) files be modified when I move or delete > emails using mdbox+gzip. Will the resulting gzipped mdbox files be > rsync-able or will they need a full re-upload? > > If I plan on using rsync for backups, am I better off not using the gzip > feature (if i can spare the extra storage)??? gzipping is irrelevant, the behavior is the same with and without gzip. The purging step recreates new mail files, so the new files will need to be fully uploaded with rsync. You might want to consider using dsync instead. From btb at bitrate.net Mon Mar 12 06:00:11 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Mon, 12 Mar 2012 00:00:11 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections Message-ID: hi- i have a configuration in which i'm using different passdb/userdb settings for each of imap, lmtp and smtp [without getting too far off on a tangent, this is so ldap group membership can be used to independently control authorization for receiving [lmtp], retrieving [imap], and sending [smtp/postfix] of email. each passdb/userdb uses a different search filter.] when i use this as my config: # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic-pae i686 Ubuntu 11.10 auth_debug = yes first_valid_gid = 2000 first_valid_uid = 2000 log_timestamp = "%d.%m.%Y %H.%M.%S " login_greeting = dovecot ready mail_debug = yes protocols = " imap lmtp" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service lmtp { inet_listener lmtp { address = 127.0.0.1 ::1 port = 10026 } } ssl_cert = Hello We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib - Does anybody know of any other project to create an easy-restore for Dovecot? Regards Maria From wouter at vdschagt.com Mon Mar 12 14:09:51 2012 From: wouter at vdschagt.com (Wouter van der Schagt) Date: Mon, 12 Mar 2012 13:09:51 +0100 Subject: [Dovecot] Dovecot LDA breaking .qmail forwarding? Message-ID: <00d201cd0049$07274880$1575d980$@vdschagt.com> Good morning all, I've a problem, I'm using the Dovecot LDA in my .qmail file: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER However when I add a forward, such as: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER &wouter at vdschagt.com The mail isn't forwarded and stays in the queue on the mail server, the same is the case when I reverse the lines. The error in the log file is: @400000004f5de5f7033bc434 delivery 824026: deferral: qmail-inject:_fatal:_qq_trouble_creating_files_in_queue_(#4.3.0)/system_erro r_calling_qmail-inject/ When using another LDA, for example when specifying the Maildir, it works, but then i cannot use Sieve scripts. Any ideas? Am I doing anything wrong? Sincerely, - Wouter van der Schagt From bind at enas.net Mon Mar 12 15:02:33 2012 From: bind at enas.net (Urban Loesch) Date: Mon, 12 Mar 2012 14:02:33 +0100 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <4F5DF3E9.9030703@enas.net> Hi, perhaps the mail_log plugin is what you need. Regards Urban On 12.03.2012 12:56, Maria Arrea wrote: > Hello > > We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: > > - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib > - Does anybody know of any other project to create an easy-restore for Dovecot? > > Regards > > Maria > From tss at iki.fi Mon Mar 12 15:04:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:04:10 +0200 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <1331557450.2081.112.camel@innu> On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From giles at coochey.net Mon Mar 12 15:09:52 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:09:52 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) Message-ID: <4F5DF5A0.9000609@coochey.net> Hi, I'm looking for a quick tool that can connect to my IMAP account grab all the messages in a particular folder and dump them to a mbox format file? Anyone know a quick easy tool to do that? This is a spam folder that I'd like to do some Bayes spam learning on, but since I've migrated to mdbox I don't think I can do this directly on the mailbox. Any thoughts appreciated. -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From maria_arrea at gmx.com Mon Mar 12 15:18:01 2012 From: maria_arrea at gmx.com (Maria Arrea) Date: Mon, 12 Mar 2012 14:18:01 +0100 Subject: [Dovecot] Question about folder creation/delete and logging Message-ID: <20120312131801.134760@gmx.com> mail_log plugin is just what we need, thank you for your support. Yes, we are using bacula+doveadm import for this project. If you are interested, we have in production an home-brew message-tracking system for our end-users. We parse qmail / postfix / clamav / spamassassin/ dovecot / sieve logs and insert them in a mysql database, and a logged user in our webapp can see what happened with her mails (sent mails or messages waiting for arrival). Affero GPL software, of course. Some screenshots (Spanish only, sorry) here: https://gestionproyectos.us.es/attachments/download/321/Seguimiento_de_mensajes___Vista_detalla_de_mensaje_1331558163319.png https://gestionproyectos.us.es/attachments/download/145/Captura_de_pantalla_2011-06-05_a_las_14.03.42.png https://gestionproyectos.us.es/attachments/download/156/Seguimiento_de_mensajes___Administraci%C3%B3n_1308042340487.png URL of the project (Seguimiento, spanish word for "tracking"): https://gestionproyectos.us.es/projects/seguimiento Regards Maria ----- Original Message ----- From: Timo Sirainen Sent: 03/12/12 02:04 PM To: Maria Arrea Subject: Re: [Dovecot] Question about folder creation/delete and logging On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From tss at iki.fi Mon Mar 12 15:19:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:19:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DF5A0.9000609@coochey.net> References: <4F5DF5A0.9000609@coochey.net> Message-ID: <1331558383.2081.114.camel@innu> On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: > Hi, > > I'm looking for a quick tool that can connect to my IMAP account grab > all the messages in a particular folder and dump them to a mbox format file? > > Anyone know a quick easy tool to do that? > > This is a spam folder that I'd like to do some Bayes spam learning on, > but since I've migrated to mdbox I don't think I can do this directly on > the mailbox. > > Any thoughts appreciated. With v2.0: dsync -m spam backup mbox:~/mbox-mails/ There are also doveadm move and doveadm import commands that can do this. From giles at coochey.net Mon Mar 12 15:37:43 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:37:43 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331558383.2081.114.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> Message-ID: <4F5DFC27.4020004@coochey.net> On 12/03/2012 13:19, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: >> Hi, >> >> I'm looking for a quick tool that can connect to my IMAP account grab >> all the messages in a particular folder and dump them to a mbox format file? >> >> Anyone know a quick easy tool to do that? >> >> This is a spam folder that I'd like to do some Bayes spam learning on, >> but since I've migrated to mdbox I don't think I can do this directly on >> the mailbox. >> >> Any thoughts appreciated. > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > There are also doveadm move and doveadm import commands that can do > this. > > OK, That would do a mailbox called spam no? I have a normal user who has an IMAP folder called 'Junk E-mail' Is there a similar syntax? -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From rob0 at gmx.co.uk Mon Mar 12 15:38:49 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 12 Mar 2012 08:38:49 -0500 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: References: Message-ID: <20120312133849.GS24983@harrier.slackbuilds.org> On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: > the problem with this is that while each of the passdb/userdb > configs for the various protocols does indeed work, if a result > is not found in one of them, the global passdb appears to then > function as a catch-all. > > how can i tell dovecot it doesn't need a global passdb? each > of the protocols' passdb/userdb configs is functioning as > desired, but having dovecot look elsewhere upon failure ends > up defeating the purpose. A simple workaround: use an empty passwd-file passdb as global. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From tss at iki.fi Mon Mar 12 15:39:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:39:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DFC27.4020004@coochey.net> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> Message-ID: <1331559583.2081.115.camel@innu> On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: > >> Any thoughts appreciated. > > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > > > There are also doveadm move and doveadm import commands that can do > > this. > > > > > OK, That would do a mailbox called spam no? > I have a normal user who has an IMAP folder called 'Junk E-mail' > > Is there a similar syntax? Yeah, the -m parameter just specifies the mailbox name. So: dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ If different accounts have different names, then I guess you'll just have to run dsync twice with each name. From giles at coochey.net Mon Mar 12 15:42:10 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:42:10 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331559583.2081.115.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> <1331559583.2081.115.camel@innu> Message-ID: <4F5DFD32.3010609@coochey.net> On 12/03/2012 13:39, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: >>>> Any thoughts appreciated. >>> With v2.0: dsync -m spam backup mbox:~/mbox-mails/ >>> >>> There are also doveadm move and doveadm import commands that can do >>> this. >>> >>> >> OK, That would do a mailbox called spam no? >> I have a normal user who has an IMAP folder called 'Junk E-mail' >> >> Is there a similar syntax? > Yeah, the -m parameter just specifies the mailbox name. So: > > dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ > > If different accounts have different names, then I guess you'll just > have to run dsync twice with each name. > > Thanks - was experimenting and got it. Works a treat! -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From micah at riseup.net Mon Mar 12 17:05:06 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:05:06 -0400 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> <4F58DF31.3040203@schetterer.org> Message-ID: <87k42pyi59.fsf@algae.riseup.net> Robert Schetterer writes: > Am 08.03.2012 17:27, schrieb Micah Anderson: >> Willie Gillespie writes: >> >>> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>>> >>>> When a user makes a folder called 'x.y' it actually creates a folder >>>> called 'x' with a folder called 'y' inside, rather than a folder called >>>> 'x.y'. I'm guessing this has to do with an internal folder separator >>>> namespace configuration, but I'm a bit confused by how this works. >>> >>> Correct. >>> Similar to how in Linux, I could create a folder >>> mkdir test1/test2 >>> It will create test2 inside of test1. >>> >>> The difference being that IMAP doesn't necessarily need the parent mailbox to >>> exist, where Linux would throw an error if test1/ didn't exist first. >>> >>> So basically, as far as I know, you can't have a folder with a "." in the name >>> with the namespaces you have set up. >> >> That makes sense, however I'm not sure that I need these namespaces any >> longer if I no longer am using the maildir format (mdbox). >> >> In either case, it seems like the internal folder separator should not >> be exposed to the user like this. What is happening now is the user gets >> something other than they expect (a folder within a folder, instead of a >> folder with a dot in the name) because of some unknown internal >> configuration. >> >> If moving to mdbox is not enough to remove these namespace >> configurations that cause this, then it would be good if the user was >> unable to create such a folder, because it was prohibited, rather than >> creating something other than they expect. >> >> micah >> > > http://wiki.dovecot.org/Plugins/Listescape > may help Interesting, thanks for the pointer, although I think I prefer if users are just prohibited from making a 'folder.withadot' and told that it is prohibited right away, rather than giving them a way to do it. micah -- From micah at riseup.net Mon Mar 12 17:10:46 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:10:46 -0400 Subject: [Dovecot] mdbox + gzip and rsync References: Message-ID: <87fwddyhvt.fsf@algae.riseup.net> Jean-Daniel Beaubien writes: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely. It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely. micah -- From tss at iki.fi Mon Mar 12 17:46:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 17:46:51 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: <87fwddyhvt.fsf@algae.riseup.net> References: <87fwddyhvt.fsf@algae.riseup.net> Message-ID: <1A1E352C-2A32-4C90-9357-A35C92D98875@iki.fi> On 12.3.2012, at 17.10, Micah Anderson wrote: > Jean-Daniel Beaubien writes: > >> After reading the following paragraph from the dovecot doc, I've been >> wondering how it would affect rsync (when combined with gzip): >> >> "Expunging a message only decreases the message's refcount. The space is >> later freed in "purge" step. This is typically done in a nightly cronjob >> when there's less disk I/O activity. The purging first finds all files that >> have refcount=0 mails. Then it goes through each file and copies the >> refcount>0 mails to other mdbox files (to the same files as where newly >> saved messages would also go), updates the map index and finally deletes >> the original file. So there is never any overwriting or file truncation." > > Interesting, so it would be recommended to those using mdbox format to > run a 'dovadm purge -A' every night to clean up these unused files? It > seems like without this, mail storage usage will just grow infinitely. Yes. > It does appear that using an rsync backup process for mdbox would not be > able to detect this and backups would also grow infinitely. rsync --delete would delete the old files, right? Anyway, I'd avoid using rsync for mdbox unless you're doing it on a filesystem snapshot. dsync backup should work better. From rtroy at ScienceTools.com Mon Mar 12 19:14:09 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) Subject: [Dovecot] Trouble adding sasl support via dovecot Message-ID: Hello Folks, I've been the admin of a site that uses Postfix with Dovecot on RedHat since, oh, gosh, maybe 1996? It's been a long time. I've never built it from source, though, just used the rpms (and I wonder if maybe that's my problem now). It just works, is reliable, and lets me be a very-part-time administrator. Repeatedly over the last few years I've been asked to have our mail system "join the modern age" and provide mail sending capabilities for clients that aren't on our internal network - via their smart-phones, from home, etc. OK... Well, way back when the site was set up, smtp servers didn't do any kind of "auth", but along the way to solving this problem (trying to configure pop-before-smtp, someone mentioned that Postfix now has an auth mechanism that uses Dovecot and I should use that instead! Great! ... Except that I've spent between 16 and 20 hours on this with no joy, and while I hate having to ask for help, it's time to ask what things that are obvious to the less ignorant that I must be doing wrong... Certainly, given the solid history of Postfix and Dovecot, I must be the problem! My problem statement is simply, "it should be working", but doesn't, and I don't get any announcement of "auth" when testing connections to Postfix as per directions here: http://www.postfix.org/SASL_README.html#server_test At least I haven't broken the normal functionality! I'm building a new server on the latest Fedora Core (16), but it's lacking in some hardware and won't be ready for a while, so I'm working with FC 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth scheme (which works fine and I'd hate to change it if I don't have to). The system has been up and functional on these versions for a couple of years, and quite stable, we just can't send if we're not local. When I do "postconf-a" it indicates cyrus and dovecot, so I take it that means Postfix has been built with sasl support. (I presume this means I don't have to compile it from source.) First Dovecot. Its set up to provide all protocols, but only imaps and pop3s have ports forwarded through the firewall. Plain-text auth is disabled, ssl is set to yes, ssl_listen is not specified, and the cert and key files are in the default locations - and work. No cipher list is used. Dovecot's chrooted. The protocol sections imap and pop3 take ALL the defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as I don't think it matters. "auth default {" has mechanisms set to cram-md5, digest-md5, plain, and login, with passdb passwd-file pointing to a file in /etc where the cram data goes. It's not using pam, and there's an OLD comment in the config: # Experience says we need an empty passdb - passwd group: which is followed by passdb passwd{}. Later, there's "userdb passwd {}. All of that was configured long ago and has been functional. The changes I've made to add sasl support primarily pertain to the "socket listen section of "auth default". There, the master section remains commented out while the client section has been uncommented, the path set to /var/spool/postfix/private/auth, mode set to 0660, and the user and group have been set to postfix. ...This is all as described here: http://www.postfix.org/SASL_README.html and http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL That's it for Dovecot. Now, to Postfix itself. >From the working environ, only listening on port 25, I simply added the following (as per directions already cited above): smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_security_options = noanonymous, noplaintext smtp_sasl_security_options = noanonymous, noplaintext smtpd_sasl_tls_security_options = noanonymous smtp_sasl_tls_security_options = noanonymous And, of course, permit_sasl_authenticated was added to smtpd_recipient_restrictions. I got the impression from the baove sources that Postfix will then use Dovecot's authentication mechanism via a socket it finds in its private/auth subdirectory. NOT documented in any of those places, someone suggested I must turn on TLS. OK... The documentation found here: http://www.postfix.org/TLS_README.html claims (intimates) that it's not possible to run a site on a self-signed certificate, however, there's ZERO budget for a signed certificate, so unless I can get one for ten bucks somewhere, that could be a deal-breaker here. However, we've been using self-signed certificates for a while now and wonder why an "exception" mechanism wouldn't exist. As that web page talks about "Netscape" I suspect it's very old and may no longer apply. In any event, I tried this, too (after trying without). On the good side, an available Android phone, previously reading fine, but unable to send, no longer complained when the setup was changed to the imap username and password, same server address, TLS security type, and the server port of 25. HOWEVER, no mail has passed through it successfully, it just gives no error whatsoever, so far, while the server's log reports "Relay access denied." Notably, when setting up TLS, Postfix complained when the smtpd_tls_key_file was incorrect, but did not complain when it was provided properly, suggesting it's reading and accepting my self-signed certificate and private key. Ideas, please?! And, by the way, what's port 465 all about? Some clients propose that's what should be used to send... Thanks in advance for your help, Richard From rtroy at ScienceTools.com Mon Mar 12 20:59:01 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 11:59:01 -0700 (PDT) Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: Message-ID: Hi All, it turned out to be the order of entries in stmpd_recipient_restrictions. Regards, Richard On Mon, 12 Mar 2012, Richard Troy wrote: > Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) > From: Richard Troy > To: postfix-users at cloud9.net, dovecot at dovecot.org > Subject: [Dovecot] Trouble adding sasl support via dovecot > > > Hello Folks, > > I've been the admin of a site that uses Postfix with Dovecot on RedHat > since, oh, gosh, maybe 1996? It's been a long time. I've never built it > from source, though, just used the rpms (and I wonder if maybe that's my > problem now). It just works, is reliable, and lets me be a very-part-time > administrator. > > Repeatedly over the last few years I've been asked to have our mail system > "join the modern age" and provide mail sending capabilities for clients > that aren't on our internal network - via their smart-phones, from home, > etc. OK... Well, way back when the site was set up, smtp servers didn't do > any kind of "auth", but along the way to solving this problem (trying to > configure pop-before-smtp, someone mentioned that Postfix now has an auth > mechanism that uses Dovecot and I should use that instead! Great! ... > Except that I've spent between 16 and 20 hours on this with no joy, and > while I hate having to ask for help, it's time to ask what things that are > obvious to the less ignorant that I must be doing wrong... Certainly, > given the solid history of Postfix and Dovecot, I must be the problem! > > My problem statement is simply, "it should be working", but doesn't, and I > don't get any announcement of "auth" when testing connections to Postfix > as per directions here: > > http://www.postfix.org/SASL_README.html#server_test > > At least I haven't broken the normal functionality! > > I'm building a new server on the latest Fedora Core (16), but it's lacking > in some hardware and won't be ready for a while, so I'm working with FC > 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth > scheme (which works fine and I'd hate to change it if I don't have to). > The system has been up and functional on these versions for a couple of > years, and quite stable, we just can't send if we're not local. > > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) > > First Dovecot. Its set up to provide all protocols, but only imaps and > pop3s have ports forwarded through the firewall. Plain-text auth is > disabled, ssl is set to yes, ssl_listen is not specified, and the cert and > key files are in the default locations - and work. No cipher list is used. > Dovecot's chrooted. The protocol sections imap and pop3 take ALL the > defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as > I don't think it matters. "auth default {" has mechanisms set to cram-md5, > digest-md5, plain, and login, with passdb passwd-file pointing to a file > in /etc where the cram data goes. It's not using pam, and there's an OLD > comment in the config: > > # Experience says we need an empty passdb - passwd group: > > which is followed by passdb passwd{}. Later, there's "userdb passwd {}. > > All of that was configured long ago and has been functional. > > The changes I've made to add sasl support primarily pertain to the "socket > listen section of "auth default". There, the master section remains > commented out while the client section has been uncommented, the path set > to /var/spool/postfix/private/auth, mode set to 0660, and the user and > group have been set to postfix. ...This is all as described here: > > http://www.postfix.org/SASL_README.html > and > http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL > > That's it for Dovecot. Now, to Postfix itself. > > >From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): > > smtpd_sasl_type = dovecot > smtpd_sasl_path = private/auth > smtpd_sasl_auth_enable = yes > broken_sasl_auth_clients = yes > smtpd_sasl_security_options = noanonymous, noplaintext > smtp_sasl_security_options = noanonymous, noplaintext > smtpd_sasl_tls_security_options = noanonymous > smtp_sasl_tls_security_options = noanonymous > > And, of course, permit_sasl_authenticated was added to > smtpd_recipient_restrictions. > > > I got the impression from the baove sources that Postfix will then use > Dovecot's authentication mechanism via a socket it finds in its > private/auth subdirectory. > > NOT documented in any of those places, someone suggested I must turn on > TLS. OK... > > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, however, there's ZERO budget for a signed certificate, so > unless I can get one for ten bucks somewhere, that could be a > deal-breaker here. However, we've been using self-signed certificates for > a while now and wonder why an "exception" mechanism wouldn't exist. As > that web page talks about "Netscape" I suspect it's very old and may no > longer apply. > > In any event, I tried this, too (after trying without). On the good side, > an available Android phone, previously reading fine, but unable to send, > no longer complained when the setup was changed to the imap username and > password, same server address, TLS security type, and the server port of > 25. HOWEVER, no mail has passed through it successfully, it just gives no > error whatsoever, so far, while the server's log reports "Relay access > denied." > > Notably, when setting up TLS, Postfix complained when the > smtpd_tls_key_file was incorrect, but did not complain when it was > provided properly, suggesting it's reading and accepting my self-signed > certificate and private key. > > Ideas, please?! > > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... > > > Thanks in advance for your help, > Richard > > -- Richard Troy, Chief Scientist Science Tools Corporation 510-717-6942 rtroy at ScienceTools.com, http://ScienceTools.com/ From terry at cnysupport.com Mon Mar 12 20:57:24 2012 From: terry at cnysupport.com (Terry Carmen) Date: Mon, 12 Mar 2012 14:57:24 -0400 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> Message-ID: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> On 03/04/2012 09:58 AM, Timo Sirainen wrote: > On 4.3.2012, at 16.48, Terry Carmen wrote: > >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >>> >>> If the prefix differs, but all of the exchange server names have >>> the same length, for example 10, you can also do: >>> >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >>> There's no otherwise nice way to parse this string. >> >> >> If by prefix, you mean the >> "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, >> they're different. > > OK, so if the prefix or suffix isn't always the same length you > can't do the above. > >> I could export the data to a text file as >> username:homeexchangeserver (or whatever other format is needed). >> >> homeservers.txt: >> user1:exch1.example.com >> user2:exch1.example.com >> user3:exch1.example.com >> user4:exch2.example.com >> >> Is it possible to do a lookup in a text file to get this? > > > If you can use userdb passwd-file and export the data to that file, > it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile > > Example line: > > user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com > > Note that you can't then return any userdb fields from passdb ldap lookup. That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. What would be perfect is if I could do something like this: //////////////////////////// http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying Proxy only server . . . In this document I assume that Dovecot is installed under /opt/dovecot, by default it is installed under /usr/local when compiling from source. Examples in this document are for MySQL but configs do not differ much with PostgreSQL. SQL table structure Create SQL table like CREATE TABLE proxy ( user varchar(255) NOT NULL, host varchar(16) default NULL, destuser varchar(255) default NULL, PRIMARY KEY (user) ); ////////////////////////////////////// All I really need is a way to lookup the user's home IMAP server when given the username, as above. Does imapproxy still support this 1.x feature? Thanks! Terry From CMarcus at Media-Brokers.com Mon Mar 12 21:02:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 12 Mar 2012 15:02:55 -0400 Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: References: Message-ID: <4F5E485F.40207@Media-Brokers.com> Since you got it working, I'll just comment on a couple of things... On Mon, 12 Mar 2012, Richard Troy wrote: > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) Correct... > From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): You really should separate AUTH to the port that is designed for it: port 587 (aka the 'submission' port/service)... just uncomment it (and its attendant lines) in master.cf > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, Where does it state any such thing? I've been using self-signed certs for 8+years with postfix... You do have to 'accept' the certs in the clients though, and that cn scare some users. I've had zero problems with this in Android, and none in recent versions of iOS, although earlier versions required you to install the cert manually (could be done using Safari on the iPhone)... Also, Outlook provides no simple way to Accept a Cert and store it permanently (Thunderbird does), so unless/until Outlook users import the Cert, they'll have to accept it each time they fire up Outlook and check mail. > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... It is the *deprecated* SMTPS (smtp over SSL). All modern clients can use the submission service, but some older versions of Outlook/Outlook Express can only use 465. It doesn't hurt anything to have it enabled, but you shoiuld absolutely tell all other clients to use the normal submissions service (STARTTLS on port 587). -- Best regards, Charles From andrei.michescu at miau.ca Tue Mar 13 07:41:53 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 01:41:53 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 Message-ID: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Hello, I'm using dovecot 2.1.1 with vpopmail 5.4.30 with multiples domains and I have problems setting up synchronization in between multiple computers. All act like master (my clients can connect to any of the them and read their emails either via POP3 either via IMAP, inbound email gets on any of the machines). Each machine is on a different continent, there is no shared drive in between and the synchronization is supposed to be asynchronous via cron scripts. To simplify the case, let's consider 2 machines (mx1.a and mx2.a) with 3 virtual domains (a and b and c). On both machine domain a is the default domain (needs only username to connect to imap). Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. I tried any of the following commands to synchronize the 2 machines: mx1.a$ doveadm -Dv sync -u user1 at a -f ssh mx2.a doveadm dsync-server -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a ssh mx2.a doveadm -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a user1 at a The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22). Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u and attempts to sync the first email in the default domain. Please advise. # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From nick.z.edwards at gmail.com Tue Mar 13 09:27:28 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 17:27:28 +1000 Subject: [Dovecot] upgrade convert omissions Message-ID: It did not convert over some things: protocol imap { listen = *:143 mail_plugins = quota imap_quota imap_client_workarounds = outlook-idle ssl_listen = *:993 } protocol pop3 { pop3_uidl_format = %f mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { mail_plugins = quota cmusieve quota_full_tempfail = no log_path = /var/log/dovecot/deliver.log deliver_log_format = msgid=%m: from=%f: %$ auth_socket_path = /var/run/dovecot/auth-master } I ended up with service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } It looks like service-foobar is replacing stuff inside protocol {} but protocol still exists in examples, kind of confusing. Oh what about service-pop3, different than examples and service-imap, where's pop3s ? I guess I'll just copy the service pop3 section from an example file. Does all this mean the protocol section is not needed? or everything inside of them I had is no longer valid? No LDA specific logging converted, it did not include the quota stuff in where it is needed (mail_plugins went MIA), but did inside converted plugin section. I did note the conversion warned that workarounds = outlook-idle is no longer needed, but nothing about the other stuff. Just concerns me if it ignored some needed stuff, what else did it ignore. Nik (who is rather reluctant to use series 2 and break everything and stay with 1.2) Below is the new conf file: auth_cache_negative_ttl = 0 auth_cache_ttl = 5 mins auth_mechanisms = plain login auth_verbose = yes disable_plaintext_auth = no first_valid_uid = 95 last_valid_uid = 95 listen = *,:: log_path = /var/log/dovecot/pop3.log login_log_format_elements = user=<%u> method=%m rip=%r %c mail_location = maildir:/vmail/%d/%n/Maildir mail_nfs_index = yes mail_nfs_storage = yes maildir_very_dirty_syncs = yes mmap_disable = yes passdb { args = /etc/dovecot-sql.conf driver = sql } plugin { quota = maildir quota_rule = *:storage=1000M quota_rule2 = Trash:storage=100M quota_rule3 = Junk:ignore quota_rule4 = Spam:ignore quota_warning = storage=90%% /usr/local/bin/quotawarn-90.sh quota_warning2 = storage=75%% /usr/local/bin/quotawarn-75.sh } pop3_lock_session = yes protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } user = vmail } service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } shutdown_clients = no ssl_cert = References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Message-ID: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> On 13.3.2012, at 7.41, Michescu Andrei wrote: > Initially I synchronize mx1.a with mx2.a using rsync. I check that I can > login using dovecot. .. > The only thing that happens is that the on each machine the folders get > doubled with some random extension (eg. Inbox becomes > Inbox_3e3ff3g3gb3bb3b22). This is kind of a feature. Currently if two mailboxes have a same name, but different GUID, dsync doesn't even try to merge them but instead renames one of them. So don't do initial sync with rsync, but with dsync. Alternatively you need to first get each mailbox assigned a GUID, for example: doveadm -A mailbox status guid '*' > Also, another bug, if there is a domain setup as default > (auth_default_realm) dsync simply ignores the specified -u and > attempts to sync the first email in the default domain. That can't be possible, something else is happening. What does dsync and auth log with debugs enabled when this happens? From tss at iki.fi Tue Mar 13 09:56:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 09:56:12 +0200 Subject: [Dovecot] upgrade convert omissions In-Reply-To: References: Message-ID: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> On 13.3.2012, at 9.27, Nick Edwards wrote: > It did not convert over some things: .. What Dovecot version did you use? In my test it converted everything (v2.0.18). Copy&pasting your config to a new file, adding ssl_cert + ssl_key and it produces output that converted everything (although cmusieve should be replaced with sieve): service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } } ssl_cert = It looks like service-foobar is replacing stuff inside protocol {} but > protocol still exists in examples, kind of confusing. Services have replaced some settings, not protocol itself. http://wiki2.dovecot.org/Services > Oh what about service-pop3, different than examples and service-imap, You had explicitly set listen/ssl_listen only for imap, not for pop3, so that's what the conversion did. pop3 uses the defaults. > where's pop3s ? It's enabled by default. But pop3s isn't a really a "protocol", so it's no longer treated specially. > I guess I'll just copy the service pop3 section from > an example file. No need to, the defaults are fine. Same with service imap actually, you could just remove it. From tss at iki.fi Tue Mar 13 10:06:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 10:06:23 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: On 12.3.2012, at 20.57, Terry Carmen wrote: >> If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile >> >> Example line: >> >> user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com >> >> Note that you can't then return any userdb fields from passdb ldap lookup. > > That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. Well, you could allow users to log in with any password and then let it just fail later at imapc login, but that's a bit ugly. You could also use passdb imap {} + userdb passwd-file {} with some extra work. The authentication would be done against the remote imap server, while the userdb_imapc_host would be looked up from the passwd-file. > What would be perfect is if I could do something like this: > > //////////////////////////// > > http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying > Proxy only server .. > All I really need is a way to lookup the user's home IMAP server when given the username, as above. > > Does imapproxy still support this 1.x feature? This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy From nick.z.edwards at gmail.com Tue Mar 13 10:15:09 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 18:15:09 +1000 Subject: [Dovecot] upgrade convert omissions In-Reply-To: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> References: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> Message-ID: On 3/13/12, Timo Sirainen wrote: > On 13.3.2012, at 9.27, Nick Edwards wrote: > >> It did not convert over some things: > .. > > What Dovecot version did you use? In my test it converted everything 1.2.17 -> 2.1.1 > > protocol pop3 { > mail_plugins = quota > pop3_client_workarounds = outlook-no-nuls oe-ns-eoh > pop3_uidl_format = %f > } > protocol lda { > auth_socket_path = /var/run/dovecot/auth-master > deliver_log_format = msgid=%m: from=%f: %$ > log_path = /var/log/dovecot/deliver.log > mail_plugins = quota cmusieve > quota_full_tempfail = no > } > >> It l any idea why it never copied over the protocol stuff? > Services have replaced some settings, not protocol itself. > http://wiki2.dovecot.org/Services thanks > >> I guess I'll just copy the service pop3 section from >> an example file. > > No need to, the defaults are fine. Same with service imap actually, you > could just remove it. OK Thanks From schut at sarvision.nl Tue Mar 13 10:46:04 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 09:46:04 +0100 Subject: [Dovecot] invalid mailbox name Message-ID: Hi, while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Migration is from dovecot 1.1.11 to 1.2.15. Thanks, Vincent. From tss at iki.fi Tue Mar 13 11:00:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:00:19 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 13.3.2012, at 10.46, Vincent Schut wrote: > Hi, > > while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: > > ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) > > I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... > > Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as "&-". From schut at sarvision.nl Tue Mar 13 11:10:48 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:10:48 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 03/13/2012 10:00 AM, Timo Sirainen wrote: > On 13.3.2012, at 10.46, Vincent Schut wrote: > >> Hi, >> >> while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: >> >> ERROR: Folder 'Organisations.RS Env& IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env& IJRS']) >> >> I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... >> >> Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? > > Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. > > Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". > Timo, thanks for the quick follow-up. However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: - when I want to create a folder with a "&" using a imap client, I have to type "&-"? or: - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) or: - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? or did you mean something else altogether ("don't use folders with & in their names, they're evil")? Vincent. From tss at iki.fi Tue Mar 13 11:15:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:15:45 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <66BBAF62-FBC2-452C-8A97-0A505515EC7E@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: >> Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. >> >> Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". >> > > Timo, > > thanks for the quick follow-up. > However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: > > - when I want to create a folder with a "&" using a imap client, I have to type "&-"? no. > or: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) no. > or: > - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? yes. > or did you mean something else altogether ("don't use folders with & in their names, they're evil")? When user types "&", the IMAP client should translate it to "&-". Also in filesystem it should be shown as "&-" (although this will be configurable in future). On input IMAP client will of course also see it as "&-" and should translate it back to "&" before making it visible to user. offlineimap works correctly in that it doesn't really need to do any kind of translation or validation, since it was originally IMAP client's fault for creating it and secondarily Dovecot's fault for allowing its creation. Although I guess offlineimap could have detected that this is an invalid mailbox name and translated it to something valid (this is how Dovecot v2.x's dsync works). From tss at iki.fi Tue Mar 13 11:20:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:20:47 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the & character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change & in there to &-, it would end up showing broken to user. Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. From schut at sarvision.nl Tue Mar 13 11:33:59 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:33:59 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> References: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> Message-ID: On 03/13/2012 10:20 AM, Timo Sirainen wrote: > On 13.3.2012, at 11.10, Vincent Schut wrote: > >> - the imap client (offlineimap in this case) should translate the "&" into"&-" on the fly (I can do that, you can give folder translation functions in offlineimap) > > Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the& character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change& in there to&-, it would end up showing broken to user. > > Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. > > Thanks Timo. Everything's clear now. There are only a few of these, I'll just rename them (and their line in the subscriptions file). Vincent. From marcio.merlone at a1.ind.br Tue Mar 13 16:36:32 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 11:36:32 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage Message-ID: <4F5F5B70.1020304@a1.ind.br> Hi people, When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is find -type d -name .Archives\* -print0 | \ while read -d $'\0' archive; do \ mv "$archive" /dead/Emails/jhon.doe/; \ ln -s "/dead/Emails/jhon.doe/$archive" .; done It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? Thanks in advance, best regards. -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 18:49:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 18:49:54 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F5B70.1020304@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> Message-ID: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> On 13.3.2012, at 16.36, Marcio Merlone wrote: > Hi people, > > When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is > > find -type d -name .Archives\* -print0 | \ > while read -d $'\0' archive; do \ > mv "$archive" /dead/Emails/jhon.doe/; \ > ln -s "/dead/Emails/jhon.doe/$archive" .; done > > It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I'm not sure what you mean by locks. dovecot-uidlist.lock? Anyway, the above is safe only if the IMAP client doesn't try to access the mailboxes during the move. Otherwise it can become confused. > I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? A perfectly working solution would be to (upgrade to v2.x and) switch to sdbox or mdbox format with alt storage enabled, then you could simply do: doveadm altmove -A mailbox 'Archives*' all From btb at bitrate.net Tue Mar 13 19:56:08 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Tue, 13 Mar 2012 13:56:08 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: <20120312133849.GS24983@harrier.slackbuilds.org> References: <20120312133849.GS24983@harrier.slackbuilds.org> Message-ID: <467D7FF4-02AB-4AB2-B1BA-26D50CC5A145@bitrate.net> On Mar 12, 2012, at 09.38, /dev/rob0 wrote: > On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: >> the problem with this is that while each of the passdb/userdb >> configs for the various protocols does indeed work, if a result >> is not found in one of them, the global passdb appears to then >> function as a catch-all. >> >> how can i tell dovecot it doesn't need a global passdb? each >> of the protocols' passdb/userdb configs is functioning as >> desired, but having dovecot look elsewhere upon failure ends >> up defeating the purpose. > > A simple workaround: use an empty passwd-file passdb as global. thanks, yeah. i've got what effectively accomplishes the same thing - a global ldap passdb within which the search filter always returns nothing. it sure seems like there would be a more logical method than this though. i looked briefly at the static password database, but at the moment it's intended application appears to be the opposite. if there were some argument such as deny=y or similar, it could be used. but ultimately, it would seem to make more sense to be able to simply tell dovecot that it doesn't need a global passdb, since obviously it doesn't. -b From andrei.michescu at miau.ca Tue Mar 13 20:22:16 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 14:22:16 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: Hello, Thank you for your reply. I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1 at b but they belong to first_user at a (which is NOT involved in this sync), BUT a is the default domain. Also for the first suggestion: 1) how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids 2) if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine Thank you, Andrei > On 13.3.2012, at 7.41, Michescu Andrei wrote: > >> Initially I synchronize mx1.a with mx2.a using rsync. I check that I can >> login using dovecot. > .. >> The only thing that happens is that the on each machine the folders get >> doubled with some random extension (eg. Inbox becomes >> Inbox_3e3ff3g3gb3bb3b22). > > This is kind of a feature. Currently if two mailboxes have a same name, > but different GUID, dsync doesn't even try to merge them but instead > renames one of them. > > So don't do initial sync with rsync, but with dsync. Alternatively you > need to first get each mailbox assigned a GUID, for example: doveadm -A > mailbox status guid '*' > >> Also, another bug, if there is a domain setup as default >> (auth_default_realm) dsync simply ignores the specified -u and >> attempts to sync the first email in the default domain. > > That can't be possible, something else is happening. What does dsync and > auth log with debugs enabled when this happens? > > > !DSPAM:4f5efb4c315461389012818! > > -------------- next part -------------- A non-text attachment was scrubbed... Name: output_doveadm.odt Type: application/vnd.oasis.opendocument.text Size: 16377 bytes Desc: not available URL: From e-frog at gmx.de Tue Mar 13 20:42:41 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 19:42:41 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors Message-ID: <4F5F9521.2060206@gmx.de> Hello Timo, I'm experimenting with 'doveadm backup' on 2.1.1 (latest hg, full dovecot -n output attached) and haven't managed to get it working. This is what I have done: 1. Create the directory /tmp/backup which is empty 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ Then I see the following errors: doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in dest (guid=9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=bd05451f2fbb574d40600000ec8d17cd) dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox 9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox bd05451f2fbb574d40600000ec8d17cd dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) It somehow finds INBOX in destination however the backup directory is newly created and empty. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: doveconf-n.txt URL: From tss at iki.fi Tue Mar 13 20:51:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 20:51:13 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> On 13.3.2012, at 20.22, Michescu Andrei wrote: > I'm attaching you the output for the 2nd bug. All the folders that you see > in there does not exists in user1 at b but they belong to first_user at a (which > is NOT involved in this sync), BUT a is the default domain. The output showed debug output from the local dsync, but not from the remote. I think you'll see that if you do: sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv dsync-server -u user1 at b Also it's possible that in v2.1.1 there was some bug related to this.. You could try the latest nightly snapshot that has several fixes related to dsync: http://www.dovecot.org/nightly/ > Also for the first suggestion: > > 1) how do you sync initially the 2 machines? Because if you create the > account on both machines, already the Inbox has 2 different guids doveadm sync should be run before the destination Maildir exists at all. If vpopmail creates that, I guess it would just have to be deleted manually.. > 2) if you know the guid, how do you change them? Because then I can do the > rsync and after I can correct the guid on the other machine The GUID is generated the first time it's used, which normally means when you run dsync for the first time. Alternatively you can also use doveadm to ask for the mailbox's GUID and it gets generated: doveadm mailbox status -u user at domain guid '*' Running rsync after this is done also copies the GUID (it's stored in dovecot-uidlist). From marcio.merlone at a1.ind.br Tue Mar 13 20:58:59 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 15:58:59 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> Message-ID: <4F5F98F3.1090601@a1.ind.br> Em 13-03-2012 13:49, Timo Sirainen escreveu: > On 13.3.2012, at 16.36, Marcio Merlone wrote: >> It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. > I'm not sure what you mean by locks. Operating system lock of open files (lsof). >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Sounds really nice. There are no 2.0 packages for Lucid tough, so will take a look at the latest stable Ubuntu, or perhaps give the beta a try (until it comes stable). Thanks for your prompt reply. :) -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 21:16:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 21:16:54 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5F9521.2060206@gmx.de> References: <4F5F9521.2060206@gmx.de> Message-ID: On 13.3.2012, at 20.42, e-frog wrote: > This is what I have done: > 1. Create the directory /tmp/backup which is empty > 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ .. > Then I see the following errors: > > doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. Try without mailbox_list_index=yes From e-frog at gmx.de Tue Mar 13 22:19:53 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 21:19:53 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: References: <4F5F9521.2060206@gmx.de> Message-ID: <4F5FABE9.3080200@gmx.de> On 13.03.2012 20:16, wrote Timo Sirainen: > On 13.3.2012, at 20.42, e-frog wrote: > >> This is what I have done: >> 1. Create the directory /tmp/backup which is empty >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > .. >> Then I see the following errors: >> >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > Try without mailbox_list_index=yes Yes, after disabling list indexes it works. From terry at cnysupport.com Tue Mar 13 23:44:40 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 17:44:40 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: <4F5FBFC8.3060306@cnysupport.com> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a regular dummy proxying setup. Sure you could still do > that, but it's not imapc proxying. > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. Thanks for the help! Terry From tss at iki.fi Wed Mar 14 00:05:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 00:05:14 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F5FBFC8.3060306@cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> Message-ID: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> On 13.3.2012, at 23.44, Terry Carmen wrote: > On 03/13/2012 04:06 AM, Timo Sirainen wrote: >> This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. If you find out that IMAP clients still don't work nicely with Exchange (apparently they have random problems, especially with shared mailboxes/accounts), you can still put imapc proxy in front of your currently working Dovecot proxy. :) From terry at cnysupport.com Wed Mar 14 00:29:09 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 18:29:09 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> Message-ID: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> ----- Message from Timo Sirainen --------- ? ? Date: Wed, 14 Mar 2012 00:05:14 +0200 ? ? From: Timo Sirainen Reply-To: Dovecot Mailing List Subject: Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location ? ? ? To: Terry Carmen ? ? ? Cc: dovecot at dovecot.org > On 13.3.2012, at 23.44, Terry Carmen wrote: >> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a >> regular dummy proxying setup. Sure you could still do that, but >> it's not imapc proxying. >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy >> The above URL worked beautifully and Dovecot is now running as a >> proxy for a dozen older Exchange servers on a private network. > If you find out that IMAP clients still don't work nicely with > Exchange (apparently they have random problems, especially with > shared mailboxes/accounts), you can still put imapc proxy in front > of your currently working Dovecot proxy. :) I'm going to hope everything is OK for a while, since my goal is to retire all the old Exchange servers and move all the users to dovecot/maildir within the next couple of months. However it's always nice to know there are options. 8-) Terry From andrei.michescu at miau.ca Wed Mar 14 07:25:09 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 01:25:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: Hello, Thank you very much... Using the nightly build and a combination of mailbox status + rsync + dsync made it happen. So the *full* procedure was: for every domain dom for every user u in dom doveadm mailbox status -u user at domain guid '*' rsync /home/vpopmail/domains/$dom/$u mx2.a:/home/vpopmail/domains/$dom/$u doveadm -Dv sync -u $u@$dom -f ssh mx2.a doveadm dsync-server -u $u@$dom loop $u loop $dom Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes). The problem comes when I start using the master-master model: emails starts getting duplicate with different ids. Here is the example: mx1.a receives an email for user1 (next line is the ls on the Maildir/new on mx1.a): -rw------- 1 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 mx2.a receives another email for user1: -rw------- 1 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx2,S\=273 cron job starts on mx1.a: sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 dsync-local(user1 at b): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at b): Info: INBOX: Couldn't keep all uids dsync-local(user1 at b): Warning: Mailbox changes caused a desync. You may want to run dsync again. --due to desync it runs again (via the cron script): sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 The expected result would be that in user1's Mailbox on both mx1.a and mx2.a there would be 2 files... which is not happening ls on mx1.a -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 -rw------- 1 vpopmail vchkpw 273 Mar 14 01:05 1331701504.32564.mx2,S\=273 -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331702193.M868989P24524.mx1,S\=278 ls on mx2.a -rw------- 1 vpopmail vchkpw 278 Mar 14 07:04 1331701451.24233.mx1,S\=278 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx1,S\=273 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331702193.M798223P32571.mx2,S\=273 As you can see on every machine the original email gets duplicated. Please advise on how I can fix this issue. Thank you and Best regards, Andrei > On 13.3.2012, at 20.22, Michescu Andrei wrote: > >> I'm attaching you the output for the 2nd bug. All the folders that you >> see >> in there does not exists in user1 at b but they belong to first_user at a >> (which >> is NOT involved in this sync), BUT a is the default domain. > > The output showed debug output from the local dsync, but not from the > remote. I think you'll see that if you do: > > sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv > dsync-server -u user1 at b > > Also it's possible that in v2.1.1 there was some bug related to this.. You > could try the latest nightly snapshot that has several fixes related to > dsync: http://www.dovecot.org/nightly/ > >> Also for the first suggestion: >> >> 1) how do you sync initially the 2 machines? Because if you create the >> account on both machines, already the Inbox has 2 different guids > > doveadm sync should be run before the destination Maildir exists at all. > If vpopmail creates that, I guess it would just have to be deleted > manually.. > >> 2) if you know the guid, how do you change them? Because then I can do >> the >> rsync and after I can correct the guid on the other machine > > The GUID is generated the first time it's used, which normally means when > you run dsync for the first time. Alternatively you can also use doveadm > to ask for the mailbox's GUID and it gets generated: > > doveadm mailbox status -u user at domain guid '*' > > Running rsync after this is done also copies the GUID (it's stored in > dovecot-uidlist). > !DSPAM:4f5f972f80146209382307! > > From CMarcus at Media-Brokers.com Wed Mar 14 12:58:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 06:58:54 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> Message-ID: <4F6079EE.4000201@Media-Brokers.com> On 2012-03-13 6:29 PM, Terry Carmen wrote: > I'm going to hope everything is OK for a while, since my goal is to retire > all the old Exchange servers and move all the users to dovecot/maildir > within the next couple of months. > > However it's always nice to know there are options. 8-) I'm currently looking at rolling out SOGo as part of a major reworking of their current infrastructure (will also include converting their old Courier-IMAP to dovecot 2.1.x among other things)... SOGo, as far as I can tell, is the best truly free and open source 'exchange clone' available that works extremely well with Thunderbird+Lightning (which is what my Client uses currently, but they are very dissatisfied with using Google Calendar for Shared calendars), Outlook and Apple Apps, as well as Android, Blackberry and Apple mobile devices - and their upcoming v2 (in beta now) will not only provide native Outlook support (no plugin needed), it will also (optionally) provide a Samba4 Active Directory server in my main Client's office - all with absolutely no licenses required. Commercial support is available from Inverse, the company created by the developers to provide said support services. I also learned something very interesting yesterday concerning SOGo and dovecot during a sales call with a SOGo rep, but I'll wait and see if Timo cares to chime in on this one... ;) -- Best regards, Charles From marcio.merlone at a1.ind.br Wed Mar 14 13:51:30 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 08:51:30 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F98F3.1090601@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> Message-ID: <4F608642.5060707@a1.ind.br> Em 13-03-2012 15:58, Marcio Merlone escreveu: > Em 13-03-2012 13:49, Timo Sirainen escreveu: >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Should this command be run every time a new folder (.Archives.2012 for example) is created or can this be automated, something like instructing deliver to do this to any Archives* is created? In other words, to cron or not to cron? Best regards. -- *Marcio Merlone* From tss at iki.fi Wed Mar 14 14:19:31 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 14:19:31 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F608642.5060707@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> Message-ID: <1331727571.2081.126.camel@innu> On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: > Em 13-03-2012 15:58, Marcio Merlone escreveu: > > Em 13-03-2012 13:49, Timo Sirainen escreveu: > >> A perfectly working solution would be to (upgrade to v2.x and) switch > >> to sdbox or mdbox format with alt storage enabled, then you could > >> simply do: doveadm altmove -A mailbox 'Archives*' all > Should this command be run every time a new folder (.Archives.2012 for > example) is created or can this be automated, something like instructing > deliver to do this to any Archives* is created? In other words, to cron > or not to cron? It's not deliver that creates it, it's the IMAP client. And it would have to be done when IMAP client copies mails there. But there's currently no easy way to automate that, so you'd need to do it in cron. Another possibility could be to add a new feature (plugin) that always immediately saves mails in Archives* mailboxes to alt storage. From tss at iki.fi Wed Mar 14 15:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 15:41:30 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5FABE9.3080200@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> Message-ID: <1331732490.2081.127.camel@innu> On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: > On 13.03.2012 20:16, wrote Timo Sirainen: > > On 13.3.2012, at 20.42, e-frog wrote: > > > >> This is what I have done: > >> 1. Create the directory /tmp/backup which is empty > >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > > .. > >> Then I see the following errors: > >> > >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > > > Try without mailbox_list_index=yes > > Yes, after disabling list indexes it works. With latest hg version it should work. From tss at iki.fi Wed Mar 14 16:09:49 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:09:49 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: <1331734189.2081.137.camel@innu> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync > keeps them in sync (cron job every 5 minutes). > > The problem comes when I start using the master-master model: emails > starts getting duplicate with different ids. I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign), but anyway: Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers. Each time dsync notices that both sides have had new mails, it needs to reassign new IMAP UIDs for the messages, which means that IMAP clients may need to redownload the mails. Better to give one MX a higher priority so mails typically are delivered through it. From tss at iki.fi Wed Mar 14 16:29:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:29:15 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: Message-ID: <1331735355.2081.140.camel@innu> On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: > Heya, > > We are expiriencing issues with dovecot 2.1.1 on Linux with weird > filenames in home directory of username. We are using mbox IMAP > folders, with no special changes (mail_location = mbox:~/:INBOX=% > h/.mailbox). > > Mar 6 13:37:17 machine dovecot: imap(username): Panic: file > mail-storage.c: line 628 (mailbox_alloc): assertion failed: > (uni_utf8_str_is_valid(vname)) .. > AFAIK, the problem lies in processing the file list of home folder, > which can contain filenames that do not have proper UTF-8 encoding of > filenames, which causes dovecot to crash. Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This should fix it by renaming such mailboxes: http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 > On the other hand, UTF-8 filenames created on the system by hand > (using touch), are not displayed in IMAP LIST command (sample is > included in the folder structure; single letter file). This is a bit trickier problem. The mailbox names are currently stored in filesystem as IMAP's modified-UTF7. So it's not really even currently supposed to work, although it's not very nice that the mailboxes aren't visible either. Maybe I'll do something smart in future for this, like allowing both mUTF-7 and UTF-8 and remembering per-mailbox which formatting it is in. From tss at iki.fi Wed Mar 14 16:40:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:40:05 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F55BA0C.5090606@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> <4F55BA0C.5090606@one.com> Message-ID: <1331736005.2081.144.camel@innu> On Tue, 2012-03-06 at 08:17 +0100, Peter Mogensen wrote: > On 2012-03-05 16:36, Timo Sirainen wrote: > >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > > > Neither behavior is wrong, just different. :) > > Ok... I were in doubt if I had missed something from the RFC. > However... for testing, I tried to create "INBOX.INBOX" on dovecot. > But then dovecot answers NO and complains that the folder already > exists. Though it's still not on disk and dovecot still doesn't list it > with "*". This is a bit problematic when you have prefix="INBOX." namespace. There if you access "INBOX.INBOX", its internal storage name is "INBOX". And INBOX's internal storage name is also "INBOX". So in some parts of the code they are treated as if they were both the same mailbox. Maybe I can get this fixed for v2.2. From campbell at cnpapers.com Wed Mar 14 16:46:58 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 10:46:58 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions Message-ID: <4F60AF62.80702@cnpapers.com> I've mostly finished a conversion from an old Centos 3 UW-Imap server to a new Centos 6 dovecot server. I did not copy the old ~/.mailboxlist file to ~/mail/.subscriptions file, but notice some users have the latter file now. These are all mbox folders on the old and new server. I'm getting ready to do the same to another old/new pair of servers and I'm wondering if there is an advantage of doing the copy. I'm assuming the .subscription files are created when they access their account through our webmail application, but I'm not sure if it was automatic or due to a "subscribe" action done manually. There are 49 accounts with a .mailboxlist file and only 4 with the new .subscriptions file. So either our webmail application isn't being used a lot or there's a problem with it due to the missing .subscriptions file, but the phone usually rings pretty quickly when problems arise. One last question, please. Over the years, some imap accounts had their folders directly in their home directory and the contents of the .mailboxlist file would have an entry with just the name of the folder in it (Trash, eg), and most had the folders in their ~/mail folder with an entry like "mail/Trash". Our webmail app, Horde/Imp, always seemed to take care of this. If I create the .subscription file for the users during the move to the new server, should I move the folders to the mail directory and amend their .subscriptions file to reflect that change on these odd ball accounts, and will that affect how their client is seeing these? The first server conversion was a bear due to my lack of dovecot knowledge. I've since learned a little more, and mostly found out that dovecot is a more complex application than the old imap application. There's so much more that can be done with dovecot, whereas the old imap server was mostly just load-and-go. Seems like no matter how much I read, the more I discovered I didn't know. Anyway, thanks for all the past help and any opinions anyone might decide to offer on this post. steve campbell From marcio.merlone at a1.ind.br Wed Mar 14 17:00:49 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 12:00:49 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <1331727571.2081.126.camel@innu> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> <1331727571.2081.126.camel@innu> Message-ID: <4F60B2A1.7060104@a1.ind.br> Em 14-03-2012 09:19, Timo Sirainen escreveu: > On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: >> Em 13-03-2012 15:58, Marcio Merlone escreveu: >>> Em 13-03-2012 13:49, Timo Sirainen escreveu: >>>> A perfectly working solution would be to (upgrade to v2.x and) switch >>>> to sdbox or mdbox format with alt storage enabled, then you could >>>> simply do: doveadm altmove -A mailbox 'Archives*' all >> Should this command be run every time a new folder (.Archives.2012 for >> example) is created or can this be automated, something like instructing >> deliver to do this to any Archives* is created? In other words, to cron >> or not to cron? > It's not deliver that creates it, it's the IMAP client. And it would I first though about deliver since this seems the guy who could do that, but don't matter. > have to be done when IMAP client copies mails there. But there's > currently no easy way to automate that, so you'd need to do it in cron. Just imagine that: protocol imap{ ... x_alternate_storage_always = Archives, Spam, Trash x_alternate_storage_size = 20MB x_alternate_storage_age = 1y ... } The client (Thunderbird?) sends imap commands to dovecot create such folder or to move a message to such folder. Dovecot obeys and check the config to see if the folder in question is one of those listed on x_alternate_storage_always. If the folder is not on alternate storage yet, create/move it there and them proceed what was asked to do. I imagine that an age based action would depend of a cron job in order to not overload the server each time it performs any imap command and check old messages age... > Another possibility could be to add a new feature (plugin) that always > immediately saves mails in Archives* mailboxes to alt storage. I don't know if the duck is male, I want the egg! :) I know nothing about the inner workings of dovecot, and very little about the outer working. A well designed and integrated plugin could be, but seems to be a nice core feature for dovecot, based on folder name, size or age. Many thanks for your time. Please be kind considering those ideas, I am just a poor man's server admin. :) Best regards. -- *Marcio Merlone* From campbell at cnpapers.com Wed Mar 14 17:48:43 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 11:48:43 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60BDDB.8000103@cnpapers.com> On 3/14/2012 10:46 AM, Steve Campbell wrote: > > > One last question, please. > > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". > Our webmail app, Horde/Imp, always seemed to take care of this. If I > create the .subscription file for the users during the move to the new > server, should I move the folders to the mail directory and amend > their .subscriptions file to reflect that change on these odd ball > accounts, and will that affect how their client is seeing these? > > The first server conversion was a bear due to my lack of dovecot > knowledge. I've since learned a little more, and mostly found out that > dovecot is a more complex application than the old imap application. > There's so much more that can be done with dovecot, whereas the old > imap server was mostly just load-and-go. Seems like no matter how much > I read, the more I discovered I didn't know. > > Anyway, thanks for all the past help and any opinions anyone might > decide to offer on this post. > > steve campbell I've discovered another situation. This may not be a problem, but I've got to deal with it at any rate. I find that some users have a .mailboxlist which points to folders in their home directory, and have folders in their mail directory as well. For the most part, this situation involved horde/imp "sent-mail" folders which are created when users send mail through our webmail but they more than likely have a client on either their phone or desktop that is configured as imap. The horde/imp "sent-mail" is not listed in their .mailboxlist file. So I'm guessing this will be all right to leave as is or to modify the resultant .subscription file to point to moved folders. So many things to consider for so many different situations. Thanks steve > > From terry at cnysupport.com Wed Mar 14 17:59:26 2012 From: terry at cnysupport.com (Terry Carmen) Date: Wed, 14 Mar 2012 11:59:26 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F60C05E.1090803@cnysupport.com> On 03/14/2012 06:58 AM, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. Looks interesting. I have currently have horde/imp/kronolith running with postfix/dovecot/mysql on the back end and it's been working nicely with all the clients and devices except for outlook. I'll have to take a look at sogo, because I'd really like to keep outlook for the users that want it, to cut down on support and complaints. Thanks! Terry From arnaud.abelard at univ-nantes.fr Wed Mar 14 18:06:32 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Wed, 14 Mar 2012 17:06:32 +0100 Subject: [Dovecot] RECENT status always 0 Message-ID: <4F60C208.6010304@univ-nantes.fr> Hello, we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) then 2 minutes later: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) How can the UNSEEN count change without the RECENT count changing accordingly? Thanks in advance, Arnaud -- Arnaud Ab?lard (jabber: arnaud.abelard at univ-nantes.fr) Administrateur Syst?me - Responsable Services Web Direction des Syst?mes d'Informations Universit? de Nantes - ne pas utiliser: trapemail at univ-nantes.fr From tss at iki.fi Wed Mar 14 18:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:27:09 +0200 Subject: [Dovecot] RECENT status always 0 In-Reply-To: <4F60C208.6010304@univ-nantes.fr> References: <4F60C208.6010304@univ-nantes.fr> Message-ID: <1AE988E0-F799-45AA-A098-A3462DC13340@iki.fi> On 14.3.2012, at 18.06, Arnaud Ab?lard wrote: > Hello, > > we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) > > then 2 minutes later: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) > > How can the UNSEEN count change without the RECENT count changing accordingly? If any client has the INBOX opened, the recent count for other connections stays at 0. I guess you're expecting recent flags to work differently than how IMAP RFC specifies them. (It's entirely possible that Courier implemented them in the wrong way.) From jom at grosjo.net Wed Mar 14 18:36:55 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 22:21:55 +0545 Subject: [Dovecot] FTS crash Message-ID: Hi Timo, I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. When I remove the plugin from the dovecot.conf, all works fine. No debug available here, but let me know how can I help you. Joan From andrei.michescu at miau.ca Wed Mar 14 18:45:07 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 12:45:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Hello, Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment. Servers are having different priorities, but this was a basic scenario to test the master-master synchronization. Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster... One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options: - the DB model = each server has a set of ids that can give (either ranges, either increment with step different then 1) - the vpopmail/qmail model = append the server name (as you saw in the previous email in the listings the email files contained the hostname mx1.a and mx2.a) Thank you, Andrei > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync >> keeps them in sync (cron job every 5 minutes). >> >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Each time dsync > notices that both sides have had new mails, it needs to reassign new > IMAP UIDs for the messages, which means that IMAP clients may need to > redownload the mails. Better to give one MX a higher priority so mails > typically are delivered through it. > > > > !DSPAM:4f60a6b137151972926802! > > From tss at iki.fi Wed Mar 14 18:56:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:56:58 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: On 14.3.2012, at 18.45, Michescu Andrei wrote: > Nope dsync was not running during the email delivery on that account. I've > simulated in a controlled environment. How? You mean simply deliver mail to server A and to server B and run dsync and it duplicates it? I can't reproduce it that way, only if I run dsync during a flood of new mails. > Think that for incoming SMTP I can even restrict which server is the > master (forcing all other to redeliver to this one). BUT, for a > distributed IMAP cluster there is no way to restrict users to perform > changes on only one server. This would defeat the model and the purpose of > a distributed cluster... For IMAP it's not much of a problem, because user typically still uses only one client actively, so clients aren't uploading mails to multiple servers at the same time. > One idea might be to have the IDs dependent on server where they appear > first time so that they keep the ID once they get replicated. Here there > are many options: The messages have GUIDs that stay the same always, but IMAP UIDs are required to be ascending from client's point of view, and several clients rely on that, so when UID conflict happens the only safe thing to do is to assign new UIDs for all of the conflicting mails. From tss at iki.fi Wed Mar 14 18:58:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:58:36 +0200 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: On 14.3.2012, at 18.36, Joan Moreau wrote: > I updated my 2.1 from HG, and now the FTS (Squat) plugin > makes a segmentation fault. To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > No debug available here, but let me know > how can I help you. gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From CMarcus at Media-Brokers.com Wed Mar 14 19:00:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 13:00:25 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60CEA9.3080008@Media-Brokers.com> On 2012-03-14 10:46 AM, Steve Campbell wrote: > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes... dovecot doesn't like it when stuff other than mail is in the home folder: http://wiki2.dovecot.org/VirtualUsers/Home -- Best regards, Charles From andrei.michescu at miau.ca Wed Mar 14 19:26:42 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 13:26:42 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: <2c95e6565a8b3783b05b2eafc2d4833e.squirrel@web.miau.ca> > On 14.3.2012, at 18.45, Michescu Andrei wrote: > >> Nope dsync was not running during the email delivery on that account. >> I've >> simulated in a controlled environment. > > How? You mean simply deliver mail to server A and to server B and run > dsync and it duplicates it? I can't reproduce it that way, only if I run > dsync during a flood of new mails. > YES. simply deliver mail to server A and then to server B (to the same user_1). After run dsync and you get exactly what you saw in my previous email. That's why I included the ls for the both servers, so that you can see the email files too. because each server duplicates only its own email (so brings the email from the other server and duplicates its own email). >> Think that for incoming SMTP I can even restrict which server is the >> master (forcing all other to redeliver to this one). BUT, for a >> distributed IMAP cluster there is no way to restrict users to perform >> changes on only one server. This would defeat the model and the purpose >> of >> a distributed cluster... > > For IMAP it's not much of a problem, because user typically still uses > only one client actively, so clients aren't uploading mails to multiple > servers at the same time. > hehe... one would think so, but when you have road-warriors that roam you can not insure that the server where they connect for IMAP (closest based on geo-ip) is the same as the server that you have picked for inbound SMTP. So you already have 2 servers that mess-up with user's mailbox. The second case where you can not control this is for mobile devices that flip-in/out of wi-fi (my iPhone is in Canada when it is on 3G and in Europe when it is WiFi due to vpn tunneling, and this can change every couple of minutes... :( ) >> One idea might be to have the IDs dependent on server where they appear >> first time so that they keep the ID once they get replicated. Here there >> are many options: > > The messages have GUIDs that stay the same always, but IMAP UIDs are > required to be ascending from client's point of view, and several clients > rely on that, so when UID conflict happens the only safe thing to do is to > assign new UIDs for all of the conflicting mails. well I don't know much about IMAP standard (you guys are the experts :)! here). If the GUID stays the same then this can be used to prevent the duplication error. Also, as you can detect if the email is new or not (a client has already seen it or not): in the case that no one has seen it then it is safe to assign any UIDs that fits. In case that on only one server it has been seen then you can give it that UIDs on all servers, and reassign all the unseen ones. So the only messed-up case is if on both servers the message has been seen with different UIDs :( Thank you very much for your time and patience. I know that our setup is pretty atypical. And think that this model with only 2 servers I'm showing you is only for simplicity as the real deployment has multiple servers geographically sparse connected by slow intercontinental internet links... :)) Otherwise we'd use a distribute file system and have only a unified storage :P Best regards. Andrei From jom at grosjo.net Wed Mar 14 19:31:47 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 23:16:47 +0545 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: I have been unlucky in deed. Problem solved with recent changes Le 14/03/2012 22:43, Timo Sirainen a ?crit : > On 14.3.2012, at 18.36, Joan Moreau wrote: > >> I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. > > To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > >> No debug available here, but let me know how can I help you. > > gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From e-frog at gmx.de Wed Mar 14 21:33:49 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 14 Mar 2012 20:33:49 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <1331732490.2081.127.camel@innu> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> Message-ID: <4F60F29D.2010409@gmx.de> On 14.03.2012 14:41, wrote Timo Sirainen: > On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: >> On 13.03.2012 20:16, wrote Timo Sirainen: >>> On 13.3.2012, at 20.42, e-frog wrote: >>> >>>> This is what I have done: >>>> 1. Create the directory /tmp/backup which is empty >>>> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>> .. >>>> Then I see the following errors: >>>> >>>> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>>> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. >>> >>> Try without mailbox_list_index=yes >> >> Yes, after disabling list indexes it works. > > With latest hg version it should work. > Hi Timo, The "can't delete mailbox INBOX" error is gone now with changeset c077ca9bc306 and it's working successfully on the account from yesterday where it also worked with mailbox_list_index=no. However using a different account (more mail and mailboxes) I'm seeing dbox corruption errors. I have tested with mailbox_list_index=yes and no and it's the same for both. So this might be unrelated to this setting. Attached are logs from doveadm backup runs. First to an empty directory and 2 consecutive runs. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: log.txt URL: From campbell at cnpapers.com Wed Mar 14 21:53:00 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 15:53:00 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60CEA9.3080008@Media-Brokers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> Message-ID: <4F60F71C.8090306@cnpapers.com> On 3/14/2012 1:00 PM, Charles Marcus wrote: > On 2012-03-14 10:46 AM, Steve Campbell wrote: >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > > Yes... dovecot doesn't like it when stuff other than mail is in the > home folder: > > http://wiki2.dovecot.org/VirtualUsers/Home I'm not sure these are virtual users, so that link may have confused me. All accounts on these servers have real unix accounts. Their inbox is /var/spool/mail/unix-user-name. Their imap folders, the ones that they create using an imap client or webmail, are either in ~ or ~/mail. Their original .mailboxlist is always in ~. Based on that, I should probably copy any imap folders not in ~/mail to that folder, duplicate ~/.mailboxlist to the file ~/mail/.subscriptions, and amend any .subscriptions file contents to just have the name of the folders (without any "mail/folder" reference in it). My example would then be as follows /home/steve = folder /home/steve/Drafts = original folder /home/steve/AnyFolder = original folder /home/steve/.mailboxlist = original file /home/steve/mail = folder (either original or created) /home/steve/mail/.subscriptions = copied contents of .mailboxlist file /home/steve/mail/Drafts = copied folder of original /home/steve/mail/AnyFolder = copied folder of original Contents of original .mailboxlist and new .subscriptions: Drafts AnyFolder If the imap folders were in ~/mail, then the original .mailboxlist would have been mail/Drafts mail/AnyFolder but after the corrections to the .subscriptions file, they would be as above (without reference to the mail folder). Is this correct? thanks for the help steve From trashcan at odo.in-berlin.de Wed Mar 14 22:36:30 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 21:36:30 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Hi -- On 14.03.2012, at 15:09, Timo Sirainen wrote: > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), Good to hear ;-) > but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ... > Better to give one MX a higher priority so mails typically are delivered > through it. That's what I did. Now dsync/replicator is performing great, if the mail volume is rather low. I'm very satisfied, because this is the best performance ever. (Before I was running unison and dsync 2.0.) But, whenever the high priority server will show delays during stress situations like huge mail loads, the low priority server will receive loads of mails as well. A dsync/replicator setup will then most probably produce duplicates (and multiples). That is a rather unrealistic scenario for my little severs, but others might have more difficulties. And spammers don't care about mx priorities at all :-( Regards, Michael From tss at iki.fi Wed Mar 14 23:14:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 23:14:10 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: On 14.3.2012, at 22.36, Michael Grimm wrote: > And spammers don't care about mx priorities at all :-( But spams go to spam mailbox where duplicates don't really matter. :) From trashcan at odo.in-berlin.de Wed Mar 14 23:26:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:26:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Hi - On 14.03.2012, at 22:14, Timo Sirainen wrote: > On 14.3.2012, at 22.36, Michael Grimm wrote: >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) True ;-) But spam mails might interfere with syncing of legitimate mail arriving at the same time. Regards, Michael From trashcan at odo.in-berlin.de Wed Mar 14 23:30:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:30:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Message-ID: <5EE60F3E-A25F-4E6D-98E9-2246B75ACA10@odo.in-berlin.de> Hi -- On 14.03.2012, at 22:26, Michael Grimm wrote: > But spam mails might interfere with syncing of legitimate > mail arriving at the same time. Forget about that part, I was wrong because duplicates are produced in corresponding mailboxes, only. Sorry for the noise, Michael From andrei.michescu at miau.ca Wed Mar 14 23:32:40 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 17:32:40 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: > On 14.3.2012, at 22.36, Michael Grimm wrote: > >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) In an ideal world yes... or no. In our deployment spam is simply header tagged and left in INBOX. Each user can decide after what they want to do with it (client side rules). And, in the end, it is the same discussion, because the spam mailbox get replicated too and if the spam gets duplicated we are in the worst scenarios... knowing that spam represents 95% of all email traffic (in a real-world public-facing system). ;) hehe... in the meanwhile I looked a little on the ietf and there are different RFCs out there on disconnected clients and UIDPLUS and other nice features ;) let me know if you are interested to get some help in implementing it :D From andrei at lctax.ro Wed Mar 14 22:58:09 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Wed, 14 Mar 2012 16:58:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> > Hi -- > > On 14.03.2012, at 15:09, Timo Sirainen wrote: >> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > >>> The problem comes when I start using the master-master model: emails >>> starts getting duplicate with different ids. > >> I was testing this a bit, and I guess in your tests dsync was running >> during a mail delivery, which seems to make it duplicate mails >> sometimes. I'll probably fix this at some point (I've actually been >> thinking about a larger dsync redesign), > > Good to hear ;-) > >> but anyway: >> >> Even if dsync worked perfectly and didn't duplicate mails, it's not a >> great idea to do deliver mails randomly to both servers. > > Sometimes croncobs are running on both servers at the same time > producing locally delivered mails simultaneously, though. Ok, one > can modify run times accordingly ... Why do you run the crontab on all the servers? You can run a start-based system where only one ("main"-master) syncs all the other masters. And like this you avoid the time synch'ing of crontabs (especially if you don't always know how longer it will take for a dsync to finish). > >> Better to give one MX a higher priority so mails typically are delivered >> through it. > > That's what I did. Now dsync/replicator is performing great, if the > mail volume is rather low. I'm very satisfied, because this is the > best performance ever. (Before I was running unison and dsync 2.0.) > > But, whenever the high priority server will show delays during stress > situations like huge mail loads, the low priority server will receive > loads of mails as well. A dsync/replicator setup will then most probably > produce duplicates (and multiples). That is a rather unrealistic > scenario for my little severs, but others might have more difficulties. This is another nice case to "motivate" Timo to look for solutions ;) I tried to push 2 ideas in the same direction earlier :P Especially that he confirmed that every single email has a unique GID (which should help prevent duplication/multiplication)... > > And spammers don't care about mx priorities at all :-( Actually, statistically speaking, spammers select the low priority ones. > > Regards, > Michael > Nice to hear that we are not the only ones out there to try to run something like this over dovecot :P Thnx, Andrei From trashcan at odo.in-berlin.de Wed Mar 14 23:51:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:51:22 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> Message-ID: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Hi -- On 14.03.2012, at 21:58, Michescu Andrei wrote: >> Sometimes croncobs are running on both servers at the same time >> producing locally delivered mails simultaneously, though. Ok, one >> can modify run times accordingly ... > > Why do you run the crontab on all the servers? You can run a start-based > system where only one ("main"-master) syncs all the other masters. You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times. >> And spammers don't care about mx priorities at all :-( > > Actually, statistically speaking, spammers select the low priority ones. Actually: you are right ;-) > Nice to hear that we are not the only ones out there to try to run > something like this over dovecot :P Yes. I never loved the idea of a clusterfs for my small mail servers, I always considered such clusterfs an overkill. Well, my servers do reside in the same housing building, thus it could be done without performance loss. But a scenario of worldwide distributed mail servers desires a dsync/replicator scheme, IMHO ;-) Regards, Michael From jtam.home at gmail.com Thu Mar 15 01:33:23 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 14 Mar 2012 16:33:23 -0700 (PDT) Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: Steve Campbell writes: > Their imap folders, the ones that they create using an imap client or > webmail, are either in ~ or ~/mail. Their original .mailboxlist is > always in ~. Based on that, I should probably copy any imap folders not > in ~/mail to that folder, duplicate ~/.mailboxlist to the file > ~/mail/.subscriptions, and amend any .subscriptions file contents to > just have the name of the folders (without any "mail/folder" reference > in it). > > My example would then be as follows > > /home/steve = folder > /home/steve/Drafts = original folder > /home/steve/AnyFolder = original folder > /home/steve/.mailboxlist = original file > /home/steve/mail = folder (either original or created) > /home/steve/mail/.subscriptions = copied contents of .mailboxlist > file > /home/steve/mail/Drafts = copied folder of original > /home/steve/mail/AnyFolder = copied folder of original > > Contents of original .mailboxlist and new .subscriptions: > > Drafts > AnyFolder > > If the imap folders were in ~/mail, then the original .mailboxlist would > have been > > mail/Drafts > mail/AnyFolder > > but after the corrections to the .subscriptions file, they would be as > above (without reference to the mail folder). > > Is this correct? That depends -- are you aliasing namespaces so that prefix={"", "mail/", etc.} all map to a user's ~/mail folder? You may be creating a confusing situation where a client with a null IMAP prefix has 2 copies of a mailbox. Joseph Tam From kgc at corp.sonic.net Thu Mar 15 01:51:38 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Wed, 14 Mar 2012 16:51:38 -0700 Subject: [Dovecot] Just in time AV scanning Message-ID: <20120314235138.GE39671@corp.sonic.net> I'm curious if anyone has any plugins for AV integration directly into dovecot. Our old pop servers have been scanning messges as they're moved from new->cur in the inbox and, at least where user's aren't poping every few seconds, there is occasionally enough time between scanning through the MXs to message retreval to snag a few more virues with updated definitions before they reach customers. Anyone doing anything similar? -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtl+dovecot at uvm.edu Thu Mar 15 03:24:34 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 14 Mar 2012 21:24:34 -0400 Subject: [Dovecot] director lmtp -> smtp problem Message-ID: <4F6144D2.2080900@uvm.edu> Hi Timo & Dovecot users, We have a 2-node director setup which front-ends for 4 nodes which share a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. Approximately 40k users, but typically only a few thousand active at any time. The director nodes run sendmail, which deliver mail "locally" using LMTP to the director, which then feeds to SMTP on the real servers (also sendmail.) Why sendmail? Because procmail is used for mail filtering and as the delivery agent. Here's the problem, on the director: Mar 14 20:40:08 imapdir2 dovecot: lmtp(10692): Connect from local Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input ->eof) Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3d99a) [0x7f79156c499a] -> /usr/lib/doveco t/libdovecot.so.0(+0x3d9e6) [0x7f79156c49e6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f791569df8f] -> dovecot/lmtp() [0x406e77] -> /usr/l ib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f79156d0044] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f79156d 0c3b] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f79156cfca8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7915 6bdfc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7914ef8cdd] -> dovecot/lmtp() [0x403d69] Mar 14 20:40:38 imapdir2 sendmail[6905]: q2D8KodI018432: SYSERR(root): timeout writing message to localhost: Broken pipe Most mail goes through OK, but some messages do not and end up queued until they run into the queue time limit. So far as I have been able to tell, all of the messages have this failure when the following conversation takes place between sendmail (on director), the Dovecot LMTP proxy, and sendmail on the backend node (SMTP): (names mangled to protect the guilty) (first, sendmail -> director LMTP) > [root at imapdir2 ~]# sendmail -qIq2EFZt1p004708 -v > > Running /var/spool/mqueue/qd2/q2EFZt1p004708 (sequence 1 of 1) > ... Connecting to > /var/lib/dovecot/lmtp-socket via cyrusv2... > 220 imapdir2.uvm.edu Dovecot LMTP ready > >>> LHLO imapdir2.uvm.edu > 250-imapdir2.uvm.edu > 250-8BITMIME > 250-ENHANCEDSTATUSCODES > 250 PIPELINING > >>> MAIL From: > 250 2.1.0 OK > >>> RCPT To: > >>> DATA > 250 2.1.5 OK > 354 OK > timeout writing message to localhost: Broken pipe > ... Deferred > Closing connection to localhost The conversation between the director (LMTP) and the backend (sendmail SMTP) goes like this: > 250-penguinc.uvm.edu Hello imapdir2.uvm.edu [132.198.100.150], pleased > to meet you > 250-ENHANCEDSTATUSCODES > 250-PIPELINING > 250-8BITMIME > 250-SIZE 10485760 > 250-ETRN > 250-AUTH DIGEST-MD5 CRAM-MD5 > 250-DELIVERBY > 250 HELP > MAIL FROM: > 250 2.1.0 ... > Sender ok > RCPT TO: > 552 5.2.2 User ntssdfwe mailbox is full At this point Dovecot should return the failed RCPT TO: status back to sendmail over LMTP, but instead it sits there (waiting for a timeout to expire?) and eventually dies. doveconf -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.2.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.2 (Santiago) base_dir = /var/run/dovecot/ default_client_limit = 6000 default_process_limit = 10240 director_mail_servers = penguina.uvm.edu penguinb.uvm.edu penguinc.uvm.edu penguind.uvm.edu director_servers = imapdir1.uvm.edu imapdir2.uvm.edu lmtp_proxy = yes login_trusted_networks = [REDACTED] passdb { args = proxy=y nopassword=y protocol=smtp driver = static } service anvil { client_limit = 40000 } service auth { client_limit = 45960 unix_listener auth-userdb { group = mail mode = 0660 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director service_count = 0 } service imap { process_limit = 10240 vsz_limit = 1 G } service lmtp { client_limit = 1 inet_listener lmtp { port = 24 } unix_listener /var/lib/dovecot/lmtp-socket { group = root mode = 0600 user = root } } service pop3-login { executable = pop3-login director service_count = 0 } service pop3 { process_limit = 5000 } shutdown_clients = no ssl_cert = <[REDACTED].pem ssl_key = <[REDACTED].key userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol lmtp { auth_socket_path = director-userdb } protocol imap { mail_max_userip_connections = 100 } Hope you can help, Jim Lawson From alexis.lelion at gmail.com Thu Mar 15 11:48:52 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 10:48:52 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules Message-ID: Hello, In my current setup, I have a spam filter upstream that adds a specific header - X-Spam-Level on every incoming mail. Based on this level, the mail will be moved to the user spam folder using sieve by doing "fileinto :create 'spam';" Unfortunately, some legitimate email may end up in this spam folder, so I have kind of a whitelist that performs an explicit keep over specific trusted domains. So, my complete spam filtering rule is : if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ]{ ??? keep; elseif header :contains "X-Spam-Level" ["0","1","2"] { ??? fileinto :create "__spam__"; } This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf This works as expected except that it doesn't take into account users filtering for domains that were matched for the explicit keep. For example, I have the following rule : if address :domain "From" "trusted.tld" { ??? fileinto "trusted" } But mail coming from that domain are still delivered in my mailbox. Is there something I'm missing here? I guess yes, otherwise it would work as I want ^_^ Any help/comment is appreciated Thanks! Alexis From tss at iki.fi Thu Mar 15 12:02:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:02:16 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F6144D2.2080900@uvm.edu> References: <4F6144D2.2080900@uvm.edu> Message-ID: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Hi, On 15.3.2012, at 3.24, Jim Lawson wrote: > We have a 2-node director setup which front-ends for 4 nodes which share > a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. .. > Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > (proxy->data_input > ->eof) I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. From paul at actionlans.com Thu Mar 15 02:29:19 2012 From: paul at actionlans.com (paul) Date: Thu, 15 Mar 2012 10:59:19 +1030 Subject: [Dovecot] firefox dovecot-sieve Message-ID: <1331771362.1955.1.camel@paul15.localdomain> Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org On 14.3.2012, at 4.45, paul wrote: > HI. I have just started to play with sieve and everything seems ok when > logging on using telnet localhost 4190 and an encoded username/password. > If I try to connect with firefox at localhost:4190 I get > "IMPLEMENTATION" "Dovecot Pigeonhole" > "SIEVE" "fileinto reject envelope encoded-character vacation subaddress > comparator-i;ascii-numeric relational regex imap4flags copy include > variables body enotify environment mailbox date ihave" > "NOTIFY" "mailto" > "SASL" "PLAIN LOGIN" > "STARTTLS" > "VERSION" "1.0" > OK "Dovecot ready." > NO "Error in MANAGESIEVE command received by server." > NO "Error in MANAGESIEVE command received by server." > NO "Invalid characters in atom" > BYE "Too many invalid MANAGESIEVE commands." > my dovecot -n shows > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.42.9-1.fc15.i686.PAE i686 Fedora release 15 (Lovelock) > auth_mechanisms = plain login > disable_plaintext_auth = no > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mail_privileged_group = mail > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date ihave > mbox_write_locks = fcntl > passdb { > driver = pam > } > plugin { > sieve = ~/.dovecot.sieve > sieve_dir = ~/sieve > } > protocols = imap pop3 lmtp sieve > service managesieve-login { > inet_listener sieve { > port = 4190 > } > } > ssl_cert = ssl_key = userdb { > driver = passwd > } > Have I missed something obvious or does Firefox clash with Managesieve? > Thanks. Paul > From aydin.demirel at endersys.com Thu Mar 15 12:09:11 2012 From: aydin.demirel at endersys.com (=?UTF-8?B?QXlkxLFuIERlbWlyZWw=?=) Date: Thu, 15 Mar 2012 12:09:11 +0200 Subject: [Dovecot] Login Failed Message-ID: <4F61BFC7.1000607@endersys.com> Hi; We are using scripts for login successes. Is there a feature for login failed status or can it be developed? Regards -- *Ayd?n Demirel Endersys Ltd. Sistem Destek M?hendisi/ System Support Engineer* * *<> Phone : +90 216 470 9423 | GSM : +90 530 401 8203 Fax : +90 216 470 9508 | Web : http://www.endersys.com Blog : http://blog.endersys.com Twitter : http://www.twitter.com/endersys LPI : The #1 Linux Certification for IT Professionals LPI (Linux Professional Institute) Turkey http://www.lpi-turkey.com From CMarcus at Media-Brokers.com Thu Mar 15 12:20:38 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:20:38 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60F71C.8090306@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> <4F60F71C.8090306@cnpapers.com> Message-ID: <4F61C276.90309@Media-Brokers.com> On 2012-03-14 3:53 PM, Steve Campbell wrote: > I'm not sure these are virtual users, so that link may have confused me. > All accounts on these servers have real unix accounts. Their inbox is > /var/spool/mail/unix-user-name. Doesn't matter, the same thing applies... don't put mail directly in their 'home' folder, put it in a subfolder (ie, /home/user/mail)... You *will* have problems if you leave those as they are... -- Best regards, Charles From tss at iki.fi Thu Mar 15 12:29:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:29:55 +0200 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <1331807395.10319.3.camel@innu> On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: > I've mostly finished a conversion from an old Centos 3 UW-Imap server to > a new Centos 6 dovecot server. This is messy stuff to do. There are ways you could make Dovecot behave identically to UW-IMAP (mail_full_filesystem_access=yes), but for future and for security it's better if you don't do that. > I did not copy the old ~/.mailboxlist > file to ~/mail/.subscriptions file, but notice some users have the > latter file now. These are all mbox folders on the old and new server. Copying it for users who haven't already readded their subscriptions would be a good idea. > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes, move all of the mboxes to mail/ directory. With the compatibility namespaces it should work so that clients don't notice changes: http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility There are also a few old mailing list threads detailing all kinds of issues and solutions related to UW-IMAP -> Dovecot migration.. From tss at iki.fi Thu Mar 15 12:31:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:31:40 +0200 Subject: [Dovecot] Login Failed In-Reply-To: <4F61BFC7.1000607@endersys.com> References: <4F61BFC7.1000607@endersys.com> Message-ID: <1331807500.10319.4.camel@innu> On Thu, 2012-03-15 at 12:09 +0200, Ayd?n Demirel wrote: > We are using scripts for login successes. Is there a feature for login > failed status or can it be developed? Login failures are only visible in auth and login processes. Probably better to implement it in auth process. And there it depends on what passdb you use. You could for example switch to passdb checkpassword, which allows you to easily run scripts for both success and failure. From tss at iki.fi Thu Mar 15 12:33:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:33:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <20120314235138.GE39671@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> Message-ID: <1331807624.10319.6.camel@innu> On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: > I'm curious if anyone has any plugins for AV integration directly into > dovecot. > > Our old pop servers have been scanning messges as they're moved from > new->cur in the inbox and, at least where user's aren't poping every > few seconds, there is occasionally enough time between scanning through > the MXs to message retreval to snag a few more virues with updated > definitions before they reach customers. > > Anyone doing anything similar? http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a script that modifies a mail while it's being read. You could make it run a virus check, and if that happens you could change the virus MIME part to be full of spaces (better not to change message size, line count or MIME structure). From stephan at rename-it.nl Thu Mar 15 12:43:05 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 11:43:05 +0100 Subject: [Dovecot] firefox dovecot-sieve In-Reply-To: <1331771362.1955.1.camel@paul15.localdomain> References: <1331771362.1955.1.camel@paul15.localdomain> Message-ID: <4F61C7B9.9050605@rename-it.nl> Op 3/15/2012 1:29 AM, paul schreef: > Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org > > On 14.3.2012, at 4.45, paul wrote: > >> HI. I have just started to play with sieve and everything seems ok when logging on using telnet localhost 4190 and an encoded username/password. If I try to connect with firefox at localhost:4190 I get >> "IMPLEMENTATION" "Dovecot Pigeonhole" >> "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave" >> "NOTIFY" "mailto" >> "SASL" "PLAIN LOGIN" >> "STARTTLS" >> "VERSION" "1.0" >> OK "Dovecot ready." >> NO "Error in MANAGESIEVE command received by server." >> NO "Error in MANAGESIEVE command received by server." >> NO "Invalid characters in atom" >> BYE "Too many invalid MANAGESIEVE commands." >> Have I missed something obvious or does Firefox clash with Managesieve? Yes you have :). Firefox speaks HTTP (and quite a few other protocols), but not ManageSieve. You'll need to run a Sieve editor on your webserver if you want to edit Sieve scripts using your browser. Regards, Stephan. From Ralf.Hildebrandt at charite.de Thu Mar 15 12:44:21 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 11:44:21 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? Message-ID: <20120315104421.GW21113@charite.de> Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:47:26 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:47:26 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:51:01 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:51:01 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 during that time, I wasn't able to access the mailbox using imap. I then issued doveadm kick username and all over sudden the mailbox was accessible (via IMAP) # doveadm kick username kicked connections from the following users: username # /usr/local/scripts/find_abnormal_imap Mar 15 11:38:48 postamt dovecot: imap: Warning: Killed with signal 15 (by pid=24545 uid=0 code=kill) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 12:47:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:47:07 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315104421.GW21113@charite.de> References: <20120315104421.GW21113@charite.de> Message-ID: <1331808427.10319.7.camel@innu> On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > during that time, I wasn't able to access the mailbox using imap. I > then issued Maildir? > doveadm kick username > > and all over sudden the mailbox was accessible (via IMAP) So one of them had the INBOX locked. Do you have pop3_lock_session=yes? From CMarcus at Media-Brokers.com Thu Mar 15 12:51:11 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:51:11 -0400 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F502485.9070503@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> Message-ID: <4F61C99F.2040409@Media-Brokers.com> On 2012-03-01 8:38 PM, Stan Hoeppner wrote: > Get yourself a qualified network architect. Pay for a full network > traffic analysis. He'll attach sniffers at multiple points in your > network to gather traffic/error/etc data. Then you'll discuss the new > office, which employees/types with move there, and you'll be able to > know almost precisely the average and peak bandwidth needs over the MAN > link. He'll very likely tell you the same thing I have, that a single > gigabit MAN link is plenty. If you hire him to do the work, he'll > program the proper QOS setup to match the traffic patterns gleaned from > the sniffers. Finally had time to properly review your answers here Stan. The time you took for the in-depth reply is very much appreciated - and I'm sure you got a kick out of the level of my ignorance... ;) As for hiring a network architect, I will absolutely be doing as you recommend (was already planning on it), but with the information I'm now armed with, at least I'll have a better chance of knowing if they know what they are doing/talking about... I'm still planning for the two physical servers (one at each location), but you have convinced me that trying to run two live mail systems is an unnecessary and even unwanted level of complexity. The DC VM will still be hot (it is always best to have two DCs in a windows domain environment anyway) so I'll get automatic real time off site backup of all of the users data (since it will all be on DFS), but for the mail services, I'll just designate one as live, and one as the hot/standby that is kept in sync using dsync. This way I'll automatically get off site back up for each site for the users data stored in the DFS, and have a second mail system ready to go if something happens to the primary. Again, thanks Stan... I am constantly amazed at the level of expertise and quality of advice available *for free* in the open source world, as is available on these lists. -- Best regards, Charles From Ralf.Hildebrandt at charite.de Thu Mar 15 13:01:18 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 12:01:18 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331808427.10319.7.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> Message-ID: <20120315110118.GX21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > > during that time, I wasn't able to access the mailbox using imap. I > > then issued > > Maildir? Yep. > > doveadm kick username > > > > and all over sudden the mailbox was accessible (via IMAP) > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > Yes. # makes Dovecot lock the mailbox for the whole session pop3_lock_session = yes Shouldn't be doing this I guess? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 13:04:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 13:04:43 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315110118.GX21113@charite.de> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> Message-ID: <1331809483.10319.9.camel@innu> On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > Yes. > # makes Dovecot lock the mailbox for the whole session > pop3_lock_session = yes > > Shouldn't be doing this I guess? If you do it then a single POP3 session can keep the mailbox locked pretty much forever. If you don't do it, you're violating POP3 RFC, but I don't think anyone really cares about that.. I guess this setting should really use a separate POP3-only lock when it's enabled. From stephan at rename-it.nl Thu Mar 15 13:11:16 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 12:11:16 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: Message-ID: <4F61CE54.4010607@rename-it.nl> Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > Hello, > > In my current setup, I have a spam filter upstream that adds a > specific header - X-Spam-Level on every incoming mail. Based on this > level, the mail will be moved to the user spam folder using sieve by > doing "fileinto :create 'spam';" > Unfortunately, some legitimate email may end up in this spam folder, > so I have kind of a whitelist that performs an explicit keep over > specific trusted domains. So, my complete spam filtering rule is : require ["fileinto", "mailbox"]; if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { keep; } elsif header :contains "X-Spam-Level" ["0","1","2"] { fileinto :create "__spam__"; } Fixed a few syntax issues there before I could test this. > This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is > my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf What version are you using? The above statement hints that it is recent, probably Dovecot v2.1 with matching Pigeonhole. > This works as expected except that it doesn't take into account users > filtering for domains that were matched for the explicit keep. For > example, I have the following rule : > if address :domain "From" "trusted.tld" { > fileinto "trusted" > } > But mail coming from that domain are still delivered in my mailbox. At my end, this is correctly delivered in the "trusted" folder, provided that this folder exists. Are you sure that the user's personal script even executes correctly? For example, the above script omits a ';'. The script also fails when there is n no "trusted" folder. Check the log files for errors. The default action in the event of an error is to store the message in INBOX, which may well be what you're seeing here. Regards, Stephan. From CMarcus at Media-Brokers.com Thu Mar 15 13:21:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 07:21:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Message-ID: <4F61D0A3.7070503@Media-Brokers.com> On 2012-03-14 5:51 PM, Michael Grimm wrote: > You misunderstood. I was referring to system cronjob's mail reports > from cron.daily jobs like security reports et al. Those reports > normally run at identical times. But are these really 'duplicate' mails? It sounds to me like they are individual to each system. I'm also confused - are you actually delivering the exact *same* mail to two (or multiple) *different* servers simultaneously? If only one copy of the mail gets delivered, regardless of which server it gets delivered to, when dsync runs, there would be no duplicates, right? I'm asking for clarification because I was considering a similar setup. -- Best regards, Charles From campbell at cnpapers.com Thu Mar 15 13:21:17 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 07:21:17 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: <4F61D0AD.1050705@cnpapers.com> On 3/14/2012 7:33 PM, Joseph Tam wrote: > Steve Campbell writes: > >> Their imap folders, the ones that they create using an imap client or >> webmail, are either in ~ or ~/mail. Their original .mailboxlist is >> always in ~. Based on that, I should probably copy any imap folders not >> in ~/mail to that folder, duplicate ~/.mailboxlist to the file >> ~/mail/.subscriptions, and amend any .subscriptions file contents to >> just have the name of the folders (without any "mail/folder" reference >> in it). >> >> My example would then be as follows >> >> /home/steve = folder >> /home/steve/Drafts = original folder >> /home/steve/AnyFolder = original folder >> /home/steve/.mailboxlist = original file >> /home/steve/mail = folder (either original or created) >> /home/steve/mail/.subscriptions = copied contents of .mailboxlist >> file >> /home/steve/mail/Drafts = copied folder of original >> /home/steve/mail/AnyFolder = copied folder of original >> >> Contents of original .mailboxlist and new .subscriptions: >> >> Drafts >> AnyFolder >> >> If the imap folders were in ~/mail, then the original .mailboxlist would >> have been >> >> mail/Drafts >> mail/AnyFolder >> >> but after the corrections to the .subscriptions file, they would be as >> above (without reference to the mail folder). >> >> Is this correct? > > That depends -- are you aliasing namespaces so that prefix={"", > "mail/", etc.} all map to a user's ~/mail folder? You may be creating a > confusing situation where a client with a null IMAP prefix has 2 copies > of a mailbox. > > Joseph Tam I have the following set: mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u namespace { type = private separator = / prefix = "#mbox/" location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } namespace { type = private separator = / prefix = mail/ hidden = yes list = no # for v1.1+ } namespace { type = private separator = / prefix = ~/mail/ hidden = yes list = yes # for v1.1+ location = mbox:~/mail:INBOX=/var/mail/%u } namespace { type = private separator = / prefix = ~%u/mail/ hidden = yes list = no # for v1.1+ } These are mostly what's defined as the "Backward Compatability" namespaces in the wiki. Are you saying that I should probably have something like the following then: namespace { type = private separator = / prefix = location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } And is the multiple "inbox = yes" in the differing namespaces a no-no? Based on the comments in the 10-mail.conf file, it seems to say it is a problem, but if a user has any prefix defined, even the blank prefix, wouldn't that mean they use only that set of parameters defined in the namespace being used? So far, I've only changed one prefix in the building to the #mbox prefix and that was because of the weird layout of files they had. I'm hoping one day to understand all of this. Dovecot, as I stated before, is much more complex that the imap server used previously. It allows one to use all of the facilities of the imap protocol, and much more, but unfortunately, for admins like me that are just moving to these new imap servers, most of those extras were either unknown to me or unused. Again, thanks all for the patience and help. steve From alexis.lelion at gmail.com Thu Mar 15 13:42:14 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 12:42:14 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61CE54.4010607@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> Message-ID: Hello Stephan, Thanks for your answer, and sorry for forgetting to specify which dovecot version I was using :-/ I'm using Dovecot 2.0.15, with PigeonHole. The syntax issues are some typos I made while writing this email, I double checked, and indeed, my production script was slightly different from what I wrote in the first place. I can confirm that the scripts compile properly with sievec, and also that the folder does exist, but just to be sure this is not an issue, I added the ":create" option to the user's fileinto. I have no errors in my logs, the only thing displayed is tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' Is there any way to increase verbosity for sieve only? Thanks On Thu, Mar 15, 2012 at 12:11 PM, Stephan Bosch wrote: > Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > >> Hello, >> >> In my current setup, I have a spam filter upstream that adds a >> specific header - X-Spam-Level on every incoming mail. Based on this >> level, the mail will be moved to the user spam folder using sieve by >> doing "fileinto :create 'spam';" >> Unfortunately, some legitimate email may end up in this spam folder, >> so I have kind of a whitelist that performs an explicit keep over >> specific trusted domains. So, my complete spam filtering rule is : > > require ["fileinto", "mailbox"]; > > if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { > ? ?keep; > } elsif header :contains "X-Spam-Level" ["0","1","2"] { > ? ?fileinto :create "__spam__"; > } > > Fixed a few syntax issues there before I could test this. > > >> This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is >> my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf > > What version are you using? The above statement hints that it is recent, > probably Dovecot v2.1 with matching Pigeonhole. > > >> This works as expected except that it doesn't take into account users >> filtering for domains that were matched for the explicit keep. For >> example, I have the following rule : >> if address :domain "From" "trusted.tld" { >> ? ? fileinto "trusted" >> } >> But mail coming from that domain are still delivered in my mailbox. > > > At my end, this is correctly delivered in the "trusted" folder, provided > that this folder exists. Are you sure that the user's personal script even > executes correctly? For example, the above script omits a ';'. The script > also fails when there is n no "trusted" folder. Check the log files for > errors. The default action in the event of an error is to store the message > in INBOX, which may well be what you're seeing here. > > Regards, > > Stephan. From jtl+dovecot at uvm.edu Thu Mar 15 13:50:22 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 07:50:22 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Message-ID: <4F61D77E.8020805@uvm.edu> On 3/15/12 6:02 AM, Timo Sirainen wrote: > Hi, > > On 15.3.2012, at 3.24, Jim Lawson wrote: >> We have a 2-node director setup which front-ends for 4 nodes which share >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > .. >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >> (proxy->data_input >> ->eof) > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > I'll give it a shot. For the purposes of doing a rolling upgrade, is it reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for the duration, or should I split-brain them during the upgrade? Jim From Ralf.Hildebrandt at charite.de Thu Mar 15 14:00:40 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 13:00:40 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331809483.10319.9.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> <1331809483.10319.9.camel@innu> Message-ID: <20120315120040.GA21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > > > Yes. > > # makes Dovecot lock the mailbox for the whole session > > pop3_lock_session = yes > > > > Shouldn't be doing this I guess? > > If you do it then a single POP3 session can keep the mailbox locked > pretty much forever. If you don't do it, you're violating POP3 RFC, but > I don't think anyone really cares about that.. Indeed. All I care about is that the user gets his/her mail. Which he didn't. I disabled it. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From campbell at cnpapers.com Thu Mar 15 14:15:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 08:15:39 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <1331807395.10319.3.camel@innu> References: <4F60AF62.80702@cnpapers.com> <1331807395.10319.3.camel@innu> Message-ID: <4F61DD6B.2020606@cnpapers.com> On 3/15/2012 6:29 AM, Timo Sirainen wrote: > On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: >> I've mostly finished a conversion from an old Centos 3 UW-Imap server to >> a new Centos 6 dovecot server. > This is messy stuff to do. There are ways you could make Dovecot behave > identically to UW-IMAP (mail_full_filesystem_access=yes), but for future > and for security it's better if you don't do that. > >> I did not copy the old ~/.mailboxlist >> file to ~/mail/.subscriptions file, but notice some users have the >> latter file now. These are all mbox folders on the old and new server. > Copying it for users who haven't already readded their subscriptions > would be a good idea. > >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > Yes, move all of the mboxes to mail/ directory. With the compatibility > namespaces it should work so that clients don't notice changes: > > http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility > > There are also a few old mailing list threads detailing all kinds of > issues and solutions related to UW-IMAP -> Dovecot migration.. I'd replied to an earlier thread, and in it, I'd asked a question about a "blank" prefix namespace and the backward compatability namespaces. I'm not sure whether my "mail_location" takes precedence over namespaces (with or without a "location" parm), especially since I don't define a "blank" prefix defined. It's been working, or at least I'm not getting calls, so maybe I'm OK. In any event, I believe if I move all of these folders to ~/mail, ensure the .subscriptions file is matching, that at least people using Thunderbird will re-read the file and set their folders properly. Not sure about other clients. Thanks for the help. steve > > From tss at iki.fi Thu Mar 15 14:22:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:22:11 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: <1331814131.10319.15.camel@innu> On Thu, 2012-03-08 at 21:36 +0200, Timo Sirainen wrote: > On 8.3.2012, at 21.18, Markus Petri wrote: > > > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > > once per user sharing an folder in LIST "" "%". > > > > I also noticed, that with 2.1 the user folder (Shared/) is no > > longer tagged as \NoSelect. > > > > Is this the intended behaviour and mutt simply cannot cope with it or > > is it a dovecot problem? > > Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even > if it did. This is a bit difficult to fix. I'll probably leave it until v2.2. > Also Dovecot probably should add \Noselect, especially if the mailbox > isn't really selectable (there's some weirdness between shared/user > being equal to shared/user/INBOX, but I'm not sure what to do about > it). These should fix this: http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 From tss at iki.fi Thu Mar 15 14:25:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:25:12 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F61D77E.8020805@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> Message-ID: <1331814312.10319.18.camel@innu> On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: > On 3/15/12 6:02 AM, Timo Sirainen wrote: > > Hi, > > > > On 15.3.2012, at 3.24, Jim Lawson wrote: > >> We have a 2-node director setup which front-ends for 4 nodes which share > >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > > .. > >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > >> (proxy->data_input > >> ->eof) > > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > > > I'll give it a shot. For the purposes of doing a rolling upgrade, is it > reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for > the duration, or should I split-brain them during the upgrade? I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The current hg version has some extra features, but it doesn't use them until all of the directors have upgraded to the new version. From tss at iki.fi Thu Mar 15 14:58:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:58:06 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: Message-ID: <1331816286.10319.23.camel@innu> On Mon, 2012-03-05 at 00:32 +0000, Andreas M. Kirchwitz wrote: > Thanks for this patch. I've applied it to the dovecot-20120303 > nightly snapshot. The good news is, compilation works fine. > The bad news is, the libraries and binaries don't work because > they don't find the custom SSL libraries. .. > $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch > $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install You would have needed to run autogen.sh again. It works with me now that I tried in a test server with OpenSSL in non-standard dir. From mhlavink at redhat.com Thu Mar 15 15:34:32 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Thu, 15 Mar 2012 14:34:32 +0100 Subject: [Dovecot] dovecot and systemd Message-ID: <4F61EFE8.1000901@redhat.com> Hi all, dovecot supports systemd socket activation. Together with standard unit activation (like old sysv init script), there are two ways how to configure dovecot(only interface:port, not whole configuration). This can result in situation where those configurations does not say the same. Question is what should happen then? For example, lets have dovecot configured to listen for imap(s) and lets have systemd dovecot socket configured to listen for all protocols - pop3(s) and imap(s). When dovecot is configured to start on boot, systemd will start it and dovecot will listen on imap(s) ports. But when dovecot.socket is enabled, it'll listen on pop3(s) too and when new pop3 connection comes, it'll pass it to dovecot and dovecot will serve it. The question is: Should this happen? What exactly should happen when dovecot.conf does not match dovecot.socket configuration? Michal From markus at mpetri.org Thu Mar 15 15:46:27 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 15 Mar 2012 14:46:27 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <1331814131.10319.15.camel@innu> References: <20120308201812.2932e90c@legolas.home.ceotex.de> <1331814131.10319.15.camel@innu> Message-ID: <20120315144627.6173dc44@legolas.home.ceotex.de> On Thu, 15 Mar 2012 14:22:11 +0200 Timo Sirainen wrote: > > Also Dovecot probably should add \Noselect, especially if the > > mailbox isn't really selectable (there's some weirdness between > > shared/user being equal to shared/user/INBOX, but I'm not sure what > > to do about it). > > These should fix this: > > http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c > http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 > Yes, those fix the problem. Thanks. From trashcan at odo.in-berlin.de Thu Mar 15 15:46:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 14:46:56 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61D0A3.7070503@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> Message-ID: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Hi -- On 15.03.2012 12:21, Charles Marcus wrote: > On 2012-03-14 5:51 PM, Michael Grimm > wrote: >> You misunderstood. I was referring to system cronjob's mail reports >> from cron.daily jobs like security reports et al. Those reports >> normally run at identical times. > > But are these really 'duplicate' mails? It sounds to me like they are > individual to each system. > > I'm also confused - are you actually delivering the exact *same* mail > to two (or multiple) *different* servers simultaneously? If only one > copy of the mail gets delivered, regardless of which server it gets > delivered to, when dsync runs, there would be no duplicates, right? Well, let me explain it in more detail: Given there are two servers called mx1 and mx2. They both have cron.daily jobs running, and let's say those cronjobs are meant to create at 3:00 a postfix-logwatch report on every server. Thus, the cronjob at mx1 sends his final report to the admin of mx1, and the one at mx2 to the admin of mx2. I happen to be the one who will finally receive those reports, and therefore I did tell sieve to drop them into some folder of mine, let's say REPORTS. Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report". I had dsync running every minute. Thus at 3:00 the final sync has been initiated, and at 3:01 dsync will find two mails to sync in REPORTS. In 99.9% of all synchronizations the final result at both server's REPORTS mailbox is as expected and as follows: mx1-report 3:01 mx2-report 3:01 But occasionally, and what I refer to as duplicates, I did find either ... mx1-report 3:01 mx1-report 3:01 mx2-report 3:01 ... or ... mx1-report 3:01 mx2-report 3:01 mx2-report 3:01 Actually, that was when I started to investigate how dsync will behave when many mails arrive at two servers simultaneously with identical final mailboxes. The day I switched to the new replicator/dsync technique, those duplicates are history, but I'm still able to produce duplicates (and multiples) if I simultaneously produce *many* mails at every server with identical mailbox destinations in a minute (see my other report a couple of days ago). Timo is suspecting the combination of arriving mails while running dsync to be a possible cause of such duplicates, if I didn't get him wrong. Again, if your servers aren't receiving loads of mails for the very same mailboxes within very short time, the current dsync/replicator works great. HTH and regards, Michael From tss at iki.fi Thu Mar 15 15:59:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 15:59:38 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <1331819978.10319.29.camel@innu> On Mon, 2012-03-05 at 18:01 -0800, Joseph Tam wrote: > On Sun, 4 Mar 2012, Timo Sirainen writes: > > > > I would like to run various doveadm commands that involves all (mail) users like > > > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > > > but any doveadm command that uses "-A" to iterate through all users will > > > stop processing at the first account with UID > > > What userdb are you using? userdb passwd should already skip users that > > aren't in the valid range. And what Dovecot version are you using? > > passwd-file under dovecot 2.0.16. Ah. The skipping only works in v2.1. Also you mean you're using passwd-file for /etc/passwd? You shouldn't really be doing that. From CMarcus at Media-Brokers.com Thu Mar 15 16:04:01 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 10:04:01 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4F61F6D1.9010703@Media-Brokers.com> On 2012-03-15 9:46 AM, Michael Grimm wrote: > Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder > REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the > mailfolder REPORTS. Important: both mails are different but they arrive > in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. > And, let's call the report from mx1 cronjob "mx1-report" and that from > mx2 "mx2-report". so these are LOCAL mails delivered to local user accounts? The easiest thing to do for this is simply alias the local address(es) so that they all go to one single server/account (I would use only virtual, but you can do it with system accounts too). I see lots of potential problems doing it the way you are doing it. -- Best regards, Charles From tss at iki.fi Thu Mar 15 16:05:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:05:29 +0200 Subject: [Dovecot] dovecot and systemd In-Reply-To: <4F61EFE8.1000901@redhat.com> References: <4F61EFE8.1000901@redhat.com> Message-ID: <1331820329.10319.32.camel@innu> On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: > What exactly should happen when > dovecot.conf does not match dovecot.socket configuration? Dovecot's systemd code was written by one of you Redhat guys. I had some similar thoughts when I applied the patch, but didn't really know what to do about it, so I didn't do anything. So: I don't know. Maybe some other project has solved this somehow already? Dovecot anyway needs its own internal UNIX listeners. Should all internal inet listeners be disabled? Could Dovecot somehow talk to systemd and ask what listeners it's using for Dovecot and log warnings if they don't match? From stephan at rename-it.nl Thu Mar 15 16:17:55 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 15:17:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: <4F61CE54.4010607@rename-it.nl> Message-ID: <4F61FA13.5060204@rename-it.nl> On 3/15/2012 12:42 PM, Alexis Lelion wrote: > Hello Stephan, > > Thanks for your answer, and sorry for forgetting to specify which > dovecot version I was using :-/ > I'm using Dovecot 2.0.15, with PigeonHole. > > The syntax issues are some typos I made while writing this email, I > double checked, and indeed, my production script was slightly > different from what I wrote in the first place. I can confirm that the > scripts compile properly with sievec, and also that the folder does > exist, but just to be sure this is not an issue, I added the ":create" > option to the user's fileinto. > > I have no errors in my logs, the only thing displayed is > tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into > mailbox 'INBOX' > > Is there any way to increase verbosity for sieve only? You can test Sieve outside normal delivery using the sieve-test tool; include the global sieve_before script using a -s argument. Alternatively, you can use the vnd.dovecot.debug extension as follows: require ["fileinto", "mailbox", "vnd.dovecot.debug"]; if address :domain "From" "trusted.tld" { fileinto :create "trusted"; debug_log "Tried to save in \"trusted\""; } You need to add the vnd.dovecot.debug extension to sieve_extensions in your 90-sieve.conf, e.g.: sieve_extensions = +vnd.dovecot.debug This will produce the following output in the user's personal sieve log (typically ~/.dovecot.sieve.log): sieve: info: started log at Mar 15 15:13:29. main_script: line 5: info: DEBUG: Tried to save in "trusted". info: msgid=unspecified: stored mail into mailbox 'trusted'. If the DEBUG line is missing at your end, the fileinto is not executed at all. If it is, and things are still delivered in INBOX, something else is going on. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 15 16:24:01 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 15:24:01 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61F6D1.9010703@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4F61F6D1.9010703@Media-Brokers.com> Message-ID: Hi -- On 15.03.2012 15:04, Charles Marcus wrote: > On 2012-03-15 9:46 AM, Michael Grimm > wrote: >> Thus, at 3:01 one report from mx1 will be delivered at mx1 into >> mailfolder >> REPORTS and at 3:01 one report from mx2 will be delivered at mx2 >> into the >> mailfolder REPORTS. Important: both mails are different but they >> arrive >> in the mailfolder REPORTS at the same time, one at mx1 the other at >> mx2. >> And, let's call the report from mx1 cronjob "mx1-report" and that >> from >> mx2 "mx2-report". > > so these are LOCAL mails delivered to local user accounts? All locally produced mails are aliased to the very same virtual user, namely myself. > The easiest thing to do for this is simply alias the local > address(es) > so that they all go to one single server/account (I would use only > virtual, but you can do it with system accounts too). That is exactly what I'm doing, I'm running virtual, only. No local user accounts here. Every locally produced system mail end in virtual mailboxes of myself. In the given example "mx1-report" is delivered to REPORTS at mx1 and "mx2-report" to REPORTS at mx2. Now, I want to access them via IMAP for instance at my mx1 mail account. Without dsync I would only be able to access "mx1-report", thus I do need to sync REPORTS to see both at mx1. > I see lots of potential problems doing it the way you are doing it. Hmm, now, I don't understand you. Regards, Michael From alexis.lelion at gmail.com Thu Mar 15 16:38:55 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 15:38:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61FA13.5060204@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> <4F61FA13.5060204@rename-it.nl> Message-ID: Thanks for this useful information, I will give it a try On Thu, Mar 15, 2012 at 3:17 PM, Stephan Bosch wrote: > On 3/15/2012 12:42 PM, Alexis Lelion wrote: >> >> Hello Stephan, >> >> Thanks for your answer, and sorry for forgetting to specify which >> dovecot version I was using :-/ >> I'm using Dovecot 2.0.15, with PigeonHole. >> >> The syntax issues are some typos I made while writing this email, I >> double checked, and indeed, my production script was slightly >> different from what I wrote in the first place. I can confirm that the >> scripts compile properly with sievec, and also that the folder does >> exist, but just to be sure this is not an issue, I added the ":create" >> option ?to the user's fileinto. >> >> I have no errors in my logs, the only thing displayed is >> tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into >> mailbox 'INBOX' >> >> Is there any way to increase verbosity for sieve only? > > > You can test Sieve outside normal delivery using the sieve-test tool; > include the global sieve_before script using a -s argument. > > Alternatively, you can use the vnd.dovecot.debug extension as follows: > > require ["fileinto", "mailbox", "vnd.dovecot.debug"]; > > > if address :domain "From" "trusted.tld" { > ?fileinto :create "trusted"; > ?debug_log "Tried to save in \"trusted\""; > } > > You need to add the vnd.dovecot.debug extension to sieve_extensions in your > 90-sieve.conf, e.g.: > > sieve_extensions = +vnd.dovecot.debug > > This will produce the following output in the user's personal sieve log > (typically ~/.dovecot.sieve.log): > > sieve: info: started log at Mar 15 15:13:29. > main_script: line 5: info: DEBUG: Tried to save in "trusted". > info: msgid=unspecified: stored mail into mailbox 'trusted'. > > If the DEBUG line is missing at your end, the fileinto is not executed at > all. If it is, and things are still delivered in INBOX, something else is > going on. > > Regards, > > Stephan. > > From tss at iki.fi Thu Mar 15 16:53:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:53:53 +0200 Subject: [Dovecot] v2.1.2 released Message-ID: <1331823233.10319.40.camel@innu> http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig There are a ton of proxying related improvements in this release. You should now be able to do pretty much anything you want with Dovecot proxy/director. This release also includes the initial version of dsync-based replication. I'm already successfully using it for @dovecot.fi mails, but it still has some problems. See http://dovecot.org/list/dovecot/2012-March/064243.html for some details how to configure it. + Initial implementation of dsync-based replication. For now this should be used only on non-critical systems. + Proxying: POP3 now supports sending remote IP+port from proxy to backend server via Dovecot-specific XCLIENT extension. + Proxying: proxy_maybe=yes with host= (instead of IP) works now properly. + Proxying: Added auth_proxy_self setting + Proxying: Added proxy_always extra field (see wiki docs) + Added director_username_hash setting to specify what part of the username is hashed. This can be used to implement per-domain backends (which allows safely accessing shared mailboxes within domain). + Added a "session ID" string for imap/pop3 connections, available in %{session} variable. The session ID passes through Dovecot IMAP/POP3 proxying to backend server. The same session ID is can be reused after a long time (currently a bit under 9 years). + passdb checkpassword: Support "credentials lookups" (for non-plaintext auth and for lmtp_proxy lookups) + fts: Added fts_index_timeout setting to abort search if indexing hasn't finished by then (default is to wait forever). - doveadm sync: If mailbox was expunged empty, messages may have become back instead of also being expunged in the other side. - director: If user logged into two directors while near user expiration, the directors might have redirected the user to two different backends. - imap_id_* settings were ignored before login. - Several fixes to mailbox_list_index=yes - Previous v2.1.x didn't log all messages at shutdown. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tss at iki.fi Thu Mar 15 17:23:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 17:23:50 +0200 Subject: [Dovecot] v2.0.19 released Message-ID: <1331825030.10319.42.camel@innu> http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz.sig Hopefully one of the last v2.0.x releases. - IMAP: ENABLE CONDSTORE/QRESYNC + STATUS for a mailbox might not have seen latest external changes to it, like new mails. - imap_id_* settings were ignored before login. - doveadm altmove did too much work sometimes, retrying moves it had already done. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tom at talpey.com Thu Mar 15 18:04:44 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 12:04:44 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 Message-ID: <4F62131C.2090008@talpey.com> I'm seeing a few warnings emitted when building for x86. They're pretty obvious, but if you want the configure options etc, I can provide those. In Dovecot 2.1.2 (I also see some of these in 2.1.1): 1) src/lib-index/mail-cache-fields.c (comparison between two last_used fields) mail-cache-fields.c: In function 'mail_cache_header_fields_read': mail-cache-fields.c:406: warning: comparison between signed and unsigned 2) src/director/user-directory.c (comparison with ioloop_time) user-directory.c: In function 'user_directory_user_is_recently_updated': user-directory.c:147: warning: comparison between signed and unsigned 3) src/replication/replicator/replicator-brain.c (comparison with ioloop_time) replicator-brain.c: In function 'doveadm_replicate': replicator-brain.c:113: warning: comparison between signed and unsigned 4) src/replication/replicator/replicator-queue.c (comparison with ioloop_time) replicator-queue.c: In function 'replicator_queue_pop': replicator-queue.c:201: warning: comparison between signed and unsigned In Pigeonhole 0.3.0: 5) src/managesieve-login/client-authenticate.c (passing size_t * not uoff_t *) client-authenticate.c: In function 'managesieve_client_auth_read_response': client-authenticate.c:214: warning: passing argument 3 of 'i_stream_get_size' from incompatible pointer type From tss at iki.fi Thu Mar 15 18:25:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 18:25:21 +0200 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F62131C.2090008@talpey.com> References: <4F62131C.2090008@talpey.com> Message-ID: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> On 15.3.2012, at 18.04, Tom Talpey wrote: > I'm seeing a few warnings emitted when building for x86. They're pretty > obvious, but if you want the configure options etc, I can provide those. > > In Dovecot 2.1.2 (I also see some of these in 2.1.1): Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. From dluke at geeklair.net Thu Mar 15 18:33:20 2012 From: dluke at geeklair.net (Daniel J. Luke) Date: Thu, 15 Mar 2012 12:33:20 -0400 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: On Mac OS X 10.5.8 / darwin 9.8.0, I'm getting this error on startup again: dovecot[74267]: master: Fatal: kevent(EV_ADD, READ, 19) failed: Invalid argument dovecot.conf contains: service stats { fifo_listener stats-mail { mode = 0 } } which fixed the issue with 2.1.1 adding the following seems to have fixed things: service aggregator { fifo_listener replication-notify-fifo { mode = 0 } } from looking at config/all-settings.c it looks like I should maybe also add the following (but I have not tried it). service director { fifo_listener login/proxy-notify { mode = 0 } } It would be really nice if this failed more gracefully so the config tweaks weren't necessary. (I can work on a patch if it's something that would be accepted and if someone can point me in the right direction). -- Daniel J. Luke +========================================================+ | *---------------- dluke at geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ From mcazzador at gmail.com Thu Mar 15 18:42:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 17:42:03 +0100 Subject: [Dovecot] replication howto Message-ID: Hello, excuse me but there is some documentation about replication now? I dont' understand where i must put the lines below (dovecot.conf? , 20-imap?) Excuse but it's not so clear for me cause i'm a new dovecot user. Another question, i use virtual users on mysql backend , so for replication i need to give ssh at every virtual users? Or i can use a only use a system ssh user? Thank's service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From trashcan at odo.in-berlin.de Thu Mar 15 19:09:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:09:21 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi -- On 15.03.2012, at 17:42, Matteo Cazzador wrote: > Hello, excuse me but there is some documentation about replication now? Not that I'm aware of. > I dont' understand where i must put the lines below (dovecot.conf? , > 20-imap?) You can put them wherever you wish, as long as you include that part of your configuration. Myself, I'm still using a single dovecot.conf, only. > Another question, i use virtual users on mysql backend , so for > replication i need to give ssh at every virtual users? > Or i can use a only use a system ssh user? If I'm not mistaken, you can use a single ssh user, and you could use the vmail user for instance. That's what I do, and I'm using sqlite for userdb. Here's my configuration: ----------------------------------------------------------------------- If you choose to run ssh on a different port from the default one, you need: ## ssh command line used in dsync replication (ssh port added) # dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} If not, you can start here: ## --- DSYNC REPLICATION ---------------------------------------- # # aggregator, replicator, doveadm, and config needed, and # dsync_remote_cmd if running ssh via non-default port # service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } The following part is for server 1, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server2.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } The following part is for server 2, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server1.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } HTH, Michael From mcazzador at gmail.com Thu Mar 15 19:16:17 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:16:17 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi, thank's a lot! for your detailed answer. About ssh (excuse for my english) i think you correctly understand what is "my problem" with virtual user (i have no system user ) and there are not ssh account. So i must use a dedicate account for replication (ssh) that must act sync for all virtual mail account. Thank' s i try you suggest now! Il 15 marzo 2012 18:09, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 17:42, Matteo Cazzador wrote: > >> Hello, excuse me but there is some documentation about replication now? > > Not that I'm aware of. > >> I dont' understand where i must put the lines below (dovecot.conf? , >> 20-imap?) > > You can put them wherever you wish, as long as you include that part > of your configuration. Myself, I'm still using a single dovecot.conf, > only. > >> Another question, i use virtual users on mysql backend , so for >> replication i need to give ssh at every virtual users? >> Or i can use a only use a system ssh user? > > If I'm not mistaken, you can use a single ssh user, and you could use > the vmail user for instance. That's what I do, and I'm using sqlite for > userdb. > > Here's my configuration: > ----------------------------------------------------------------------- > > If you choose to run ssh on a different port from the default one, you need: > > ? ## ssh command line used in dsync replication (ssh port added) > ? # > ? dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > > > If not, you can start here: > > ? ## --- DSYNC REPLICATION ---------------------------------------- > ? # > ? # aggregator, replicator, doveadm, and config needed, and > ? # dsync_remote_cmd if running ssh via non-default port > ? # > ? service aggregator { > ? ? ? ?# give enough permissions for mail processes > ? ? ? ?# > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? } > ? service replicator { > ? ? ? ?# start replication at startup > ? ? ? ?# > ? ? ? ?process_min_avail = 1 > ? } > ? service doveadm { > ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail > ? ? ? ?# (not root) > ? ? ? ?# > ? ? ? ?user = vmail > ? } > ? service config { > ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm > ? ? ? ?# > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > ? } > > > > The following part is for server 1, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server2.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > > > The following part is for server 2, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server1.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > HTH, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jtl+dovecot at uvm.edu Thu Mar 15 19:23:01 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:23:01 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <1331814312.10319.18.camel@innu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> Message-ID: <4F622575.7050405@uvm.edu> On 3/15/12 8:25 AM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: >> On 3/15/12 6:02 AM, Timo Sirainen wrote: >>> Hi, >>> >>> On 15.3.2012, at 3.24, Jim Lawson wrote: >>>> We have a 2-node director setup which front-ends for 4 nodes which share >>>> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. >>> .. >>>> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >>>> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >>>> (proxy->data_input >>>> ->eof) >>> I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. >>> >> I'll give it a shot. For the purposes of doing a rolling upgrade, is it >> reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for >> the duration, or should I split-brain them during the upgrade? > I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The > current hg version has some extra features, but it doesn't use them > until all of the directors have upgraded to the new version. > Trying with v2.1.2 (peer is v2.0.18): Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left == NULL && dir->right == NULL)) Mar 15 13:15:53 imapdir2 dovecot: director: Fatal: master: service(director): child 513 killed with signal 6 (core not dumped) Mar 15 13:15:53 imapdir2 dovecot: director: Error: Director 132.198.100.149:9090/right disconnected Which is OK, I can run them split-brained (rules in iptables to prevent directors from talking) while I move users around. It'll mean poor performance for GFS for the duration, but that's better than an outage. The good news is, the lmtp problem I wrote about above appears to be fixed. Thanks !!! Jim From trashcan at odo.in-berlin.de Thu Mar 15 19:28:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:28:37 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:16, Matteo Cazzador wrote: > with virtual user (i have no system user ) and there are not ssh > account. So i must use a dedicate account for replication (ssh) > that must act sync for all virtual mail account. Yes, that's what I use. I did create a dedicated account for vmail with all the necessary ssh stuff in ~vmail/.ssh One remark I forgot to mention in my last mail: >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> # >> user = vmail >> } This part is only needed, if you choose to run device doveadm as user vmail like I do. >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> # >> unix_listener config { >> user = vmail >> } >> } Regards, Michael From andrei at lctax.ro Thu Mar 15 19:49:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 13:49:58 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: > The day I switched to the new replicator/dsync technique, those > duplicates > are history, but I'm still able to produce duplicates (and multiples) > if Hello, Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated. I only do replication using the doveadm sync command. My servers are geographically distributed as you might remember from previous posts so I run doveadm every 5 minutes, and only 1 instance of doveadm runs at any given times (so let's say that due to a HUGE volume the doveamd take 30 minutes to complete, then all in-between 5minutes are skipped). Thnx, Andrei From tss at iki.fi Thu Mar 15 19:52:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:52:58 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F622575.7050405@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> Message-ID: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> On 15.3.2012, at 19.23, Jim Lawson wrote: >> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >> current hg version has some extra features, but it doesn't use them >> until all of the directors have upgraded to the new version. >> > Trying with v2.1.2 (peer is v2.0.18): > > Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line > 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left > == NULL && dir->right == NULL)) This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? From tss at iki.fi Thu Mar 15 19:53:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:53:57 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: On 15.3.2012, at 19.49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. > > I only do replication using the doveadm sync command. Try at least v2.1.2 first, since it has some fixes. Also post your doveconf -n output. From jtl+dovecot at uvm.edu Thu Mar 15 19:55:57 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:55:57 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> Message-ID: <4F622D2D.80802@uvm.edu> On 3/15/12 1:52 PM, Timo Sirainen wrote: > On 15.3.2012, at 19.23, Jim Lawson wrote: > >>> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >>> current hg version has some extra features, but it doesn't use them >>> until all of the directors have upgraded to the new version. >>> >> Trying with v2.1.2 (peer is v2.0.18): >> >> Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line >> 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left >> == NULL && dir->right == NULL)) > This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? > That's correct. Configs are the same between directors (same as I sent in the original msg) Jim From mcazzador at gmail.com Thu Mar 15 19:57:22 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:57:22 +0100 Subject: [Dovecot] replication howto In-Reply-To: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi, yes it'a good idea but i'm using now root i hope this not invalid all I obtain this error but maybe i need some pause Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF Thank's a lot! Il 15 marzo 2012 18:28, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 18:16, Matteo Cazzador wrote: > >> with virtual user (i have no system user ) and there are not ssh >> account. So i must use a dedicate account for replication (ssh) >> that must act sync for all virtual mail account. > > Yes, that's what I use. I did create a dedicated account for vmail > with all the necessary ssh stuff in ~vmail/.ssh > > One remark I forgot to mention in my last mail: > >>> ? service doveadm { >>> ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail >>> ? ? ? ?# (not root) >>> ? ? ? ?# >>> ? ? ? ?user = vmail >>> ? } > > This part is only needed, if you choose to run device doveadm as user > vmail like I do. > >>> ? service config { >>> ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm >>> ? ? ? ?# >>> ? ? ? ?unix_listener config { >>> ? ? ? ? ? ? ? ?user = vmail >>> ? ? ? ?} >>> ? } > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From campbell at cnpapers.com Thu Mar 15 21:06:31 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 15:06:31 -0400 Subject: [Dovecot] Lack of external documentation? Message-ID: <4F623DB7.9060707@cnpapers.com> Firstly, this isn't meant to be critical, and I realize the subject line probably suggest criticism, so... I was sort of forced into using dovecot as my imap/pop server due to upgrading 3 versions of OS on my mail servers. So far, that's not bad. What surprises me is that one of the first things I usually do whenever I start using different software is to purchase a book that seems to suit me. Searching all of the common places like amazon, ebay, etc for manuals turned up little to nothing on dovecot. I'm wondering why and is this so new that people just haven't written books about it yet? The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Thanks for all the help I've received so far and I think I'm really going to like dovecot. Once I get the hang of it, I'll probably reduce the amount of noise on the list by half. steve campbell From terry at cnysupport.com Thu Mar 15 21:27:37 2012 From: terry at cnysupport.com (Terry Carmen) Date: Thu, 15 Mar 2012 15:27:37 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <4F6242A9.6090209@cnysupport.com> On 03/15/2012 03:06 PM, Steve Campbell wrote: > Firstly, this isn't meant to be critical, and I realize the subject > line probably suggest criticism, so... > > I was sort of forced into using dovecot as my imap/pop server due to > upgrading 3 versions of OS on my mail servers. So far, that's not bad. > What surprises me is that one of the first things I usually do > whenever I start using different software is to purchase a book that > seems to suit me. Searching all of the common places like amazon, > ebay, etc for manuals turned up little to nothing on dovecot. > > I'm wondering why and is this so new that people just haven't written > books about it yet? > > The one thing I'm a little critical of, though, is that trying to make > heads or tails of dovecot by following the online documentation is a > little problematic. I'm constantly jumping to another page and then > back to the original page, and for the most part, I just don't know > enough about it all yet to know what I'm looking for. The best docs are on the wiki and this mailing list. If you find the information in the wiki to be lacking, the best thing you can do is find the solution yourself and/or on this mailing list, and then make a wiki entry so the next person will know how to solve the same problem you had. Dovecot is a complex piece of software, and understanding some functionality requires reading the wiki, asking on the mailing list and/or examining the source code. You can also obtain paid support from these companies: http://dovecot.org/support.html I'll be the first to admit that complex and specialized configurations are sometimes difficult to figure out, however this list has always been a tremendous amount of help. Terry From stan at hardwarefreak.com Thu Mar 15 21:44:57 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 15 Mar 2012 14:44:57 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F61C99F.2040409@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> <4F61C99F.2040409@Media-Brokers.com> Message-ID: <4F6246B9.5080309@hardwarefreak.com> On 3/15/2012 5:51 AM, Charles Marcus wrote: > On 2012-03-01 8:38 PM, Stan Hoeppner wrote: >> Get yourself a qualified network architect. Pay for a full network >> traffic analysis. He'll attach sniffers at multiple points in your >> network to gather traffic/error/etc data. Then you'll discuss the new >> office, which employees/types with move there, and you'll be able to >> know almost precisely the average and peak bandwidth needs over the MAN >> link. He'll very likely tell you the same thing I have, that a single >> gigabit MAN link is plenty. If you hire him to do the work, he'll >> program the proper QOS setup to match the traffic patterns gleaned from >> the sniffers. > > Finally had time to properly review your answers here Stan. > > The time you took for the in-depth reply is very much appreciated - and Multi-site setups can be tricky as they often temp folks to do unnecessary things they otherwise would not. Just trying to help keep your sails pointed in the right direction. :) #1 rule when building a multi-site network: only duplicate hardware and services at the remote site(s) when absolutely necessary. > I'm sure you got a kick out of the level of my ignorance... ;) Not at all. I'm sure there is some subject or another where you would demonstrate my ignorance. From another perspective, if there was no ignorance left on the planet then there would be nothing left for anyone to learn. That would make for a boring world. > As for hiring a network architect, I will absolutely be doing as you > recommend (was already planning on it), but with the information I'm now > armed with, at least I'll have a better chance of knowing if they know > what they are doing/talking about... Now that you are aware of network analysis using sniffers, allow me to throw you a curve ball. For a network of your size, less than 70 users IIRC, with a typical application mix but with SMB/NFS traffic/file sizes a little above 'average', a qualified engineer probably won't need to plug sniffers into your network to determine the size MAN pipe and what traffic shaping you'll need. He'll have already done a near identical setup dozens of times. The good news is this saves you a few grand. Analysis with sniffers ain't cheap, even for small networks. And sniffers are normally only deployed to identify the cause of network problems, not very often for architectural or capacity planning. But, asking him about doing a full analysis using sniffers, and hearing his response, may lead to a valuable discussion nonetheless. Have your MAN and internet providers' (if not the same company) pricing sheet(s) in hand when you meet with the engineer. Depending on fast ethernet MAN, GbE MAN, and internet pipe pricing, he may have some compelling options/recommendations for you, possibly quite different, less costly, and more redundant than what you have been considering up to this point. > I'm still planning for the two physical servers (one at each location), Again, if you don't _need_ hardware and services at the 2nd site to achieve the current service level at the primary site, do not add these things to the 2nd site. I really want to put a bunch of exclamation points here but I hate exclamation points in technical emails--actually I just hate them, period. ;) > but you have convinced me that trying to run two live mail systems is an > unnecessary and even unwanted level of complexity. Running an active/active Dovecot cluster doesn't guarantee an unnecessary nor unwanted additional complexity. The need for clustering should go through a justification process just like anything else: what's the benefit, total 'cost', what's the ROI, etc. Lots of people here do active/active clustering every day with great success. Connecting the cluster nodes over a MAN link, however, does introduce unnecessary complexity. Locating one node in another building many blocks away is unnecessary. Putting the nodes in the same rack/room is smart, and easily accomplished in your environment, gives you the redundancy above, but without the potentially problematic MAN link as the cluster interconnect. Granted you'll need to build two new (preferably identical) systems from scratch and setup shared storage (DRBD or a SAN array) and GFS2 or OCFS, etc. Given your environment, there are only two valid reasons for locating equipment and duplicating data and services at a remote site: 1. Unrecoverable network failure (due to single MAN link) 2. Unrecoverable primary site failure (natural or man made disaster) #1 is taken care of by redundant MAN links #2 you've never planned for to this date (probability is *low*) and you need _everything_ duplicated at the remote site Duplicating servers for high(er) user throughput/lower latency to/from servers isn't a valid reason for remote site duplication in your case because you are able to afford plenty of bandwidth and link redundancy between the sites. The relative low cost and high bandwidth of the MAN link outweighs any benefit of service replication due to the latter's complexity level. Here are some other 'rules': 1. Don't duplicate servers at remote sites to mitigate network link failure when sites are close and redundant bandwidth is afforadable 2. Do duplicate network links to mitigate link failure when sites are close and bandwidth is affordable 3. Implement and test a true disaster avoidance and recovery plan > The DC VM will still > be hot (it is always best to have two DCs in a windows domain > environment anyway) so I'll get automatic real time off site backup of > all of the users data (since it will all be on DFS), but for the mail > services, I'll just designate one as live, and one as the hot/standby > that is kept in sync using dsync. This way I'll automatically get off > site back up for each site for the users data stored in the DFS, and > have a second mail system ready to go if something happens to the primary. Again, you're not looking at this network design from the proper perspective. See rules 1-3 above. Off site backups/replication are used exclusively to mitigate data loss due to catastrophic facility failure, not server failure, enabling rapid system recovery when new equipment has arrived. Many business insurers have catastrophic IT equipment replacement plans and relationships with the big 5 hardware vendors, enabling you to get new new equipment racked and begin your restore from offsite tape, within as little as 24 hours of notification. Think of how FEMA stages emergency supplies all around the country. Now think 10 times better, faster. Such services increase your premiums, but if you're serious about disaster avoidance and recovery, this is the only way to go. IBM, HP, maybe Dell, Sun (used to anyway), have dedicated account reps for disaster recovery. They work with you to keep an inventory of all of your systems and storage. Your records are constantly updated when your products are EOL'd or superseded or you replace or add hardware, and a list is maintained of current hardware best matched to replace all of your now burned, flooded, tornado shredded, hurricane blasted equipment, right down to bare metal restore capability, if possible/applicable. You plan to replicate filesystem user data and mailbox data to a 2nd site to mitigate single server failures. Why does that need to be done to an offsite location/system? It doesn't. There is no benefit whatsoever. You can accomplish this in the same rack/room and get by with a smaller MAN pipe saving time, money, and administrative burden. The restore procedure will be faster if all machines are in the same rack/room and you're using tape, and you won't slow users down with restore traffic going over the MAN link. If you really want off-site backup, for what it's meant to accomplish, get a network attached tape library/silo, or a speedy high cap LTO-4/5 tape drive in each server, put a real backup rotation and restore plan in place, and store backup tapes in a secure facility. A remote "hot site" is great when it's in a different city, better yet region, or in a hardened facility in any locale. Your hot site is only a few blocks away. If your primary site it taken out by anything other than fire, such as a tornado, earthquake, hurricane being more likely in your case, chances are your hot site may go down soon after the primary. If you want/need a real off site backup solution, rotate tapes to an everything-proof facility. Here are 3 companies in the Atlanta area that offer media rotation storage services. Watch the Offsite Tape Vaulting video at IronMountain: http://www.ironmountain.com/Knowledge-Center/Reference-Library/View-by-Document-Type/Demonstrations-Videos/Tours/Offsite-Tape-Vaulting.aspx http://www.askads.net/media-rotation/ http://www.adamsdatamanagement.com/tape-rotation-atlanta-ga.htm > Again, thanks Stan... I am constantly amazed at the level of expertise > and quality of advice available *for free* in the open source world, as > is available on these lists. Always glad to assist my brethren in this digital kingdom. Whichever architecture/topology you choose, remote replicated systems or not, I hope my input has given you some good information on which to base your decisions. -- Stan From list at airstreamcomm.net Thu Mar 15 21:48:57 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 15 Mar 2012 14:48:57 -0500 Subject: [Dovecot] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> On Thu, 15 Mar 2012 16:53:53 +0200, Timo Sirainen wrote: > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. Are there any performance metrics around dsync replication, such as how many users this has been tested on, or how long the replication take to occur? Also I have not been able to determine from reading the mailinglist whether or not dsync replication works with different types of mailboxes (maildir, dbox, mbox), what is supported? From tss at iki.fi Thu Mar 15 21:55:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 21:55:57 +0200 Subject: [Dovecot] v2.1.2 released In-Reply-To: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> References: <1331823233.10319.40.camel@innu> <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> Message-ID: <6489C385-E8C3-425E-8D2D-B3A242A6E0AF@iki.fi> On 15.3.2012, at 21.48, wrote: > Are there any performance metrics around dsync replication, such as how > many users this has been tested on, or how long the replication take to > occur? The performance isn't optimal yet. You can probably replicate some hundreds of users ok, maybe thousands, but depends. > Also I have not been able to determine from reading the mailinglist > whether or not dsync replication works with different types of mailboxes > (maildir, dbox, mbox), what is supported? Maildir and dbox is supported, mbox probably works okayish but since it doesn't have proper message GUIDs you could run into trouble. From tom at talpey.com Thu Mar 15 22:23:19 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 16:23:19 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> Message-ID: <4F624FB7.9000408@talpey.com> On 3/15/2012 12:25 PM, Timo Sirainen wrote: > On 15.3.2012, at 18.04, Tom Talpey wrote: > >> I'm seeing a few warnings emitted when building for x86. They're pretty >> obvious, but if you want the configure options etc, I can provide those. >> >> In Dovecot 2.1.2 (I also see some of these in 2.1.1): > > Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. Confirmed, Dovecot builds cleanly for me now. Thanks Timo! The pigeonhole warning appears to be harmless and I'll wait for Stefan to confirm/address. From giles at coochey.net Thu Mar 15 22:30:19 2012 From: giles at coochey.net (Giles Coochey) Date: Thu, 15 Mar 2012 20:30:19 +0000 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F62515B.1050207@coochey.net> On 15/03/2012 19:27, Terry Carmen wrote: > On 03/15/2012 03:06 PM, Steve Campbell wrote: >> Firstly, this isn't meant to be critical, and I realize the subject >> line probably suggest criticism, so... >> >> I was sort of forced into using dovecot as my imap/pop server due to >> upgrading 3 versions of OS on my mail servers. So far, that's not >> bad. What surprises me is that one of the first things I usually do >> whenever I start using different software is to purchase a book that >> seems to suit me. Searching all of the common places like amazon, >> ebay, etc for manuals turned up little to nothing on dovecot. >> >> I'm wondering why and is this so new that people just haven't written >> books about it yet? >> >> The one thing I'm a little critical of, though, is that trying to >> make heads or tails of dovecot by following the online documentation >> is a little problematic. I'm constantly jumping to another page and >> then back to the original page, and for the most part, I just don't >> know enough about it all yet to know what I'm looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always > been a tremendous amount of help. > > Terry > > > > > > What he said +1. I don't want to be-little IMAP software or the work that Timo has done to get dovecot to the IMAP server world, but IMAP in general is a small enough subject to only really warrant two maybe three books - the most recent of which was written 5-7 years ago. The original release of dovecot was around 2002, but I don't think it became as widely adopted as Courier / Cyrus until around 2010. I wouldn't be surprised that if there is a next edition of "The Book of IMAP" or the O'reilly "Managing IMAP" that there would probably be an equal share section on dovecot than any other server out there. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From jerry at seibercom.net Thu Mar 15 22:46:18 2012 From: jerry at seibercom.net (Jerry) Date: Thu, 15 Mar 2012 16:46:18 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <20120315164618.705ca356@scorpio> On Thu, 15 Mar 2012 15:27:37 -0400 Terry Carmen articulated: > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > Firstly, this isn't meant to be critical, and I realize the subject > > line probably suggest criticism, so... > > > > I was sort of forced into using dovecot as my imap/pop server due > > to upgrading 3 versions of OS on my mail servers. So far, that's > > not bad. What surprises me is that one of the first things I > > usually do whenever I start using different software is to purchase > > a book that seems to suit me. Searching all of the common places > > like amazon, ebay, etc for manuals turned up little to nothing on > > dovecot. > > > > I'm wondering why and is this so new that people just haven't > > written books about it yet? > > > > The one thing I'm a little critical of, though, is that trying to > > make heads or tails of dovecot by following the online > > documentation is a little problematic. I'm constantly jumping to > > another page and then back to the original page, and for the most > > part, I just don't know enough about it all yet to know what I'm > > looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized > configurations are sometimes difficult to figure out, however this > list has always been a tremendous amount of help. The lack of truly informative documentation has been the Achilles' heel of open-source software since its inception. I feel your pain. I have always loved a hard copy, i.e. book documenting the subject I am studying. Jumping from screen to screen sucks, plus how do I highlight a passage on the monitor for future reference? There have been a few books written to document Postfix, but to the best of my knowledge, none exist for Dovecot. -- Jerry ? Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __________________________________________________________________ From e-frog at gmx.de Thu Mar 15 22:46:22 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 15 Mar 2012 21:46:22 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F60F29D.2010409@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> <4F60F29D.2010409@gmx.de> Message-ID: <4F62551E.1000102@gmx.de> On 14.03.2012 20:33, wrote e-frog: > On 14.03.2012 14:41, wrote Timo Sirainen: >> >> With latest hg version it should work. >> > > Hi Timo, > > The "can't delete mailbox INBOX" error is gone now with changeset > c077ca9bc306 and it's working successfully on the account from yesterday > where it also worked with mailbox_list_index=no. > > However using a different account (more mail and mailboxes) I'm seeing > dbox corruption errors. I have tested with mailbox_list_index=yes and no > and it's the same for both. So this might be unrelated to this setting. > Attached are logs from doveadm backup runs. First to an empty directory > and 2 consecutive runs. > Further testing (now with 2.1.2) shows it only seems to work for a single mailbox. e.g. doveadm -v backup -u testuser at ubuntu-test.localdomain -m 'INBOX' mdbox:/tmp/backup dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=c63f581c030b774b572a0000ec8d17cd) -> no errors This works for every single mailbox in this account. The errors only occur without -m 'mailbox'. Using maildir as destination format however seems to work fine on the whole account. doveadm -v backup -u testuser at ubuntu-test.localdomain maildir:/tmp/backup -> no errors Thanks, e-frog From trashcan at odo.in-berlin.de Thu Mar 15 22:48:31 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:48:31 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi -- On 15.03.2012, at 18:57, Matteo Cazzador wrote: > Hi, yes it'a good idea but i'm using now root i hope this not > invalid all Actually it's a bad idea to use root for ssh from a security point of view. A hacked root account isn't fun. Thus, normally one needs to explicitly change the config of the sshd daemon to allow root logins (at least with FreeBSD what I'm using). Thus, I do recommend to use an unprivileged user like vmail. > I obtain this error but maybe i need some pause ;-) > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found root doesn't not find doveadm at the remote server. As mentioned above you better create an account for vmail and allow that user to find doveadm in its path. > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF That's an error due to not finding doveadm at the remote site. Regards, Michael From trashcan at odo.in-berlin.de Thu Mar 15 22:55:17 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:55:17 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html and http://www.dovecot.org/img/dsync-director-replication-ssh.png helped me a lot understand the idea behind it. > As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. As Timo recommended already, you better upgrade to 2.1.2 first. I can confirm that he fixed a lot compared to older dsync versions. Regards, Michael From tss at iki.fi Thu Mar 15 23:01:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:01:39 +0200 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <67E4C4F8-A9CE-4912-9B3F-05770041C383@iki.fi> On 15.3.2012, at 21.06, Steve Campbell wrote: > The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Perhaps it would be helpful to have some more talkative howtos for some of the typical configurations, that don't only list the options that are given but actually talks about why things are done the way they are? I've tried to avoid duplication of text in wiki, because if something changes it's difficult to update it everywhere, but in howtos I guess it wouldn't be too bad. Or maybe the wiki could be restructured in some way to make it easier to follow. I think I'm the worst possible person to figure out anything like that, because I don't know what the difficult parts are. I'd think the Dovecot wiki is good if you know what you want to do and just want to know some specifics, but I guess it can be difficult to figure things out otherwise. > Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? A few people have talked about writing a Dovecot book and I've promised to help them, but no one's actually written one as far as I know. > So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Features that aren't yet even fully implemented don't really have documentation for them. From tss at iki.fi Thu Mar 15 23:05:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:05:54 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> On 15.3.2012, at 22.48, Michael Grimm wrote: > On 15.03.2012, at 18:57, Matteo Cazzador wrote: > >> Hi, yes it'a good idea but i'm using now root i hope this not >> invalid all > > Actually it's a bad idea to use root for ssh from a security point > of view. A hacked root account isn't fun. Thus, normally one needs > to explicitly change the config of the sshd daemon to allow root > logins (at least with FreeBSD what I'm using). Thus, I do recommend > to use an unprivileged user like vmail. Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. From hoogendyk at bio.umass.edu Thu Mar 15 23:13:34 2012 From: hoogendyk at bio.umass.edu (Chris Hoogendyk) Date: Thu, 15 Mar 2012 17:13:34 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <4F625B7E.5060902@bio.umass.edu> On 3/15/12 4:46 PM, Jerry wrote: > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > >> On 03/15/2012 03:06 PM, Steve Campbell wrote: >>> Firstly, this isn't meant to be critical, and I realize the subject >>> line probably suggest criticism, so... >>> >>> I was sort of forced into using dovecot as my imap/pop server due >>> to upgrading 3 versions of OS on my mail servers. So far, that's >>> not bad. What surprises me is that one of the first things I >>> usually do whenever I start using different software is to purchase >>> a book that seems to suit me. Searching all of the common places >>> like amazon, ebay, etc for manuals turned up little to nothing on >>> dovecot. >>> >>> I'm wondering why and is this so new that people just haven't >>> written books about it yet? >>> >>> The one thing I'm a little critical of, though, is that trying to >>> make heads or tails of dovecot by following the online >>> documentation is a little problematic. I'm constantly jumping to >>> another page and then back to the original page, and for the most >>> part, I just don't know enough about it all yet to know what I'm >>> looking for. >> The best docs are on the wiki and this mailing list. If you find the >> information in the wiki to be lacking, the best thing you can do is >> find the solution yourself and/or on this mailing list, and then make >> a wiki entry so the next person will know how to solve the same >> problem you had. >> >> Dovecot is a complex piece of software, and understanding some >> functionality requires reading the wiki, asking on the mailing list >> and/or examining the source code. You can also obtain paid support >> from these companies: http://dovecot.org/support.html >> >> I'll be the first to admit that complex and specialized >> configurations are sometimes difficult to figure out, however this >> list has always been a tremendous amount of help. > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. I like books, but, especially in the case of actively developed software such as Dovecot, they become outdated very quickly. I have two editions of the Unix System Administration Handbook (can't remember the last time I looked at them), and two editions of Backup & Recovery. I use Amanda for backup. It has been developed actively over the last several years, and the Backup & Recovery chapter on Amanda is sorely out of date. The wiki, the users mailing list, and the man pages are the only way to really be up-to-date. With the book, you won't know anything about any changes or additions since the book was written, which would have been at least many months before it was published. I'm into online documentation every day. I'm a Solaris admin, but I've been jumping from Solaris 10 to Ubuntu without any books, and I've been jumping from ZFS to LVM without any books. That's a significant transition. But it seems I can find almost everything online. Sometimes another admin gives me an explanation and a link. It's just the way things are. The digital world is moving too fast to be frozen in print. -- --------------- Chris Hoogendyk - O__ ---- Systems Administrator c/ /'_ --- Biology& Geology Departments (*) \(*) -- 140 Morrill Science Center ~~~~~~~~~~ - University of Massachusetts, Amherst --------------- Erd?s 4 From amk at spamfence.net Thu Mar 15 23:43:02 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Thu, 15 Mar 2012 21:43:02 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: <1331816286.10319.23.camel@innu.invalid> Message-ID: Timo Sirainen wrote: >> $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch >> $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install > > You would have needed to run autogen.sh again. It works with me now that > I tried in a test server with OpenSSL in non-standard dir. Sorry, I didn't know that with "autogen.sh". Just grabbed Dovecot 2.1.2 (which is all properly set up - so I couldn't do anything wrong ;-) and compiled it. Compilation works. Great! The binaries find all their libraries. But two libraries are not quite okay. They don't find their SSL libs: libdovecot-lda.so libdovecot-storage.so Since libdovecot-lda.so doesn't contain the words libssl or libcrypto, I guess that ldd just complains because it uses libdovecot-storage.so. Thus, libdovecot-storage.so is the (only) one left with an incomplete library search path. Luckily, all binaries use some additional libraries which come with a proper library path. So the whole things works, but it's more like some kind of magic. It would be great if libdovecot-storage.so could be fixed as well to make things finally perfect. Thanks for all your effort. (I know this isn't top priority as most people use precompiled stuff and never run into such kind of things.) Greetings, Andreas From p at state-of-mind.de Thu Mar 15 23:46:59 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Thu, 15 Mar 2012 22:46:59 +0100 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <20120315214658.GC3750@state-of-mind.de> * Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. Dovecot is a moving target and it is hard to produce any print that represents what Dovecot can do when the print finally will be released. I know, because I am one of the two authors who wrote "The Book of Postfix" and we found it hard if almost impossible to keep up with Wietse's pace when he wrote major parts of Postfix. For now, I believe, the wiki and the mailing list is as good as it gets. Later when Dovecot settles a book might be something to write and something to spend money on because it lasts for a while. man pages would be a good thing, but given Dovecots configuration syntax and flexibility this might be an even harder task. Its probably easier to describe certain aspects of configuration or use cases than list all options and their possible occurences. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 From david at blue-labs.org Thu Mar 15 23:49:54 2012 From: david at blue-labs.org (David Ford) Date: Thu, 15 Mar 2012 17:49:54 -0400 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <4F626402.4030606@blue-labs.org> in ~privilgeduser/.ssh/authorized keys: from= cmd=dsync.sh pubkey... On 03/15/2012 05:05 PM, Timo Sirainen wrote: > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. And if you allow ssh > login only with public key authentication I don't think there are much > security issues. And finally, it would be possible to write a small > wrapper that allows the root's public key auth to only execute > dsync-user.sh script that can't do anything except sync a specified > user's mails. From andrei at lctax.ro Thu Mar 15 23:52:39 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 17:52:39 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <64c9b1a4813862ad254f591c6a5ffc02.squirrel@web.miau.ca> Hello Timo, I have update the repository with hg pull -u, recompiled and redeployed and somehow the dovecot -n still shows 2.1.1... :( I ran exactly the same test: starting for 1 clean user1, I create 2 emails, one on mx1.a and one on mx2.a and I sync them with doveadm. The output is exactly as previously sent :( Here is my conf: # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } Thank you, Andrei > On 15.3.2012, at 19.49, Michescu Andrei wrote: > >> Can you get a little bit more in details about this replicator/dsync >> techique? As my main problem is that EVERYTHING (that gets created on >> different servers in the same time) gets duplicated. >> >> I only do replication using the doveadm sync command. > > Try at least v2.1.2 first, since it has some fixes. Also post your > doveconf -n output. > > > !DSPAM:4f622cb881591647615726! > > From tss at iki.fi Thu Mar 15 23:55:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:55:26 +0200 Subject: [Dovecot] replication howto In-Reply-To: <4F626402.4030606@blue-labs.org> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Plus the scripts that 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) and 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. On 15.3.2012, at 23.49, David Ford wrote: > in ~privilgeduser/.ssh/authorized keys: > > from= cmd=dsync.sh pubkey... > > On 03/15/2012 05:05 PM, Timo Sirainen wrote: >> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. > From stephan at rename-it.nl Fri Mar 16 00:17:40 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 23:17:40 +0100 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F624FB7.9000408@talpey.com> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> <4F624FB7.9000408@talpey.com> Message-ID: <4F626A84.1070705@rename-it.nl> On 3/15/2012 9:23 PM, Tom Talpey wrote: > On 3/15/2012 12:25 PM, Timo Sirainen wrote: >> On 15.3.2012, at 18.04, Tom Talpey wrote: >> >>> I'm seeing a few warnings emitted when building for x86. They're pretty >>> obvious, but if you want the configure options etc, I can provide >>> those. >>> >>> In Dovecot 2.1.2 (I also see some of these in 2.1.1): >> >> Thanks, fixed in hg. I guess I should add x86 vm building these >> nightly as well.. > > Confirmed, Dovecot builds cleanly for me now. Thanks Timo! > > The pigeonhole warning appears to be harmless and I'll wait for Stefan > to confirm/address. Thanks, fixed: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/75c1a2fd9b26 Regards, Stephan. From andrei at lctax.ro Fri Mar 16 01:20:00 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 19:20:00 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Message-ID: hello, So I upgraded to 2.1.2 (not from repository because that one still says 2.1.1, but from the release). I ran exactly the same test with exactly the same behaviour. (new account, synced successfully on 2 servers, deliver 1 email to each server, run doveadm sync)... Please find below the dovecot-uidlists: on mx1.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852540.19862.mx2,S=272 2 G1331852540.19862.mx2,S=272 :1331852573.M89342P19877.mx2,S=272 3 :1331852488.30409.mx1,S=268 on mx2.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852488.30409.mx1,S=268 2 :1331852540.19862.mx2,S=272 3 G1331852488.30409.mx1,S=268 :1331852572.M622052P30410.mx1,S=268 As you can see both servers duplicated the email that was delivered first to them (1 in both cases, because the user1 is a clean account). There is the same effect in the folders: initial there is only one file on each server and after sync there are 3 files instead of only 2... Also, after the sync, there should be 2 new emails (N2 if I interpret correctly that N1 means only one new). Thank you. Andrei PS: also I need to run dsync twice, because first time I receive: dsync-local(user1 at a): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at a): Info: INBOX: Couldn't keep all uids dsync-local(user1 at a): Warning: Mailbox changes caused a desync. You may want to run dsync again. The config is below: # 2.1.2: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From gedalya at gedalya.net Fri Mar 16 01:55:09 2012 From: gedalya at gedalya.net (Gedalya) Date: Thu, 15 Mar 2012 19:55:09 -0400 Subject: [Dovecot] IMAP to Maildir Migration preserving UIDs? In-Reply-To: <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> Message-ID: <4F62815D.7020002@gedalya.net> On 01/28/2012 12:45 PM, Timo Sirainen wrote: > On 27.1.2012, at 2.00, Gedalya wrote: > >> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> >> Program received signal SIGSEGV, Segmentation fault. >> mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 >> 213 mailbox-log.c: No such file or directory. >> in mailbox-log.c > This crash is now fixed, so there's no need to give /tmp/imapc path anymore: > http://hg.dovecot.org/dovecot-2.1/rev/7b94d1c8a6e7 > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0. From ml at smtp.fakessh.eu Fri Mar 16 02:31:59 2012 From: ml at smtp.fakessh.eu (ml) Date: Fri, 16 Mar 2012 01:31:59 +0100 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <1331857919.16694.5.camel@localhost> Le jeudi 15 mars 2012 ? 16:53 +0200, Timo Sirainen a ?crit : > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. > > i build with succes the last release for centos 5 work fine and best ns.fakessh.eu/rpms/dovecot-2.1.2-1.centme.el5.src.rpm thanks Timo -- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC2626742 gpg --keyserver pgp.mit.edu --recv-key C2626742 http://urlshort.eu fakessh @ http://gplus.to/sshfake http://gplus.to/sshswilting http://gplus.to/john.swilting https://lists.fakessh.eu/mailman/ This list is moderated by me, but all applications will be accepted provided they receive a note of presentation -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From campbell at cnpapers.com Fri Mar 16 03:08:15 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 21:08:15 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <1331860095.4f62927f9acd0@perdition.cnpapers.net> Quoting Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. > > -- > Jerry ??? > > Disclaimer: off-list followups get on-list replies or get ignored. > Please do not ignore the Reply-To header. > __________________________________________________________________ > So many great replies, but I'll pick this one to use as my reply-to since it mirrors mostly how I feel about my experiences so far when it comes to learning Dovecot. I installed a new server, going from Centos 3 to Centos 6. I found that Postfix was the preferred SMTP server and Dovecot was the preferred imap/pop server. I gave Postfix my best shot, but didn't really have it tested well enough to stick with it, so I dropped back to Sendmail, something I'm somewhat familiar with. I've read multiple versions of O'Reilly's Sendmail books along with the Sendmail Cookbook. I have to admit that it was these books that made me realize the power of Sendmail. Post l website to further learn, but I had to get the basics first to do what needed to be done to get the job into a working server. Dovecot is an application that probably would work out of the box for me if I didn't have to use data from the previous server. So I had to use more than the standard options to make this work. Finding those options was the main gripe I had with the wiki - there are just so many options to make Dovecot the complete server. That's a good thing. Just remember, us noobies-to-Dovecot have to discover all of those options. I mentioned that I was happy with the wiki and the list when it comes to answering my questions. But I'm sure the list will get tired of me asking what must appear to be redundant, simple, obnoxious questions. The index-like wiki page is most helpful. I knew dovecot has been around for a while, but didn't know how mature it was. The fact that Centos/Red Hat uses it as a default says quite a bit about it's reliability, so I'll stick with it. One of the the things I was planning on doing was combining two servers, which services one domain on one server and services two other domains on the other, into one server, and have the other as a server-in-waiting. So along comes this dsync thread, and now it appears that Dovecot might make that all easier. I see all the potential Dovecot has, but learning it is a little difficult for us new users. Once I get the hang of it, I'm sure I want need to search for the things I need to find, but for now, a good book would have been nice and a lot easier. I give all the praise to Timo that he deserves. (I'm guessing he's either the developer, the lead guru on the list or something of that stature). I like what I'm seeing, I'm just not always seeing what I need. Again, this is not critical in nature. I'm just stating what this particular rookie is conveying to the list about my experience (and lack of experience) in getting where I need to be with Dovecot. Thanks for such a great application, all the great replies and help so far. steve ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From lists at wiesinger.com Fri Mar 16 08:02:10 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Fri, 16 Mar 2012 07:02:10 +0100 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage Message-ID: <4F62D762.7080607@wiesinger.com> Hello, After fixing configuration and other issues I'm still having one problem with imap executable and pine: less .pinerc # Changed config: #rsh-command=/usr/sbin/dovecot --exec-mail imap rsh-command=/usr/local/bin/imap Calling imap still fails as non root: imap /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied collect2: ld returned 1 exit statusn Any ideas to fix it? BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Thnx. Ciao, Gerhard From tss at iki.fi Fri Mar 16 10:05:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 10:05:20 +0200 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <4F62D762.7080607@wiesinger.com> References: <4F62D762.7080607@wiesinger.com> Message-ID: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: > After fixing configuration and other issues I'm still having one problem with imap executable and pine: > less .pinerc > # Changed config: > #rsh-command=/usr/sbin/dovecot --exec-mail imap > rsh-command=/usr/local/bin/imap That's correct. > Calling imap still fails as non root: > imap > /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied > collect2: ld returned 1 exit statusn Huh? That looks like imap is running ld to link something. It shouldn't be doing that. > BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Maildir for reliability, sdbox/mdbox for performance. From mcazzador at gmail.com Fri Mar 16 10:37:47 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 09:37:47 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, thank's everybody, today afternoon i apply the suggest and i test solution. I post the actual configuration that i will test: vmail users is present too, i create ssh-keygen for users vmail and relative home directory and permit ssh with no password with user vmail on two servers. Then i use the configuration below i leave comment the line below or i need to active it excuse but i don't understand clear cause my terrible english? #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} and apply this on two servers service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } Thank's everyboy Il 15 marzo 2012 22:55, Timo Sirainen ha scritto: > Plus the scripts that > > 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) > > and > > 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself > > Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. > > On 15.3.2012, at 23.49, David Ford wrote: > >> in ~privilgeduser/.ssh/authorized keys: >> >> from= cmd=dsync.sh pubkey... >> >> On 03/15/2012 05:05 PM, Timo Sirainen wrote: >>> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. >> > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jernej.porenta at arnes.si Fri Mar 16 11:09:07 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Fri, 16 Mar 2012 10:09:07 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1331735355.2081.140.camel@innu> References: <1331735355.2081.140.camel@innu> Message-ID: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> On Mar 14, 2012, at 3:29 PM, Timo Sirainen wrote: > On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: >> Heya, >> >> We are expiriencing issues with dovecot 2.1.1 on Linux with weird >> filenames in home directory of username. We are using mbox IMAP >> folders, with no special changes (mail_location = mbox:~/:INBOX=% >> h/.mailbox). >> >> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >> (uni_utf8_str_is_valid(vname)) > .. >> AFAIK, the problem lies in processing the file list of home folder, >> which can contain filenames that do not have proper UTF-8 encoding of >> filenames, which causes dovecot to crash. > > Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This > should fix it by renaming such mailboxes: > http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. Whenever . LIST "" "*" is issued, dovecot crashes: Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b5466f3119c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Any clues? > >> On the other hand, UTF-8 filenames created on the system by hand >> (using touch), are not displayed in IMAP LIST command (sample is >> included in the folder structure; single letter file). > > This is a bit trickier problem. The mailbox names are currently stored > in filesystem as IMAP's modified-UTF7. So it's not really even currently > supposed to work, although it's not very nice that the mailboxes aren't > visible either. Maybe I'll do something smart in future for this, like > allowing both mUTF-7 and UTF-8 and remembering per-mailbox which > formatting it is in. I think we can leave this issue out, since I don't believe the users will be creating folders directly from interactive SSH sessions and rather use IMAP to create folders. So, this use-case is not very likely to occur. Thank you for your help... Cheers, Jernej From nmilas at noa.gr Fri Mar 16 11:26:45 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 11:26:45 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x Message-ID: <4F630755.7070909@noa.gr> Hi, A quick question: Are there any incompatibilities in config settings among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require changes of any config settings (as upgrading from v1.x to 2.0.x did)? Thanks, Nick From mstevens at imt-systems.com Fri Mar 16 11:39:53 2012 From: mstevens at imt-systems.com (Morten Stevens) Date: Fri, 16 Mar 2012 10:39:53 +0100 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <4F630755.7070909@noa.gr> References: <4F630755.7070909@noa.gr> Message-ID: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> On 16.03.2012 10:26, Nikolaos Milas wrote: > Hi, > > A quick question: Are there any incompatibilities in config settings > among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? > > That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require > changes of any config settings (as upgrading from v1.x to 2.0.x did)? Hi, See: http://wiki2.dovecot.org/Upgrading/2.1 Best regards, Morten From nicku at nicku.org Fri Mar 16 12:08:36 2012 From: nicku at nicku.org (Nick Urbanik) Date: Fri, 16 Mar 2012 21:08:36 +1100 Subject: [Dovecot] imaptest: performance testing Message-ID: <20120316100836.GA12049@nicku.org> Dear Folks, Using head of imaptest with dovecot 2.1, I am attempting to put a heavy load on the server, with insufficient success. I made 15000 user accounts, put them in to a file, one per line. Then I ran imaptest with ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 pass=SECRETPASSWORD But CPU load is only reaching 20. I put an mbox containing 67 messages into ~/mail/dovecot-crlf So what is the best way to put a really heavy load on an imap server? The documentation describes testing correctness of the server without really emphasising performance. I would like to do something like dnsperf, which adds a linearly increasing load until the server is unable to cope. This machine has 24G RAM and 8 cores. I'll be grateful for any constructive suggestions. -- Nick Urbanik http://nicku.org nicku at nicku.org GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24 From odhiambo at gmail.com Fri Mar 16 12:51:46 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 13:51:46 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 Message-ID: I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall. I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook pops a screen saying "IMAP session state is inconsistent, please relogin". Dovecot says: Mar 16 13:30:26 jaribu dovecot: master: Warning: Killed with signal 15 (by pid=72242 uid=0 code=kill) Mar 16 13:30:29 master: Info: Dovecot v2.1.2 starting up Mar 16 13:31:15 auth-worker(72594): Info: mysql(localhost): Connected to database exim4u Mar 16 13:31:15 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72595 Mar 16 13:31:15 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:15 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72646 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid=-1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72647 Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around offset=894): msg header has bad magic value Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage: rebuilding indexes Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: /var/spool/virtual/kictanet.or.ke/wash/mdbox/mailboxes/SentItems/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 16 13:33:23 imap(wash at kictanet.or.ke): Info: Disconnected: IMAP session state is inconsistent, please relogin. bytes=13816863/907529 My doveconf output is here -> http://pastebin.com/6yNP5ygt -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From mlopez at gtdinternet.com Fri Mar 16 13:07:15 2012 From: mlopez at gtdinternet.com (=?ISO-8859-1?Q?Mauricio_L=F3pez_Riffo?=) Date: Fri, 16 Mar 2012 08:07:15 -0300 Subject: [Dovecot] POP3 Performance Message-ID: <4F631EE3.40806@gtdinternet.com> Hi, We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith) Attach of dovecot -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail at mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert = References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F631F27.2050506@Media-Brokers.com> On 2012-03-15 3:27 PM, Terry Carmen wrote: > On 2012-03-15 3:06 PM, Steve Campbell wrote: >> Does anyone know of any manuals/books that have been written that >> might introduce me to most of the stuff in dovecot? > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always been > a tremendous amount of help. I agree completely. The ability to come to places like this and get answers directly from the software developer(s) is one of the main reasons I love open source software. And I will also say that Timo (yes, Steve, he is *the* dovecot developer, although he has had some excellent help for a while now) and this list is one of the most civil & respectful of any list I've been on, and the quality of support/answers is second to none. The postfix list is imo just as good as far as the quality of support, but they are very strict on 'form' - ie, no top-posting, you're expected (and often reminded) to read the instructions in the welcome message as to 'How to report a problem' and to actually follow those instructions - and quite often their replies seem harsh and unfriendly. I'd actually like to see dovecot have a similarly detailed welcome message (complete with a link to a detailed wiki page on 'How to Report a Problem' along with some helpful troubleshooting tips), but as much as I dislike top-posters (especially those who blindly quote the entire message they are replying to), I'm glad that this list is a bit less strict on form, and just seems more friendly. I for one would *love* to see some kind of 'The Book of Dovecot' (like 'The Book of Postfix'), but one reason I can see that would keep someone from wanting to write one is that dovecot (like most popular open source software) is still a very fast moving target as compared to the useful life of a book. Maybe his commercial support company can provide the resources for writing one once the target slows down a bit - or maybe even start off writing [a][some] smaller 'Basic Configuration' guide[s] for the things that aren't such fast moving targets that could eventually become chapters in a more comprehensive book. That would I think be a (admittedly probably fairly small) revenue generator, but hopefully at least enough to pay for itself and maybe provide a small profit. Another option I can think of would be for Timo to provide a method for people to pay a small fee for his support company to write up a custom 'How-To' for someone based on a list of requirements. I would imagine this as a web page that is put together with the appropriate questions, the answers for which are necessary to accomplish the goal. Of course, the other option is for other people to step up and 'fix the wiki' or 'write the Book' (or How-Tos), instead of just complaining about the lack (no offense, your 'complaint' wasn't all that bad). Yeah, I know this is the standard answer on free/open source software support lists, but it is the standard answer for a reason. On 2012-03-15 9:08 PM, Steve Campbell wrote: > I found that Postfix was the preferred SMTP server and Dovecot was > the preferred imap/pop server. I gave Postfix my best shot, but > didn't really have it tested well enough to stick with it, so I > dropped back to Sendmail, something I'm somewhat familiar with. I understand the argument for sticking with something you're familiar with, but I don't think you gave postfix a fair shot either - and it *does* have a number of excellent books written for it, so you don't have that excuse for postfix... ;). It is *much* easier to configure and run than sendmail, is much more performant and supposedly much more secure (just going by what I've read), and can do most anything that sendmail does (even supports milters). > Dovecot is an application that probably would work out of the box for > me if I didn't have to use data from the previous server. So I had to > use more than the standard options to make this work. Finding those > options was the main gripe I had with the wiki - there are just so > many options to make Dovecot the complete server. That's a good > thing. Just remember, us noobies-to-Dovecot have to discover all of > those options. You always have the option to get commercial support for fast resolutions to complex problems like this... ;) -- Best regards, Charles From robert at schetterer.org Fri Mar 16 14:10:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Fri, 16 Mar 2012 13:10:49 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F632DC9.4070108@schetterer.org> Am 16.03.2012 12:07, schrieb Mauricio L?pez Riffo: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see that > the performance is very poor and receive complaint about delays of > autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection (betwen > mtas and nfs storage) All account information work in LDAP plataform > (two servers in replicated mode, no high average or delays detected in > this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all of > service suffer a high load average (about 8 - 20 load average in each > dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, > autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, like > a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english > looks like you need to play with some config stuff and do more debug on your possible bottlenecks, what did you allready played with dovecot to high performance guess Timo will help about config settings after all for short to read http://wiki.dovecot.org/Authentication/Caching http://wiki.dovecot.org/LoginProcess http://wiki2.dovecot.org/Services any reasons for that mail_full_filesystem_access = yes ? general nfs may not optimal, but that a long story also virtual machines have their pros and contras maildir is not so good in performance -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From janfrode at tanso.net Fri Mar 16 14:11:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 16 Mar 2012 13:11:07 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <20120316121107.GA23566@dibs.tanso.net> One quick fix to try, if it's the login-time that's killing you, is to enable auth caching: http://wiki2.dovecot.org/Authentication/Caching that should offload your backend LDAP-servers from doing bind() on ever login, had a huge login performance impact for us. We use "auth_cache_size = 100 M", which gives us 99% cache hits: dovecot: auth: Authentication cache hits 3654591/3669119 (99%) dovecot: auth: Authentication cache inserts: positive: 588030 80931909B, negative: 912 49888B -jf From mcazzador at gmail.com Fri Mar 16 14:12:40 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 13:12:40 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, i obtain the same error Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF i've create vmail users (i've virtual domain netlite.locale (postfix), mysql backend i receive ana send mail correctly i use imap protocol), get ssh connection with publick key, i verify that with su - vmail , vmail find doveadm, i post my dovecot.conf file because i don't know what is wrong vmail exist on every server with publick key (server one => 10.0.0.118 server two => 10.0.0.122) dovecot.conf -> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm unix_listener config { user = vmail } } plugin { # this host replicates to remote host # mail_replica = remote:vmail at 10.0.0.122 # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hour } idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 thank's -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Fri Mar 16 14:49:55 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 14:49:55 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> References: <4F630755.7070909@noa.gr> <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> Message-ID: <4F6336F3.6040601@noa.gr> On 16/3/2012 11:39 ??, Morten Stevens wrote: > See: http://wiki2.dovecot.org/Upgrading/2.1 Thank you Morten. This was exactly what I was looking for. By the way, searching in the wiki2 for "Upgrade" does not locate the upgrade pages, except "Upgrading/1.0". The pages are found when searching for "upgrading". I am wondering whether it would be possible to add keyword(s) to the respective articles, in order to provide better search results. Thanks again, Nick From campbell at cnpapers.com Fri Mar 16 14:54:21 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Fri, 16 Mar 2012 08:54:21 -0400 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F6337FD.4070404@cnpapers.com> On 3/16/2012 7:07 AM, Mauricio L?pez Riffo wrote: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see > that the performance is very poor and receive complaint about delays > of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection > (betwen mtas and nfs storage) All account information work in LDAP > plataform (two servers in replicated mode, no high average or delays > detected in this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all > of service suffer a high load average (about 8 - 20 load average in > each dovecot) and authenticacion takes about 2 -10 seconds (in low > traffic, autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, > like a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = > maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english It doesn't seem to matter what type of hardware you might have, NFS can cause real bottlenecks, even to the point that your machine may report disk errors. Unfortunately, it's an evil necessity in some shops, but any way to eliminate NFS when large throughput is occurring will definitely help. Make sure you're running the latest version of NFS on all machines since V3 and V4 don't always like each other. I don't have a solution for it's replacement other than expensive hardware solutions. steve From mcazzador at gmail.com Fri Mar 16 15:02:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:02:03 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, with this changes first step is passed: I decomment this #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} i active and add absolute path of doveadm dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} But now from server 1 obtain Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no home directory Note: if i send a mail from server2 mail goes correcly in local (server 2) home virtual directory I note that when i launch manually from server1 sync, mysql on server 2 make correct sql to find home user dir On server 2 SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE username = 'matteo at netlite.locale' result are: +------------------------+------+------+------------------------------------------+ | maildir | uid | gid | mail | +------------------------+------+------+------------------------------------------+ | netlite.locale/matteo/ | 1000 | 1000 | /home/domini-posta/netlite.locale/matteo | +------------------------+------+------+------------------------------------------+ this configuration is teh same for server 1 and 2 thank's Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: > Hi, i obtain the same error > > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF > > > i've create vmail users (i've virtual domain netlite.locale (postfix), > mysql backend i receive ana send mail correctly i use imap protocol), > get ssh connection with publick key, > i verify that with su - vmail , vmail find doveadm, i post my > dovecot.conf file because i don't know what is wrong > > vmail exist on every server with publick key > > (server one => 10.0.0.118 > > server two => 10.0.0.122) > > dovecot.conf -> > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > service aggregator { > # give enough permissions for mail processes > # > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > } > service replicator { > # start replication at startup > # > ? ? ? ?process_min_avail = 1 > } > > > service doveadm { > # if you're using a single virtual user, set this to start ssh as vmail > # (not root) > > ? ? ? ?user = vmail > } > > service config { > # needed to grant access to /var/run/dovecot/config for service doveadm > > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > } > > plugin { > ? ? ? # this host replicates to remote host > # > ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 > > # run full synchronization mode every other hour > # (default is every 24 hours) > # > ? ? ? ?replication_full_sync_interval = 1 hour > } > > > idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 > > thank's > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:07:24 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:07:24 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) What does the CPU usage and NFS IOPS usage look like during those times? Meaning is the problem related to disk usage or something else? Note that for POP3 connections you don't get the "OK Logged in" reply until all of the message sizes have been read into memory. > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Are you randomly redirecting users to different Dovecot servers? Dovecot director would work better: http://wiki2.dovecot.org/Director > auth_worker_max_count = 100 Auth workers are irrelevant with LDAP. > lock_method = dotlock fcntl would be faster, if your NFS setup can handle it. > mail_cache_min_mail_count = 5 I'm not really sure if it's a good idea to ever set this anything else than 0. Of course if you have detected that this actually decreases disk IO I'd be interested to see numbers. > mail_full_filesystem_access = yes If your users are sharing the same UID, this means all the users can access each others' mails now! Even if they have different UIDs this is unlikely to be helpful. > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes Is /data also on NFS? Or does each server have its own local indexes? > maildir_copy_with_hardlinks = no This makes IMAP COPY command slower. Is there a reason why you've disabled it? > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } post-login script makes logins slower. What do you do in it? Why only for IMAP, not POP3? > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } These also slow logins down a little bit. v2.1 fixes that. > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } pop3_uidl_format=%u is a little bad, since it doesn't include %v. And you can improve pop3 performance with: pop3_no_flag_changes=yes And if the maildir filenames don't contain S=1234 sizes, this also makes a huge difference: pop3_fast_size_lookups=yes From mcazzador at gmail.com Fri Mar 16 15:10:04 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:10:04 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, Solved! i add at my sql SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail, '/home/domini-posta/netlite.locale/matteo' as home FROM mailbox WHERE username = 'matteo at netlite.locale' Now i've see first replication going!!! thank's everybody I hope my test help someone. Now i proceedd at use the replication system. Il 16 marzo 2012 14:02, Matteo Cazzador ha scritto: > Hi, with this changes first step is passed: > > I decomment this > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > i active and add absolute path of doveadm > > dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm > dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > But now ?from server 1 obtain > > Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no > home directory > > Note: if i send a mail from server2 mail goes correcly in local > (server 2) home virtual directory > > I note that when i launch manually from server1 sync, mysql on server > 2 make correct sql to find home user dir > > On server 2 > > SELECT maildir, 1000 AS uid, 1000 AS gid, > '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE > username = 'matteo at netlite.locale' > > result are: > > +------------------------+------+------+------------------------------------------+ > | maildir ? ? ? ? ? ? ? ?| uid ?| gid ?| mail > ? ? ? ? ? | > +------------------------+------+------+------------------------------------------+ > | netlite.locale/matteo/ | 1000 | 1000 | > /home/domini-posta/netlite.locale/matteo | > +------------------------+------+------+------------------------------------------+ > > this configuration is teh same for server 1 and 2 > > thank's > > Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: >> Hi, i obtain the same error >> >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: >> command not found >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: read() from worker server >> failed: EOF >> >> >> i've create vmail users (i've virtual domain netlite.locale (postfix), >> mysql backend i receive ana send mail correctly i use imap protocol), >> get ssh connection with publick key, >> i verify that with su - vmail , vmail find doveadm, i post my >> dovecot.conf file because i don't know what is wrong >> >> vmail exist on every server with publick key >> >> (server one => 10.0.0.118 >> >> server two => 10.0.0.122) >> >> dovecot.conf -> >> >> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server >> -u%u -l%{lock_timeout} -n%{namespace} >> >> service aggregator { >> # give enough permissions for mail processes >> # >> ? ? ? ?fifo_listener replication-notify-fifo { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> ? ? ? ?unix_listener replication-notify { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> } >> service replicator { >> # start replication at startup >> # >> ? ? ? ?process_min_avail = 1 >> } >> >> >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> >> ? ? ? ?user = vmail >> } >> >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> >> ? ? ? ?unix_listener config { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ?} >> } >> >> plugin { >> ? ? ? # this host replicates to remote host >> # >> ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 >> >> # run full synchronization mode every other hour >> # (default is every 24 hours) >> # >> ? ? ? ?replication_full_sync_interval = 1 hour >> } >> >> >> idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 >> >> thank's >> >> >> -- >> Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. >> ****************************************** >> Ing. Matteo Cazzador >> Email: mcazzador at gmail.com >> ****************************************** > > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:14:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:14:12 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> Message-ID: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> On 16.3.2012, at 11.09, Jernej Porenta wrote: >>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>> (uni_utf8_str_is_valid(vname)) >> .. > We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. > > Whenever . LIST "" "*" is issued, dovecot crashes: > Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot I don't think this is the same Panic as the original one? What is the Panic message now? From tss at iki.fi Fri Mar 16 15:17:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:17:50 +0200 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <20120316100836.GA12049@nicku.org> References: <20120316100836.GA12049@nicku.org> Message-ID: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> On 16.3.2012, at 12.08, Nick Urbanik wrote: > Using head of imaptest with dovecot 2.1, I am attempting to put a > heavy load on the server, with insufficient success. > > I made 15000 user accounts, put them in to a file, one per line. Then > I ran imaptest with > > ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 > pass=SECRETPASSWORD A single imaptest process can't handle that many simultaneous clients. You'd need to run multiple imaptests in parallel. > So what is the best way to put a really heavy load on an imap server? > The documentation describes testing correctness of the server without > really emphasising performance. Yes, imaptest is mainly meant to test server correctness (i.e. for me to test that Dovecot is bugfree). It spends a lot of time checking and tracking things that is irrelevant when you simply want to load the server. You could add no_tracking parameter to get rid of some of it. From tss at iki.fi Fri Mar 16 15:24:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:24:53 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: Message-ID: <1331904293.26095.2.camel@innu> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) > to Apple Mail. Having looked at all options, I have resorted to the, > perhaps, most difficult way: Create folders on the IMAP server, copy > e-mails into them from Outlook, connect Apple Mail and do the reverse. > However, I have hit a wall. > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items > folder, but the process dies at some point. When that happens, Outlook pops > a screen saying "IMAP session state is inconsistent, please relogin". .. > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around > offset=894): msg header has bad magic value Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)? What happens if you now run: doveadm force-resync -u wash at kictanet.or.ke INBOX Does it show any errors? If not, and if you try to copy the mails again, does it still fail? It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage. From nmilas at noa.gr Fri Mar 16 15:39:38 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 15:39:38 +0200 Subject: [Dovecot] ldapi support Message-ID: <4F63429A.6040304@noa.gr> Hi, Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) connections for LDAP lookups (user, password, auth etc.) or only ldap/ldaps (over TCP)? If yes, how do we specify ldapi://localhost in Dovecot configuration files? Thanks, Nick From odhiambo at gmail.com Fri Mar 16 15:51:18 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 16:51:18 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > Yes, it was pretty much empty..actually, I just configured the Dovecot instance yesterday evening and only tested for "correct operation" by sending a test mail to myself and login to POP3/IMAP. Pretty much pristine. And yes, if is FreeBSD ufs. [wash at jaribu ~]$ mount /dev/label/rootfs0 on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) procfs on /proc (procfs, local) linprocfs on /compat/linux/proc (linprocfs, local) /dev/ada1s1a on /disk2 (ufs, local, noatime, soft-updates) > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > Unfortunately, the user has taken the MacBook now so I am unable to test copying again. > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > I should be able to test this again at some point next week, but just to add, I changed the storage to Maildir and I was able to copy all the mails to the IMAP folder without any issue so I think it's something with SIS. PS: I wish I could test this with Outlook running on Windows, but I guess that introduces a completely different environment than what I had on the MacBook, right? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From odhiambo at gmail.com Fri Mar 16 16:00:41 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:00:41 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > > Ok. I have been able to reproduce it anyway. Environment: Windows 8 Consumer Preview, Outlook 2010. I had 415 e-mails in the Inbox. While copying, Outlook popped up an error: IMAP session state is inconsistent, please relogin. Protocol: IMAP Server: 192.168.40.252 Port: 143 Error Code: 0x800CCCDD ...and dovecot.log details at that time can be found here - http://196.200.26.114/~wash/dovecot.log.txt And the output of the force-resync command is: [root at jaribu] /usr/home/wash# cp /var/log/dovecot.log ~wash/public_html/dovecot.log.txt [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.5 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.5 doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.10 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.10 -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From CMarcus at Media-Brokers.com Fri Mar 16 16:03:12 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 10:03:12 -0400 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> References: <20120316100836.GA12049@nicku.org> <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> Message-ID: <4F634820.5040605@Media-Brokers.com> On 2012-03-16 9:17 AM, Timo Sirainen wrote: > imaptest is mainly meant to test server correctness (i.e. for me to > test that Dovecot is bugfree). It spends a lot of time checking and > tracking things that is irrelevant when you simply want to load the > server. You could add no_tracking parameter to get rid of some of it. Maybe imaptest could be duped+modified somehow to produce a new imap_load_test utility...? -- Best regards, Charles From giles at coochey.net Fri Mar 16 16:11:40 2012 From: giles at coochey.net (Giles Coochey) Date: Fri, 16 Mar 2012 14:11:40 +0000 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <4F634A1C.8060501@coochey.net> On 16/03/2012 14:00, Odhiambo Washington wrote: > On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> I have a situation where I need to migrate e-mails from Outlook 2011 >> (Mac) >> Personally I would just use readpst to export the standard Outlook personal storage folders to mbox format... -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From mhlavink at redhat.com Fri Mar 16 16:48:00 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Fri, 16 Mar 2012 15:48:00 +0100 Subject: [Dovecot] dovecot and systemd In-Reply-To: <1331820329.10319.32.camel@innu> References: <4F61EFE8.1000901@redhat.com> <1331820329.10319.32.camel@innu> Message-ID: <4F6352A0.5020200@redhat.com> On 03/15/2012 03:05 PM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: >> What exactly should happen when >> dovecot.conf does not match dovecot.socket configuration? > > Dovecot's systemd code was written by one of you Redhat guys. I had some > similar thoughts when I applied the patch, but didn't really know what > to do about it, so I didn't do anything. So: I don't know. Maybe some > other project has solved this somehow already? > > Dovecot anyway needs its own internal UNIX listeners. Should all > internal inet listeners be disabled? Could Dovecot somehow talk to > systemd and ask what listeners it's using for Dovecot and log warnings > if they don't match? I don't know that match about systemd. I'll forward this to systemd mailing list and I will let you know once I know more. From odhiambo at gmail.com Fri Mar 16 16:50:50 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:50:50 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F634A1C.8060501@coochey.net> References: <1331904293.26095.2.camel@innu> <4F634A1C.8060501@coochey.net> Message-ID: On Fri, Mar 16, 2012 at 17:11, Giles Coochey wrote: > On 16/03/2012 14:00, Odhiambo Washington wrote: > >> On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: >> >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> >>>> I have a situation where I need to migrate e-mails from Outlook 2011 >>>> >>> (Mac) >>> >>> Personally I would just use readpst to export the standard Outlook > personal storage folders to mbox format... > > Outlook 2011 (Mac OS X - Lion) can export everything into (!pst) .olm I haven't no clue whether .olm and .pst are one and the same, but I highly doubt. With Outlook 2011, the guys at Redmond intended to lock the user to Outlook! I have seen Outlook->Apple Mail migration procedures fraught with e-mail loses. Well, not quite a loss because you still have the e-mail inside Outlook, but that idea of running Outlook side-by-side with Apple Mail is not what I want to subject the user to. I also don't want to make them start searching for the e-mails that might not have been migrated and forward them to themselves. Migration should be complete & safe - no losses. There are commercial software out there that's said to be good at this process, but I wanted the "free"method. If there is *readpst *that can migrate from .olm to mbox, then I am willing to give it a shot, but I also want to see if Dovecot gets a fix for whatever problem I am facing. -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 17:05:42 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:05:42 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <1331910342.26095.34.camel@innu> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > Corrupted > > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > > offset=894): msg header has bad magic value > > > Ok. I have been able to reproduce it anyway. It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails. Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest Simply run it for an empty test account as: imaptest host=localhost user=testuser pass=testpass Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful.. From tss at iki.fi Fri Mar 16 17:09:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:09:14 +0200 Subject: [Dovecot] ldapi support In-Reply-To: <4F63429A.6040304@noa.gr> References: <4F63429A.6040304@noa.gr> Message-ID: <1331910554.26095.35.camel@innu> On Fri, 2012-03-16 at 15:39 +0200, Nikolaos Milas wrote: > Hi, > > Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) > connections for LDAP lookups (user, password, auth etc.) or only > ldap/ldaps (over TCP)? > > If yes, how do we specify ldapi://localhost in Dovecot configuration files? OpenLDAP library handles the connections internally. It probably works the same way in Dovecot as in other software that uses OpenLDAP. So I don't know, try ldapi://. From lists at wildgooses.com Fri Mar 16 17:22:07 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:22:07 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F635A9F.2020406@wildgooses.com> On 14/03/2012 10:58, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. > > I also learned something very interesting yesterday concerning SOGo > and dovecot during a sales call with a SOGo rep, but I'll wait and see > if Timo cares to chime in on this one... ;) > If the answer is that he will write a Z-Push/Activesync module for SOGo then I'm all ears! I have been watching SOGo for some time and the main thing I would miss is that every phone I have ever owned has largely limited/broken Funambol based sync and annoyingly working Activesync capability (I own a stream of Nokias...). It seems that although I don't like it, I need activesync support if I want my contacts/calendar on my phone... (I think I can do caldav on some of them, but not cardav on my N9) Apart from that it's a very neat system! Ed W From tss at iki.fi Fri Mar 16 17:26:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:26:33 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <1331911593.26095.47.camel@innu> On Fri, 2012-03-16 at 15:22 +0000, Ed W wrote: > > I also learned something very interesting yesterday concerning SOGo > > and dovecot during a sales call with a SOGo rep, but I'll wait and see > > if Timo cares to chime in on this one... ;) > > > > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) We're also very much wishing for SOGo Activesync, but I'm not planning on writing it myself (but maybe we'll hire someone who will). Annoyingly Microsoft has patented Activesync, so I guess it can't be legally used at least in USA without paying MS. From lists at wildgooses.com Fri Mar 16 17:30:42 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:30:42 +0000 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <1331807624.10319.6.camel@innu> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> Message-ID: <4F635CA2.8080401@wildgooses.com> On 15/03/2012 10:33, Timo Sirainen wrote: > On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: >> I'm curious if anyone has any plugins for AV integration directly into >> dovecot. >> >> Our old pop servers have been scanning messges as they're moved from >> new->cur in the inbox and, at least where user's aren't poping every >> few seconds, there is occasionally enough time between scanning through >> the MXs to message retreval to snag a few more virues with updated >> definitions before they reach customers. >> >> Anyone doing anything similar? > http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a > script that modifies a mail while it's being read. You could make it run > a virus check, and if that happens you could change the virus MIME part > to be full of spaces (better not to change message size, line count or > MIME structure). > > Couple of other ideas: 1) Could use one of the (buggy and variously unsupported) on access virus scanners. I think Dazuko is now abandoned, but this is a new one mentioned via the Clamav site: http://www.fsl.cs.sunysb.edu/docs/avfs-security04/index.html 2) Extremely racey, but if you were on maildir you could use some kind of pre-login scripting to kick off a scan on login. Touch some lock file so that you can tell when last scanned and only scan if the definitions have been updated since you last scanned? 3) There are some POP proxies which offer inline virus scanning. Could place one in front of your mail server. Presumably this will expose you to all the bugs in that proxy... Good luck Ed W From CMarcus at Media-Brokers.com Fri Mar 16 17:45:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 11:45:58 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <4F636036.4020604@Media-Brokers.com> On 2012-03-16 11:22 AM, Ed W wrote: > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) While I agree it would be nice, why not just switch to a supported phone and be done with it? ;) When we roll out SOGo, we'll only be supporting the officially supported mobile clients (android, iphone/ipad, blackberry and windows mobile)... -- Best regards, Charles From odhiambo at gmail.com Fri Mar 16 17:57:29 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 18:57:29 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 18:05, Timo Sirainen wrote: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > > Corrupted > > > > dbox file /var/spool/virtual/ > kictanet.or.ke/wash/mdbox/storage/m.4(around > > > > offset=894): msg header has bad magic value > > > > > Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > > > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I had to do that, but things don't appear good at all. [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest host=localhost user=wash at kictanet.or.ke pass=XXX Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox You are welcome to access my box if you like, but I must warn you, things will be slow. Internationa links are slow as a result of fiber cuts in MSA, KE! -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 18:09:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:09:58 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <9AD845E2-7922-4E4A-A7ED-F1C2EE55F31A@iki.fi> On 16.3.2012, at 17.57, Odhiambo Washington wrote: >> imaptest host=localhost user=testuser pass=testpass >> >> Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else >> can report that they can reproduce this problem that would be helpful.. >> > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I > had to do that, but things don't appear good at all. No, you can also specify it as mbox=path parameter. > [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest > host=localhost user=wash at kictanet.or.ke pass=XXX > Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox But don't point the mbox there! You'll need to download http://www.dovecot.org/tmp/dovecot-crlf file and point the mbox to that. It's used to upload mails. From e-frog at gmx.de Fri Mar 16 18:16:49 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 17:16:49 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <4F636771.1000308@gmx.de> On 16.03.2012 16:05, wrote Timo Sirainen: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>> Corrupted >>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>> offset=894): msg header has bad magic value >>> This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related. http://www.dovecot.org/list/dovecot/2012-March/064462.html >> Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > From lists at wildgooses.com Fri Mar 16 18:17:19 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 16:17:19 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F636036.4020604@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> <4F636036.4020604@Media-Brokers.com> Message-ID: <4F63678F.9080104@wildgooses.com> On 16/03/2012 15:45, Charles Marcus wrote: > On 2012-03-16 11:22 AM, Ed W wrote: >> If the answer is that he will write a Z-Push/Activesync module for SOGo >> then I'm all ears! I have been watching SOGo for some time and the main >> thing I would miss is that every phone I have ever owned has largely >> limited/broken Funambol based sync and annoyingly working Activesync >> capability (I own a stream of Nokias...). It seems that although I >> don't like it, I need activesync support if I want my contacts/calendar >> on my phone... (I think I can do caldav on some of them, but not cardav >> on my N9) > > While I agree it would be nice, why not just switch to a supported > phone and be done with it? ;) > > When we roll out SOGo, we'll only be supporting the officially > supported mobile clients (android, iphone/ipad, blackberry and windows > mobile)... > That implies you will be using cardav/caldav on those phones? I thought Android support was quite weak for those? I definitely don't like the idea of supporting activesync, but it seems like the only widely supported solution to pushing calendar and contacts updates to clients? Caldav gets you part of the way there, but cardav seems badly supported and there is no push support with either... Out of curiousity, what kind of performance are you getting out of the web interface and any tricks you used to improve "perceived" performance? My quick testing gave something circa 150-200ms response times from SOGo (forget exactly now) and as a result it was perceivable and just very slightly laggy (versus a desktop mail program!!). I get slightly better perceived performance from Roundcube (which also seems more amenable to building extension plugins) Seems a bit of a surprise that a compiled language delivers results slightly less quickly than PHP... Did you find any magic knobs to twist to get performance up there with gmail? Cheers Ed W From tss at iki.fi Fri Mar 16 18:57:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:57:57 +0200 Subject: [Dovecot] v2.1.3 released Message-ID: <1235E4DC-130A-4CE7-9C22-C6180062D914@iki.fi> http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz.sig Do not use v2.1.2 with multi-dbox format, it's broken. I didn't notice that a small optimization I did a few days ago broke mdbox in some situations (when mdbox first created a new m.X file, and later in same session saved another message to it). It's quite a high priority for me to run some automated testing before releasing new versions, for example a small imaptest run with mdbox would have caught this. Perhaps the next release will already have the automated testing. From tss at iki.fi Fri Mar 16 19:00:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:00:57 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F636771.1000308@gmx.de> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: On 16.3.2012, at 18.16, e-frog wrote: > On 16.03.2012 16:05, wrote Timo Sirainen: >> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>> Corrupted >>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>> offset=894): msg header has bad magic value >>>> > > This is kind of the same error message I'm getting with doveadm backup. > I can reproduce this at will. Not sure however if this is related. > > http://www.dovecot.org/list/dovecot/2012-March/064462.html Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. From e-frog at gmx.de Fri Mar 16 19:24:38 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 18:24:38 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: <4F637756.3020707@gmx.de> On 16.03.2012 18:00, wrote Timo Sirainen: > On 16.3.2012, at 18.16, e-frog wrote: > >> On 16.03.2012 16:05, wrote Timo Sirainen: >>> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>>> Corrupted >>>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>>> offset=894): msg header has bad magic value >>>>> >> >> This is kind of the same error message I'm getting with doveadm backup. >> I can reproduce this at will. Not sure however if this is related. >> >> http://www.dovecot.org/list/dovecot/2012-March/064462.html > > Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. Ok, just tested with 2.1.3 and it works again. Thanks Timo! From kgc at corp.sonic.net Fri Mar 16 19:49:46 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:49:46 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <4F637D3A.1000301@corp.sonic.net> On 03/16/12 06:07, Timo Sirainen wrote: > Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. Timo, can you explain why Maildir isn't a good for POP3 in this context? Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. -K From kgc at corp.sonic.net Fri Mar 16 19:52:58 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:52:58 -0700 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F635CA2.8080401@wildgooses.com> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> Message-ID: <4F637DFA.9070800@corp.sonic.net> On 03/16/12 08:30, Ed W wrote: > 2) Extremely racey, but if you were on maildir you could use some kind > of pre-login scripting to kick off a scan on login. Touch some lock file > so that you can tell when last scanned and only scan if the definitions > have been updated since you last scanned? I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Still need to take a look at Timo's patch set. -K From tss at iki.fi Fri Mar 16 19:54:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:54:06 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F637D3A.1000301@corp.sonic.net> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> Message-ID: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> On 16.3.2012, at 19.49, Kelsey Cummings wrote: > On 03/16/12 06:07, Timo Sirainen wrote: >> Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > > Timo, can you explain why Maildir isn't a good for POP3 in this context? Compared to mbox/mdbox: It needs to read and delete multiple separate files, which is typically much slower than reading and deleting a single file. > Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. You mean deleting the messages after +OK, instead of before? Does it really make a difference?.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. From tss at iki.fi Fri Mar 16 19:55:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:55:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F637DFA.9070800@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> <4F637DFA.9070800@corp.sonic.net> Message-ID: <044DBE43-7DEA-45D5-A34A-41F7E0FB8345@iki.fi> On 16.3.2012, at 19.52, Kelsey Cummings wrote: > On 03/16/12 08:30, Ed W wrote: >> 2) Extremely racey, but if you were on maildir you could use some kind >> of pre-login scripting to kick off a scan on login. Touch some lock file >> so that you can tell when last scanned and only scan if the definitions >> have been updated since you last scanned? > > I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Whatever you do: Don't modify existing message files (without renaming them so they appear as new mails). IMAP (and Dovecot) require that messages never change. From kgc at corp.sonic.net Fri Mar 16 21:04:19 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 12:04:19 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> Message-ID: <4F638EB3.6040802@corp.sonic.net> On 03/16/12 10:54, Timo Sirainen wrote: >> Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. > > You mean deleting the messages after +OK, instead of before? Does it really make a difference? In the context of a clients "send and receive" phase taking a (small) fraction of a second less time, perhaps, but it is a small difference in any case. It was one of many small changes we made to try to improve interactive performance. >.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. True, we decided that loosing that ability didn't really matter. (Like not counting newlines as two bytes in the message size.) -K From alex.handle at gmail.com Fri Mar 16 22:00:05 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:00:05 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection Message-ID: Hi Dovecot-list! My setup consists of a dovecot server with lmtp delivery and 3 postfix mta servers in front. Previously the mtas authenticated (SASL) through the courier-authdaemond software to our mysql database. To get support for more password formats i migrated to dovecot for SASL authentification. Our postfix mtas connect to dovecot through a tcp-socket smtpd_sasl_auth_enable = yes smtpd_sasl_type = dovecot smtpd_sasl_path = inet:10.11.100.230:12345 smtpd_sasl_security_options = noanonymous smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = broken_sasl_auth_clients = yes 10.11.100.230 is our dovecot server. Please look at the attached doveconf.log to see my auth service configuration. I did the configuration according to the postfix SASL README. http://www.postfix.org/SASL_README.html#server_dovecot I tested the setup and everything worked fine but after 2 days i noticed these error messages in my mail.log: dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of existing connection and also these messages from postfix: SASL LOGIN authentication failed: Connection lost to authentication server I get the dovecot error message about 3000 times a day and postfix message about 270 times. Please see my attached mail.log for a detailed trace. Thank you for your help :) Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: doveconf.log Type: application/octet-stream Size: 14009 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mail.log Type: application/octet-stream Size: 3120 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 22:14:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 22:14:28 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: Message-ID: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> On 16.3.2012, at 22.00, Alex Ha wrote: > dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of > existing connection Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. -------------- next part -------------- A non-text attachment was scrubbed... Name: tcp-auth.diff Type: application/octet-stream Size: 718 bytes Desc: not available URL: From alex.handle at gmail.com Fri Mar 16 22:39:42 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:39:42 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 22.00, Alex Ha wrote: > >> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >> existing connection > > Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. > Thanks Timo! I will try the patch and report to you. Alex From lists at wiesinger.com Sat Mar 17 08:31:12 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Sat, 17 Mar 2012 07:31:12 +0100 (CET) Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> References: <4F62D762.7080607@wiesinger.com> <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> Message-ID: On Fri, 16 Mar 2012, Timo Sirainen wrote: > On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: >> Calling imap still fails as non root: >> imap >> /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied >> collect2: ld returned 1 exit statusn > > Huh? That looks like imap is running ld to link something. It shouldn't be doing that. After starting it once as root the following files are created and it works also as non root: ls -l /usr/local/bin/.libs/ total 1160 -rwxr-xr-x 1 root root 235848 Aug 25 2010 lt-doveconf -rwxr-xr-x 1 root root 938454 Mar 16 07:03 lt-imap Before only one of these files was generated (I think lt-doveconf). Strange. Any ideas? Ciao, Gerhard -- http://www.wiesinger.com/ From hsn at filez.com Sat Mar 17 08:36:22 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 07:36:22 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs Message-ID: <4F6430E6.6040100@filez.com> Is there way to import old plain mboxes via dsync? It complains about lack of index files: ponto:(admin)~>dsync mirror mbox:~/mail dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files From tss at iki.fi Sat Mar 17 12:32:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 12:32:06 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F6430E6.6040100@filez.com> References: <4F6430E6.6040100@filez.com> Message-ID: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> On 17.3.2012, at 8.36, Radim Kolar wrote: > Is there way to import old plain mboxes via dsync? It complains about lack of index files: > > ponto:(admin)~>dsync mirror mbox:~/mail > dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? From varia at e-healthexpert.org Sat Mar 17 16:14:24 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Sat, 17 Mar 2012 14:14:24 +0000 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <20120317141424.1c629e46@e-healthexpert.org> On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: > On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > pop3_no_flag_changes=yes Is it the same as pop3_no_flag_updates=yes ? M. From tss at iki.fi Sat Mar 17 16:40:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 16:40:59 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <20120317141424.1c629e46@e-healthexpert.org> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <20120317141424.1c629e46@e-healthexpert.org> Message-ID: <1FE9581E-498E-4AE7-800D-4038DA32BB73@iki.fi> On 17.3.2012, at 16.14, Mark Alan wrote: > On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: >> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > >> pop3_no_flag_changes=yes > > Is it the same as pop3_no_flag_updates=yes ? Yeah. I wrote it from my memory. From hsn at filez.com Sat Mar 17 19:29:07 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 18:29:07 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> Message-ID: <4F64C9E3.7080102@filez.com> > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? indexes never existed because these mboxes were never used by dovecot, its not conversion from one format to another, its import. Maybe open bug to add feature "dsync import" which will not depend on existing indexes? From kayasaman at gmail.com Sat Mar 17 20:31:33 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 18:31:33 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice Message-ID: <4F64D885.8000501@gmail.com> Hi, I am currently in the process of setting up an IMAP repository for round 100 users.... Currently the user authentication method is being handled via a Windows Domain Controller. The host OS for Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? Can anyone give me a hand with this? Regards, Kaya From sven at svenhartge.de Sat Mar 17 21:36:15 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 20:36:15 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> Message-ID: <88l5d3on0ev8@mids.svenhartge.de> Kaya Saman wrote: > I am currently in the process of setting up an IMAP repository for > round 100 users.... > Currently the user authentication method is being handled via a > Windows Domain Controller. > The host OS for Dovecot will either be FreeBSD or CentOS. > Would Dovecot be able to authenticate to either the DC directly or > would we need to go through LDAP?? Why not join the server to the domain and simply use PAM? Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you. > Additionally what would be the best method to store the **mail** > information? - as in MySQL database or Maildir format; coinciding with > this what is the best backup method in order to be able to do 'dump' > backups or restore single emails?? Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress. To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots. Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore. Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself. Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory. Some, like mbox or mdbox are a little bit more complex, but with the correct doveadm command you are nevertheless able to restore single mails. Gr??e, Sven. -- Sigmentation fault. Core dumped. From pstm.spain at gmail.com Sat Mar 17 21:42:27 2012 From: pstm.spain at gmail.com (PSTM) Date: Sat, 17 Mar 2012 20:42:27 +0100 Subject: [Dovecot] Problem managing mbox Message-ID: <4F64E923.5060401@gmail.com> Hello, I have a problem with dovecot. seems that do not erase mail that mail client request to be erased. And I have this errors: > Error: Next message unexpectedly corrupted in mbox file Info: > dovecot-2.1.1-2.0.cf.fc16.i686 > root 5979 0.0 0.1 3208 1260 ? Ss 20:18 0:00 > /usr/sbin/dovecot -F > dovenull 5985 0.0 0.2 7060 2280 ? S 20:18 0:00 > dovecot/imap-login > vmail 5988 0.0 0.1 7888 1848 ? S 20:18 0:00 > dovecot/imap permissions on mail dir: > total 4 > drwxr-xr-x 9 vmail mail 4096 ene 21 21:43 vmail Any suggestion? Regards, -- -- http://www.0pc.eu/ From kayasaman at gmail.com Sat Mar 17 21:55:35 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 19:55:35 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <88l5d3on0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> Message-ID: <4F64EC37.5010309@gmail.com> On 03/17/2012 07:36 PM, Sven Hartge wrote: > Kaya Saman wrote: > >> I am currently in the process of setting up an IMAP repository for >> round 100 users.... >> Currently the user authentication method is being handled via a >> Windows Domain Controller. >> The host OS for Dovecot will either be FreeBSD or CentOS. >> Would Dovecot be able to authenticate to either the DC directly or >> would we need to go through LDAP?? > Why not join the server to the domain and simply use PAM? > > Using ActiveDirectory through LDAP is a bit of a pain so I would avoid > this if I where you. Danke Sven :-) I don't actually have much AD/LDAP integration experience so I will try your method! >> Additionally what would be the best method to store the **mail** >> information? - as in MySQL database or Maildir format; coinciding with >> this what is the best backup method in order to be able to do 'dump' >> backups or restore single emails?? > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. > > To have a consistent backup, your mail storage should be able to > snapshot the volume the mail is stored on, so use LVM or an external > storage unit capable of snapshots. Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. > > Then backup the content of the snapshot using any program you like. > I use Bacula for long-term offsite storage and a local rsnapshot to keep > 7 days worth of mail for a quick restore. To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server. > > Whether you are able to restore single mails or the complete storage is > no property or feature of the mailbox format itself. > > Some formats are simpler to handle, like Maildir++, where you just drop > the file containing a mail into a directory. You mention Maildir++... is this Maildir format or something new which I haven't heard about yet? > > Some, like mbox or mdbox are a little bit more complex, but with the > correct doveadm command you are nevertheless able to restore single > mails. > > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sat Mar 17 23:03:21 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 22:03:21 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> Message-ID: <98l5i9bn0ev8@mids.svenhartge.de> Kaya Saman wrote: > On 03/17/2012 07:36 PM, Sven Hartge wrote: >> Kaya Saman wrote: >>> I am currently in the process of setting up an IMAP repository for >>> round 100 users.... Currently the user authentication method is >>> being handled via a Windows Domain Controller. The host OS for >>> Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to >>> authenticate to either the DC directly or would we need to go >>> through LDAP?? >> Why not join the server to the domain and simply use PAM? >> Using ActiveDirectory through LDAP is a bit of a pain so I would >> avoid this if I where you. > I don't actually have much AD/LDAP integration experience so I will > try your method! Question: do you need public or shared folders? Using samba and winbindd to join a domain creates real users on your server and as far as I know configuring shared folders with real users is a bit of a pain, especially of you need shared flags (like Seen, Replied, etc.) (Someone [Timo?] please correct me.) >>> Additionally what would be the best method to store the **mail** >>> information? - as in MySQL database or Maildir format; coinciding >>> with this what is the best backup method in order to be able to do >>> 'dump' backups or restore single emails?? >> Storing mails inside SQL? Not supported by dovecot and not very wise, >> IMHO. DBmail does this, but to be honest, I never heard any good >> feedback from admins using that product. From what I have been told, you >> need quite the beefy server to get a decent performance out of DBmail, >> compared to the needs of a "traditional" setup like with dovecot or >> courier-mail, but I digress. >> >> To have a consistent backup, your mail storage should be able to >> snapshot the volume the mail is stored on, so use LVM or an external >> storage unit capable of snapshots. > Hmm..... so FreeBSD coupled together with a ZFS repo for mail should > take care of 'Snapshot' issues. Yes. Or using LVM on Linux. >> Then backup the content of the snapshot using any program you like. >> I use Bacula for long-term offsite storage and a local rsnapshot to >> keep 7 days worth of mail for a quick restore. > To be honest I was considering rsync'ing the dir containing users > mailboxes to either another storage pool or server. No need to rsync, if you use ZFS. Just create a new snapshot and you are done. Bet thing about ZFS: you get deduplication for free, so the needed space to store the backups will not grow as fast. But you still may want to store the mails offsite/offserver for desaster recovery. Either use doveadm backup for that purpose or use rsnapshot, again gaining you deduplication on the target server. >> Whether you are able to restore single mails or the complete storage is >> no property or feature of the mailbox format itself. >> >> Some formats are simpler to handle, like Maildir++, where you just drop >> the file containing a mail into a directory. > You mention Maildir++... is this Maildir format or something new which I > haven't heard about yet? Maildir++ extends the original Maildir with things like Quota and ACLs and was first implemented in Courier. http://www.courier-mta.org/imap/README.maildirquota.html All current MTAs and POP3/IMAP servers implement this variant. Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are). In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands. I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever". Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sat Mar 17 23:24:25 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:24:25 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <98l5i9bn0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> Message-ID: <4F650109.7090702@gmail.com> Thanks so much Sven for your indepth and complete responses! > Question: do you need public or shared folders? I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage. Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange. To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc. > > Using samba and winbindd to join a domain creates real users on your > server and as far as I know configuring shared folders with real users > is a bit of a pain, especially of you need shared flags (like Seen, > Replied, etc.) (Someone [Timo?] please correct me.) Actually we might have an LDAP server already taking care of the AD<->UNIX integration..... I don't know yet it's only my first week :-) > >>>> Additionally what would be the best method to store the **mail** >>>> information? - as in MySQL database or Maildir format; coinciding >>>> with this what is the best backup method in order to be able to do >>>> 'dump' backups or restore single emails?? >>> Storing mails inside SQL? Not supported by dovecot and not very wise, >>> IMHO. DBmail does this, but to be honest, I never heard any good >>> feedback from admins using that product. From what I have been told, you >>> need quite the beefy server to get a decent performance out of DBmail, >>> compared to the needs of a "traditional" setup like with dovecot or >>> courier-mail, but I digress. >>> >>> To have a consistent backup, your mail storage should be able to >>> snapshot the volume the mail is stored on, so use LVM or an external >>> storage unit capable of snapshots. >> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >> take care of 'Snapshot' issues. > Yes. Or using LVM on Linux. Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too...... > >>> Then backup the content of the snapshot using any program you like. >>> I use Bacula for long-term offsite storage and a local rsnapshot to >>> keep 7 days worth of mail for a quick restore. >> To be honest I was considering rsync'ing the dir containing users >> mailboxes to either another storage pool or server. > No need to rsync, if you use ZFS. Just create a new snapshot and you are > done. Bet thing about ZFS: you get deduplication for free, so the needed > space to store the backups will not grow as fast. Ok so that solves that! :-) > But you still may want to store the mails offsite/offserver for desaster > recovery. They are currently being stored on the parent company mail server so this will be the/off-site/ disaster recovery system in a way :-P > > Either use doveadm backup for that purpose or use rsnapshot, again > gaining you deduplication on the target server. I will research this - thank you for that info :-) > >>> Whether you are able to restore single mails or the complete storage is >>> no property or feature of the mailbox format itself. >>> >>> Some formats are simpler to handle, like Maildir++, where you just drop >>> the file containing a mail into a directory. >> You mention Maildir++... is this Maildir format or something new which I >> haven't heard about yet? > Maildir++ extends the original Maildir with things like Quota and ACLs > and was first implemented in Courier. > http://www.courier-mta.org/imap/README.maildirquota.html > > All current MTAs and POP3/IMAP servers implement this variant. > > Depending on the amount of mail a user collects inside a folder, Maildir > is not the best storage format. You may want to check into mdbox, if > your users are kind of "mail hoarders" (like some of my users are). > > In my opinion, Maildir has outlived its usefullnes. It was fine when > users had 1,000 mails in some 10 folders, but today, users collect over > 100,000 mails a year and Maildir is causing serious I/O trouble and the > need to heavily fine tune your storage and filesystems to cope with > those demands. > > I cannot thank Timo enough for inventing mdbox, as this format breaks > this viciuos cycle and, as someone else said "it ends the battle at the > I/O front forever". So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!} Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability. However, if like you say mdbox is the way to go then I will put a strong case together! > > Gr??e, > Sven. > Regards, Kaya From stsiol at yahoo.co.uk Sat Mar 17 23:51:28 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sat, 17 Mar 2012 21:51:28 +0000 (GMT) Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F64D885.8000501@gmail.com> References: <4F64D885.8000501@gmail.com> Message-ID: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> >Hi, > >I am currently in the process of setting up an IMAP repository for round 100 users.... > >Currently the user authentication method is being handled via a Windows Domain Controller. > >The host OS for Dovecot will either be FreeBSD or CentOS. > > >Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? > > >Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? > > >Can anyone give me a hand with this? > > >Regards, > > >Kaya Hi Kaya, I can't force you to follow a specific path. All I can do, is tell you my experience on this. Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for Webmail, OpenLDAP for LDAP (no windows software there) and CentOS v5.5 32-bit onwards. User base is about 30 users. System uptime without a glitch reached at some point (had to reboot the server for maintenance reasons) about 200 days. I am sure it would go beyond 365 days. Hope this helps, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From kayasaman at gmail.com Sat Mar 17 23:53:44 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:53:44 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> References: <4F64D885.8000501@gmail.com> <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> Message-ID: <4F6507E8.4060204@gmail.com> On 03/17/2012 09:51 PM, Spyros Tsiolis wrote: >> Hi, >> >> I am currently in the process of setting up an IMAP repository for round 100 users.... >> >> Currently the user authentication method is being handled via a Windows Domain Controller. >> >> The host OS for Dovecot will either be FreeBSD or CentOS. >> >> >> Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? >> >> >> Additionally > what would be the best method to store the **mail** information? - as > in MySQL database or Maildir format; coinciding with this what is the > best backup method in order to be able to do 'dump' backups or restore > single emails?? >> >> Can anyone give me a hand with this? >> >> >> Regards, >> >> >> Kaya > > > > Hi Kaya, > > I can't force you to follow a specific path. > All I can do, is tell you my experience on this. > > Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for > Webmail, OpenLDAP for LDAP (no windows software > there) and CentOS v5.5 32-bit onwards. > User base is about 30 users. > System uptime without a glitch reached at some point > (had to reboot the server for maintenance reasons) about > 200 days. I am sure it would go beyond 365 days. > > Hope this helps, > > spyros > > > > ---- > "I merely function as a channel that filters > music through the chaos of noise" > - Vangelis Thanks for that Spyros! Regards, Kaya From sven at svenhartge.de Sun Mar 18 00:28:04 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 23:28:04 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: Kaya Saman wrote: > >> Question: do you need public or shared folders? > I don't need anything apart from an IMAP storage solution. I don't > intend to tie in Dovecot with an MTA either as I will simply be using > this for storage. > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. So, how do you plan to get the mails into this storage? offlineimap? imapsync? mbsync? fetchmail? >>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>> take care of 'Snapshot' issues. >> Yes. Or using LVM on Linux. > Yeah.... true but I specified ZFS as I'm a fan and also am quite > comfortable with Solaris/*BSD too...... If you know ZFS and are familiar with it, then, by all means, go for it. >> Depending on the amount of mail a user collects inside a folder, >> Maildir is not the best storage format. You may want to check into >> mdbox, if your users are kind of "mail hoarders" (like some of my >> users are). >> >> In my opinion, Maildir has outlived its usefullnes. It was fine when >> users had 1,000 mails in some 10 folders, but today, users collect >> over 100,000 mails a year and Maildir is causing serious I/O trouble >> and the need to heavily fine tune your storage and filesystems to >> cope with those demands. >> >> I cannot thank Timo enough for inventing mdbox, as this format breaks >> this viciuos cycle and, as someone else said "it ends the battle at >> the I/O front forever". > So mdbox is a 'new' mailbox standard? ie. one can replace Maildir > format with this and use mdbox instead. {Note to self: time to > browse!} mdbox is a format invented by Timo for dovecot. But dovecot can use nearly all common mailbox formats (except MH, but no one uses that one today). > Since where I'm implementing this is mainly an MS based environment > they are concerned about /flat/ files.... which MS seems to typically > do (although never used MS before so I wouldn't know). So there is > some concern over performance, efficiency and manageability. Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no doubt about this. But even with this crappy format, dovecot is able to deliver astounding performance by use of separete index files which allow it to access the storage in an efficient manner. mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten. mdbox solves this problem by a) storing all meta-data in the index and b) by only ever appending to a mdbox storage file, c) never truncating an existing mdbox storage file and d) using more than one mdbox storage file. Max size and TTL are configurable. But this also means deleted mails are still inside a mdbox storage file and need to be finally removed by copying all remaining files into a new file. This process has to be manually run during low traffic hours, for example using a cronjob. You can say, mdbox is like mbox on steroids. ;) Flat files are not evil or bad or slow per se, but you have to use them the right way. > However, if like you say mdbox is the way to go then I will put a > strong case together! You may want to start with something familiar and convert later, which is no problem with dovecot. Gr??e, Sven -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 00:35:37 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 22:35:37 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F6511B9.1020801@gmail.com> On 03/17/2012 10:28 PM, Sven Hartge wrote: > Kaya Saman wrote: >> >>> Question: do you need public or shared folders? >> I don't need anything apart from an IMAP storage solution. I don't >> intend to tie in Dovecot with an MTA either as I will simply be using >> this for storage. >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > So, how do you plan to get the mails into this storage? offlineimap? > imapsync? mbsync? fetchmail? Since everything is blocked at the Exchange end, users will have to manually transfer for now through MS Outlook. Currently that's what they're doing to their PST's.... > >>>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>>> take care of 'Snapshot' issues. >>> Yes. Or using LVM on Linux. >> Yeah.... true but I specified ZFS as I'm a fan and also am quite >> comfortable with Solaris/*BSD too...... > If you know ZFS and are familiar with it, then, by all means, go for it. :-) > >>> Depending on the amount of mail a user collects inside a folder, >>> Maildir is not the best storage format. You may want to check into >>> mdbox, if your users are kind of "mail hoarders" (like some of my >>> users are). >>> >>> In my opinion, Maildir has outlived its usefullnes. It was fine when >>> users had 1,000 mails in some 10 folders, but today, users collect >>> over 100,000 mails a year and Maildir is causing serious I/O trouble >>> and the need to heavily fine tune your storage and filesystems to >>> cope with those demands. >>> >>> I cannot thank Timo enough for inventing mdbox, as this format breaks >>> this viciuos cycle and, as someone else said "it ends the battle at >>> the I/O front forever". >> So mdbox is a 'new' mailbox standard? ie. one can replace Maildir >> format with this and use mdbox instead. {Note to self: time to >> browse!} > mdbox is a format invented by Timo for dovecot. But dovecot can use > nearly all common mailbox formats (except MH, but no one uses that one > today). Ok so if you claim that mdbox is the 'best' mailbox storage solution then I'll look at implementing this. > >> Since where I'm implementing this is mainly an MS based environment >> they are concerned about /flat/ files.... which MS seems to typically >> do (although never used MS before so I wouldn't know). So there is >> some concern over performance, efficiency and manageability. > Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no > doubt about this. > > But even with this crappy format, dovecot is able to deliver astounding > performance by use of separete index files which allow it to access the > storage in an efficient manner. > > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. > > mdbox solves this problem by a) storing all meta-data in the index and > b) by only ever appending to a mdbox storage file, c) never > truncating an existing mdbox storage file and d) using more than one > mdbox storage file. Max size and TTL are configurable. > > But this also means deleted mails are still inside a mdbox storage file > and need to be finally removed by copying all remaining files into a new > file. This process has to be manually run during low traffic hours, for > example using a cronjob. > > You can say, mdbox is like mbox on steroids. ;) > > Flat files are not evil or bad or slow per se, but you have to use them > the right way. Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-( Need to make a strong case! > >> However, if like you say mdbox is the way to go then I will put a >> strong case together! > You may want to start with something familiar and convert later, which > is no problem with dovecot. Maildir is what I'm familiar with currently and mbox format - though only use mbox as an unfortunate side product of /system mail/ accounts. Works well with Alpine client though! > > Gr??e, > Sven > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:04:22 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:04:22 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: Kaya Saman wrote: >> Flat files are not evil or bad or slow per se, but you have to use >> them the right way. > Thanks a lot for that info. I will research more into this but I maybe > overridden at some point :-( > Need to make a strong case! Hmm. Just because Microsofts way of usage of flat file database sucks does not mean any usage of flat files is bad or evil or slow, if done right. Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox But as I wrote before, it is quite easy to convert from one format to the other: http://wiki2.dovecot.org/Migration/MailFormat Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:18:30 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:18:30 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: <4F6529D6.60609@gmail.com> On 03/18/2012 12:04 AM, Sven Hartge wrote: > Kaya Saman wrote: > >>> Flat files are not evil or bad or slow per se, but you have to use >>> them the right way. >> Thanks a lot for that info. I will research more into this but I maybe >> overridden at some point :-( >> Need to make a strong case! > Hmm. > > Just because Microsofts way of usage of flat file database sucks does > not mean any usage of flat files is bad or evil or slow, if done right. Coming from a UNIX background I deal quite a lot with this kind of stuff so there's not problem for me. However, where I'm trying to deploy this system is a primarily MS based enterprise meaning that as the only UNIX engineer onsite and the newest addition to the team I have to convince people of working with UNIX technologies or somehow increase UNIX awareness. As a bi-product I know nothing about MS tech. only what it told to me by my colleagues :-) > > Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox I checked that out after your last email... I started Google'ing a little. :-) Looks like it would be a good solution! > > But as I wrote before, it is quite easy to convert from one format to > the other: http://wiki2.dovecot.org/Migration/MailFormat Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system??? To be honest, I run Zimbra @home for my OpenSource work and really enjoy it; in conjunction with Dovecot on FreeBSD which I run imapsync to backup **all** emails to. It works really well...... :-) I have messed around with Postfix, Dovecot and Horde3 in the past which also was really nice. > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:32:41 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:32:41 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: Kaya Saman wrote: > Once we get setup this may come in quite handy! Not sure what's going > on currently as everyone above me is still quite set in using an SQL > DB as a mail storage system??? RDBMS where not designed for such a task. Using a relational database as a storage method for big chunks of data is very unwise, in my opinion. It degrades them to just being some sort of filing cabinet. Now, wouldn't it be nice, if we had something like that, a filing cabinet where we can store large chunks of data and randomly read and write them in a fast manner? Oh yes, I remember, it is called a "filesystem". Let's use some of those to store the mail data. It will be soooo awesome! ;-) Ok, back being serious: there is nothing wrong with using a RDBMS in the way it was intented, to store user credentials, quota values, account settings, forwarding addresses, address book data, bookmarks, etc. Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:42:29 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:42:29 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F652F75.7060901@gmail.com> On 03/18/2012 12:32 AM, Sven Hartge wrote: > Kaya Saman wrote: > >> Once we get setup this may come in quite handy! Not sure what's going >> on currently as everyone above me is still quite set in using an SQL >> DB as a mail storage system??? > RDBMS where not designed for such a task. Using a relational database > as a storage method for big chunks of data is very unwise, in my > opinion. It degrades them to just being some sort of filing cabinet. > > Now, wouldn't it be nice, if we had something like that, a filing > cabinet where we can store large chunks of data and randomly read and > write them in a fast manner? > > Oh yes, I remember, it is called a "filesystem". Let's use some of those > to store the mail data. It will be soooo awesome! ;-) I think for the serious engineer there's Linux if even more serious there's UNIX and for the rest there's MS..... Actually as a medical term MS is something not that great to have; why does that also equate to IT/Computing too ;-P > > > Ok, back being serious: there is nothing wrong with using a RDBMS in the > way it was intented, to store user credentials, quota values, account > settings, forwarding addresses, address book data, bookmarks, etc. I agree! My humble opinion for a personal preference setup in this instance: FreeBSD 8.2 x64 as base OS UFS2 running on root drive Create ZFS pools for storage Have users mailboxes on the ZFS pools Enable ZFS caching and snapshots Dovecot to manage IMAPv4 --- Get rid of MS altogether! ....Then start working a really cool implementation of UNIX/Linux only infrastructure :-) > > > Gr??e, > Sven. > Regards, Kaya From stan at hardwarefreak.com Sun Mar 18 11:16:55 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 18 Mar 2012 04:16:55 -0500 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F650109.7090702@gmail.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F65A807.9020206@hardwarefreak.com> On 3/17/2012 4:24 PM, Kaya Saman wrote: > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. It's been many years since I used, or supported, MS Outlook. That said, for the 10+ years I did support it, ~1996-2006, the corporate version of Outlook, not to be confused with Outlook Express, did not store any mail in local .PST files unless specifically configured to do so. By default it keeps all mail in the user account in the Exchange server store. Thus I would assume these Outlook clients have been manually configured to use .PST files to keep copies of mail locally, for faster access and to keep inefficient MS Exchange (MAPI) traffic off the WAN link? Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off? If the problem isn't with the .PST format for storing the emails, why not simply setup a local Samba server and configure the Outlook clients to store users' PSTs on Samba shares? Better yet, if you already have a file server for home directories, simply use a folder redirection policy to put the PST files in folders in users' home directories. This is an extremely common practice in the MS world because all Microsoft Windows apps store everything in the user profile directory by default, which again, causes big problems with roaming profiles, which many/most enterprises use. -- Stan From kayasaman at gmail.com Sun Mar 18 11:46:13 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 09:46:13 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F65AEE5.30208@gmail.com> On 03/18/2012 09:16 AM, Stan Hoeppner wrote: > On 3/17/2012 4:24 PM, Kaya Saman wrote: > >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > It's been many years since I used, or supported, MS Outlook. That said, > for the 10+ years I did support it, ~1996-2006, the corporate version of > Outlook, not to be confused with Outlook Express, did not store any mail > in local .PST files unless specifically configured to do so. By default > it keeps all mail in the user account in the Exchange server store. > > Thus I would assume these Outlook clients have been manually configured > to use .PST files to keep copies of mail locally, for faster access and > to keep inefficient MS Exchange (MAPI) traffic off the WAN link? > > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? > > If the problem isn't with the .PST format for storing the emails, why > not simply setup a local Samba server and configure the Outlook clients > to store users' PSTs on Samba shares? > > Better yet, if you already have a file server for home directories, > simply use a folder redirection policy to put the PST files in folders > in users' home directories. This is an extremely common practice in the > MS world because all Microsoft Windows apps store everything in the user > profile directory by default, which again, causes big problems with > roaming profiles, which many/most enterprises use. > Thanks Stan for this, currently our users have about 270MB space located on the Exchange server which we have no control over. Therefor users are currently manually backing up their information to .pst. Since all contact/calendar/other information is already stored on the server the IMAP solution is a better one. It also means that a more UNIX/Linux centric approach is being ask for/tolerated which is where I come in being the only full-bread UNIX engineer on site. I couldn't comment on the MS side of things as I have never really used MS stuff before but my plan using ZFS and FreeBSD should be the best. Again it's going to be **scalable** storage which is perfect! -Also easy to maintain; otherwise I don't think there would be anyone left who will be able to admin the SMB/local directory method (I can't as I don't understand MS) - additionally there isn't much space available and bandwidth either meaning we would purchase a dedicated server or build a dedicated server for this (well I would :-) ). Regards, Kaya From tss at iki.fi Sun Mar 18 17:15:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 17:15:46 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> On 18.3.2012, at 0.28, Sven Hartge wrote: > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. Dovecot moves only minimal amount of data within mbox. A flag change writes only a few bytes to mbox file, it doesn't rewrite it the whole file. Only time when the entire mbox file is rewritten is when you expunge the first message. From sven at svenhartge.de Sun Mar 18 17:19:38 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 16:19:38 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: Timo Sirainen wrote: > On 18.3.2012, at 0.28, Sven Hartge wrote: >> mbox has big problems with concurrent writes, the bigger the mbox is, >> the more problems you get. This is mainly caused by the meta-data of >> a message (meaning flags, status, etc.) which is stored inside the >> mbox file itself. Flagging a message as read or replied causes the >> whole mbox file to be rewritten. > Dovecot moves only minimal amount of data within mbox. A flag change > writes only a few bytes to mbox file, it doesn't rewrite it the whole > file. Only time when the entire mbox file is rewritten is when you > expunge the first message. OK, then my memory was clouded by my exposure to the brokenness of uw-imapd and uw-popd, who rewrite an mbox file very often, resulting in abysmal performance. Gr??e, Sven. -- Sigmentation fault. Core dumped. From CMarcus at Media-Brokers.com Sun Mar 18 17:32:21 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:32:21 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F660005.8070902@Media-Brokers.com> On 2012-03-18 5:16 AM, Stan Hoeppner wrote: > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? If so, using redirected folders (if you're not using them, you should be) would alleviate this problem nicely, even in a large environment. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 18 17:36:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:36:25 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: <4F6600F9.1010604@Media-Brokers.com> On 2012-03-18 11:15 AM, Timo Sirainen wrote: > Only time when the entire mbox file is rewritten is when you > expunge the first message. Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Although I have no desire to use mbox (planning on using mdbox), so it isn't important to me... ;) -- Best regards, Charles From janfrode at tanso.net Sun Mar 18 19:52:49 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 18 Mar 2012 18:52:49 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <20120318175249.GA15524@dibs.tanso.net> On Sun, Mar 18, 2012 at 11:36:25AM -0400, Charles Marcus wrote: > > Hmmm... wonder if there would be a way to add some kind of 'dummy' > first message that dovecot would simply ignore (not show to the > user), that would prevent that bevaior? That's what uw-imap does. It creates a message with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA", which is very annoying if your users has direct access to the mbox's... http://www.washington.edu/imap/IMAP-FAQs/index.html#6.14 -jf From tss at iki.fi Sun Mar 18 20:07:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 20:07:34 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <50F4C47F-EE04-4384-9465-89AFCF5C8B67@iki.fi> On 18.3.2012, at 17.36, Charles Marcus wrote: > On 2012-03-18 11:15 AM, Timo Sirainen wrote: >> Only time when the entire mbox file is rewritten is when you >> expunge the first message. > > Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Dovecot has such a dummy first message exactly like UW-IMAP. But what I meant is that if you expunge the first message and you want to free the space used by it, there's no other choice than to rewrite all of the messages after it. (And of course the first message isn't special in any way, there's just about as much of rewriting if you delete the 2nd or 3rd or other messages from the beginning of the mbox.) From arekm at maven.pl Sun Mar 18 23:00:35 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:00:35 +0100 Subject: [Dovecot] mdbox and filesystem quota Message-ID: <201203182200.36011.arekm@maven.pl> http://wiki2.dovecot.org/MailboxFormat/dbox "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. " What happens if there is filesystem hard quota that is exceeded? Will dovecot allow to delete mails to free space without a need to wait for cronjob to do the purge? -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:45:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:45:54 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182200.36011.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> Message-ID: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > http://wiki2.dovecot.org/MailboxFormat/dbox > > "Expunging a message only decreases the message's refcount. The space is later > freed in "purge" step. This is typically done in a nightly cronjob when > there's less disk I/O activity. " > > What happens if there is filesystem hard quota that is exceeded? Will dovecot > allow to delete mails to free space without a need to wait for cronjob to do > the purge? No. Also the purging itself won't work, because it needs to write new data first before it can delete old data. Don't run out of disk space! From arekm at maven.pl Sun Mar 18 23:52:38 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:52:38 +0100 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> Message-ID: <201203182252.38995.arekm@maven.pl> On Sunday 18 of March 2012, Timo Sirainen wrote: > On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > > http://wiki2.dovecot.org/MailboxFormat/dbox > > > > "Expunging a message only decreases the message's refcount. The space is > > later freed in "purge" step. This is typically done in a nightly cronjob > > when there's less disk I/O activity. " > > > > What happens if there is filesystem hard quota that is exceeded? Will > > dovecot allow to delete mails to free space without a need to wait for > > cronjob to do the purge? > > No. Also the purging itself won't work, because it needs to write new data > first before it can delete old data. Don't run out of disk space! Can dovecot treat soft quota like hard quota for user then? Or better enforce quota based on filesystem quot information. With xfs I can set quota but turn enforcement off. All fs quota counters work but no enforcement is being made by xfs itself. -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:56:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:56:48 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182252.38995.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> <201203182252.38995.arekm@maven.pl> Message-ID: <4BC1C8AE-4AD5-4A51-8954-FFAA4B84F35D@iki.fi> On 18.3.2012, at 23.52, Arkadiusz Mi?kiewicz wrote: >>> "Expunging a message only decreases the message's refcount. The space is >>> later freed in "purge" step. This is typically done in a nightly cronjob >>> when there's less disk I/O activity. " >>> >>> What happens if there is filesystem hard quota that is exceeded? Will >>> dovecot allow to delete mails to free space without a need to wait for >>> cronjob to do the purge? >> >> No. Also the purging itself won't work, because it needs to write new data >> first before it can delete old data. Don't run out of disk space! > > Can dovecot treat soft quota like hard quota for user then? > > Or better enforce quota based on filesystem quot information. With xfs I can > set quota but turn enforcement off. All fs quota counters work but no > enforcement is being made by xfs itself. Yes, Dovecot does the quota enforcement itself. I'm not entirely sure if it uses soft or hard quota though. From trashcan at odo.in-berlin.de Mon Mar 19 10:35:34 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Mon, 19 Mar 2012 09:35:34 +0100 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi -- On 15.03.2012 22:05, Timo Sirainen wrote: > On 15.3.2012, at 22.48, Michael Grimm wrote: >> Actually it's a bad idea to use root for ssh from a security point >> of view. A hacked root account isn't fun. Thus, normally one needs >> to explicitly change the config of the sshd daemon to allow root >> logins (at least with FreeBSD what I'm using). Thus, I do recommend >> to use an unprivileged user like vmail. > > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. Root has access to everyone's mail as well. > And if you allow ssh login only with public key authentication I > don't think there are much security issues. And finally, it would > be possible to write a small wrapper that allows the root's public > key auth to only execute dsync-user.sh script that can't do anything > except sync a specified user's mails. All those safety measures can be applied for the vmail user as well. Actually, that's what I did in my case, plus allowing ssh only between both mail servers (firewall rule). Regards, Michael From lcaron at unix-scripts.info Mon Mar 19 11:06:20 2012 From: lcaron at unix-scripts.info (Laurent CARON) Date: Mon, 19 Mar 2012 10:06:20 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot Message-ID: <20120319095939.maneexuo@trusted.unix-scripts.info> Hi, I'm currently having a fairly simple setup: - users (real, not virtual) - Maildir storage (over NFS) - 1 namespace I'm currently trying to render the storage snapshots available through dovecot (to allow my users to browse their mail history). dovecot.conf: namespace { inbox = yes location = prefix = INBOX. type = private } I did the following modifications: dovecot.conf: namespace snaps-h0 { prefix = INBOX.EmailBackup.h0. hidden = no list = yes inbox = no location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u type = private } Problem: I don't see the content of the inbox folder contained in the snapshots (subfolders are perfectly viewed). Do any of you have a clue on how to render it visible ? Thanks Laurent From jernej.porenta at arnes.si Mon Mar 19 11:58:56 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 10:58:56 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> Message-ID: <3974AB53-476A-4945-A828-11425C667165@arnes.si> On Mar 16, 2012, at 2:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 11.09, Jernej Porenta wrote: > >>>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>>> (uni_utf8_str_is_valid(vname)) >>> .. >> We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. >> >> Whenever . LIST "" "*" is issued, dovecot crashes: >> Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot > > I don't think this is the same Panic as the original one? What is the Panic message now? Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) It is the same. We will try 2.1.3 today and report the results... Regards, Jernej From mcazzador at gmail.com Mon Mar 19 12:50:39 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Mon, 19 Mar 2012 11:50:39 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi, i've a simple question, what do you mean for dovecot director setup? 'i've a doubt. The solution that i'm testing is using 3 mail server in different geoghrapic locations. An user can travel in varius location, and i want his imap mail reside on mail server in every locations. S? i use you solution about replication. First server (by dns record) that receive mail sync it on the other servers, and when user consult is mail by imap protocol everything is sync on all servers. Do you suggest to use a horizontal structure for it like i explain or is better to have a single node external mail server and customer locations server like slave? Thank's Il 19 marzo 2012 09:35, Michael Grimm ha scritto: > Hi -- > > > On 15.03.2012 22:05, Timo Sirainen wrote: >> >> On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>> Actually it's a bad idea to use root for ssh from a security point >>> of view. A hacked root account isn't fun. Thus, normally one needs >>> to explicitly change the config of the sshd daemon to allow root >>> logins (at least with FreeBSD what I'm using). Thus, I do recommend >>> to use an unprivileged user like vmail. >> >> >> Then again it's safer to use system user accounts than a single vmail >> account that has access to everyone's emails. > > > Root has access to everyone's mail as well. > > >> And if you allow ssh login only with public key authentication I >> don't think there are much security issues. And finally, it would >> be possible to write a small wrapper that allows the root's public >> key auth to only execute dsync-user.sh script that can't do anything >> except sync a specified user's mails. > > > All those safety measures can be applied for the vmail user as well. > Actually, that's what I did in my case, plus allowing ssh only between > both mail servers (firewall rule). > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Mon Mar 19 13:20:22 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 13:20:22 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages Message-ID: <4F671676.3060809@noa.gr> Hi, We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've been using LTB OpenLDAP packages (http://ltb-project.org/wiki/download#openldap), which get installed in non-standard file system locations. So, I would like to re-build Dovecot packages based on these OpenLDAP libraries, esp. because I see that dovecot RPM packages are built using OpenLDAP v2.3 libraries. I am not much experienced in building RPMs and preparing spec files. In http://dl.atrpms.net/all/dovecot.spec I see: ------------------------------------------------ BuildRequires: openldap-devel, cyrus-sasl-devel ... Obsoletes: %{name}-pgsql < %{epoch}:%{version}-%{release}, %{name}-mysql < %{epoch}:%{version}-%{release}, %{name}-sqlite < %{epoch}:%{version}-%{release}, %{name}-ldap < %{epoch}:%{version}-%{release}, $ Conflicts: %{name}-pgsql > %{epoch}:%{version}-%{release}, %{name}-mysql > %{epoch}:%{version}-%{release}, %{name}-sqlite > %{epoch}:%{version}-%{release}, %{name}-ldap > %{epoch}:%{version}-%{release}, $ ------------------------------------------------ So, I can change the former reference (openldap-devel) to: openldap-ltb-debuginfo, cyrus-sasl-devel Question 1: What other changes should we make in order to specify that we will be using LDAP libraries from: /usr/local/openldap/lib64 and include files from: /usr/local/openldap/include (rather than from /usr/lib64 and /usr/include, respectively, which are the standard file paths used in openldap-devel) Question 2: How the Obsoletes and Conflicts lines should be changed? Question 3: It seems to me (by reading the spec file) that the final Dovecot RPM (and the included executables) does not need any LDAP dynamic library in order to run with LDAP support (because I don't see any dependencies on openldap package). Can somebody please confirm? Any other associated info would be appreciated. Thanks, Nick From rob0 at gmx.co.uk Mon Mar 19 14:20:50 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:20:50 -0500 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <20120319122050.GM4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 09:35:34AM +0100, Michael Grimm wrote: > On 15.03.2012 22:05, Timo Sirainen wrote: > >On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>Actually it's a bad idea to use root for ssh from a security > >>point of view. A hacked root account isn't fun. Thus, normally > >>one needs to explicitly change the config of the sshd daemon to > >>to allow root logins (at least with FreeBSD what I'm using). > >>Thus, I do recommend to use an unprivileged user like vmail. > > > >Then again it's safer to use system user accounts than a single > >vmail account that has access to everyone's emails. > > Root has access to everyone's mail as well. I think you are missing the point, that being: if all your mail are belong to vmail, somebody set up us the bomb if the vmail account is compromised. (Obviously that's true with a root compromise as well, but that is unavoidable. Effects of a root compromise can be limited with technologies like Apparmor and SELinux, but that is difficult to configure properly and only provides limited benefit: compromised root can do everything real root was allowed to do.) The point is: vmail has added a SECOND vulnerable point from which disaster can ensue. If mailbox ownership is distributed among multiple UID/GID, compromise of any one of those only endangers the mails to which it had access. > >And if you allow ssh login only with public key authentication I > >don't think there are much security issues. And finally, it would > >be possible to write a small wrapper that allows the root's public > >key auth to only execute dsync-user.sh script that can't do > >anything except sync a specified user's mails. > > All those safety measures can be applied for the vmail user as > well. Actually, that's what I did in my case, plus allowing ssh > only between both mail servers (firewall rule). Sure, but there too, all your email eggs are in the vmail basket. No, disaster is not imminent nor even likely to ensue, but the fact stands that you and millions of other virtual-only sites do have this additional potential vulnerability. It is well supported in Dovecot to be able to use a unique UID and GID for every virtual mailbox, but management of such a system presents more challenges than the single-vmail-user approach. Consequently the popular virtual frontends don't support it. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From cjeanneret at internux.ch Mon Mar 19 12:28:34 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Mon, 19 Mar 2012 12:28:34 +0200 Subject: [Dovecot] Problem with sieve Message-ID: Hello List! I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA doesn't run sieve, and thus doesn't filter my emails. Here's the sieve configuration: plugin { # Used by both the Sieve plugin and the ManageSieve protocol sieve=/var/local/vmail/%n/dovecot.sieve sieve_dir=/var/local/vmail/%n/sieve sieve_extensions = +notify +imapflags } The managesiege: protocol managesieve { # Specify an alternative address:port the daemon must listen on # (default: *:2000) listen = localhost:2000 managesieve_logout_format = bytes ( in=%i : out=%o ) } (this one is working fine, I can edit the filters through roundcube webmail, and the correct file (/var/local/vmail/%n/dovecot.sieve) is edited) the lda part: protocol lda { postmaster_address = foo at bar.com mail_plugins = sieve } I think all is in place to allow dovecot to use sieve... ? One more thing: dovecot --version 1.2.15 Any help will be welcomed :). Thanks in advance ! Cheers, C. From ngu.antoine at gmail.com Mon Mar 19 14:27:08 2012 From: ngu.antoine at gmail.com (Antoine Nguyen) Date: Mon, 19 Mar 2012 13:27:08 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: 2012/3/19 C?dric Jeanneret > Hello List! > > I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > doesn't run sieve, and thus doesn't filter my emails. > > Here's the sieve configuration: > > plugin { > # Used by both the Sieve plugin and the ManageSieve protocol > sieve=/var/local/vmail/%n/**dovecot.sieve > sieve_dir=/var/local/vmail/%n/**sieve > sieve_extensions = +notify +imapflags > } > > The managesiege: > > protocol managesieve { > # Specify an alternative address:port the daemon must listen on > # (default: *:2000) > listen = localhost:2000 > managesieve_logout_format = bytes ( in=%i : out=%o ) > } > (this one is working fine, I can edit the filters through roundcube > webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > edited) > > the lda part: > > protocol lda { > postmaster_address = foo at bar.com > mail_plugins = sieve > } > > I think all is in place to allow dovecot to use sieve... ? > > One more thing: > > dovecot --version > 1.2.15 > > > Any help will be welcomed :). > > Thanks in advance ! > > Cheers, > > C. > Have you checked the MTA configuration. Does it use dovecot's LDA ? Antoine From rob0 at gmx.co.uk Mon Mar 19 14:32:14 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:32:14 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F671676.3060809@noa.gr> References: <4F671676.3060809@noa.gr> Message-ID: <20120319123213.GN4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 01:20:22PM +0200, Nikolaos Milas wrote: > We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / > RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've > been using LTB OpenLDAP packages > (http://ltb-project.org/wiki/download#openldap), which get > installed in non-standard file system locations. ISTM that herein lies the whole problem. Why did you not rpmbuild your OpenLDAP? That would have avoided all further fuss. Another observation I can offer, unwelcome as it may be: your OS choice was not a good one when you want the features of recent software. Perhaps you should rethink that choice. You have invested much effort in this task. > So, I would like to re-build Dovecot packages based on these > OpenLDAP libraries, esp. because I see that dovecot RPM packages > are built using OpenLDAP v2.3 libraries. > > I am not much experienced in building RPMs and preparing spec > files. And that is really more a question for a CentOS forum than here. > In http://dl.atrpms.net/all/dovecot.spec I see: > > ------------------------------------------------ > BuildRequires: openldap-devel, cyrus-sasl-devel The latter requirement seems curious to me. In what way does Dovecot use Cyrus SASL? -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From jernej.porenta at arnes.si Mon Mar 19 15:27:06 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 14:27:06 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <3974AB53-476A-4945-A828-11425C667165@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: On Mar 19, 2012, at 10:58 AM, Jernej Porenta wrote: > Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt > Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) > > It is the same. We will try 2.1.3 today and report the results... Same thing with 2.1.3 (. LIST "" "*"): Mar 19 14:08:59 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=28438, secured Mar 19 14:09:04 server dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 14:09:04 server dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071811870] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae0718118c6] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071810d83] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae0715500c5] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d6ac] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d826] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2ae07155e221] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2ae07155717c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 14:09:04 server dovecot: imap(username): Fatal: master: service(imap): child 28438 killed with signal 6 (core dumps disabled) The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz # dovecot -n # 2.1.3: /opt/dovecot-2.1.3/etc/dovecot/dovecot.conf # OS: Linux 2.6.18-274.17.1.el5 x86_64 CentOS release 5.7 (Final) default_login_user = nobody disable_plaintext_auth = no login_greeting = Server ready. login_trusted_networks = x.y.z.p/32 mail_location = mbox:~/:INBOX=%h/.mailbox:INDEX=/opt/dovecot2-indexes/%1u/%u mail_plugins = quota mail_privileged_group = dovecot mbox_write_locks = fcntl namespace inbox { inbox = yes location = prefix = separator = / type = private } passdb { args = session=yes dovecot driver = pam } plugin { quota = fs } protocols = imap pop3 service imap-login { inet_listener imap { port = 143 } inet_listener imaps { ssl = no } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { ssl = no } } ssl = no userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_max_userip_connections = 15 mail_plugins = quota imap_quota } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } Regards, Jernej From nmilas at noa.gr Mon Mar 19 15:47:24 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 15:47:24 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319123213.GN4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> Message-ID: <4F6738EC.5050200@noa.gr> On 19/3/2012 2:32 ??, /dev/rob0 wrote: > ISTM that herein lies the whole problem. Why did you not rpmbuild > your OpenLDAP? That would have avoided all further fuss. Thanks for the reply. First, how would I rpmbuild my openldap v2.4.x as a standard CentOS 5 package (i.e. replacing native openldap-2.3.43-25)? If I were more experienced, I could have tried to engineer openldap-2.3.43-25.el5.src.rpm to upgrade the system to use 2.4.x... But still, I haven't seen any OpenLDAP packages attempting to do so, probably because of the tight integration of CentOS with some openldap v2.3 libraries. I think it's good that third-party packages (even of the same software) give the ability to not mess with standard system. The same is true for reputable Symas OpenLDAP packages. So, I simply use LTB OpenLDAP, even though it's installed at non-standard locations. (This has an added benefit of easy migration. You can setup any/all of those on the same system and decide which one to enable at any time.) > Another observation I can offer, unwelcome as it may be: your OS > choice was not a good one when you want the features of recent > software. Perhaps you should rethink that choice. You have invested > much effort in this task. I like CentOS from many aspects as an enterprise server OS. I wouldn't change it. Yet, it's important to me to be able to build/combine non-standard packages. Even with CentOS 6, I would still continue to use LTB OpenLDAP for a number of reasons. It's true that I've invested much effort in this task, but mostly because my knowledge on this subject is very basic. Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. We just try to make things better (and make our life a bit more difficult) :-) > > And that is really more a question for a CentOS forum than here. > True, but I am hoping that there might be some Dovecot RHEL/CentOS packagers in this list, and that would help resolve issues more effectively, as it is a Dovecot-specific (even if for a package thereof) question. So, any help will be appreciated! > The latter requirement seems curious to me. In what way does Dovecot > use Cyrus SASL? Hmm, I can't tell. I hope atrpm packager(s), if present on this list, can provide some feedback. Thanks again, Nick From tss at iki.fi Mon Mar 19 15:53:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:53:40 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: <1332165220.26095.71.camel@innu> On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: > > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > > > > It is the same. We will try 2.1.3 today and report the results... > The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d From tss at iki.fi Mon Mar 19 15:57:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:57:58 +0200 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <20120319095939.maneexuo@trusted.unix-scripts.info> References: <20120319095939.maneexuo@trusted.unix-scripts.info> Message-ID: <1332165478.26095.73.camel@innu> On Mon, 2012-03-19 at 10:06 +0100, Laurent CARON wrote: > I did the following modifications: > dovecot.conf: > namespace snaps-h0 { > prefix = INBOX.EmailBackup.h0. > hidden = no > list = yes > inbox = no > location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u > type = private > } > > Problem: > I don't see the content of the inbox folder contained in the snapshots > (subfolders are perfectly viewed). > > Do any of you have a clue on how to render it visible ? So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| new} directories? The INBOX should be accessible via the "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a newer Dovecot version. From tss at iki.fi Mon Mar 19 16:00:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:00:00 +0200 Subject: [Dovecot] Problem managing mbox In-Reply-To: <4F64E923.5060401@gmail.com> References: <4F64E923.5060401@gmail.com> Message-ID: <1332165600.26095.75.camel@innu> On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: > Hello, > > I have a problem with dovecot. seems that do not erase mail that mail > client request to be erased. Are you sure the clients have actually issued the EXPUNGE command, rather than simply marked the mail with \Deleted flag? > And I have this errors: > > Error: Next message unexpectedly corrupted in mbox file > Info: mbox code isn't perfect, but if this doesn't happen often it shouldn't matter much. doveconf -n output might have been helpful in giving more suggestions. From tss at iki.fi Mon Mar 19 16:02:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:02:43 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F64C9E3.7080102@filez.com> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> <4F64C9E3.7080102@filez.com> Message-ID: <1332165763.26095.77.camel@innu> On Sat, 2012-03-17 at 18:29 +0100, Radim Kolar wrote: > > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? > indexes never existed because these mboxes were never used by dovecot, > its not conversion from one format to another, its import. > > Maybe open bug to add feature "dsync import" which will not depend on > existing indexes? dsync doesn't need existing indexes, it wants to create indexes. If it can't because of e.g. permission issues, you should be able to work around it with: dsync mirror mbox:~/mail:INDEX=/tmp/indexes I might change dsync at some point to work even without permanent mailbox GUIDs, but there are many other more important things to do. From lcaron at lncsa.com Mon Mar 19 16:05:38 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 15:05:38 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <1332165478.26095.73.camel@innu> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> Message-ID: <4F673D32.2060300@lncsa.com> On 19/03/2012 14:57, Timo Sirainen wrote: > So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| > new} directories? The INBOX should be accessible via the > "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a > newer Dovecot version. > It's not actually. Sorry for the obvoius info I didn't give. I'm currently using dovecot 2.0.7 Regards, Laurent From rob0 at gmx.co.uk Mon Mar 19 17:28:20 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 10:28:20 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F6738EC.5050200@noa.gr> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> Message-ID: <20120319152820.GP4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 03:47:24PM +0200, Nikolaos Milas wrote: > On 19/3/2012 2:32 ??, /dev/rob0 wrote: > > >ISTM that herein lies the whole problem. Why did you not rpmbuild > >your OpenLDAP? That would have avoided all further fuss. > > Thanks for the reply. > > First, how would I rpmbuild my openldap v2.4.x as a standard CentOS > 5 package (i.e. replacing native openldap-2.3.43-25)? If I were > more experienced, I could have tried to engineer > openldap-2.3.43-25.el5.src.rpm to upgrade the system to use That's what I would have tried. > 2.4.x... But still, I haven't seen any OpenLDAP packages attempting > to do so, probably because of the tight integration of CentOS with > some openldap v2.3 libraries. I don't have anything to tell you there, and I note that we are now fully off-topic. :) > I think it's good that third-party packages (even of the same > software) give the ability to not mess with standard system. The same > is true for reputable Symas OpenLDAP packages. > > So, I simply use LTB OpenLDAP, even though it's installed at > non-standard locations. Failing the SRPM translation, why not just install into the CentOS standard locations? ... oops, I typed too fast ... > (This has an added benefit of easy migration. You can setup any/all > of those on the same system and decide which one to enable at any > time.) So you are in fact using both the CentOS OpenLDAP and your own version? This does not sound good at all. :( > >Another observation I can offer, unwelcome as it may be: your > >OS choice was not a good one when you want the features of > >recent software. Perhaps you should rethink that choice. You > >have invested much effort in this task. > > I like CentOS from many aspects as an enterprise server OS. I > wouldn't change it. I don't doubt that CentOS/RHEL offers many benefits, but my point here is that in this endeavor you are seeing the drawbacks. > Yet, it's important to me to be able to build/combine non-standard > packages. Even with CentOS 6, I would still continue to use LTB > OpenLDAP for a number of reasons. > > It's true that I've invested much effort in this task, but mostly > because my knowledge on this subject is very basic. And there too, the better forum, with more of the skills you need, would be the CentOS one. :) > Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), > i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. > We just try to make things better (and make our life a bit more > difficult) :-) > > > > >And that is really more a question for a CentOS forum than here. > > > > True, but I am hoping that there might be some Dovecot RHEL/CentOS > packagers in this list, and that would help resolve issues more > effectively, as it is a Dovecot-specific (even if for a package > thereof) question. > > So, any help will be appreciated! > > >The latter requirement seems curious to me. In what way does > >Dovecot use Cyrus SASL? > > Hmm, I can't tell. I hope atrpm packager(s), if present on this > list, can provide some feedback. I was thinking maybe Timo would know. As far as I can tell it doesn't. I do see in configure.in's check for LDAP, a search for sasl.h or sasl/sasl.h, so it appears that Cyrus SASL might be required to build Dovecot's LDAP support. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From nmilas at noa.gr Mon Mar 19 18:01:01 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 18:01:01 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319152820.GP4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> <20120319152820.GP4603@harrier.slackbuilds.org> Message-ID: <4F67583D.2000309@noa.gr> On 19/3/2012 5:28 ??, /dev/rob0 wrote: > So you are in fact using both the CentOS OpenLDAP and your own > version? This does not sound good at all. :( I talked about migration, didn't I? (Helps in test environments too!) > And there too, the better forum, with more of the skills you need, > would be the CentOS one. :) > > ... > > And that is really more a question for a CentOS forum than here. > I guess I might have to subscribe to CentOS forum/mailing list... (I haven't needed to yet.) Thanks anyway, Nick From hsn at filez.com Mon Mar 19 18:01:18 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:01:18 +0100 Subject: [Dovecot] INBOX cant be created Message-ID: <4F67584E.7030309@filez.com> Inbox does not exists on disk, but following command sequence will not create it. 2 select INBOX 2 NO Mailbox doesn't exist: INBOX 3 create INBOX 3 NO [ALREADYEXISTS] Mailbox already exists: INBOX i think its bug From lcaron at lncsa.com Mon Mar 19 18:11:04 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 17:11:04 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <4F673D32.2060300@lncsa.com> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> <4F673D32.2060300@lncsa.com> Message-ID: <4F675A98.10302@lncsa.com> On 19/03/2012 15:05, Laurent CARON wrote: > On 19/03/2012 14:57, Timo Sirainen wrote: >> So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| >> new} directories? The INBOX should be accessible via the >> "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a >> newer Dovecot version. >> > > It's not actually. > > Sorry for the obvoius info I didn't give. > > I'm currently using dovecot 2.0.7 > > Regards, > > Laurent Upgrading did the trick. Thanks From hsn at filez.com Mon Mar 19 18:22:44 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:22:44 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67584E.7030309@filez.com> References: <4F67584E.7030309@filez.com> Message-ID: <4F675D54.4020203@filez.com> doveadm does not works too: sudo doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied sudo doveadm mailbox create -u admin INBOX.2 (works) From patrickdk at patrickdk.com Mon Mar 19 18:37:28 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Mon, 19 Mar 2012 12:37:28 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> I'm having this problem also, with a very very few users. But in my case the email isn't double gzip, just single like normal. Error: read(.../.Deleted Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output error (uid=250) All I have to do is rename the file to add back the lost S= part and all is fine. This has happened in the inbox, deleted, and trash folders so far. and always after a change, the S= exists for new emails. It's like it's loosing it on adding the read flag, and mailbox moves But out of millions of emails, only a very few are like this, that I know of, around 6 emails. I manually fixed them, will be looking to see if this issue comes back. Quoting Timo Sirainen : > On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: > >>> Alternatively you can just tell Dovecot not to care about it: >>> maildir_broken_filename_sizes=yes. Although you probably can't do >>> that if you have compressed mails. >> >> In the case above that mail was gzipped twice :( > > Yes, looks like Dovecot can't correctly fix the wrong S size for > gzipped mails. I don't know if I should bother fixing it, especially > since in your case the doubly-gzipped mails will look corrupted to > user.. From alexwbaule at gmail.com Mon Mar 19 19:04:12 2012 From: alexwbaule at gmail.com (Alex Baule) Date: Mon, 19 Mar 2012 14:04:12 -0300 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F675D54.4020203@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: doveadm(admin): Error: Can't create mailbox INBOX: Permission denied The INBOX exists but has a wrong owner. Em 19 de mar?o de 2012 13:22, Radim Kolar escreveu: > doveadm does not works too: > > sudo doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > sudo doveadm mailbox create -u admin INBOX.2 > (works) > From hsn at filez.com Mon Mar 19 20:23:04 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 19:23:04 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: <4F677988.9080403@filez.com> > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > > The INBOX exists but has a wrong owner. nope ponto# cd /var/mail ponto# mv admin/ admin.X ponto# doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied but it might be that ordinary user admin cant create directories in /var/mail message from IMAP reply is wrong for sure because mailbox does not exists: ponto# cd /var/mail ponto# mv admin admin.x ponto# telnet localhost imap 3 select inbox 3 NO Mailbox doesn't exist: INBOX 4 create INBOX 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX From Ralf.Hildebrandt at charite.de Mon Mar 19 20:35:47 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Mon, 19 Mar 2012 19:35:47 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> Message-ID: <20120319183547.GA28363@charite.de> * Patrick Domack : > I'm having this problem also, with a very very few users. > > But in my case the email isn't double gzip, just single like normal. > > Error: read(.../.Deleted > Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output > error (uid=250) > > All I have to do is rename the file to add back the lost S= part and > all is fine. > This has happened in the inbox, deleted, and trash folders so far. > and always after a change, the S= exists for new emails. It's like > it's loosing it on adding the read flag, and mailbox moves Yes, I'm also seeing it now with mailboxes where no mail is doubly gzipped. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From alex.handle at gmail.com Mon Mar 19 21:16:58 2012 From: alex.handle at gmail.com (Alex Ha) Date: Mon, 19 Mar 2012 20:16:58 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:39 PM, Alex Ha wrote: > On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: >> On 16.3.2012, at 22.00, Alex Ha wrote: >> >>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>> existing connection >> >> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >> > > Thanks Timo! I will try the patch and report to you. > Hi Timo! I tried the patch with 2.0.19 and the dovecot error messages disappeared. I still get a lot of this postfix warnings: SASL LOGIN authentication failed: Connection lost to authentication server but only for ips which tried a sasl brute force attack. "Connection lost to authentication server" could this be because of the dovecot auth penalties? so far i did not get any complaints from users. Thanks for your help! Alex From dovecot at r.paypc.com Mon Mar 19 23:11:25 2012 From: dovecot at r.paypc.com (Robin) Date: Mon, 19 Mar 2012 14:11:25 -0700 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F67A0FD.8050900@r.paypc.com> On 3/17/2012 12:36 PM, Sven Hartge wrote: > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. Ugh, I've tried the product. It works pretty well, until you move more than a small handful of users and email hives to it, and you hit some hard walls pretty fast with how many inbound emails/second it can handle for even burly server configurations. Those hard walls occur at too low a threshold for me. The product's mailing list is supportive and there are many dedicated DBMail users who step in an answer questions, but be prepared for "BUY MORE RAM" as the answer to concerns about performance. When 128GB of RAM is needed for a small organisation's email setup to perform well, I am strongly inclined to move on to the next product. Best practices for it seem to revolve around being able to have your ENTIRE email + index content resident in RAM. Well, gosh. Why didn't I think of that before instead of wasting all of this time worrying about design and efficiency? And if you're hoping that it will make text searches "automagically" fast, think again. Timo's FTS_SQUAT blows it out of the water by orders of magnitude, even with mailbox sizes of around 300K emails (20GB), let alone something like Lucene or Solr. I understand why it seems like a great idea to store email this way, but realise that the bulk of email is NOT structured or inherently relational. =R= From jsimmons at goblin.punk.net Tue Mar 20 01:04:29 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:04:29 -0700 Subject: [Dovecot] Using plaintext auth and SSL Message-ID: <201203191604.29407.jsimmons@goblin.punk.net> I'm working with a company that presently has a Linux mailserver which all users have (no shell) accounts on. Mail is accessed via pop3 with plaintext authentication. They want to move to a system using imap with SSL. I'm building them a new server. I'd like to offer both for a while so we can work the bugs out and migrate users over to SSL imap over time. It appears that in order to limit the imap connections to SSL I will need to run two separate instances of Dovecot. Is this correct? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From jsimmons at goblin.punk.net Tue Mar 20 01:37:05 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:37:05 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67BE5E.4000501@knutejohnson.com> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> Message-ID: <201203191637.05129.jsimmons@goblin.punk.net> On Monday, March 19, 2012 04:16:46 pm you wrote: > On 3/19/2012 4:04 PM, Jeff Simmons wrote: > > I'm working with a company that presently has a Linux mailserver which > > all users have (no shell) accounts on. Mail is accessed via pop3 with > > plaintext authentication. They want to move to a system using imap with > > SSL. I'm building them a new server. I'd like to offer both for a while > > so we can work the bugs out and migrate users over to SSL imap over > > time. It appears that in order to limit the imap connections to SSL I > > will need to run two separate instances of Dovecot. Is this correct? > > I only have SSL or TLS connections enabled and I only have one copy of > Dovecot running. Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls only authentication imap. The 'allow plaintext authentication' configuration directive appears to be global, meaning I will need to run two instances of dovecot for a while. Is that correct, or can this be done on a single instance of dovecot? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From dovecot at knutejohnson.com Tue Mar 20 01:56:01 2012 From: dovecot at knutejohnson.com (Knute Johnson) Date: Mon, 19 Mar 2012 16:56:01 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67C791.2000609@knutejohnson.com> On 3/19/2012 4:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > I'm pretty sure if you set disable_plain_text_auth = no that you can log in to the appropriate ports with SSL or without. Sorry I sent the first reply to you, wasn't paying attention. -- Knute Johnson From eliezer at ngtech.co.il Tue Mar 20 02:16:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:16:21 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67CC55.3070706@ngtech.co.il> On 20/03/2012 01:37, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > there is no connection between the plaintext auth to the ssl\tls layer. you can just change the in the service section of the 10-master.conf file of the imap to no imap at all and use only imaps listener with port for your choose such as 143 or 993 and you will have a only imap over ssl. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:18:39 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:18:39 +0200 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F677988.9080403@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> Message-ID: <4F67CCDF.2010309@ngtech.co.il> On 19/03/2012 20:23, Radim Kolar wrote: > >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >> >> The INBOX exists but has a wrong owner. > nope > ponto# cd /var/mail > ponto# mv admin/ admin.X > ponto# doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied get into the maildir folder and use: ls -la to see all the directories and permissions. it might be with a starting "." what will make it "invisible" to regular ls. Regards, Eliezer > > but it might be that ordinary user admin cant create directories in > /var/mail > message from IMAP reply is wrong for sure because mailbox does not exists: > > ponto# cd /var/mail > ponto# mv admin admin.x > ponto# telnet localhost imap > 3 select inbox > 3 NO Mailbox doesn't exist: INBOX > 4 create INBOX > 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:45:16 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:45:16 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67D31C.2030302@ngtech.co.il> On 20/03/2012 02:16, Eliezer Croitoru wrote: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. one mistake, change the imap service to port 0 and port 143 will be disabled with regular imap service > > Regards, > Eliezer > -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From florob at babelmonkeys.de Tue Mar 20 04:26:12 2012 From: florob at babelmonkeys.de (Florian Zeitz) Date: Tue, 20 Mar 2012 03:26:12 +0100 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67EAC4.1050208@babelmonkeys.de> Am 20.03.2012 01:16, schrieb Eliezer Croitoru: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. > Because it is going to drive me insane if I don't ask: Is there really no way to archive this with a modern (aka. STARTTLS based) IMAP setup? From gedalya at gedalya.net Tue Mar 20 05:19:42 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:19:42 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67F74E.10403@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > This is all you have to do: protocol imap { ssl=required } See: http://wiki2.dovecot.org/SSL Globally, you can leave disable_plaintext_auth = no, and leave protocol pop3 {} alone. Your clients will be able to log in to pop3 with any authentication mechanism you have enabled, and imap will be accessible only with SSL/TLS, either over port 143 with STARTTLS or over port 993 with implicit SSL. I actually took the trouble to verify this on my local server before posting, and it turns out the wiki didn't lie. From gedalya at gedalya.net Tue Mar 20 05:42:38 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:42:38 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67FCAE.9060205@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > Turns out you can also use the disable_plaintext_auth = yes directive under protocol imap {}, but as noted by others previously, this is related specifically to plaintext authentication methods, and is not the same as requiring SSL/TLS for the entire session. If my understanding is correct, disable_plaintext_auth means your clients can authenticate with non-plaintext e.g. with CRAM-MD5 and proceed with an unsecured session. From jeetuindian at gmail.com Tue Mar 20 06:33:47 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 10:03:47 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hi guys, I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix in 2 system both system is having same configuration and os. I want to use dsync in mirror mode via ssh but I am not able to do it. When I followed the dsync wiki then I an unable to find the username : *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com dsync -u username* In the above command what will b user name and id_dsa.dovecot I am unable to understand. -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andrei.michescu at miau.ca Tue Mar 20 07:19:47 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 20 Mar 2012 01:19:47 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: <679071689c2e93f66654f318d277e8fc.squirrel@web.miau.ca> Hello Jitendra, I'm trying to do the same thing but I hit a wall as the sync seems to be doubling some emails in my case. My issues is still open with Timo (I HOPE!) As you seem to be stuck in an earlier stage here are my advices: 1) add the id_rsa to the .ssh folder to the user that runs the sync to test that this step is ok you should be able to $ ssh mailuser at example.com without being asked for a certificate / password (if you wonder how to achieve this here is a sample tutorial http://jaybyjayfresh.com/2009/02/04/logging-in-without-a-password-certificates-ssh/) 2) username is the user that has the email (if you work in a virtual environment generally is user at domain). 3) the syntax that I found to be working for me is the following: doveadm -Dv sync -u user1 at dom1 -f ssh mx1.a doveadm dsync-server -u user1 at dom1 To explain a little bit clearer the setup: - you start with 2 server mx1.a and mx2.a. On both servers you have vpopmail as the virtual user management for the virtual domain dom1 - from mx1.a you can ssh vpopmail at mx2.a directly without being prompted for a certificate or password) - user1 at dom1 is a virtual user defined both on mx1.a and on mx2.a (which means that you can deliver emails to this user both at mx1.a and mx2.a and you can also read them through imap on both servers). Hope this makes it a little bit more clear. Have fun, Andrei > Hi guys, > > I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix > in 2 system both system is having same configuration and os. I want to use > dsync in mirror mode via ssh but I am not able to do it. When I followed > the dsync wiki then I an unable to find the username : > > *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com > dsync -u username* > > In the above command what will b user name and id_dsa.dovecot I am unable > to understand. > > > > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > From achekalin at lazurit.com Tue Mar 20 07:28:30 2012 From: achekalin at lazurit.com (Alexander Chekalin) Date: Tue, 20 Mar 2012 08:28:30 +0300 Subject: [Dovecot] Per-user IMAP enable - is it possible? Message-ID: <4F68157E.5090806@lazurit.com> Just wonder if it is possible to enable/disable IMAP4 on Dovecot (2.0.x as far) on per-user basis? The deal is simple: our policy is not to store a lot of mailing on mailserver (the user should store it locally), thus the 'use POP3' approach, but for a vary few users it is permitted to use IMAP4. But users sometimes simple miss the point that some mail clients (e.g. TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of mailings and no local store of it on user's workstation. Sound too complicated, but setting up two Dovecots is not something I'd love to do as well. Thank you for any ideas, Alexander From gedalya at gedalya.net Tue Mar 20 07:43:10 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 01:43:10 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F68157E.5090806@lazurit.com> References: <4F68157E.5090806@lazurit.com> Message-ID: <4F6818EE.6090801@gedalya.net> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: > Just wonder if it is possible to enable/disable IMAP4 on Dovecot > (2.0.x as far) on per-user basis? > > The deal is simple: our policy is not to store a lot of mailing on > mailserver (the user should store it locally), thus the 'use POP3' > approach, but for a vary few users it is permitted to use IMAP4. But > users sometimes simple miss the point that some mail clients (e.g. TB) > 'prefer' to use IMAP4 first, and afterward I see mailbox full of > mailings and no local store of it on user's workstation. > > Sound too complicated, but setting up two Dovecots is not something > I'd love to do as well. > > Thank you for any ideas, > Alexander There would be various ways to do this, the specifics would depend on what kind of passdb you use. If you happen to be using a SQL database, you could do something like this: Add an allow_imap column, and change the password_query in dovecot-sql.conf.ext to something like this: password_query = SELECT password FROM user WHERE username = '%n' AND domain = '%d' \ AND ('%s' != 'imap' or allow_imap=1) This would make the user appear to not exist when trying to log in via IMAP. http://wiki2.dovecot.org/Variables From gedalya at gedalya.net Tue Mar 20 08:18:12 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 02:18:12 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6818EE.6090801@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> Message-ID: <4F682124.4010406@gedalya.net> On 3/20/2012 1:43 AM, Gedalya wrote: > On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >> (2.0.x as far) on per-user basis? >> >> The deal is simple: our policy is not to store a lot of mailing on >> mailserver (the user should store it locally), thus the 'use POP3' >> approach, but for a vary few users it is permitted to use IMAP4. But >> users sometimes simple miss the point that some mail clients (e.g. >> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >> mailings and no local store of it on user's workstation. >> >> Sound too complicated, but setting up two Dovecots is not something >> I'd love to do as well. >> >> Thank you for any ideas, >> Alexander > There would be various ways to do this, the specifics would depend on > what kind of passdb you use. > > If you happen to be using a SQL database, you could do something like > this: Add an allow_imap column, and change the password_query in > dovecot-sql.conf.ext to something like this: > > password_query = SELECT password FROM user WHERE username = '%n' AND > domain = '%d' \ > AND ('%s' != 'imap' or allow_imap=1) > > This would make the user appear to not exist when trying to log in via > IMAP. > > http://wiki2.dovecot.org/Variables > Or like this, might be more appropriate. password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, NULL, 'y') as nologin \ FROM user WHERE username = '%n' AND domain = '%d' http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin From cjeanneret at internux.ch Tue Mar 20 08:49:10 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Tue, 20 Mar 2012 08:49:10 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: On 19.03.2012 14:27, Antoine Nguyen wrote: > 2012/3/19 C?dric Jeanneret > >> Hello List! >> >> I have a tiny-teeny problem with dovecot + sieve: it seems that the >> LDA >> doesn't run sieve, and thus doesn't filter my emails. >> >> Here's the sieve configuration: >> >> plugin { >> # Used by both the Sieve plugin and the ManageSieve protocol >> sieve=/var/local/vmail/%n/**dovecot.sieve >> sieve_dir=/var/local/vmail/%n/**sieve >> sieve_extensions = +notify +imapflags >> } >> >> The managesiege: >> >> protocol managesieve { >> # Specify an alternative address:port the daemon must listen on >> # (default: *:2000) >> listen = localhost:2000 >> managesieve_logout_format = bytes ( in=%i : out=%o ) >> } >> (this one is working fine, I can edit the filters through roundcube >> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) >> is >> edited) >> >> the lda part: >> >> protocol lda { >> postmaster_address = foo at bar.com >> mail_plugins = sieve >> } >> >> I think all is in place to allow dovecot to use sieve... ? >> >> One more thing: >> >> dovecot --version >> 1.2.15 >> >> >> Any help will be welcomed :). >> >> Thanks in advance ! >> >> Cheers, >> >> C. >> > > Have you checked the MTA configuration. Does it use dovecot's LDA ? > > Antoine Hello Antoine (and List), Well, it should use dovecot, as the mails are delivered to the user inbox - and I don't think postfix knows about them.. How may I be sure otherwise that postfix really uses dovecot? Cheers, C. From evocage at gmail.com Tue Mar 20 09:06:42 2012 From: evocage at gmail.com (evolution age) Date: Tue, 20 Mar 2012 12:36:42 +0530 Subject: [Dovecot] Dovecot with postfix setup Message-ID: Hey frnds, Could you suggest me any ebook or documentation for the setup of dovecot with postfix on centos 5.7 . I need it. -- Warm Regards Jitendra Kumar Bhaskar cell :- +91-8886742555 From gedalya at gedalya.net Tue Mar 20 09:12:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:12:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: <4F682DF3.2030409@gedalya.net> On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > On 19.03.2012 14:27, Antoine Nguyen wrote: >> 2012/3/19 C?dric Jeanneret >> >>> Hello List! >>> >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA >>> doesn't run sieve, and thus doesn't filter my emails. >>> >>> Here's the sieve configuration: >>> >>> plugin { >>> # Used by both the Sieve plugin and the ManageSieve protocol >>> sieve=/var/local/vmail/%n/**dovecot.sieve >>> sieve_dir=/var/local/vmail/%n/**sieve >>> sieve_extensions = +notify +imapflags >>> } >>> >>> The managesiege: >>> >>> protocol managesieve { >>> # Specify an alternative address:port the daemon must listen on >>> # (default: *:2000) >>> listen = localhost:2000 >>> managesieve_logout_format = bytes ( in=%i : out=%o ) >>> } >>> (this one is working fine, I can edit the filters through roundcube >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is >>> edited) >>> >>> the lda part: >>> >>> protocol lda { >>> postmaster_address = foo at bar.com >>> mail_plugins = sieve >>> } >>> >>> I think all is in place to allow dovecot to use sieve... ? >>> >>> One more thing: >>> >>> dovecot --version >>> 1.2.15 >>> >>> >>> Any help will be welcomed :). >>> >>> Thanks in advance ! >>> >>> Cheers, >>> >>> C. >>> >> >> Have you checked the MTA configuration. Does it use dovecot's LDA ? >> >> Antoine > > Hello Antoine (and List), > > Well, it should use dovecot, as the mails are delivered to the user > inbox - and I don't think postfix knows about them.. > How may I be sure otherwise that postfix really uses dovecot? > > Cheers, > > C. If you are using dovecot 1.2, I think lda should show lines like the following in your log. Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: msgid=: stored mail into mailbox 'INBOX' 'deliver' refers to lda. From cjeanneret at internux.ch Tue Mar 20 09:20:41 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:20:41 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F682DF3.2030409@gedalya.net> References: <4F682DF3.2030409@gedalya.net> Message-ID: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:12:51 -0400 Gedalya wrote: > On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > > On 19.03.2012 14:27, Antoine Nguyen wrote: > >> 2012/3/19 C?dric Jeanneret > >> > >>> Hello List! > >>> > >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > >>> doesn't run sieve, and thus doesn't filter my emails. > >>> > >>> Here's the sieve configuration: > >>> > >>> plugin { > >>> # Used by both the Sieve plugin and the ManageSieve protocol > >>> sieve=/var/local/vmail/%n/**dovecot.sieve > >>> sieve_dir=/var/local/vmail/%n/**sieve > >>> sieve_extensions = +notify +imapflags > >>> } > >>> > >>> The managesiege: > >>> > >>> protocol managesieve { > >>> # Specify an alternative address:port the daemon must listen on > >>> # (default: *:2000) > >>> listen = localhost:2000 > >>> managesieve_logout_format = bytes ( in=%i : out=%o ) > >>> } > >>> (this one is working fine, I can edit the filters through roundcube > >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > >>> edited) > >>> > >>> the lda part: > >>> > >>> protocol lda { > >>> postmaster_address = foo at bar.com > >>> mail_plugins = sieve > >>> } > >>> > >>> I think all is in place to allow dovecot to use sieve... ? > >>> > >>> One more thing: > >>> > >>> dovecot --version > >>> 1.2.15 > >>> > >>> > >>> Any help will be welcomed :). > >>> > >>> Thanks in advance ! > >>> > >>> Cheers, > >>> > >>> C. > >>> > >> > >> Have you checked the MTA configuration. Does it use dovecot's LDA ? > >> > >> Antoine > > > > Hello Antoine (and List), > > > > Well, it should use dovecot, as the mails are delivered to the user > > inbox - and I don't think postfix knows about them.. > > How may I be sure otherwise that postfix really uses dovecot? > > > > Cheers, > > > > C. > > If you are using dovecot 1.2, I think lda should show lines like the > following in your log. > > Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: > msgid=: stored mail into mailbox 'INBOX' > > 'deliver' refers to lda. > Hmm... strange, doesn't show up like that in logs: Mar 20 08:14:54 sqdf3 postfix/smtpd[27509]: connect from host.foo.bar[...] Mar 20 08:15:16 sqdf3 postfix/smtpd[27509]: 892335659F4: client=host.foo.bar[...] Mar 20 08:15:35 sqdf3 postfix/cleanup[27516]: 892335659F4: message-id=<> Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 08:15:35 sqdf3 postfix/virtual[27518]: 892335659F4: to=, relay=virtual, delay=27, delays=27/0.03/0/0.14, dsn=2.0.0, status=sent (delivered to maildir) Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: removed Maybe the problem is there (postfix main.cf): virtual_transport = virtual I have virtual users and a mailman running on the same domain :/. If postfix doesn't use dovecot, how comes that mails are put in the right place ? From 24x7server at 24x7server.net Tue Mar 20 09:35:56 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Tue, 20 Mar 2012 13:05:56 +0530 (Asi) Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 Message-ID: <.120.61.90.33.1332228956.squirrel@24x7server.net> hi my system is a centos 5 with qmailtoaster i migrated my email server with around 5000 users from dovecot version 1.2 to version 2 i have two separate 2 tb hdd's storing webmail data of these users. the load on the server goes very high over 100 during peak load times and the imap connections get dropped frequently, webmail becomes very slow. however pop3 download works ok in the dovecot log file i get errors as such Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 expunge attempts) i had configured dovecot 1.2 using source but i had installed version 2 using qmailtoaster rpm i am a bit confused as to what settings are to be done for a very busy server note that the cpu system usage and ram usage are less but IO wait goes to around 80-90 percent can you please guide me or post some dovecot version 2 config file settings that are relevant to a busy server. rajesh From gedalya at gedalya.net Tue Mar 20 09:36:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:36:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683393.60403@gedalya.net> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > status=sent (delivered to maildir) Your log clearly says postfix is delivering directly to the maildir. Postfix must be configured to find the user's maildir using things like virtual_mailbox_base and / or virtual_mailbox_maps ? From cjeanneret at internux.ch Tue Mar 20 09:44:58 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:44:58 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683393.60403@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> Message-ID: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:36:51 -0400 Gedalya wrote: > On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > > status=sent (delivered to maildir) > Your log clearly says postfix is delivering directly to the maildir. > Postfix must be configured to find the user's maildir using things like > virtual_mailbox_base and / or virtual_mailbox_maps ? > Geez.. right - taking it from ldap... should only comment out the following lines: virtual_mailbox_base = / virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf ? From gedalya at gedalya.net Tue Mar 20 10:01:08 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:01:08 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683944.2030408@gedalya.net> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 03:36:51 -0400 > Gedalya wrote: > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>> status=sent (delivered to maildir) >> Your log clearly says postfix is delivering directly to the maildir. >> Postfix must be configured to find the user's maildir using things like >> virtual_mailbox_base and / or virtual_mailbox_maps ? >> > Geez.. right - taking it from ldap... should only comment out the following lines: > virtual_mailbox_base = / > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > ? If you want postfix to use the dovecot LDA then you have to set it up, you can get a general idea here http://wiki.dovecot.org/LDA/Postfix and adapt it to your circumstances. Basically you have to add the appropriate lines to master.cf and put virtual_transport = dovecot in main.cf. From cjeanneret at internux.ch Tue Mar 20 10:23:30 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 09:23:30 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683944.2030408@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> Message-ID: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 04:01:08 -0400 Gedalya wrote: > On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > > On Tue, 20 Mar 2012 03:36:51 -0400 > > Gedalya wrote: > > > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > >>> status=sent (delivered to maildir) > >> Your log clearly says postfix is delivering directly to the maildir. > >> Postfix must be configured to find the user's maildir using things like > >> virtual_mailbox_base and / or virtual_mailbox_maps ? > >> > > Geez.. right - taking it from ldap... should only comment out the following lines: > > virtual_mailbox_base = / > > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > > > ? > If you want postfix to use the dovecot LDA then you have to set it up, > you can get a general idea here > http://wiki.dovecot.org/LDA/Postfix > and adapt it to your circumstances. > > Basically you have to add the appropriate lines to master.cf and put > virtual_transport = dovecot in main.cf. > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. It _should_ take transport_maps = hash:/etc/postfix/transport which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( From gedalya at gedalya.net Tue Mar 20 10:50:33 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:50:33 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6844D9.7050504@gedalya.net> On 3/20/2012 4:23 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 04:01:08 -0400 > Gedalya wrote: > >> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: >>> On Tue, 20 Mar 2012 03:36:51 -0400 >>> Gedalya wrote: >>> >>>> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>>>> status=sent (delivered to maildir) >>>> Your log clearly says postfix is delivering directly to the maildir. >>>> Postfix must be configured to find the user's maildir using things like >>>> virtual_mailbox_base and / or virtual_mailbox_maps ? >>>> >>> Geez.. right - taking it from ldap... should only comment out the following lines: >>> virtual_mailbox_base = / >>> virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf >>> >>> ? >> If you want postfix to use the dovecot LDA then you have to set it up, >> you can get a general idea here >> http://wiki.dovecot.org/LDA/Postfix >> and adapt it to your circumstances. >> >> Basically you have to add the appropriate lines to master.cf and put >> virtual_transport = dovecot in main.cf. >> > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > It _should_ take > transport_maps = hash:/etc/postfix/transport > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( Once you have dovecot set up in your master.cf you should be able to put as the last line simply domain.com dovecot What do your logs look like when this is set? From cjeanneret at internux.ch Tue Mar 20 11:08:01 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 10:08:01 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6844D9.7050504@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> Message-ID: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> > > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > > It _should_ take > > transport_maps = hash:/etc/postfix/transport > > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( > Once you have dovecot set up in your master.cf you should be able to put > as the last line simply > domain.com dovecot > What do your logs look like when this is set? > I guess it may be easier if I paste my whole config in here: postfix main.cf: smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases mydestination = public_fqdn, localhost mynetworks = 127.0.0.0/8 inet_interfaces = all recipient_delimiter = + smtpd_sasl_security_options = noanonymous myorigin = domain.ltd smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_sasl_path = private/auth smtpd_sasl_type = dovecot owner_request_special = no smtpd_tls_CAfile = /etc/postfix/ssl/ca.crt smtpd_tls_cert_file = /etc/postfix/ssl/cert.crt smtpd_tls_key_file = /etc/postfix/ssl/keyForApache2.key smtpd_use_tls = yes smtpd_sasl_tls_security_options = $smtpd_sasl_security_options smtpd_sasl_auth_enable = yes default_process_limit = 5 smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/access, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unlisted_recipient smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache broken_sasl_auth_clients = yes transport_maps = hash:/etc/postfix/transport virtual_transport = dovecot transport_maps = hash:/etc/postfix/transport virtual_alias_maps = ldap:/etc/postfix/ldap-aliases.cf, hash:/var/lib/mailman/data/virtual-mailman virtual_gid_maps = static:104 virtual_minimum_uid = 8 virtual_uid_maps = static:8 virtual_mailbox_base = / mailman_destination_recipient_limit = 1 virtual_mailbox_domains = avocats-ch.ch virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf With this configuration, here's what I get: sending to a virtual user: Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: DC9285659F4: client=remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/cleanup[1528]: DC9285659F4: message-id=<> Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 10:02:49 sqdf3 dovecot: deliver(camptocamp): msgid=: saved mail to INBOX Mar 20 10:02:49 sqdf3 postfix/pipe[1529]: DC9285659F4: to=, relay=dovecot, delay=0.45, delays=0.18/0.01/0/0.26, dsn=2.0.0, status=sent (delivered via dovecot service) Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: removed so it uses dovecot. but mail is not filtered as it should :(( And now, sending a mail to a mailman list: Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: NOQUEUE: reject: RCPT from remote.host.ltd[...]: 550 5.1.1 : Recipient address rejected: User unknown in virtual mailbox table; from= to= proto=ESMTP helo= Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: warning: non-SMTP command from remote.host.ltd[...]: Subject: testing mailman Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: disconnect from remote.host.ltd[...] may I cry ? :( From luca.palazzo at unict.it Tue Mar 20 11:09:12 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 20 Mar 2012 10:09:12 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash Message-ID: <4F684938.9000208@unict.it> Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. Log lines are like: Mar 20 10:05:45 mailgw-lb dovecot: pop3-login: Fatal: master: service(pop3-login): child 27764 killed with signal 11 (core dumps disabled) Mar 20 10:06:17 mailgw-lb dovecot: imap-login: Fatal: master: service(imap-login): child 28468 killed with signal 11 (core dumps disabled) Stack trace of a crashed process is like: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1220163904 (LWP 27764)] 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 #4 0xb77ca9b5 in server_input (proxy=0x0) at login-proxy.c:93 #5 0xb7793762 in io_loop_call_io (io=0x8094180) at ioloop.c:380 #6 0xb7794cc9 in io_loop_handler_run (ioloop=0x8055480) at ioloop-epoll.c:213 #7 0xb77936f9 in io_loop_run (ioloop=0x8055480) at ioloop.c:399 #8 0xb777e4c8 in master_service_run (service=0x80553b0, callback=0xb77cc110 ) at master-service.c:544 #9 0xb77cbcee in login_binary_run (binary=0x804ad80, argc=2, argv=0x80551c0) at main.c:406 #10 0x08049812 in main (argc=0, argv=0x0) at client.c:303 The strange part of the story is that not all process crash. I'm trying to figure out if only TLS/SSL process crash. Any idea? Thanks Luca From eliezer at ngtech.co.il Tue Mar 20 11:42:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:42:21 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6850FD.9010602@ngtech.co.il> On 20/03/2012 11:08, Cedric Jeanneret wrote: > I guess it may be easier if I paste my whole config in here: you didnt sent the virtual_transport file content. i will quote from the man pages of the transport: [quote] user at domain transport:nexthop Deliver mail for user at domain through transport to nexthop. [\quote] means you can specify specific transport such as maliman to specific user. but because you are using the virtual maps table\lookup you also must have a vaild ldap user with the same name for the list. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 11:45:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:45:21 +0200 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F682124.4010406@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> Message-ID: <4F6851B1.4030509@ngtech.co.il> On 20/03/2012 08:18, Gedalya wrote: > On 3/20/2012 1:43 AM, Gedalya wrote: >> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>> (2.0.x as far) on per-user basis? >>> >>> The deal is simple: our policy is not to store a lot of mailing on >>> mailserver (the user should store it locally), thus the 'use POP3' >>> approach, but for a vary few users it is permitted to use IMAP4. But >>> users sometimes simple miss the point that some mail clients (e.g. >>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>> mailings and no local store of it on user's workstation. >>> >>> Sound too complicated, but setting up two Dovecots is not something >>> I'd love to do as well. >>> >>> Thank you for any ideas, >>> Alexander >> There would be various ways to do this, the specifics would depend on >> what kind of passdb you use. >> >> If you happen to be using a SQL database, you could do something like >> this: Add an allow_imap column, and change the password_query in >> dovecot-sql.conf.ext to something like this: >> >> password_query = SELECT password FROM user WHERE username = '%n' AND >> domain = '%d' \ >> AND ('%s' != 'imap' or allow_imap=1) >> >> This would make the user appear to not exist when trying to log in via >> IMAP. >> >> http://wiki2.dovecot.org/Variables >> > Or like this, might be more appropriate. > > password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, > NULL, 'y') as nologin \ > FROM user WHERE username = '%n' AND domain = '%d' > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin > but this will disallow also pop3... Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From gedalya at gedalya.net Tue Mar 20 11:49:45 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 05:49:45 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6851B1.4030509@ngtech.co.il> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> <4F6851B1.4030509@ngtech.co.il> Message-ID: <4F6852B9.1050809@gedalya.net> On 3/20/2012 5:45 AM, Eliezer Croitoru wrote: > On 20/03/2012 08:18, Gedalya wrote: >> On 3/20/2012 1:43 AM, Gedalya wrote: >>> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>>> (2.0.x as far) on per-user basis? >>>> >>>> The deal is simple: our policy is not to store a lot of mailing on >>>> mailserver (the user should store it locally), thus the 'use POP3' >>>> approach, but for a vary few users it is permitted to use IMAP4. But >>>> users sometimes simple miss the point that some mail clients (e.g. >>>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>>> mailings and no local store of it on user's workstation. >>>> >>>> Sound too complicated, but setting up two Dovecots is not something >>>> I'd love to do as well. >>>> >>>> Thank you for any ideas, >>>> Alexander >>> There would be various ways to do this, the specifics would depend on >>> what kind of passdb you use. >>> >>> If you happen to be using a SQL database, you could do something like >>> this: Add an allow_imap column, and change the password_query in >>> dovecot-sql.conf.ext to something like this: >>> >>> password_query = SELECT password FROM user WHERE username = '%n' AND >>> domain = '%d' \ >>> AND ('%s' != 'imap' or allow_imap=1) >>> >>> This would make the user appear to not exist when trying to log in via >>> IMAP. >>> >>> http://wiki2.dovecot.org/Variables >>> >> Or like this, might be more appropriate. >> >> password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, >> NULL, 'y') as nologin \ >> FROM user WHERE username = '%n' AND domain = '%d' >> >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin >> > but this will disallow also pop3... > > > Eliezer > No. It will return NULL unless the service is 'imap' and allow_imap != 1. nologin=NULL has no effect, so everything is allowed. From cjeanneret at internux.ch Tue Mar 20 12:00:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 11:00:21 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6850FD.9010602@ngtech.co.il> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> <4F6850FD.9010602@ngtech.co.il> Message-ID: <20120320110021.529fba41@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 11:42:21 +0200 Eliezer Croitoru wrote: > On 20/03/2012 11:08, Cedric Jeanneret wrote: > > I guess it may be easier if I paste my whole config in here: > you didnt sent the virtual_transport file content. > i will quote from the man pages of the transport: > [quote] > user at domain transport:nexthop > Deliver mail for user at domain through transport to nexthop. > [\quote] > means you can specify specific transport such as maliman to specific user. > but because you are using the virtual maps table\lookup you also must > have a vaild ldap user with the same name for the list. > > Regards, > Eliezer > Hello, while trying to remove/add options to my postfix, this part is now working now - it uses dovecot for virtual users, and mailman for lists :). Now that's good, I'll go back to sieve and ensure there's no missing configuration in dovecot. I had to remove "virtual_transport" option, and remove an "@" in my transport map. I'll come back on this thread if I still have problem with sieve. Cheers, C. From cjeanneret at internux.ch Tue Mar 20 13:05:28 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 12:05:28 +0100 Subject: [Dovecot] dovecot, sieve and vacation Message-ID: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Hi there ! Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to after searching a bit on the net, I stumbled on another (old) thread: http://www.mail-archive.com/dovecot at dovecot.org/msg25955.html I'm not really sure it's the same problem, as I'm sending the mail from another host (via telnet, for testing purpose). Other sieve rules (such as flagging, moving and so on) work fine. Only vacation is crapy. Here's the roundcube generated rule: require ["vacation"]; if true { vacation :days 2 "on holidays!"; } Any help welcome :) Cheers, C. Informations: # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-6-pve i686 Debian 6.0.4 simfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): localhost:2000 ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login first_valid_uid: 8 mail_privileged_group: mail mail_uid: mail mail_gid: mail mail_location: maildir:/var/local/vmail/%n mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve managesieve_logout_format(default): bytes=%i/%o managesieve_logout_format(imap): bytes=%i/%o managesieve_logout_format(pop3): bytes=%i/%o managesieve_logout_format(managesieve): bytes ( in=%i : out=%o ) lda: postmaster_address: foo at bar.com mail_plugins: sieve auth default: mechanisms: plain login user: mail passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 432 user: postfix group: mail plugin: home: /var/local/vmail/%u sieve: /var/local/vmail/%n/.dovecot.sieve sieve_dir: /var/local/vmail/%n/sieve sieve_extensions: +notify +imapflags From jeetuindian at gmail.com Tue Mar 20 13:10:46 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 16:40:46 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hey Frnds, Could you tell me about the error : # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com doveadm(root): Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, home=/home/example1.com/jitendra.b doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:/home/ example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b doveadm(jitendra.b at example.com): Debug: fs: root=/home/ example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ example.com/jitendra.b, alt= dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 doveadm(jitendra.b): Fatal: User doesn't exist dsync-local(jitendra.b at example.com): Error: read() from worker server failed: EOF -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andreas.a.lamprecht at atos.net Tue Mar 20 13:16:33 2012 From: andreas.a.lamprecht at atos.net (Lamprecht, Andreas) Date: Tue, 20 Mar 2012 11:16:33 +0000 Subject: [Dovecot] IMAP and POP3 per SSL Message-ID: Hi! I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set SSLHonorCipherOrder On in apache config. This results in the following C-Code being executed: SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. Is there a way to implement such a setting into Dovecot, too? I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 *************** *** 924,930 **** X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); --- 924,930 ---- X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... So, maybe you have the time to look over it and implement a final solution for the BEAST problem. Greetings Andreas lamprecht From support at palatineweb.com Tue Mar 20 13:26:56 2012 From: support at palatineweb.com (Palatine Support) Date: Tue, 20 Mar 2012 11:26:56 +0000 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67CCDF.2010309@ngtech.co.il> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> <4F67CCDF.2010309@ngtech.co.il> Message-ID: <4F686980.5040600@palatineweb.com> I have tried to unsubscribe from this mailing list 10 times now. Remove my email address please asap. Thanks Paul On 20/03/2012 00:18, Eliezer Croitoru wrote: > On 19/03/2012 20:23, Radim Kolar wrote: >> >>> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >>> >>> The INBOX exists but has a wrong owner. >> nope >> ponto# cd /var/mail >> ponto# mv admin/ admin.X >> ponto# doveadm mailbox create -u admin INBOX >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > get into the maildir folder and use: > ls -la to see all the directories and permissions. > it might be with a starting "." what will make it "invisible" to > regular ls. > > Regards, > Eliezer > >> >> but it might be that ordinary user admin cant create directories in >> /var/mail >> message from IMAP reply is wrong for sure because mailbox does not >> exists: >> >> ponto# cd /var/mail >> ponto# mv admin admin.x >> ponto# telnet localhost imap >> 3 select inbox >> 3 NO Mailbox doesn't exist: INBOX >> 4 create INBOX >> 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX > > From robert at schetterer.org Tue Mar 20 13:32:04 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:32:04 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: References: Message-ID: <4F686AB4.3070506@schetterer.org> Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: > Hi! > > I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. > > Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. > The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 > > "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set > > SSLHonorCipherOrder On > > in apache config. This results in the following C-Code being executed: > > SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); > > This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. > > Is there a way to implement such a setting into Dovecot, too? > > I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. > This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: > > *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 > --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 > *************** > *** 924,930 **** > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > --- 924,930 ---- > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > > > Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... > > So, maybe you have the time to look over it and implement a final solution for the BEAST problem. > > Greetings > Andreas lamprecht > perhaps look at http://wiki2.dovecot.org/SSL/DovecotConfiguration -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From robert at schetterer.org Tue Mar 20 13:34:58 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:34:58 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: <4F686AB4.3070506@schetterer.org> References: <4F686AB4.3070506@schetterer.org> Message-ID: <4F686B62.2050205@schetterer.org> Am 20.03.2012 12:32, schrieb Robert Schetterer: > Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: >> Hi! >> >> I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. >> >> Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. >> The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 >> >> "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set >> >> SSLHonorCipherOrder On >> >> in apache config. This results in the following C-Code being executed: >> >> SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); >> >> This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. >> >> Is there a way to implement such a setting into Dovecot, too? >> >> I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. >> This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: >> >> *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 >> --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 >> *************** >> *** 924,930 **** >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> --- 924,930 ---- >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> >> >> Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... >> >> So, maybe you have the time to look over it and implement a final solution for the BEAST problem. >> >> Greetings >> Andreas lamprecht >> > > perhaps look at > > http://wiki2.dovecot.org/SSL/DovecotConfiguration > and perhaps have a look at http://hg.dovecot.org/dovecot-2.0/rev/e3d46fd04105 and upgrade your dove version to dovecot 2.0.18 -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From CMarcus at Media-Brokers.com Tue Mar 20 14:22:59 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 20 Mar 2012 08:22:59 -0400 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F6876A3.2040409@Media-Brokers.com> On 2012-03-20 3:06 AM, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. Distro specific questions should be directed to your distro support lists. -- Best regards, Charles From nmilas at noa.gr Tue Mar 20 15:12:31 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 20 Mar 2012 15:12:31 +0200 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F68823F.4040905@noa.gr> On 20/3/2012 9:06 ??, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. You have not provided any info on your requirements, so it's hard to provide assistance. If you are only now starting the design (you should devote at least some time to it - design is the cornerstone) and you want LDAP, you may want to check projects like: GOsa (https://oss.gonicus.de/labs/gosa/) Or use a packaged solution, if it's OK with your requirements: http://www.iredmail.org/ (I prefer to install/control packages personally.) If you go manually, it shouldn't be difficult to find one of the many guides on the web. First, you need to find packages supporting the features you need (because the CentOS standard packages are very old). Of course you can start with CentOS standard Postfix package (supports ldap, pcre, SASL, TLS), but make sure you upgrade soon esp. if it's a production system! For Postfix, check that the package offers support for whatever you want (e.g. LDAP, mysql, SASL auth, TLS, pcre etc.). Otherwise, you may need to build your own RPM. You may want to read: http://tech.groups.yahoo.com/group/postfix-users/message/284530 http://tech.groups.yahoo.com/group/postfix-users/message/284359 Dovecot RPMs from here: http://packages.atrpms.net/dist/el5/dovecot/ are fine (I think they support anything you would possibly want). Then, find a guide like: http://www.howtoforge.com/linux_postfix_virtual_hosting or http://wanderingbarque.com/howtos/mailserver/mailserver.html depending on the type of setup you want (e.g. multiple domains, local vs virtual users, etc.) Good luck! Nick From nicolas.kowalski at gmail.com Tue Mar 20 15:55:12 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Tue, 20 Mar 2012 14:55:12 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? Message-ID: <20120320135512.GD28951@petole.demisel.net> Hello, Does dovecot support the subject Alternative Name email value [1] as ssl_cert_username_field? If so, how should it be specified in the configuration? Thanks. [1] http://www.openssl.org/docs/apps/x509v3_config.html#Subject_Alternative_Name_ -- Nicolas From stephan at rename-it.nl Tue Mar 20 16:16:21 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 20 Mar 2012 15:16:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F689135.6020602@rename-it.nl> Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > Hi there ! > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to The vacation action will not send a response when the envelope-to address (in your case virtual_user at hostname) is not contained in the To: or Cc: headers of the message itself; the message needs to be explicitly addressed to the recipient. For the version you are using this needs to match the final recipient as passed to Dovecot. In newer versions of the Pigeonhole Sieve implementation the original SMTP envelope recipient (i.e. before local rewrites) can also be used instead. Alternatively, new versions allow disabling this behavior entirely, although this is not recommended. Regards, Stephan. From andrei at lctax.ro Tue Mar 20 16:46:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Tue, 20 Mar 2012 10:46:58 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: Hello, As log as example.com resolves to 192.0.43.10 (which I suppose it is a host that you don't own) this will not work. dsync seems to be resolving example.com and trying to connect there via ssh using the current user. It is better to test on domains that you own, on on domains that don't exists and you adjust your hosts file accordingly. Also seems that you have some issues with the configuration file. Can you post your dovecot -n output. Thnx, Andrei > Hey Frnds, > > Could you tell me about the error : > > # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com > > > > doveadm(root): Debug: Loading modules from directory: > /usr/local/lib/dovecot/doveadm > doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: > undefined symbol: acl_user_module (this is usually intentional, so just > ignore this message) > doveadm(root): Debug: Skipping module doveadm_expire_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined > symbol: expire_set_lookup (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_quota_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined > symbol: quota_user_module (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because > dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: > undefined symbol: i_stream_create_deflate (this is usually intentional, so > just ignore this message) > doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: > undefined symbol: fts_list_backend (this is usually intentional, so just > ignore this message) > doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, > home=/home/example1.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, > prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes > location=mbox:/home/ > example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: fs: root=/home/ > example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ > example.com/jitendra.b, alt= > dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions > from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 > doveadm(jitendra.b): Fatal: User doesn't exist > dsync-local(jitendra.b at example.com): Error: read() from worker server > failed: EOF > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > > !DSPAM:4f6865bf72822789337279! > From cjeanneret at internux.ch Tue Mar 20 16:48:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 15:48:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <4F689135.6020602@rename-it.nl> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> <4F689135.6020602@rename-it.nl> Message-ID: <20120320154821.11c80a31@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 15:16:21 +0100 Stephan Bosch wrote: > Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > > Hi there ! > > > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to > > The vacation action will not send a response when the envelope-to > address (in your case virtual_user at hostname) is not contained in the To: > or Cc: headers of the message itself; the message needs to be explicitly > addressed to the recipient. For the version you are using this needs to > match the final recipient as passed to Dovecot. In newer versions of the > Pigeonhole Sieve implementation the original SMTP envelope recipient > (i.e. before local rewrites) can also be used instead. Alternatively, > new versions allow disabling this behavior entirely, although this is > not recommended. > > Regards, > > Stephan. Hello, thanks for the hint. In fact, I have to add the address aliases in the rule, and it works. Not really cool, but it works like that :). Cheers, C. From patrickdk at patrickdk.com Tue Mar 20 16:55:38 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Tue, 20 Mar 2012 10:55:38 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319183547.GA28363@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> Message-ID: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> And found two more users with this issue, but while looking at it, I see another related issue, but it's not throwing an error. all email in the INBOX/new and /cur are correct but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa This is happening for all folder moves. the Sent folder isn't affected, but I assume cause an email wasn't moved in that case. Quoting Ralf Hildebrandt : > * Patrick Domack : >> I'm having this problem also, with a very very few users. >> >> But in my case the email isn't double gzip, just single like normal. >> >> Error: read(.../.Deleted >> Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output >> error (uid=250) >> >> All I have to do is rename the file to add back the lost S= part and >> all is fine. >> This has happened in the inbox, deleted, and trash folders so far. >> and always after a change, the S= exists for new emails. It's like >> it's loosing it on adding the read flag, and mailbox moves > > Yes, I'm also seeing it now with mailboxes where no mail is doubly > gzipped. > > -- > Ralf Hildebrandt > Gesch?ftsbereich IT | Abteilung Netzwerk > Charit? - Universit?tsmedizin Berlin > Campus Benjamin Franklin > Hindenburgdamm 30 | D-12203 Berlin > Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 > ralf.hildebrandt at charite.de | http://www.charite.de From ka at pacific.net Tue Mar 20 17:26:01 2012 From: ka at pacific.net (Ken A) Date: Tue, 20 Mar 2012 10:26:01 -0500 Subject: [Dovecot] mdbox and pop3 locking Message-ID: <4F68A189.2010800@pacific.net> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Thanks, Ken -- Ken Anderson Pacific Internet - http://www.pacific.net From Ralf.Hildebrandt at charite.de Tue Mar 20 17:33:20 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Tue, 20 Mar 2012 16:33:20 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120320153320.GD26616@charite.de> * Patrick Domack : > And found two more users with this issue, but while looking at it, I > see another related issue, but it's not throwing an error. > > all email in the INBOX/new and /cur are correct > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have > double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Yes indeed: postamt:/home/h/a/happel/Maildir/.Trash/cur# ll total 16 -rw------- 1 happel users 7541 Mar 20 15:23 1332253428.M342974P5666.postamt.charite.de,S=37641,W=38197,S=37641,W=38197:2,Se -rw------- 1 happel users 6378 Mar 20 15:42 1332254568.M9552P591.postamt.charite.de,S=27486,W=28188,S=27486,W=28188:2,Se -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From micah at riseup.net Tue Mar 20 17:40:43 2012 From: micah at riseup.net (Micah Anderson) Date: Tue, 20 Mar 2012 11:40:43 -0400 Subject: [Dovecot] Antispam plugin not compatible with Dovecot 2.1 References: <4F155670.6010905@gmail.com> <1326897258.11500.53.camel@innu> <1326904309.11500.83.camel@innu> Message-ID: <877gyfp9fo.fsf@algae.riseup.net> "Eugene Paskevich" writes: > On Wed, 18 Jan 2012 18:31:49 +0200, Timo Sirainen wrote: > >> On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: >>> >> mailbox.c: In function 'antispam_save_begin': >>> >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >>> >> 'copying' >>> > >>> > The "copying" should be changed to "copying_via_save". >>> >>> Thank you, Timo. >>> Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? >> >> Where do you expect to find such macro? ;) Hm. Perhaps I should try to >> add one. > > Heh. That's Johannes' package private macro... :) I notice that Johannes hasn't made a 2.1 version of the anti-spam plugin, Eugene were you able to build one successfully? If so, would you be willing to share your changes that were required to make it work? thanks, micah From jernej.porenta at arnes.si Tue Mar 20 19:44:26 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 20 Mar 2012 18:44:26 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1332165220.26095.71.camel@innu> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> <1332165220.26095.71.camel@innu> Message-ID: <79D375C1-1009-46B3-A383-A33DD0A699E8@arnes.si> On Mar 19, 2012, at 2:53 PM, Timo Sirainen wrote: > On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: >>> Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) >>> >>> It is the same. We will try 2.1.3 today and report the results... > >> The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz > > Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d > Confirmed working? Thank you again, cheers, Jernej From mjeghers at Brocade.com Tue Mar 20 20:29:56 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 11:29:56 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> All, Below is my config. When I run dovecot from xinetd, I get these errors in the log: Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ total 248656 drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root users 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim Any ideas what is wrong? What is different running under xinetd? All the process run under the same user ids... Thanks, /Mark My config --------------------------------------------------------------- # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.7.1.el6.centos.plus.i686 i686 CentOS release 6.2 (Final) ext4 auth_debug = yes auth_verbose = yes auth_verbose_passwords = plain disable_plaintext_auth = no doveadm_worker_count = 4 mail_debug = yes mail_gid = users mail_location = mbox:/var/spool/mailpop3:INBOX=/var/spool/mailpop3/%u mail_uid = root mbox_write_locks = fcntl passdb { args = /etc/passwd.dovecot driver = passwd-file } passdb { driver = shadow } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size from subject vsize flags mail_log_group_events = yes } protocols = pop3 ssl_cert = References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F6943D6.1000600@hardwarefreak.com> On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From mjeghers at Brocade.com Wed Mar 21 06:26:23 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 21:26:23 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <4F6943D6.1000600@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Hi Stan Afraid it did not help. Here is what I got: *** entered into a telnet session... user ann +OK pass ******** -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Connection closed by foreign host. [root at t4pserver2 mailpop3]# *** resulted in maillog... Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 *** file permissions... [root at t4pserver2 mailpop3]# ls -al total 248652 drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. Any other ideas? /Mark -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Stan Hoeppner Sent: Tuesday, March 20, 2012 7:59 PM To: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From nicolas.kowalski at gmail.com Wed Mar 21 08:50:49 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Wed, 21 Mar 2012 07:50:49 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? In-Reply-To: <20120320135512.GD28951@petole.demisel.net> References: <20120320135512.GD28951@petole.demisel.net> Message-ID: <20120321065049.GE28951@petole.demisel.net> On Tue, Mar 20, 2012 at 02:55:12PM +0100, Nicolas KOWALSKI wrote: > Does dovecot support the subject Alternative Name email value [1] as > ssl_cert_username_field? If so, how should it be specified in the > configuration? Well, I just found the wiki states no: "The text is looked up from subject DN's specified field" (http://wiki2.dovecot.org/SSL/DovecotConfiguration) Sorry for the noise, -- Nicolas From stan at hardwarefreak.com Wed Mar 21 10:41:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 03:41:39 -0500 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F699443.1090704@hardwarefreak.com> On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From nmilas at noa.gr Wed Mar 21 11:00:10 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Wed, 21 Mar 2012 11:00:10 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 Message-ID: <4F69989A.3000106@noa.gr> Hi, I've upgraded from 2.0.13 to 2.1.1 and when I started the service, I got the following warning: Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) I didn't see such warnings in 2.0.13. I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? (I could also set "userdb_warning_disable=yes" as advised, but I'm trying to figure out what's the real cause of the warning.) The config follows below. Thanks, Nick ============================================================= protocols = imap pop3 mail_location = maildir:~/Maildir/ mail_gid = 502 mail_uid = 502 auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes disable_plaintext_auth = no mail_plugins = quota protocol imap { imap_client_workarounds = "delay-newmail " mail_plugins = quota imap_quota } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota postmaster_address = sysadmin at example.com sendmail_path = /usr/lib/sendmail } userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap } passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u } service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root } service imap-login { service_count = 1 vsz_limit = 64 M } service pop3-login { service_count = 1 vsz_limit = 64 M } ssl_ca = References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> On 19.3.2012, at 21.16, Alex Ha wrote: >>>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>>> existing connection >>> >>> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >>> >> >> Thanks Timo! I will try the patch and report to you. >> > > Hi Timo! > > I tried the patch with 2.0.19 and the dovecot error messages disappeared. OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever gets released). > I still get a lot of this postfix warnings: > > SASL LOGIN authentication failed: Connection lost to authentication server > > but only for ips which tried a sasl brute force attack. > > "Connection lost to authentication server" could this be because of > the dovecot auth penalties? > so far i did not get any complaints from users. The auth penalties wait for max. 17 seconds I think. Looks like Postfix has a timeout of 10 seconds. You could disable auth penalties, or perhaps Postfix should use 20 second limit. From CMarcus at Media-Brokers.com Wed Mar 21 13:55:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 07:55:19 -0400 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> Message-ID: <4F69C1A7.2040601@Media-Brokers.com> On 2012-03-21 7:48 AM, Timo Sirainen wrote: > On 19.3.2012, at 21.16, Alex Ha wrote: >>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>> 7542 of existing connection >>>> Oh, right, PIDs of course aren't unique when you're using >>>> mulitiple servers. Try if the attached patch fixes your >>>> troubles. If it does, I'll commit it to hg. >>> Thanks Timo! I will try the patch and report to you. >> I tried the patch with 2.0.19 and the dovecot error messages >> disappeared. > OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever > gets released). Presumably you mean 2.1.4 (since 2.1.3 is already released)? -- Best regards, Charles From tss at iki.fi Wed Mar 21 13:57:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:57:45 +0200 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: On 20.3.2012, at 20.29, Mark Jeghers wrote: > Below is my config. When I run dovecot from xinetd, I get these errors in the log: You can't run Dovecot v2.x via inetd. You could run it via systemd though. From tss at iki.fi Wed Mar 21 13:58:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:58:09 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <4F69C1A7.2040601@Media-Brokers.com> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> <4F69C1A7.2040601@Media-Brokers.com> Message-ID: <38A53BE8-A53F-4906-996F-6CC863E537CC@iki.fi> On 21.3.2012, at 13.55, Charles Marcus wrote: > On 2012-03-21 7:48 AM, Timo Sirainen wrote: >> On 19.3.2012, at 21.16, Alex Ha wrote: >>>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>>> 7542 of existing connection > >>>>> Oh, right, PIDs of course aren't unique when you're using >>>>> mulitiple servers. Try if the attached patch fixes your >>>>> troubles. If it does, I'll commit it to hg. > >>>> Thanks Timo! I will try the patch and report to you. > >>> I tried the patch with 2.0.19 and the dovecot error messages >>> disappeared. > >> OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever >> gets released). > > Presumably you mean 2.1.4 (since 2.1.3 is already released)? Ah, yes. :) From tss at iki.fi Wed Mar 21 13:59:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:59:50 +0200 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <4F68A189.2010800@pacific.net> References: <4F68A189.2010800@pacific.net> Message-ID: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> On 20.3.2012, at 17.26, Ken A wrote: > With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? > > Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. From tss at iki.fi Wed Mar 21 14:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:06:53 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 In-Reply-To: <4F69989A.3000106@noa.gr> References: <4F69989A.3000106@noa.gr> Message-ID: On 21.3.2012, at 11.00, Nikolaos Milas wrote: > Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) > Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) > > I didn't see such warnings in 2.0.13. > > I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? Hmm. Yes, if dovecot-usrdb-ldap.conf is a separate file from dovecot-passdb-ldap.conf you can just remove it. But this reminds me that in several places I've suggested to make one of them a symlink to the other, and you can't really do it then. Perhaps I'll need to remove this warning, or maybe make it recognize the symlink case. Anyway I added it for both LDAP and SQL hoping that it would reduce questions like: "I changed user_attrs, but it doesn't do anything!" From tss at iki.fi Wed Mar 21 14:26:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:26:05 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: On 20.3.2012, at 16.55, Patrick Domack wrote: > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From patrickdk at patrickdk.com Wed Mar 21 14:47:56 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 08:47:56 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321084756.Horde.xxWqdZLnE6FPac38iyLGWYA@kishi.patrickdk.com> Thanks, applied it to 2.1.3 and going to test. You didn't even give me enough time to look at the source myself to find the issue. Quoting Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > >> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have >> double S and W tags. >> >> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >> >> This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From noel.butler at ausics.net Wed Mar 21 15:26:19 2012 From: noel.butler at ausics.net (Noel Butler) Date: Wed, 21 Mar 2012 23:26:19 +1000 Subject: [Dovecot] sysconfdir depreacted Message-ID: <1332336379.10474.5.camel@tardis> The purpose of any build scripts --sysconfdir is to tell the configuration to build in a path for its binaries configuration file(s). Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, is this a bug? if not, then I see no point of sysconfdir any more and it should be removed, if dovecot deliberately ignores what it is told to use. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Wed Mar 21 15:46:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 15:46:44 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332336379.10474.5.camel@tardis> References: <1332336379.10474.5.camel@tardis> Message-ID: On 21.3.2012, at 15.26, Noel Butler wrote: > The purpose of any build scripts --sysconfdir is to tell the > configuration to build in a path for its binaries configuration file(s). > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > is this a bug? if not, then I see no point of sysconfdir any more and it > should be removed, if dovecot deliberately ignores what it is told to > use. --sysconfdir=/etc uses /etc/dovecot/ --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ There is now always the dovecot/ suffix, but the the /etc part is still configurable. From jtl+dovecot at uvm.edu Wed Mar 21 15:53:50 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 09:53:50 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) Message-ID: <4F69DD6E.1090502@uvm.edu> Had a user who couldn't access his INBOX: > Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file > mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai > led: (offset < (uint16_t)-1) > Mar 21 09:21:17 penguina dovecot: imap([USER]): Error: Raw backtrace: > /usr/lib/dovecot/libdovecot.so.0 [0x342683c660] -> /usr > /lib/dovecot/libdovecot.so.0 [0x342683c6b6] -> > /usr/lib/dovecot/libdovecot.so.0 [0x342683bb73] -> > /usr/lib/dovecot/libdovecot > -storage.so.0 [0x3426c966a8] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_ext_intro+0x240) > [0x3426c979c0] -> / > usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_record+0x401) > [0x3426c99151] -> /usr/lib/dovecot/libdovecot-storage.s > o.0(mail_index_sync_map+0x245) [0x3426c99c55] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_map+0x71b) > [0x3426c8afbb > ] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c85d8b] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_open+0x1c > e) [0x3426c8617e] -> > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_mailbox_open+0xb5) > [0x3426c4d865] -> /usr/lib/dov > ecot/libdovecot-storage.so.0 [0x3426c75eab] -> > /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c31006] -> > dovecot/imap [hdtod > d 10.245.30.58 SELECT](cmd_ Stack trace made it look like it was the INBOX, so I deleted the index files for his INBOX and everything was OK. doveconf -n: > # OS: Linux 2.6.18-274.18.1.el5 x86_64 Red Hat Enterprise Linux Server > release 5.8 (Tikanga) > auth_gssapi_hostname = penguina.uvm.edu > auth_krb5_keytab = /etc/krb5.keytab.dovecot > auth_master_user_separator = * > auth_mechanisms = plain login gssapi > base_dir = /var/run/dovecot/ > default_process_limit = 250 > first_valid_uid = 50 > lock_method = flock > login_trusted_networks = [REDACTED] > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%1u/%1.1u/%u > mail_max_lock_timeout = 30 secs > mail_max_userip_connections = 100 > mbox_read_locks = flock > mbox_write_locks = flock > mmap_disable = yes > namespace { > inbox = yes > location = > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > args = /etc/dovecot/passwd.masterusers > driver = passwd-file > master = yes > } > passdb { > driver = pam > } > service imap { > process_limit = 4096 > } > service lmtp { > client_limit = 1 > inet_listener lmtp { > port = 24 > } > } > ssl_cert = <[REDACTED] > ssl_key = < [REDACTED] > userdb { > driver = passwd > } > verbose_proctitle = yes Any questions/suggestions welcome. Jim From tss at iki.fi Wed Mar 21 16:02:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 16:02:47 +0200 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <4F69DD6E.1090502@uvm.edu> References: <4F69DD6E.1090502@uvm.edu> Message-ID: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> On 21.3.2012, at 15.53, Jim Lawson wrote: > Had a user who couldn't access his INBOX: > >> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >> led: (offset < (uint16_t)-1) I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. > Stack trace made it look like it was the INBOX, so I deleted the index > files for his INBOX and everything was OK. If it happens again, get a copy of the indexes. From CMarcus at Media-Brokers.com Wed Mar 21 16:26:29 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 10:26:29 -0400 Subject: [Dovecot] squat not working in 2.1 In-Reply-To: <20120229143038.GX13045@charite.de> References: <6A93411B-4058-4A7D-9F94-452403AE83ED@iki.fi> <4F4DF7F7.8020405@in.tum.de> <20120229100957.GX13045@charite.de> <20120229102250.GY13045@charite.de> <20120229132718.GN13045@charite.de> <92D75C5F-46E8-4EE4-B43D-60A3261E071C@iki.fi> <46a03b5105c847df7f7491f0889ef7ec@imt-systems.com> <20120229135851.GU13045@charite.de> <1be342370509d17ae81682aede00f016@imt-systems.com> <5febb8861c0cc824b0446cb2fec98d19@imt-systems.com> <20120229143038.GX13045@charite.de> Message-ID: <4F69E515.9080904@Media-Brokers.com> On 2012-02-29 9:30 AM, Ralf Hildebrandt wrote: > * Morten Stevens: > >> This is a Fedora-specific problem, because clucene (build >> requirement) is not correctly packaged. > > Well, debian showed the same packaging (wrong place). I just attempted to update to 2.1.3 on gentoo and received the same error: /usr/include/CLucene/SharedHeader.h:18:36: fatal error: CLucene/clucene-config.h: No such file or directory So, is this also a packaging error that I need to report to gentoo? -- Best regards, Charles From Ralf.Hildebrandt at charite.de Wed Mar 21 16:33:48 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 15:33:48 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321143348.GR2789@charite.de> * Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > > > This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 That doesn't seem to work: Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename has wrong S value, renamed the file from /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S to /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) failed: Input/output error (uid=5270) It's renaming itself to itself again? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From luca at lm-net.it Wed Mar 21 16:43:14 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Wed, 21 Mar 2012 15:43:14 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup Message-ID: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Hello list. I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. It would be desirable for multiple reasons: - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) - be ready to serve traffic over IPv6 without touching our precious mailbox servers - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). I'd like to know if that plan sounds reasonable or if there's something stupid in it. Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? thanks, -- Luca Lesinigo From tss at iki.fi Wed Mar 21 17:00:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:00:26 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120321143348.GR2789@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: On 21.3.2012, at 16.33, Ralf Hildebrandt wrote: >>> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. >>> >>> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >>> >>> This is happening for all folder moves. >> >> Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 > > That doesn't seem to work: It fixed only the duplicate S= and W= values. > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename > has wrong S value, renamed the file from > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > to > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) > failed: Input/output error (uid=5270) > > It's renaming itself to itself again? Hmm. Yeah, this is a bit problematic for compressed mails. If the S=size isn't correct, Dovecot fixes it by stat()ing the file and using it as the size. And that's of course wrong. Also Dovecot can't simply remove the S=size, because the current Maildir code assumes that it always exists for compressed mails. There's no easy and efficient way to fix this.. Maybe you could just manually rename the files to have correct S=size? :) zcat file | wc should give the right size. From tss at iki.fi Wed Mar 21 17:17:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:17:56 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F684938.9000208@unict.it> References: <4F684938.9000208@unict.it> Message-ID: Hi, On 20.3.2012, at 11.09, Luca Palazzo wrote: > Hi Timo, hi all, > after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > > 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > 710 { > (gdb) bt > #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 526 bytes Desc: not available URL: From ka at pacific.net Wed Mar 21 17:19:16 2012 From: ka at pacific.net (Ken Anderson) Date: Wed, 21 Mar 2012 10:19:16 -0500 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> References: <4F68A189.2010800@pacific.net> <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> Message-ID: <4F69F174.9000501@pacific.net> On 3/21/2012 6:59 AM, Timo Sirainen wrote: > On 20.3.2012, at 17.26, Ken A wrote: > >> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? >> >> Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? > > Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. > Awesome! I haven't migrated to mdbox yet, but in testing with it on a dev server, it looks like it will solve a huge problem. Users seem to want ever larger mailboxes, and mdbox gives them that, without asking more than additional disk space. Fixing the pop locking would be an additional benefit! Thanks, Ken Pacific.Net From tss at iki.fi Wed Mar 21 17:21:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:21:43 +0200 Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 In-Reply-To: <.120.61.90.33.1332228956.squirrel@24x7server.net> References: <.120.61.90.33.1332228956.squirrel@24x7server.net> Message-ID: <7D494B74-E138-415F-8010-F1208604E246@iki.fi> On 20.3.2012, at 9.35, Rajesh M wrote: > i migrated my email server with around 5000 users from dovecot version 1.2 > to version 2 > > i have two separate 2 tb hdd's storing webmail data of these users. You mean you simply upgraded the Dovecot version, the server is exactly the same? > the load on the server goes very high over 100 during peak load times and > the imap connections get dropped frequently, webmail becomes very slow. There shouldn't be much performance difference between v1.2 and v2.x. > in the dovecot log file i get errors as such > > Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: > Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 > expunge attempts) This simply means that the disk IO usage is very high. > i am a bit confused as to what settings are to be done for a very busy server Show dovecot -n output of the new server, and if you have the old configuration available that could be helpful also to compare their differences. From tss at iki.fi Wed Mar 21 17:25:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:25:46 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: On 19.3.2012, at 12.50, Matteo Cazzador wrote: > Hi, i've a simple question, what do you mean for dovecot director setup? > 'i've a doubt. > The solution that i'm testing is using 3 mail server in different > geoghrapic locations. > An user can travel in varius location, and i want his imap mail reside > on mail server in every locations. > S? i use you solution about replication. First server (by dns record) > that receive mail sync it on the other servers, and when > user consult is mail by imap protocol everything is sync on all servers. > Do you suggest to use a horizontal structure for it like i explain or > is better to have a single node external mail server > and customer locations server like slave? Dovecot director isn't really meant to be used for geographic user distribution. Also the replication doesn't yet support more than two servers. A master-slave setup wouldn't have the UID conflict problems that multi-master dsync replication has, but the UID conflicts probably won't be a big problem. Anyway, difficult to give recommendations about an unfinished feature.. From lukas.mueller at newmedia.ch Wed Mar 21 17:45:09 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Wed, 21 Mar 2012 15:45:09 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS Message-ID: Hi, I'm stuck with a problem we have with dovecot. My suspicion is, that it has to do with accessing the same mailbox/mail stored on a NFS-share from two machines at the same time. setup We have to mail servers running, both run a Ubuntu 10.04, Postfix 2.70 and Dovecot 1.2.9. The mailboxes are stored in maildir format on a NFS-Share. In front of those to mail servers we have a load balancer. Unfortunately it can't be set up to use the same server for each domain, but it uses the same server for the same source-ip for at least 1 hour. Here is the output of dovecot -n: # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-33-server x86_64 Ubuntu 10.04.3 LTS nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/ca-bundle/SSL123_CA_Bundle.pem ssl_cert_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.crt ssl_key_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.key ssl_verify_client_cert: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_max_userip_connections: 25 mail_privileged_group: mail mail_location: maildir:/data/vmail/%d/%n:INDEX=/data/vmail/%d/%n/indexes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): auth default: passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf plugin: quota: maildir:storage=409600 sieve_global_path: /data/vmail/globalsieverc dict: quotadict: mysql:/etc/dovecot-dict-quota.conf problem the problem happens with a client's mailbox that is used by multiple users. >From time to time he cannot see any Emails in the mailbox, neither with his mail clients (Apple Mail) nor with in the webmail (Roundcube). Around this time I get the following entries in the log files: Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1524, file=1327500903.V15I5722c8M210039.mail01:2,Se) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f37602b8b5a] -> imap(+0xaebc7) [0x7f37602b8bc7] -> imap(+0xae238) [0x7f37602b8238] -> imap(+0x497d7) [0x7f37602537d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f37602545c2] -> imap(+0x4bb06) [0x7f3760255b06] -> imap(maildir_uidlist_sync_init+0x4d) [0x7f376025652d] -> imap(+0x46ed4) [0x7f3760250ed4] -> imap(maildir_storage_sync_init+0x147) [0x7f3760251557] -> imap(imap_sync_init+0x70) [0x7f376023b190] -> imap(+0x2411e) [0x7f376022e11e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f37602c069c] -> imap(io_loop_handler_run+0x60) [0x7f37602c1000] -> imap(io_loop_run+0x18) [0x7f37602c0448] -> imap(main+0x58e) [0x7f376023bc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f375f877c4d] -> imap(+0x21979) [0x7f376022b979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 16934 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7ff81b415b5a] -> imap(+0xaebc7) [0x7ff81b415bc7] -> imap(+0xae238) [0x7ff81b415238] -> imap(+0x497d7) [0x7ff81b3b07d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7ff81b3b15c2] -> imap(maildir_uidlist_sync_init+0x105) [0x7ff81b3b35e5] -> imap(+0x46ed4) [0x7ff81b3aded4] -> imap(maildir_storage_sync_force+0x52) [0x7ff81b3ae392] -> imap(maildir_file_do+0x99) [0x7ff81b3b3cb9] -> imap(+0x4d944) [0x7ff81b3b4944] -> imap(index_mail_set_seq+0x148) [0x7ff81b3c8ed8] -> imap(index_storage_search_next_nonblock+0x162) [0x7ff81b3cd622] -> imap(mailbox_search_next_nonblock+0x20) [0x7ff81b3db2c0] -> imap(mailbox_search_next+0x26) [0x7ff81b3db316] -> imap(imap_fetch_more+0x2bf) [0x7ff81b39295f] -> imap(cmd_fetch+0x36c) [0x7ff81b38a9ec] -> imap(+0x28fad) [0x7ff81b38ffad] -> imap(+0x2908d) [0x7ff81b39008d] -> imap(client_handle_input+0x135) [0x7ff81b3902c5] -> imap(client_input+0x5f) [0x7ff81b390baf] -> imap(io_loop_handler_run+0xbd) [0x7ff81b41e05d] -> imap(io_loop_run+0x18) [0x7ff81b41d448] -> Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): imap(main+0x58e) [0x7ff81b398c5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ff81a9d4c4d] -> imap(+0x21979) [0x7ff81b388979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 13712 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f4a311fcb5a] -> imap(+0xaebc7) [0x7f4a311fcbc7] -> imap(+0xae238) [0x7f4a311fc238] -> imap(+0x497d7) [0x7f4a311977d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f4a311985c2] -> imap(+0x47023) [0x7f4a31195023] -> imap(maildir_storage_sync_init+0x147) [0x7f4a31195557] -> imap(imap_sync_init+0x70) [0x7f4a3117f190] -> imap(+0x2411e) [0x7f4a3117211e] -> imap(+0x64c0e) [0x7f4a311b2c0e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f4a3120469c] -> imap(io_loop_handler_run+0x60) [0x7f4a31205000] -> imap(io_loop_run+0x18) [0x7f4a31204448] -> imap(main+0x58e) [0x7f4a3117fc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f4a307bbc4d] -> imap(+0x21979) [0x7f4a3116f979] Mar 6 08:42:31 mail01 dovecot: dovecot: child 24257 (imap) killed with signal 6 (core dumps disabled) Note: the first part is on a mail server, while the smaller second part is on the other one. Unfortunately I'm not able to reproduce this error. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. I have no idea how to solve this problem and any help is greatly appreciated. If you need further information, please say so. Mit freundlichen Gr?ssen Lukas M?ller Systems Engineer _______________________________________________ NEWMEDIA S?dostschweiz Newmedia AG Kasernenstrasse 1 Postfach 508, CH-7007 Chur http://www.newmedia.ch _______________________________________________ TYPO3 & Drupal - Wir wissen wie. Ihre professionelle Web Agentur in Chur, Ilanz, Glarus und Z?rich. From tss at iki.fi Wed Mar 21 17:50:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:50:29 +0200 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: Message-ID: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> On 21.3.2012, at 17.45, M?ller Lukas wrote: > Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. > My suspicion/speculation what happens is the following: > Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. > Somehow this leads to problems with Locks on NFS, which leads to the crash. Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > I have no idea how to solve this problem and any help is greatly appreciated. The only way to fully fix this is: http://wiki2.dovecot.org/Director From Ralf.Hildebrandt at charite.de Wed Mar 21 17:52:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 16:52:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: <20120321155245.GS2789@charite.de> * Timo Sirainen : > > It's renaming itself to itself again? > > Hmm. Yeah, this is a bit problematic for compressed mails. If the > S=size isn't correct, Dovecot fixes it by stat()ing the file and using > it as the size. And that's of course wrong. Also Dovecot can't simply > remove the S=size, because the current Maildir code assumes that it > always exists for compressed mails. There's no easy and efficient way > to fix this.. Maybe you could just manually rename the files to have > correct S=size? :) zcat file | wc should give the right size. Right now the whole system is down because nobody can acces his/her mails due to this. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From fxmulder at gmail.com Wed Mar 21 17:56:12 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 09:56:12 -0600 Subject: [Dovecot] distributed mdbox Message-ID: Anyone know how to setup dovecot with mdbox so that it can be used through shared storage from multiple hosts? I've setup a gluster volume and am sharing it between 2 test clients. I'm using postfix/dovecot LDA for delivery and I'm using postal to send mail between 40 users. In doing this, I'm seeing these errors in the logs Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count 272 -> 271 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=3768 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but next_uid = 517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=4220 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=5088 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index This is my dovecot config currently: jdevine at test-gluster-client2:~> dovecot -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 lock_method = dotlock mail_fsync = always mail_location = mdbox:~/mdbox mail_nfs_index = yes mail_nfs_storage = yes mmap_disable = yes passdb { driver = pam } protocols = " imap" ssl_cert = References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> Message-ID: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > * Timo Sirainen : > >>> It's renaming itself to itself again? >> >> Hmm. Yeah, this is a bit problematic for compressed mails. If the >> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >> it as the size. And that's of course wrong. Also Dovecot can't simply >> remove the S=size, because the current Maildir code assumes that it >> always exists for compressed mails. There's no easy and efficient way >> to fix this.. Maybe you could just manually rename the files to have >> correct S=size? :) zcat file | wc should give the right size. > > Right now the whole system is down because nobody can acces his/her > mails due to this. All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 385 bytes Desc: not available URL: From luca.palazzo at unict.it Wed Mar 21 18:04:00 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Wed, 21 Mar 2012 17:04:00 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: References: <4F684938.9000208@unict.it> Message-ID: <4F69FBF0.6090003@unict.it> It worked. We have no more sigsegv on *-login process. Thanks Luca Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > Hi, > > On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> Hi Timo, hi all, >> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >> >> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> 710 { >> (gdb) bt >> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > > Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > From tss at iki.fi Wed Mar 21 18:05:52 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:05:52 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> On 21.3.2012, at 17.56, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs Dovecot assumes that the filesystem behaves the same way as regular local filesystems. > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 Looks like gluster doesn't fit that assumption. So, the solution is the same as with NFS: http://wiki2.dovecot.org/Director From tss at iki.fi Wed Mar 21 18:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:08:07 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F69FBF0.6090003@unict.it> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> Message-ID: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e I'll figure out a proper fix soon. On 21.3.2012, at 18.04, Luca Palazzo wrote: > It worked. We have no more sigsegv on *-login process. > > Thanks > > Luca > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >> Hi, >> >> On 20.3.2012, at 11.09, Luca Palazzo wrote: >> >>> Hi Timo, hi all, >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>> >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> 710 { >>> (gdb) bt >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >> >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >> > From fxmulder at gmail.com Wed Mar 21 18:25:14 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:25:14 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > On 21.3.2012, at 17.56, James Devine wrote: > > > Anyone know how to setup dovecot with mdbox so that it can be used > through > > shared storage from multiple hosts? I've setup a gluster volume and am > > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > > delivery and I'm using postal to send mail between 40 users. In doing > > this, I'm seeing these errors in the logs > > Dovecot assumes that the filesystem behaves the same way as regular local > filesystems. > > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: > Fixed > > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: > messages_count > > 272 -> 271 > > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > > synchronization error at seq=4,offset=3768 for > > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > > next_uid = 517 > > Looks like gluster doesn't fit that assumption. So, the solution is the > same as with NFS: http://wiki2.dovecot.org/Director > > What filesystem mechanisms might not be working in this case? From fxmulder at gmail.com Wed Mar 21 18:47:53 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:47:53 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: Also I don't seem to get these errors with a single dovecot machine using the shared storage and it looks like there are multiple simultaneous delivery processes running On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > > > On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > >> On 21.3.2012, at 17.56, James Devine wrote: >> >> > Anyone know how to setup dovecot with mdbox so that it can be used >> through >> > shared storage from multiple hosts? I've setup a gluster volume and am >> > sharing it between 2 test clients. I'm using postfix/dovecot LDA for >> > delivery and I'm using postal to send mail between 40 users. In doing >> > this, I'm seeing these errors in the logs >> >> Dovecot assumes that the filesystem behaves the same way as regular local >> filesystems. >> >> > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >> Fixed >> > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >> messages_count >> > 272 -> 271 >> > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >> Log >> > synchronization error at seq=4,offset=3768 for >> > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >> but >> > next_uid = 517 >> >> Looks like gluster doesn't fit that assumption. So, the solution is the >> same as with NFS: http://wiki2.dovecot.org/Director >> >> > What filesystem mechanisms might not be working in this case? > From tss at iki.fi Wed Mar 21 19:04:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 19:04:36 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. On 21.3.2012, at 18.47, James Devine wrote: > Also I don't seem to get these errors with a single dovecot machine using > the shared storage and it looks like there are multiple simultaneous > delivery processes running > > On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > >> >> >> On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: >> >>> On 21.3.2012, at 17.56, James Devine wrote: >>> >>>> Anyone know how to setup dovecot with mdbox so that it can be used >>> through >>>> shared storage from multiple hosts? I've setup a gluster volume and am >>>> sharing it between 2 test clients. I'm using postfix/dovecot LDA for >>>> delivery and I'm using postal to send mail between 40 users. In doing >>>> this, I'm seeing these errors in the logs >>> >>> Dovecot assumes that the filesystem behaves the same way as regular local >>> filesystems. >>> >>>> Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >>> Fixed >>>> index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >>> messages_count >>>> 272 -> 271 >>>> Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >>> Log >>>> synchronization error at seq=4,offset=3768 for >>>> /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >>> but >>>> next_uid = 517 >>> >>> Looks like gluster doesn't fit that assumption. So, the solution is the >>> same as with NFS: http://wiki2.dovecot.org/Director >>> >>> >> What filesystem mechanisms might not be working in this case? >> From jtl+dovecot at uvm.edu Wed Mar 21 20:19:19 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 14:19:19 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> References: <4F69DD6E.1090502@uvm.edu> <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> Message-ID: <4F6A1BA7.7030208@uvm.edu> On 3/21/12 10:02 AM, Timo Sirainen wrote: > On 21.3.2012, at 15.53, Jim Lawson wrote: > >> Had a user who couldn't access his INBOX: >> >>> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >>> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >>> led: (offset < (uint16_t)-1) > I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. This applies cleanly against 2.0.19; should I try it on that version, or not recommended? >> Stack trace made it look like it was the INBOX, so I deleted the index >> files for his INBOX and everything was OK. > If it happens again, get a copy of the indexes. > I sent them, encrypted, to your email address/GPG key 0x40558AC9. Jim From mjeghers at Brocade.com Wed Mar 21 20:59:39 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Wed, 21 Mar 2012 11:59:39 -0700 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <4F699443.1090704@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> All, I was mistaken in how I described my problem, please forgive this dovecot newbie for describing the problem incorrectly! It is not under xinitd, it is trying to run as an init.d service. Ok, let's try again... I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? Thanks, hope this clarifies things, /Mark -----Original Message----- From: Stan Hoeppner [mailto:stan at hardwarefreak.com] Sent: Wednesday, March 21, 2012 1:42 AM To: Mark Jeghers Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From tss at iki.fi Wed Mar 21 21:16:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 21:16:51 +0200 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> Message-ID: On 21.3.2012, at 20.59, Mark Jeghers wrote: > I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? .. >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Permission errors point to SELinux being the problem. Try disabling it. From bear at rwhartzell.net Wed Mar 21 22:54:09 2012 From: bear at rwhartzell.net (Robert Hartzell) Date: Wed, 21 Mar 2012 13:54:09 -0700 Subject: [Dovecot] Dovecot 2.1.3 on solaris with mysql - make fails Message-ID: <95DD93BE-F841-4BEB-A96C-059FFF0ACF2F@rwhartzell.net> I'm trying to build 2.1.3 on solaris 11 11/11 with gcc 4.5.2 & sun studio 12.2 & 12.3 CPPFLAGS="-I/opt/openssl/include -I/usr/mysql/include/mysql" \ LDFLAGS="-L/opt/openssl/lib -L/usr/mysql/lib/mysql -R/opt/openssl/lib:/usr/mysql/lib/mysql" \ ./configure --prefix=/opt/dovecot \ --sysconfdir=/etc/opt \ --with-ssl=openssl \ --with-mysql make fails with both solaris standard openssl and my build of openssl. I'm also getting the same error using sunstudio mysql version is 5.1.37 The relevant output of make is on pastebin http://pastebin.com/aALHG0yL I have seen some reference to this with google but nothing thats very recent and no solutions. Anyone know how to get past this? Any tips on building dovecot on solaris? Pointers would be much appreciated. -- Robert From patrickdk at patrickdk.com Wed Mar 21 23:52:39 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 17:52:39 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120321175239.Horde.6cigCZLnE6FPak2nibEXzWA@kishi.patrickdk.com> Quoting Timo Sirainen : > On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > >> * Timo Sirainen : >> >>>> It's renaming itself to itself again? >>> >>> Hmm. Yeah, this is a bit problematic for compressed mails. If the >>> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >>> it as the size. And that's of course wrong. Also Dovecot can't simply >>> remove the S=size, because the current Maildir code assumes that it >>> always exists for compressed mails. There's no easy and efficient way >>> to fix this.. Maybe you could just manually rename the files to have >>> correct S=size? :) zcat file | wc should give the right size. >> >> Right now the whole system is down because nobody can acces his/her >> mails due to this. > > All of your mails are compressed and have wrong S=size in the > filename? You can disable the check with the attached patch, but I'm > not sure if there are other places where it fails. At least quota > calculations won't be correct. The issue only started happening since I upgraded to 2.1.1, it didn't exist before then, I have check my system, and files before the date of upgrade are fine, only files/emails moved after upgrading to 2.1.1 have lost the S= value. I have made something that can pretty easily fix the issue, but it only stays fixed till another email gets moved and looses it's S= value. Sorry, I haven't had time to test out 2.1.3 yet. This will print out the commands needed to fix the files though. find . -name '*hostname:*' -exec 'gzip' '-l' '{}' ';' | awk '/hostname/ {for(x=4;x References: <1332336379.10474.5.camel@tardis> Message-ID: <1332381356.4112.9.camel@tardis> On Wed, 2012-03-21 at 15:46 +0200, Timo Sirainen wrote: > On 21.3.2012, at 15.26, Noel Butler wrote: > > > The purpose of any build scripts --sysconfdir is to tell the > > configuration to build in a path for its binaries configuration file(s). > > > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > > is this a bug? if not, then I see no point of sysconfdir any more and it > > should be removed, if dovecot deliberately ignores what it is told to > > use. > > > --sysconfdir=/etc uses /etc/dovecot/ > > --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ > > There is now always the dovecot/ suffix, but the the /etc part is still configurable. > perhaps it should be renamed then, given it violates the known normal for SYSCONF dir, you've just created another form of --datadir from gnu.org: "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) "datadir" The directory for installing idiosyncratic read-only architecture-independent data files for this program. This is usually the same place as ?datarootdir?, but we use the two separate variables so that you can move these program-specific files without altering the location for Info files, man pages, etc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From gedalya at gedalya.net Thu Mar 22 04:46:20 2012 From: gedalya at gedalya.net (Gedalya) Date: Wed, 21 Mar 2012 22:46:20 -0400 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A8BAC.4000002@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> Message-ID: <4F6A927C.6010003@gedalya.net> On 3/21/2012 10:17 PM, Martin Schitter wrote: > Am 16.3.2011 20:59, schrieb Gedalya: >>> >>>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at >>>> example.com -o imapc_password=**** backup -u jedi at example.com -R >>>> imapc: >>> >> Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current >> versions are putting the body of the last message in "Sent Items" in >> place of every single email in INBOX. >> In other words, for every email that sits in INBOX in the source, I get >> a copy of the last email in "Sent Items" instead. >> This happens for every account I try to migrate. >> Very strange. I noticed this only now, and the last package I have left >> in the local apt cache which still works is 2.1.rc7-0~auto+0. > > i see the same regression (2.1.3-0~auto+4) :( > > doveadm sync/backup via impac puts the same message all over the place... Thanks Martin, I've set up a test platform to investigate this further but I've been short on time... From stan at hardwarefreak.com Thu Mar 22 06:11:19 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 23:11:19 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> Message-ID: <4F6AA667.1080908@hardwarefreak.com> On 3/21/2012 12:04 PM, Timo Sirainen wrote: > The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. > > With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could use a true cluster file system such as OCFS or GFS. Both will eliminate this problem, and without requiring Dovecot director. And you'll get better performance than with Gluster, which, BTW, isn't really suitable as a transactional filesystem, was not designed for such a use case. -- Stan From ruskie at codemages.net Thu Mar 22 08:28:40 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Thu, 22 Mar 2012 07:28:40 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332381356.4112.9.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: :2012-03-22T11:55:Noel Butler: > perhaps it should be renamed then, given it violates the known normal > for SYSCONF dir, you've just created another form of --datadir Not really. The way I see it works as expected. The sysconf dir is the root of the configuration dir. Then if the app so chooses uses it's own directory structure under that. Considering that by default dovecot uses dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Quis custodiet ipsos custodes? From noel.butler at ausics.net Thu Mar 22 10:30:27 2012 From: noel.butler at ausics.net (Noel Butler) Date: Thu, 22 Mar 2012 18:30:27 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: <1332405027.6792.7.camel@tardis> On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > :2012-03-22T11:55:Noel Butler: > > > perhaps it should be renamed then, given it violates the known normal > > for SYSCONF dir, you've just created another form of --datadir > > Not really. The way I see it works as expected. The sysconf dir is the Then you and I and a few other devs involved in other very well known bits of software that everyone likely uses, will have to agree to disagree "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) > root of the configuration dir. Then if the app so chooses uses it's own > directory structure under that. Considering that by default dovecot uses > dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. > By default as of only 2.something, not in 0.x not in 1.0.x not in 1.1.x and not in 1.2.x I've said all I'm going to say on the mater, I got three emails offlist from others here agreeing with me, shame they didn't do it on-list, but I respect their right to remain silent so as not to endure the wrath of Timo and certain other cretins well known for having nothing else better to do. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From Ralf.Hildebrandt at charite.de Thu Mar 22 10:46:10 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 22 Mar 2012 09:46:10 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120322084609.GF28323@charite.de> * Timo Sirainen : > > Right now the whole system is down because nobody can acces his/her > > mails due to this. > > All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. That patch totally saved my ass. I rolled it out today and the Mar 22 09:33:00 postamt dovecot: imap(stoffelm): Error: Maildir filename has wrong S value, renamed the file from /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd to /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd errors subsided. At the same time the users CAN access the affected folder. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From amateo at um.es Thu Mar 22 11:55:58 2012 From: amateo at um.es (Angel L. Mateo) Date: Thu, 22 Mar 2012 10:55:58 +0100 Subject: [Dovecot] dovecot-auth restaring and caching Message-ID: <4F6AF72E.9030206@um.es> Hello, I'm trying to configure dovecot (2.0.13) to cache user and pass dbs. This a mail server whose purpose is only to deliver messages through dovecot lda. My users are in a ldap server. So I have configure auth_cache_size (with 20MB) and auth_cache_ttl (with 1 day). I have checked that caching is being done, and it is. If a send a message to a user, dovecot looks for it in my ldap server. If then I send another, then it uses cache information. The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. This is the auth log for the first message: Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:29:41 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:29:41 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXX Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXXX#011gid=XXX#011home=XXXXXXXXXXXX And this is the second one, just after a few minutes: Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:41:03 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:41:03 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXXX Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXX#011gid=XXX#011home=XXXXXXXXXX This is my configuration: root at lynx10:/etc/dovecot/conf.d# doveconf -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-4-amd64 x86_64 Ubuntu 10.04.4 LTS auth_cache_size = 20 M auth_cache_ttl = 1 days auth_debug = yes auth_verbose = yes hostname = lynx10 passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +imapflags sieve_max_redirects = 15 } postmaster_address = postmaster at um.es protocols = " imap lmtp pop3" service auth { unix_listener auth-userdb { mode = 0666 } } ssl_cert = References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> Message-ID: <4F6A8BAC.4000002@mur.at> Am 16.3.2011 20:59, schrieb Gedalya: >> >>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current > versions are putting the body of the last message in "Sent Items" in > place of every single email in INBOX. > In other words, for every email that sits in INBOX in the source, I get > a copy of the last email in "Sent Items" instead. > This happens for every account I try to migrate. > Very strange. I noticed this only now, and the last package I have left > in the local apt cache which still works is 2.1.rc7-0~auto+0. i see the same regression (2.1.3-0~auto+4) :( doveadm sync/backup via impac puts the same message all over the place... From tss at iki.fi Thu Mar 22 16:05:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 16:05:34 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> Message-ID: <1332425134.26095.88.camel@innu> Hi, These should fix it properly: http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: > The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e > > I'll figure out a proper fix soon. > > On 21.3.2012, at 18.04, Luca Palazzo wrote: > > > It worked. We have no more sigsegv on *-login process. > > > > Thanks > > > > Luca > > > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > >> Hi, > >> > >> On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> > >>> Hi Timo, hi all, > >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > >>> > >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> 710 { > >>> (gdb) bt > >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > >> > >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > >> > > > From luca.palazzo at unict.it Thu Mar 22 16:16:34 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Thu, 22 Mar 2012 15:16:34 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <1332425134.26095.88.camel@innu> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> <1332425134.26095.88.camel@innu> Message-ID: <4F6B3442.8020000@unict.it> I've applied both and reverted previous one. Everything seems to run flowless. Thanks Luca On 03/22/2012 03:05 PM, Timo Sirainen wrote: > Hi, > > These should fix it properly: > > http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 > http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d > > On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: >> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e >> >> I'll figure out a proper fix soon. >> >> On 21.3.2012, at 18.04, Luca Palazzo wrote: >> >>> It worked. We have no more sigsegv on *-login process. >>> >>> Thanks >>> >>> Luca >>> >>> Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >>>> Hi, >>>> >>>> On 20.3.2012, at 11.09, Luca Palazzo wrote: >>>> >>>>> Hi Timo, hi all, >>>>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>>>> >>>>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> 710 { >>>>> (gdb) bt >>>>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>>>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>>>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >>>> >>>> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >>>> >>> >> > > From micah at riseup.net Thu Mar 22 16:38:55 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 22 Mar 2012 10:38:55 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply Message-ID: <87bonon1j4.fsf@algae.riseup.net> I've been moving users from one system to another by doing a dsync mirror operation. The first dsync mirror takes some time, because of the number of users involved, so I am doing an initial sync, and then I direct the users to the new location and do a final 'freshening' sync to get any changes that happened during the longer sync. The problem seems to be with this freshening sync, it seems quite fragile breaking with things like: dsync-local(user at example.com): Error: msg-get failed: box=Spam uid=3034 guid=1ad456015ae9694f083b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69096 guid=c22b541a71e4694fc93700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69097 guid=4b6d6b13d0e9694f505700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69098 guid=175b1c2e4aea694fc97100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69099 guid=bfb08c1b3bee694f133e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69100 guid=fa5d630c17ef694fa75f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69101 guid=7ca96011dcef694f3f0400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69102 guid=ef547107eff1694ff96700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69103 guid=5597bc0519f2694f2e7000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69104 guid=8336a53a54f5694fb21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69105 guid=96169d13c8fd694f831800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69106 guid=af21a5183f036a4f263200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69107 guid=d0fde3348e036a4ff44000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69108 guid=4ce01d1a59056a4fee2200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69109 guid=497f96066e056a4f322700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69110 guid=ef34f505c0066a4fc26b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69111 guid=81adcb2c6e076a4f751100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69112 guid=a110841e8a076a4fa21500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69113 guid=60d8e70a970d6a4fae2100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69114 guid=7c6cb41572106a4ff13c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69115 guid=aaf4d32b2f126a4ff21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69116 guid=ab52f43a58126a4ffd1800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69117 guid=eb543a2179186a4fe45800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69118 guid=cd7cb408a12a6a4f272100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69119 guid=2ec02e2ef2326a4f9e1100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69120 guid=e7a4552ff8336a4f7f6700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69121 guid=0724b023d33a6a4f1b3300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69122 guid=9985c91afe3b6a4f127100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69123 guid=9300751b913d6a4f7a4000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69124 guid=822ff806ae3f6a4f293b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69125 guid=eac8ed1f2b426a4f164200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69126 guid=4109561ae3426a4ff26700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69127 guid=30bc832e5e496a4f563600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69128 guid=c0b36410bd4b6a4f102b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69129 guid=38a9d41a534d6a4ff40200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69130 guid=b8e84d239b4d6a4fd11000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69131 guid=f060ef22154f6a4f2b5c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69132 guid=e2999c107c4f6a4f5a7600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69133 guid=8d09280aae506a4f073500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69134 guid=43d7ec3aa6556a4f963a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69135 guid=c5800130d2556a4f594200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69136 guid=83a91e08b4566a4f197100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69137 guid=50cf9721f95e6a4f7e4400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69138 guid=fda2a82886606a4f881700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69139 guid=97ee1d1ad1636a4fc94d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69140 guid=4f50671f85666a4f306100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69141 guid=0fea590fb4666a4f7a6b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69142 guid=f3210b02a5676a4ffa1f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69143 guid=8d325a06686a6a4f2b3600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69144 guid=0cbf1839f1756a4f8f6800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69145 guid=d6209a2898796a4f671a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69146 guid=576ade31da7d6a4f5f5700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69147 guid=70a15b34247e6a4f445a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69148 guid=3ff92631cd886a4ffd6300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69149 guid=3ff2081568916a4f134d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69150 guid=d1a67b0907ab6a4f546000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69151 guid=3d4cb1197ee96a4fbf5f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69152 guid=aae2542818266b4f7d1e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2077 guid=c5a0930248e8694fb77c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2078 guid=dce63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2079 guid=dde63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2080 guid=dee63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2081 guid=dfe63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2082 guid=a110b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2083 guid=a210b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2084 guid=93afeb1f7d0b6a4f694e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2085 guid=ef2bb1098a2e6a4fcf1b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2086 guid=d8ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2087 guid=d9ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2088 guid=7e5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2089 guid=7f5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2090 guid=b08c5205ba646a4f106700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2091 guid=805b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2092 guid=815b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. those final "Unexpected finish reply" and "Unexpected reply from server" are a bit surprising results. Running dsync again seems to resolve things, but there seems to be a bug here that is causing unexpected results to leak through to the dsync process? thanks, micah From jtl+dovecot at uvm.edu Thu Mar 22 18:17:10 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 22 Mar 2012 12:17:10 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6AA667.1080908@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> Message-ID: <4F6B5086.4030001@uvm.edu> On 03/22/2012 12:11 AM, Stan Hoeppner wrote: > On 3/21/2012 12:04 PM, Timo Sirainen wrote: >> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >> >> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. > If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could > use a true cluster file system such as OCFS or GFS. Both will eliminate > this problem, and without requiring Dovecot director. And you'll get > better performance than with Gluster, which, BTW, isn't really suitable > as a transactional filesystem, was not designed for such a use case. Speaking as an admin who has run Dovecot on top of GFS both with and without the director, I would never go back to a cluster without the director. The cluster performs *so* much better when glocks can be cached on a single node, and this can't happen if a single user has IMAP processes on separate nodes. No, you don't strictly need the director if you have GFS, but if you can manage it, you'll be a lot happier. Jim From ms at mur.at Thu Mar 22 19:09:13 2012 From: ms at mur.at (Martin Schitter) Date: Thu, 22 Mar 2012 18:09:13 +0100 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A927C.6010003@gedalya.net> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> Message-ID: <4F6B5CB9.9080204@mur.at> Am 2012-03-22 03:46, schrieb Gedalya: >> >> doveadm sync/backup via impac puts the same message all over the place... > > Thanks Martin, I've set up a test platform to investigate this further > but I've been short on time... after some debugging a few more remarks about this problem: the bug only appears on recursive folder hierarchies. if you specity option "-m INBOX" everything works fine. for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( has anyone a clue how to fix this problem in the source code? From tss at iki.fi Thu Mar 22 20:57:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 20:57:21 +0200 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <4F6AF72E.9030206@um.es> References: <4F6AF72E.9030206@um.es> Message-ID: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> On 22.3.2012, at 11.55, Angel L. Mateo wrote: > The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. service auth { idle_kill = 0 } From ednitido at gmail.com Thu Mar 22 23:18:12 2012 From: ednitido at gmail.com (Ed Nitido) Date: Thu, 22 Mar 2012 17:18:12 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy Message-ID: Hey all, I've upgraded from a working Dovecot 2.0.17 Proxy with a master user setup to Dovecot 2.1.3 and I've merged my conf settings from 2.0.17 into 2.1.3. I'm able to start up dovecot proxy and telnet localhost, however it creates the users home director on the proxy server instead of going to the backend dovecot server (which has already been successfully upgraded to 2.1.3 from 2.0.17). In my old 2.0.17, I had the entire namespace section commented out in 10-mail.conf. To achieve the same in 2.1.3 I had to remove the file 15-mailboxes.conf, otherwise my error log would have the following if I just commented out the namespace section and left that file in place: *Error: user edward at dev.domain.com: Initialization failed: namespace configuration error: inbox=yes namespace missing* *Error: Invalid user settings. Refer to server log for more information.* So now I'm trying to figure out why the proxy doesn't get the mail server IP from director, when doveadm has the backend server listed *# doveadm director status* *mail server ip vhosts users* *192.168.12.205 100 0* Can anyone see something I've missed? Here's my doveconf output: # 2.1.3: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-openvz-amd64 i686 Debian 6.0.4 auth_debug = yes auth_debug_passwords = yes auth_master_user_separator = * auth_mechanisms = plain login auth_socket_path = /usr/local/var/run/dovecot/auth-master auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log director_doveadm_port = 542 director_mail_servers = 192.168.12.205 director_servers = 192.168.12.209 disable_plaintext_auth = no info_log_path = /var/log/dovecot-info.log listen = * lmtp_proxy = yes log_path = /var/log/dovecot-err.log mail_debug = yes mail_gid = vmail mail_location = maildir:%h/Maildir mail_plugins = " quota" mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/home/%d/shared-mailboxes quota = maildir:User quota quota_rule = *:storage=1G sieve = %h/.dovecot.sieve sieve_dir = ~/sieve sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M sieve_quota_max_scripts = 0 sieve_quota_max_storage = 0 } postmaster_address = postmaster at dev.domain.com protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0666 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 541 } inet_listener { port = 542 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { user = postfix } } service managesieve-login { executable = managesieve-login director inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service pop3-login { executable = pop3-login director inet_listener pop3 { port = 110 } } ssl_cert = Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root at n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M /home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root at n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real 0m2.677s user 0m3.184s sys 0m1.513s [root at n24 bu]# time dsync backup -u testuser at domain.com \ mdbox:/home/bu/testuser real 1m9.519s user 1m7.592s sys 0m1.126s [root at n24 bu]# time dsync backup -u testuser at domain.com \ sdbox:/home/bu/testuser2 real 1m2.164s user 1m0.882s sys 0m0.993s [root at n24 bu]# From list at airstreamcomm.net Fri Mar 23 04:36:44 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 22 Mar 2012 21:36:44 -0500 Subject: [Dovecot] Dovecot and scalable database storage Message-ID: I saw some interesting mails from TImo back in 2009 talking about the idea of using something like Cassandra db or similar as a storage platform for both email and index/logs. I was wondering if this has been discussed since then, and if there are any plans to support something like this in the future? I have been playing with Cassandra and found that their RackAwareStrategy gives you the ability to replicate writes to as many nodes as you would like, but more importantly what nodes and one of those nodes could be defined by what rack it lives in or what data center it lives in. This means multiple sites high available storage clusters, seemingly a system that dovecot could benefit from in terms of performance and redundancy and simplicity. Any takers? From stan at hardwarefreak.com Fri Mar 23 09:13:18 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 02:13:18 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6B5086.4030001@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> Message-ID: <4F6C228E.5060902@hardwarefreak.com> On 3/22/2012 11:17 AM, Jim Lawson wrote: > On 03/22/2012 12:11 AM, Stan Hoeppner wrote: >> On 3/21/2012 12:04 PM, Timo Sirainen wrote: >>> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >>> >>> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. >> If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could >> use a true cluster file system such as OCFS or GFS. Both will eliminate >> this problem, and without requiring Dovecot director. And you'll get >> better performance than with Gluster, which, BTW, isn't really suitable >> as a transactional filesystem, was not designed for such a use case. > > Speaking as an admin who has run Dovecot on top of GFS both with and > without the director, I would never go back to a cluster without the > director. The cluster performs *so* much better when glocks can be > cached on a single node, and this can't happen if a single user has IMAP > processes on separate nodes. > > No, you don't strictly need the director if you have GFS, but if you can > manage it, you'll be a lot happier. Did/do you see the Director/glock benefit with both maildir and mdbox Jim? Do you see any noteworthy performance differences between the two formats on GFS, with and without Director? BTW, are you hitting FC or iSCSI LUNs? -- Stan From tlx at leuxner.net Fri Mar 23 11:40:52 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Fri, 23 Mar 2012 10:40:52 +0100 Subject: [Dovecot] Dovecot v2.1.3 (f30437ed63dc) Auth/Login Issues Message-ID: <20120323094052.GA9851@nihlus.leuxner.net> Hi, some change between ff5c341f8838 and f30437ed63dc seems to have broken auth: => Bad Login Mar 23 09:01:46 spectre dovecot: master: Dovecot v2.1.3 (f30437ed63dc) starting up [...] Mar 23 10:25:44 spectre dovecot: auth: Debug: auth client connected (pid=7266) Mar 23 10:25:45 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=62388#011resp= Mar 23 10:25:45 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): hit: #011userdb_quota_rule=*:storage=5G#011userdb_acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:25:45 spectre dovecot: auth: Debug: master in: REQUEST#0113958898689#0117266#0111#011bfc44f32051961b909e2b458440d645f Mar 23 10:25:45 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): hit: tlx at leuxner.net#011uid=5000#011gid=5000#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: master out: USER#0113958898689#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=7267, TLS Mar 23 10:25:45 spectre dovecot: imap(tlx at leuxner.net): Connection closed in=0 out=319uthentication/login: => Good Login Mar 23 10:26:37 spectre dovecot: master: Dovecot v2.1.3 (ff5c341f8838) starting up [...] Mar 23 10:27:18 spectre dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 10:27:18 spectre dovecot: auth: Debug: auth client connected (pid=9832) Mar 23 10:27:19 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=51647#011resp= Mar 23 10:27:19 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file /var/vmail/auth.d/leuxner.net/passwd: Read 1 users in 0 secs Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:27:19 spectre dovecot: auth: Debug: master in: REQUEST#0113656384513#0119832#0111#0114782efcbd0324b228bb85aaae916cfe6 Mar 23 10:27:19 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: master out: USER#0113656384513#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:27:19 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=9835, TLS Regards Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From tss at iki.fi Fri Mar 23 11:51:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 11:51:43 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332405027.6792.7.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> Message-ID: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> On 22.3.2012, at 10.30, Noel Butler wrote: > On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > >> :2012-03-22T11:55:Noel Butler: >> >>> perhaps it should be renamed then, given it violates the known normal >>> for SYSCONF dir, you've just created another form of --datadir >> >> Not really. The way I see it works as expected. The sysconf dir is the > > > Then you and I and a few other devs involved in other very well known > bits of software that everyone likely uses, will have to agree to > disagree A ton of software installs into /etc// directory. Most Linux distributions installed Dovecot v1.x that way as well. And of course everyone expects configuration to be under /etc. The default of sysconfdir is PREFIX/etc/. Dovecot v2.0 really shouldn't install its stuff into PREFIX/etc/ but into PREFIX/etc/dovecot/. So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? From alain.defrance at univ-evry.fr Fri Mar 23 12:20:01 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:20:01 +0100 Subject: [Dovecot] quota ldap Message-ID: <4F6C4E51.7010603@univ-evry.fr> hello all, i'm using quota + ldap with dovecot 2 in dovecot-ldap.conf.ext file i have the line : user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid,quota=quota_rule=*:storage=%$B how can i add "Trash:storage= " to have more place for deleting messages like in 90-quota.conf file ? quota_rule2 = Trash:storage thanks for help regards -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From mlists at edicom.eu Fri Mar 23 12:38:18 2012 From: mlists at edicom.eu (Miguel Tormo) Date: Fri, 23 Mar 2012 11:38:18 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <201203231138.18338.mlists@edicom.eu> El Mi?rcoles, 21 de Marzo de 2012 15:43:14 Luca Lesinigo escribi?: > Hello list. Hello, > > I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. > I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. > > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. > It would be desirable for multiple reasons: I have recently deployed a very similar setup: imap proxy, mailbox sharding... Although not exactly like yours. Comments below: > - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes This is reasonable. For example, I did this to seamless migrate lots of users from one server to another, migrating just a few of them at a time. > - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) You can do the exact same thing in the future, of course. > - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) This is fairly easy to accomplish with imap proxying. > - be ready to serve traffic over IPv6 without touching our precious mailbox servers This is doable. > - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > > Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). I didn't implement a lemonade profile nor quotas in my setup. However, I can confirm you that IMAP IDLE does work with imap proxy. > > In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). No problem with that, but this is related to the MTA configuration, not dovecot. > > I'd like to know if that plan sounds reasonable or if there's something stupid in it. > Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. From dovecot-l at fu-berlin.de Fri Mar 23 12:44:32 2012 From: dovecot-l at fu-berlin.de (Heiko Schlichting) Date: Fri, 23 Mar 2012 11:44:32 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <20120323104432.GB1353054@CIS.FU-Berlin.DE> Timo wrote: > So the only way I can think of how to change this is to add another > option to optionally remove the dovecot/ suffix from the directory, but > is this really worth the trouble? I would appreciate such option too. For large dedicated installations other schemes than /etc/dovecot are common. See http://dovecot.org/list/dovecot/2009-January/036131.html Heiko Heiko Schlichting Freie Universit?t Berlin heiko.schlichting at fu-berlin.de Zentraleinrichtung f?r Datenverarbeitung Telefon +49 30 838-54327 Fabeckstra?e 32 Telefax +49 30 838454327 14195 Berlin From nmilas at noa.gr Fri Mar 23 12:48:18 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 23 Mar 2012 12:48:18 +0200 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C4E51.7010603@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> Message-ID: <4F6C54F2.7020203@noa.gr> On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > how can i add "Trash:storage= " to have more place for deleting > messages like in > See, for example, my setup: http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html I use a single conf file (because it's small and it's more intuitive to me). Details: http://wiki2.dovecot.org/Quota/Configuration Regards, Nick From tss at iki.fi Fri Mar 23 12:50:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:50:04 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> On 21.3.2012, at 16.43, Luca Lesinigo wrote: > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. From tss at iki.fi Fri Mar 23 12:53:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:53:16 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <20120323104432.GB1353054@CIS.FU-Berlin.DE> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> Message-ID: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> On 23.3.2012, at 12.44, Heiko Schlichting wrote: > Timo wrote: >> So the only way I can think of how to change this is to add another >> option to optionally remove the dovecot/ suffix from the directory, but >> is this really worth the trouble? > > I would appreciate such option too. For large dedicated installations other > schemes than /etc/dovecot are common. > > See http://dovecot.org/list/dovecot/2009-January/036131.html Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). From alain.defrance at univ-evry.fr Fri Mar 23 12:58:09 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:58:09 +0100 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C54F2.7020203@noa.gr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> Message-ID: <4F6C5741.3000408@univ-evry.fr> thanks Nick so if i understand correctly i can mix the 2 quota_rule ? the one who came from ldap user_attrs (quota_rule=*:bytes=%$) and the other which from quota_rule2 = Trash:storage=+3%% in your case you add 3% quota more for Trash ? Am i write ? regards > On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > >> how can i add "Trash:storage= " to have more place for deleting >> messages like in >> > > See, for example, my setup: > http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html > > I use a single conf file (because it's small and it's more intuitive > to me). > > Details: http://wiki2.dovecot.org/Quota/Configuration > > Regards, > Nick > -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From jtam.home at gmail.com Fri Mar 23 12:58:36 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Fri, 23 Mar 2012 03:58:36 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 Message-ID: I ran into two issues trying to upgrade our dovecot installation (Solaris 10). 1) Does not compile with OpenSSL 0.9.7 Not a big deal, as I was able to successfully against OpenSSL 0.9.8, but does dovecot require OpenSSL >= 0.9.8 now? libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-test -std=gnu99 -O3 -fomit-frame-pointer -mcpu=ultrasparc -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -MT istream-openssl.lo -MD -MP -MF .deps/istream-openssl.Tpo -c istream-openssl.c -fPIC -DPIC -o .libs/istream-openssl.o iostream-openssl-context.c:9:28: openssl/engine.h: No such file or directory iostream-openssl-context.c: In function `ssl_iostream_deinit_global': iostream-openssl-context.c:431: warning: implicit declaration of function `ENGINE_finish' iostream-openssl-context.c:432: warning: implicit declaration of function `ENGINE_cleanup' ... 2) Dovecot's LDA does not work After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe exact same config file, local mail delivery tempfails: Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address Mar 23 02:51:51 server dovecot: auth: Error: userdb connection: Failed to get peer's credentials Mar 23 02:51:51 server dovecot: lda: Error: userdb lookup(j.tam): Disconnected unexpectedly Mar 23 02:51:51 server dovecot: lda: Fatal: Internal error occurred. Refer to server log for more information. # Sendmail reports stat=Deferred: local mailer (/var/dovecot/libexec/dovecot-lda) exited with EX_TEMPFAIL After seeing 2) in the logs, I had to revert back to 2.0.16. Any hints on what could be wrong? Joseph Tam # 2.0.16: /var/dovecot/etc/dovecot/dovecot.conf # OS: SunOS 5.10 sun4u nfs auth_cache_negative_ttl = 10 mins auth_cache_size = 64 k auth_cache_ttl = 1 days auth_failure_delay = 5 secs auth_master_user_separator = * auth_socket_path = /var/dovecot/run/auth-userdb auth_username_chars = abcdefghijklmnopqrstuvwxyz01234567890.-_ auth_worker_max_count = 1 base_dir = /var/dovecot/run default_vsz_limit = 64 M deliver_log_format = first_valid_gid = 10000 first_valid_uid = 10000 hostname = our.mail.domain last_valid_gid = 19999 last_valid_uid = 19999 lda_mailbox_autocreate = yes log_timestamp = login_greeting = Ready. mail_location = mbox:/nfs/home/%n/mail:INBOX=/nfs/mail/%n:INDEX=/data/dc-cache/%n mail_nfs_storage = yes mail_temp_dir = /var/tmp mbox_very_dirty_syncs = yes mbox_write_locks = dotlock_try fcntl namespace { inbox = yes location = prefix = separator = / } namespace { hidden = yes list = no location = prefix = / separator = / } namespace { hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace { hidden = yes list = no location = prefix = mail/ separator = / } passdb { args = /var/dovecot/etc/master-users driver = passwd-file master = yes pass = yes } passdb { args = /var/yp/etc/passwd driver = passwd-file } postmaster_address = MAILER-DAEMON at our.mail.domain protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth-worker { user = dovecot } service auth { idle_kill = 1 hours } service imap-login { process_limit = 2 service_count = 0 } service imap { process_limit = 512 } service pop3-login { process_limit = 1 service_count = 0 } service pop3 { process_limit = 64 } shutdown_clients = no ssl_cert = References: Message-ID: On Fri, 23 Mar 2012, dovecot-request at dovecot.org wrote: >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the > dovecot/ suffix from other directories as well. That might be > something worth doing (--without-package-suffix or something?). +1. I fake it now with symlinks (e.g. etc/dovecot -> .). Joseph Tam From rainer.frey at inxmail.de Fri Mar 23 13:19:45 2012 From: rainer.frey at inxmail.de (Rainer Frey) Date: Fri, 23 Mar 2012 12:19:45 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <69027C8C-A95B-41C2-B06B-824345F738DA@inxmail.de> On Mar 23, 2012, at 10:51 AM, Timo Sirainen wrote: >>> :2012-03-22T11:55:Noel Butler: >>> >>>> perhaps it should be renamed then, given it violates the known normal >>>> for SYSCONF dir, you've just created another form of --datadir >>> >>> Not really. The way I see it works as expected. >> >> The directory for installing read-only data files that pertain >> to a single machine?that is to say, files for configuring a >> host. Mailer and network configuration files, ?/etc/passwd?, and >> so forth belong here. All the files in this directory should be >> ordinary ASCII text files. This directory should normally be >> ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are >> using Autoconf, write it as ?@sysconfdir@?.) Well, I don't see that that prevents organizing the files in sysconfdir into a subdirectory. > ton of software installs into /etc// directory. [...] > So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? I really don't think so. What for? Nobody has shown a real-world problem with that subdirectory. From tss at iki.fi Fri Mar 23 13:26:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 13:26:54 +0200 Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: References: Message-ID: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> On 23.3.2012, at 12.58, Joseph Tam wrote: > I ran into two issues trying to upgrade our dovecot installation (Solaris 10). > > 1) Does not compile with OpenSSL 0.9.7 > > Not a big deal, as I was able to successfully against OpenSSL 0.9.8, > but does dovecot require OpenSSL >= 0.9.8 now? Hm. Maybe it's time by now? :) It could be fixed with some more #ifdefs but those make code more unreadable. > 2) Dovecot's LDA does not work > > After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe > exact same config file, local mail delivery tempfails: > > Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? From hsn at filez.com Fri Mar 23 13:41:24 2012 From: hsn at filez.com (Radim Kolar) Date: Fri, 23 Mar 2012 12:41:24 +0100 Subject: [Dovecot] delivering with maildrop Message-ID: <4F6C6164.2050506@filez.com> Can somebody provide maildrop syntax for using deliver-lda as final delivery program during sorting mail in user mailfilter? i mean replacement for "to" statement if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) { to $MAIL/.dovecot/ } From jtl+dovecot at uvm.edu Fri Mar 23 14:13:21 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 08:13:21 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C228E.5060902@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> Message-ID: <4F6C68E1.4030400@uvm.edu> On 3/23/12 3:13 AM, Stan Hoeppner wrote: >> Speaking as an admin who has run Dovecot on top of GFS both with and >> without the director, I would never go back to a cluster without the >> director. The cluster performs *so* much better when glocks can be >> cached on a single node, and this can't happen if a single user has IMAP >> processes on separate nodes. >> >> No, you don't strictly need the director if you have GFS, but if you can >> manage it, you'll be a lot happier. > Did/do you see the Director/glock benefit with both maildir and mdbox > Jim? Do you see any noteworthy performance differences between the two > formats on GFS, with and without Director? BTW, are you hitting FC or > iSCSI LUNs? > Actually, we're all mbox. This primarily has to do with how users do self-service mail recovery from backup: one folder = one file. I'd like to move to mdbox, but it would mean the recovery scripts will need to understand which files are associated with which folders, as well as restoring the associated index files. That's a to-do. We're using fibrechannel (IBM v7000) storage, but I would expect to see the same thing with iSCSI. It's mostly about different nodes contending over locks on the same files (although I'm sure cache locality helps a great deal, too.) If you end up with imap processes for the same folder on different nodes, or mail delivery happening on one node and imap on the other, you will feel the lag in your IMAP client. "Oh, my INBOX has been unresponsive for 10 seconds, I must be getting a lot of mail right now!" That's an exaggeration, but not by much. Jim From amateo at um.es Fri Mar 23 14:15:40 2012 From: amateo at um.es (Angel L. Mateo) Date: Fri, 23 Mar 2012 13:15:40 +0100 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> References: <4F6AF72E.9030206@um.es> <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> Message-ID: <4F6C696C.5030900@um.es> El 22/03/12 19:57, Timo Sirainen escribi?: > On 22.3.2012, at 11.55, Angel L. Mateo wrote: > >> The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. > > service auth { > idle_kill = 0 > } > In a test server I have, this have solved the problem. In my productions servers it is still being restarted. Could it be another parameter involve in this? service_count is set to 0. I have also seen that, whenever dovecot/auth is restarted, dovecot/config has also been restarted. Could be related? My config related with this service auth is: service auth { chroot = client_limit = 4096 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } -- Angel L. Mateo Mart?nez Secci?n de Telem?tica ?rea de Tecnolog?as de la Informaci?n _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 From list at airstreamcomm.net Fri Mar 23 15:39:07 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 08:39:07 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> On Wed, 21 Mar 2012 09:56:12 -0600, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=4220 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=5088 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: > fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: > fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index > > > This is my dovecot config currently: > > jdevine at test-gluster-client2:~> dovecot -n > # 2.0.13: /etc/dovecot/dovecot.conf > # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 > lock_method = dotlock > mail_fsync = always > mail_location = mdbox:~/mdbox > mail_nfs_index = yes > mail_nfs_storage = yes > mmap_disable = yes > passdb { > driver = pam > } > protocols = " imap" > ssl_cert = ssl_key = userdb { > driver = passwd > } I was able to get dovecot working across a gluster cluster a few weeks ago and it worked just fine. I would recommend using the native gluster mount option (need to install gluster software on clients), and using distributed replicated as your replication mechanism. If you're running two gluster servers you should have a replica count of two with distributed replicated. You should test first to make sure you can create a file in both mounts and see it from every mount point in the cluster, as well as interact with it. It's also very important to make sure your servers are running with synchronized clocks from an NTP server. Very bad things happen to a (dovecot or gluster) cluster out of sync with NTP. From eliezer at ngtech.co.il Fri Mar 23 15:57:30 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Fri, 23 Mar 2012 15:57:30 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <4F6C814A.2050803@ngtech.co.il> On 23/03/2012 12:53, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > >> Timo wrote: >>> So the only way I can think of how to change this is to add another >>> option to optionally remove the dovecot/ suffix from the directory, but >>> is this really worth the trouble? >> >> I would appreciate such option too. For large dedicated installations other >> schemes than /etc/dovecot are common. >> >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > well squid is using another way such as the directory you specify and without the /dovecot (squid) suffix. it's not that important. if you do change the config directory you know where you are putting it. i,m using the /opt/(service name) to install most of my self complied software so idont really care about it. but if the sysconfig directory as a directive it should be the default. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il From stan at hardwarefreak.com Fri Mar 23 16:02:41 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 09:02:41 -0500 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C6164.2050506@filez.com> References: <4F6C6164.2050506@filez.com> Message-ID: <4F6C8281.10906@hardwarefreak.com> On 3/23/2012 6:41 AM, Radim Kolar wrote: > Can somebody provide maildrop syntax for using deliver-lda as final > delivery program during sorting mail in user mailfilter? > > i mean replacement for "to" statement > > if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) > { > to $MAIL/.dovecot/ > } Dovecot's local delivery agent uses the Sieve language: http://wiki.dovecot.org/LDA/Sieve The syntax is quite different than maildrop or procmail. -- Stan From tss at iki.fi Fri Mar 23 16:06:25 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 16:06:25 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On 23.3.2012, at 15.39, wrote: > I was able to get dovecot working across a gluster cluster a few weeks ago > and it worked just fine. I would recommend using the native gluster mount > option (need to install gluster software on clients), and using distributed > replicated as your replication mechanism. Have you tried stress testing it with imaptest? Run in parallel for both servers: imaptest host=gluster1 user=testuser pass=testpass imaptest host=gluster2 user=testuser pass=testpass http://imapwiki.org/ImapTest And see if Dovecot logs any errors. From micah at riseup.net Fri Mar 23 17:52:02 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 11:52:02 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply References: <87bonon1j4.fsf@algae.riseup.net> Message-ID: <87ty1fl3h9.fsf@algae.riseup.net> Micah Anderson writes: > dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; > dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 > dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. I'm also getting similar strange results with my regular dsync backup: dsync-local(user at example.com): Error: Unexpected reply from server: 0 23bdce147b43674f8e2700002c449efa 1242 146 \Recent 1332335848 this is with 2.1.3. micah From micah at riseup.net Fri Mar 23 18:25:27 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 12:25:27 -0400 Subject: [Dovecot] doveadm user -f index Message-ID: <87pqc3l1xk.fsf@algae.riseup.net> I've configured my mail_location to have a different location for performance reasons so they aren't in the same location as the mail_location. The 'doveadm user -f home' is useful to find where a user's home directory is for various scripting purposes, but I can't seem to find a way to determine the location of the user's indexes. I can do something with the output of dovecot -a to find the mail_location and then look for a configured INDEX, but then I don't have a good way of translating the %d/%1n/%n type string formatters into their values for a user. thanks for any suggestions! micah -- From stan at hardwarefreak.com Fri Mar 23 19:11:49 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 12:11:49 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C68E1.4030400@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> Message-ID: <4F6CAED5.4000206@hardwarefreak.com> On 3/23/2012 7:13 AM, Jim Lawson wrote: > On 3/23/12 3:13 AM, Stan Hoeppner wrote: > >>> Speaking as an admin who has run Dovecot on top of GFS both with and >>> without the director, I would never go back to a cluster without the >>> director. The cluster performs *so* much better when glocks can be >>> cached on a single node, and this can't happen if a single user has IMAP >>> processes on separate nodes. >>> >>> No, you don't strictly need the director if you have GFS, but if you can >>> manage it, you'll be a lot happier. >> Did/do you see the Director/glock benefit with both maildir and mdbox >> Jim? Do you see any noteworthy performance differences between the two >> formats on GFS, with and without Director? BTW, are you hitting FC or >> iSCSI LUNs? >> > > Actually, we're all mbox. This primarily has to do with how users do > self-service mail recovery from backup: one folder = one file. Yeah, mbox isn't as dead as some people contend, but it just doesn't have legs for newer deployment architectures. > I'd like to move to mdbox, but it would mean the recovery scripts will > need to understand which files are associated with which folders, as > well as restoring the associated index files. That's a to-do. That's an easy weekend project. ;) > We're using fibrechannel (IBM v7000) storage, but I would expect to see > the same thing with iSCSI. It's mostly about different nodes contending > over locks on the same files (although I'm sure cache locality helps a > great deal, too.) If you end up with imap processes for the same folder > on different nodes, or mail delivery happening on one node and imap on > the other, you will feel the lag in your IMAP client. "Oh, my INBOX has > been unresponsive for 10 seconds, I must be getting a lot of mail right > now!" That's an exaggeration, but not by much. I was asking about your SAN storage unrelated to the locking issue. Just a curiosity thing. Note my email domain. ;) I'm an FC fan but iSCSI seems to be more popular in many circles, actually pretty much market wide these days. So when I come across another SAN user I'm naturally curious as to what hardware they use. Just so nobody gets the wrong idea, I wasn't advocating against Director earlier in the thread. I think it's fantastic and solves some critical scalability problems. As in your case, it allows one to use his mail storage format of choice with a cluster filesystem while mostly avoiding the locking headaches. In the past one pretty much had to use maildir with a cluster FS to avoid the locking performance killed. But one had to suffer the higher IOPS load on the storage. Not always a good tradeoff, especially for busy mail systems. I assume you do still have some minor locking/performance issues with the INBOX, even with Director, when LDA and the user MUA are both hitting the INBOX index and mbox files. You'll still see this with mdbox, but probably to a lesser degree if you use a smallish mdbox_rotate_size value. To mitigate this INBOX locking you could go with a dual namespaces, using maildir or sdbox for the INBOX and mdbox for the other user mail folders. -- Stan From tss at iki.fi Fri Mar 23 19:19:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 19:19:26 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <3845B569-3CE5-4C0C-BB60-B9CA91FF8B56@iki.fi> On 23.3.2012, at 19.11, Stan Hoeppner wrote: > I assume you do still have some minor locking/performance issues with > the INBOX, even with Director, when LDA and the user MUA are both > hitting the INBOX index and mbox files. You'll still see this with > mdbox, but probably to a lesser degree if you use a smallish > mdbox_rotate_size value. To mitigate this INBOX locking you could go > with a dual namespaces, using maildir or sdbox for the INBOX and mdbox > for the other user mail folders. The biggest difference is that mbox requires read locks, mdbox doesn't. mdbox lock waits are very similar to maildir's. Of course, I don't know about the cluster filesystems' internal locking, but I thought it was even worse with Maildir than with mbox because it had to get a read lock for each read file, but I guess this depends on the filesystem. From ruskie at codemages.net Fri Mar 23 19:22:11 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Fri, 23 Mar 2012 18:22:11 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: :2012-03-23T12:53:Timo Sirainen: > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). I would suggest to have a --layout=gnu|opt That would either do what it currently does(gnu) and opt to install everything into a single dir i.e.: /opt/dovecot/ With subdirs under there. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Be advised: causing a disturbance may result in fines, detainment, bodily harm, or death. Enjoy your stay. From jtl+dovecot at uvm.edu Fri Mar 23 19:33:42 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 13:33:42 -0400 Subject: [Dovecot] recovery of mdbox folders (was: Re: distributed mdbox) In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <4F6CB3F6.5010006@uvm.edu> On 3/23/12 1:11 PM, Stan Hoeppner wrote: > On 3/23/2012 7:13 AM, Jim Lawson wrote: > > >> I'd like to move to mdbox, but it would mean the recovery scripts will >> need to understand which files are associated with which folders, as >> well as restoring the associated index files. That's a to-do. > That's an easy weekend project. ;) > Out of curiosity, does anyone do self-service restoration of individual mdbox folders? If I'm going to write a script to do it, it'd be nice to avoid any pitfalls someone else has already run into. :-) We're already backing up from snapshots, so the synchronization issues are solved (at least at backup time...) Jim From list at airstreamcomm.net Fri Mar 23 19:43:34 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 12:43:34 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On Fri, 23 Mar 2012 16:06:25 +0200, Timo Sirainen wrote: > On 23.3.2012, at 15.39, > wrote: > >> I was able to get dovecot working across a gluster cluster a few weeks >> ago >> and it worked just fine. I would recommend using the native gluster >> mount >> option (need to install gluster software on clients), and using >> distributed >> replicated as your replication mechanism. > > Have you tried stress testing it with imaptest? Run in parallel for both > servers: > > imaptest host=gluster1 user=testuser pass=testpass > imaptest host=gluster2 user=testuser pass=testpass > > http://imapwiki.org/ImapTest > > And see if Dovecot logs any errors. I did stress test it, but we have developed a "mail bot net" tool for the purpose. I should mention this was tested using dovecot 1.2, as this is our current production version (hopefully will be upgrading soon). Its comprised of a control server that starts a bot network of client machines that creates pop/imap connections (smtp as well) on our test cluster of dovecot (and postfix) servers. In my test I distributed the load across a two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN storage attached to it. I actually didn't change my configuration from when I had a test NFS server connected to the test servers (mmap disabled, fcntl locking, etc), because glusterfs was an afterthought when we were stress testing our new netapp system using NFS. We have everything in VMware, including the glusterfs servers. Using five bot servers and connecting 7 times a second from each server (35 connections per second) for both pop and imap (70 total connections per second) split between two dovecot servers I was not seeing any big issues. The load average was low, and there were no errors to speak of in dovecot (or postfix). I was mounting the storage with the glusterfs native client, not using NFS (which I have not tested). I would like to do a more thorough test of glusterfs using Dovecot 2.0 on some dedicated hardware and see how much further I can push the system. From busseniu at in.tum.de Fri Mar 23 20:02:10 2012 From: busseniu at in.tum.de (=?UTF-8?B?Q2hyaXN0b3BoIEJ1w59lbml1cw==?=) Date: Fri, 23 Mar 2012 19:02:10 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: <4F6CBAA2.5020409@in.tum.de> Hi, maybe try "dsync -o mail_fsync=never". Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From luca at lm-net.it Fri Mar 23 20:12:56 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Fri, 23 Mar 2012 19:12:56 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> Message-ID: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: > Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... >> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) > Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... > I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. > However, I can confirm you that IMAP IDLE does work with imap proxy. That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. > You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. > You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). -- Luca Lesinigo From gedalya at gedalya.net Fri Mar 23 20:24:11 2012 From: gedalya at gedalya.net (Gedalya) Date: Fri, 23 Mar 2012 14:24:11 -0400 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Message-ID: <4F6CBFCB.60209@gedalya.net> On 03/23/2012 02:12 PM, Luca Lesinigo wrote: > Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. > I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. >>> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) >> Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. > Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) > > Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >>> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... >> I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. The same proxy_maybe (dummy proxy) setup works great for POP3 too. Very simple to set up, works like a charm. Nothing much to think about. > >> However, I can confirm you that IMAP IDLE does work with imap proxy. > That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. It will work as well as it was working with your existing courier server. But it will work great for accounts migrated to native dovecot. >> You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. > Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. It's a very good idea to have completely separate machines for outgoing mail. Once you have imap-only boxes, you can eliminate the need for an MTA by using the dovecot LMTP server. Your postfix transport map can send mail to either smtp:imap.yourdomain.com:25 or lmtp:imap.yourdomain.com:2525 on a per account basis, and you can get rid of the MTA in due time. >> You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. > On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Since you're sending everything back to the MX, you might as well have your MX use LMTP, looking up the correct protocol and host from the database, and spend the next couple of years telling your customers to change their mail client configuration to use a dedicated outgoing mail server. It's worth the trouble. > > Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). > > -- > Luca Lesinigo From ednitido at gmail.com Fri Mar 23 21:44:23 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 15:44:23 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are the same Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 From tss at iki.fi Fri Mar 23 21:46:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:46:53 +0200 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: On 23.3.2012, at 21.44, Ed Nitido wrote: > I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are > the same > > Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 Set auth_debug=yes. What does it log with v2.1.3? Also what's in your dovecot-ldap.conf.ext? From tss at iki.fi Fri Mar 23 21:49:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:49:38 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4F6CBFCB.60209@gedalya.net> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> <4F6CBFCB.60209@gedalya.net> Message-ID: On 23.3.2012, at 20.24, Gedalya wrote: > On 03/23/2012 02:12 PM, Luca Lesinigo wrote: >> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >>> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. >> I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. >> I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... > I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. Right, a lot of people have done migration from Courier -> Dovecot using the dummy proxying. Since v2.0 the proxying automatically handles any CAPABILITY string issues. From ednitido at gmail.com Fri Mar 23 22:26:46 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 16:26:46 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: <950E30E6-38A5-4F5F-B2D6-B12C810AB439@iki.fi> Message-ID: Ooops, didn't email the list... it working now thanks to Timo, solution below On Fri, Mar 23, 2012 at 4:14 PM, Timo Sirainen wrote: > >> On 23.3.2012, at 22.01, Ed Nitido wrote: >> >> > pass_attrs = >> uid=user,userPassword=password,=proxy,=master=doveadmin,=pass=xxxxxx >> >> I guess it doesn't like the "=proxy" part. I guess I should fix it. For >> now just set "=proxy=y". >> > > From ncjeffgus at zimage.com Fri Mar 23 22:42:23 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Fri, 23 Mar 2012 13:42:23 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6CBAA2.5020409@in.tum.de> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> Message-ID: <1332535343.5601.6.camel@sally> On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > Hi, > > maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync *should* be able to be just as fast, but it currently is spending way too much time doing something. What is it? ...Jeff From post at michael-neubert.de Fri Mar 23 22:57:28 2012 From: post at michael-neubert.de (Michael Neubert) Date: Fri, 23 Mar 2012 21:57:28 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 Message-ID: <4F6CE3B8.7020507@michael-neubert.de> Hello, I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". The config was not touched but now IMAP connections are not possible anymore (LMTP works fine). When I try to connect to a mailbox, the connect fails. Some log entries: ############################################################################################################### Mar 23 21:45:28 imap-login: Warning: SSL: where=0x10, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv2/v3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: auth client connected (pid=3431) Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server done A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write session ticket A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write change cipher spec A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x20, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: client in: AUTH 1 PLAIN service=imap secured lip=yyy.yyy.yyy.yyy rip=xxx.xxx.xxx.xxx lport=993 rport=51379 Mar 23 21:45:28 auth: Debug: client out: CONT 1 Mar 23 21:45:28 auth: Debug: client in: CONT 1 AG5lbWlAdmlzaXQtd29ybGQuZGUAUHJvNDUwLnN1 Mar 23 21:45:28 auth-worker(3433): Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 21:45:28 auth-worker(3433): Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_mysql.so Mar 23 21:45:28 auth-worker(3433): Info: mysql(zzz.zzz.zzz.zzz): Connected to database dovecot Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): query: SELECT password, 'directory' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid FROM users WHERE username = 'username' AND domain = 'domain' AND active = 'Y' Mar 23 21:45:28 auth: Debug: client out: OK 1 user=username Mar 23 21:45:28 auth: Debug: master in: REQUEST 2286813185 3394 1 4727968fd3514dd45f623ad9f944e305 Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): SELECT home, uid, gid FROM users WHERE username = 'username' AND domain = 'domain' Mar 23 21:45:28 auth: Debug: master out: USER 2286813185 username home=directory uid=8 gid=8 Mar 23 21:45:28 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=3434, TLS Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap(username): Info: Connection closed in=0 out=303 ############################################################################################################### The MySQL authentification seems to work fine, but after this the connection is closed with the SSL alert. In Dovecot 2.1.2 everything worked fine. The SSL certifcate is also correct. Any hints are welcome to identify the problem. Thanks in advance. Beste wishes Michael From tss at iki.fi Fri Mar 23 23:03:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:03:01 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> On 23.3.2012, at 19.43, wrote: >> Have you tried stress testing it with imaptest? Run in parallel for both >> servers: > I did stress test it, but we have developed a "mail bot net" tool for the > purpose. I should mention this was tested using dovecot 1.2, as this is > our current production version (hopefully will be upgrading soon). Its > comprised of a control server that starts a bot network of client machines > that creates pop/imap connections (smtp as well) on our test cluster of > dovecot (and postfix) servers. In my test I distributed the load across a > two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN > storage attached to it. I actually didn't change my configuration from > when I had a test NFS server connected to the test servers (mmap disabled, > fcntl locking, etc), because glusterfs was an afterthought when we were > stress testing our new netapp system using NFS. We have everything in > VMware, including the glusterfs servers. Using five bot servers and > connecting 7 times a second from each server (35 connections per second) > for both pop and imap (70 total connections per second) split between two > dovecot servers I was not seeing any big issues. The load average was low, > and there were no errors to speak of in dovecot (or postfix). I was > mounting the storage with the glusterfs native client, not using NFS (which > I have not tested). I would like to do a more thorough test of glusterfs > using Dovecot 2.0 on some dedicated hardware and see how much further I can > push the system. What did the bots do? Add messages and delete messages as fast as they could? I guess that's mostly enough to see if things work. imaptest anyway hammers the server as fast as it can with all kinds of commands. From tss at iki.fi Fri Mar 23 23:25:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:25:28 +0200 Subject: [Dovecot] dsync redesign Message-ID: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: http://dovecot.org/tmp/dsync-redesign-problem.txt It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. From list at airstreamcomm.net Sat Mar 24 01:39:11 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 18:39:11 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> Message-ID: <7e40b18742c5053948aeaaa51d41ceca@mail.airstreamcomm.net> On Fri, 23 Mar 2012 23:03:01 +0200, Timo Sirainen wrote: > On 23.3.2012, at 19.43, > wrote: > >>> Have you tried stress testing it with imaptest? Run in parallel for both >>> servers: >> I did stress test it, but we have developed a "mail bot net" tool for the >> purpose. I should mention this was tested using dovecot 1.2, as this is >> our current production version (hopefully will be upgrading soon). Its >> comprised of a control server that starts a bot network of client >> machines >> that creates pop/imap connections (smtp as well) on our test cluster of >> dovecot (and postfix) servers. In my test I distributed the load across >> a >> two node dovecot (/postfix) cluster back ended by glusterfs, which has >> SAN >> storage attached to it. I actually didn't change my configuration from >> when I had a test NFS server connected to the test servers (mmap >> disabled, >> fcntl locking, etc), because glusterfs was an afterthought when we were >> stress testing our new netapp system using NFS. We have everything in >> VMware, including the glusterfs servers. Using five bot servers and >> connecting 7 times a second from each server (35 connections per second) >> for both pop and imap (70 total connections per second) split between two >> dovecot servers I was not seeing any big issues. The load average was >> low, >> and there were no errors to speak of in dovecot (or postfix). I was >> mounting the storage with the glusterfs native client, not using NFS >> (which >> I have not tested). I would like to do a more thorough test of glusterfs >> using Dovecot 2.0 on some dedicated hardware and see how much further I >> can >> push the system. > > What did the bots do? Add messages and delete messages as fast as they > could? I guess that's mostly enough to see if things work. imaptest anyway > hammers the server as fast as it can with all kinds of commands. We created two python scripts on the bots that listed all the messages in the inbox then deleted all the messages in the inbox, one script doing pop and the other doing imap. The bots were also sending messages to the server simultaneously to repopulate inboxes. I didn't know about imaptest, thanks! From noel.butler at ausics.net Sat Mar 24 03:19:50 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sat, 24 Mar 2012 11:19:50 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <1332551990.11835.19.camel@tardis> On Fri, 2012-03-23 at 12:53 +0200, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > > > Timo wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't take much effort at all to change that to look for the long time default of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ No-one is suggesting putting all the individual conf files in /etc, only for existence of dovecot.conf itself. There are plenty of linux and unix systems that have been using /etc for as long as I can recall (even early redhat did), its only certain distros that build as /etc/foo/ the ones that use rpms or debs are obviously not running anything special (we all know no build config process will suite all operations) there are a large number i'm sure who use source (besides, with debian and redhat, who knows WHAT butchering they've done to upstreams code)... Which brings up another question, may I ask why some of the options to disable some passwd types were removed from build process? Systems that dont use system password files (amongst other formats) dont need to build them, that's not a criticism, 'just sayin'. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Sat Mar 24 03:50:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:50:07 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332551990.11835.19.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: On 24.3.2012, at 3.19, Noel Butler wrote: >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't > take much effort at all to change that to look for the long time default > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > No-one is suggesting putting all the individual conf files in /etc, only > for existence of dovecot.conf itself. So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Which brings up another question, may I ask why some of the options to > disable some passwd types were removed from build process? Systems that > dont use system password files (amongst other formats) dont need to > build them, that's not a criticism, 'just sayin'. There's also no harm in having that code included. They add no extra library dependencies. The only thing they do is to use a few kilobytes of more disk space, and possibly a few kilobytes of more memory (even that isn't certain). All options just increase the number of combinations that can cause things to go wrong. If I add some code to be compiled optionally, it just adds more combinations that should be tested together to see if the code still even compiles. Previously I've broken SSL code many times by not testing if after changes Dovecot builds without OpenSSL. So the less options there are, the more robust Dovecot is, and the less work I have to do to keep it working when adding new features. So I add an option only when there is a good use case for it and I expect more than one person to use it. From tss at iki.fi Sat Mar 24 03:52:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:52:56 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: On 23.3.2012, at 19.22, Andra? 'ruskie' Levstik wrote: > :2012-03-23T12:53:Timo Sirainen: > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > I would suggest to have a --layout=gnu|opt > > That would either do what it currently does(gnu) and opt to install > everything into a single dir i.e.: > /opt/dovecot/ > > With subdirs under there. Yes, --with-layout=gnu|opt could be useful. Anyone want to volunteer to implement it? :) From dovecot at tlinx.org Sat Mar 24 08:12:44 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:12:44 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6D65DC.7030304@tlinx.org> Jeff Gustafson wrote: > On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > >> Hi, >> >> maybe try "dsync -o mail_fsync=never". >> > > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? > ...Jeff > --- Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can use large buffers (~16MB gives best results on my local Gbit network), but it misses all those pesky acls and extended attrs, not to mention file perms...*sigh* Compare that to the I/O done 4k at a time by many older utils... If I'm writing to the LOCAL HD, instead of the network, then a 1GB-4GB buffer size gives best results (1GB/s raid5). Small buffers are such a PITA! From dovecot at tlinx.org Sat Mar 24 08:16:08 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:16:08 -0700 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? Message-ID: <4F6D66A8.3050208@tlinx.org> Is this redhat's version of the kernel only? Or does it apply to other linux kernels and other distros? Any idea what linux kernel versions might cause this? (from main dovecot webpage news) Thu Mar 22 14:38:53 EET 2012 Red Hat/CentOS users: A recent kernel update causes Dovecot to start failing after it has reached 1000 child processes. To fix this, downgrade your kernel until Red Hat releases a fixed kernel. From bra at fsn.hu Sat Mar 24 09:19:48 2012 From: bra at fsn.hu (Attila Nagy) Date: Sat, 24 Mar 2012 08:19:48 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <4F6D7594.10800@fsn.hu> On 03/23/12 22:25, Timo Sirainen wrote: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) It would bring Dovecot into the class of open source mail servers where there are currently no contenders. BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. For example, messages are immutable, so you won't face this issue in this area. And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). And of course there is always the problem of designing a good, consistent method which is also efficient. From jtam.home at gmail.com Sat Mar 24 11:36:33 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sat, 24 Mar 2012 02:36:33 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> References: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> Message-ID: On Fri, 23 Mar 2012, Timo Sirainen wrote: > On 23.3.2012, at 12.58, Joseph Tam wrote: > >> I ran into two issues trying to upgrade our dovecot installation (Solaris 10). >> >> 1) Does not compile with OpenSSL 0.9.7 >> >> Not a big deal, as I was able to successfully against OpenSSL 0.9.8, >> but does dovecot require OpenSSL >= 0.9.8 now? > > Hm. Maybe it's time by now? :) It could be fixed with some more > #ifdefs but those make code more unreadable. It might still compile with OpenSSL 0.9.7 if it is built with engine support (the default), but yeah, it's time to move to 0.9.8 or 1.0.0. >> 2) Dovecot's LDA does not work >> >> After stopping the the old dovecot, and starting dovecot 2.1.3 using the >> exact same config file, local mail delivery tempfails: >> >> Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address > > http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? Spot on, as usual. Thanks. Joseph Tam From janfrode at tanso.net Sat Mar 24 12:04:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 24 Mar 2012 11:04:07 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <20120324100407.GB31829@dibs.tanso.net> On Sat, Mar 24, 2012 at 08:19:48AM +0100, Attila Nagy wrote: > On 03/23/12 22:25, Timo Sirainen wrote: > > > Well, dsync is a very useful tool, but with continuous replication > it tries to solve a problem which should be handled -at least > partially- elsewhere. Storing stuff in plain file systems and > duplicating them to another one just doesn't scale. I don't see why this shouldn't scale. Mailboxes are after all changed relatively infrequently. One idea for making it more scalable might be to treat indexes/metadata and messages differently. Make index/metadata updates synchronous over the clusters/locations (with re-sync capability in case of lost synchronisation), while messages are store in one "altstorage" per cluster/location. For a two-location solution, message-data should be stored in: mail_location = mdbox:~/mdbox ALTcache=mdbox:~/mdbox-remoteip-cache ALT=dfetch://remoteip/ <-- new protocol If a message is in the index, look for it in that order: local mdbox ALTcache ALT if it finds the message in ALT, make a copy into ALTcache (or local mdbox?). Syncronizing messages could be a very low frequency job, and could be handled by simple rsync of ALT to ALTcache. No need for specialized tool for this job. Syncronizing ALTcache to local mdbox could be done with a reversed doveadm-altmove, but might not be necessary. Of course this is probably all very naive.. but you get the idea :-) -jf From stan at hardwarefreak.com Sat Mar 24 12:06:25 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 24 Mar 2012 05:06:25 -0500 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? In-Reply-To: <4F6D66A8.3050208@tlinx.org> References: <4F6D66A8.3050208@tlinx.org> Message-ID: <4F6D9CA1.9050008@hardwarefreak.com> On 3/24/2012 1:16 AM, Linda Walsh wrote: > Is this redhat's version of the kernel only? Or does it apply to other > linux kernels and other distros? > > Any idea what linux kernel versions might cause this? > > (from main dovecot webpage news) > > Thu Mar 22 14:38:53 EET 2012 > > Red Hat/CentOS users: A recent kernel update > causes Dovecot to > start failing after it has reached 1000 child processes. To fix this, > downgrade your kernel until Red Hat releases a fixed kernel. It appears to be a Red Hat centric regression. They added a patch to fix one thing and broke other things, Dovecot, in the process, because the Red Hat programmer made an incorrect assumption about what real world applications were doing, apparently without investigating such first. Note that one won't see this problem on their REHL/CentOS system if they never hit 1000 child processes. And as Timo states in the bug report it's *possible* Postfix could suffer the same problem as it uses the same pipe/epoll system. However nobody runs 1000 Postfix smtp[d]s. Few, if any, run over 200. The ones that do usually don't know how to properly tune Postfix, and they use a high smtp[d] daemon count to compensate for suboptimal configuration elsewhere in the system. A properly setup Postfix server can handle 200-300 msgs/second with the default 100 smtp[d] processes. 1000 smtp[d]s would suggest a message rate 10x that, or 2000-3000 msgs/second. The server plus disk subsystem required to queue that kind of message rate would be impressive, and expensive, for a mail server. This same message rate can typically be achieved by a much less expensive scale out farm. If anyone on the planet is running a properly tuned 1000 process Postfix server, I'd love to read about it. -- Stan From tss at iki.fi Sat Mar 24 13:49:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 13:49:36 +0200 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <0B23962D-A067-4A71-9A10-067FCA76B06D@iki.fi> On 24.3.2012, at 9.19, Attila Nagy wrote: > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. dsync solves several other problems besides replication. Even if Dovecot had a super efficient replicated storage, dsync would still exist for doing things like: - migrating between mailbox formats - migrating from other imap/pop3 servers - creating (incremental) backups - the redesign works great for super-high latency replication (USB sticks, cross-planet replication :) - and when you really just don't want any kind of a complex replicated database, just something simple So I'll need to get this working well in any case. And with the redesign the replication should be efficient enough to scale pretty well. > I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) Yes, I'm also planning on implementing that, but not yet. > It would bring Dovecot into the class of open source mail servers where there are currently no contenders. > > BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. > For example, messages are immutable, so you won't face this issue in this area. > And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. My plan was to create similar index files as currently exists in filesystem. It would work pretty much the same as you described: There's a "log" where changes are appended, and once in a while the changes are written into an "index" snapshot. When reading you first read the snapshot and then apply new changes from the log. The conflict resolution if DB supports it would work by reading the two logs in parallel and figure out a way to merge them consistently, similar to how dsync does pretty much the same thing. Hmm. Perhaps the metadata log could exist exactly as the dsync data format and have dsync code do the merging?.. > Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). This is already optionally done in v2.0+dbox. MIME attachments can be stored in plain binary form if they can be reconstructed back into their original form. It doesn't break any signed stuff. From CMarcus at Media-Brokers.com Sat Mar 24 14:01:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:01:07 -0400 Subject: [Dovecot] SIS and restoring from backups Message-ID: <4F6DB783.3050808@Media-Brokers.com> On 2012-03-24 7:49 AM, Timo Sirainen wrote: > This is already optionally done in v2.0+dbox. MIME attachments can be > stored in plain binary form if they can be reconstructed back into > their original form. It doesn't break any signed stuff. Hey Timo, Splitting this off into a separate thread... On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? Or maybe I misunderstood the problem? -- Best regards, Charles From tss at iki.fi Sat Mar 24 14:08:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 14:08:13 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DB783.3050808@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> Message-ID: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> On 24.3.2012, at 14.01, Charles Marcus wrote: > On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? > > Or maybe I misunderstood the problem? You can do full backups from a filesystem snapshot, which works "well enough" (might leave some unused attachments lying around in some rare cases, but that can also happen if Dovecot crashes/dies). The other possibility is to already use dsync (doveadm backup) to do full backups. With the redesigned dsync you would be able to do incremental backups also. In any case the solution involves de-SISing mails for backup. From CMarcus at Media-Brokers.com Sat Mar 24 14:54:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:54:14 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> Message-ID: <4F6DC3F6.70306@Media-Brokers.com> On 2012-03-24 8:08 AM, Timo Sirainen wrote: > You can do full backups from a filesystem snapshot, which works > "well enough" (might leave some unused attachments lying around in > some rare cases, but that can also happen if Dovecot crashes/dies). But the problem isn't with backups, but with restores, right? > The other possibility is to already use dsync (doveadm backup) to do > full backups. With the redesigned dsync you would be able to do > incremental backups also. In any case the solution involves > de-SISing mails for backup. So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Doesn't sound ideal... I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. Am I correct that enabling SIS as it is currently implemented would break this backup tool? I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... -- Best regards, Charles From post at michael-neubert.de Sat Mar 24 15:04:55 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 14:04:55 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6DC677.1000100@michael-neubert.de> The problem starts just after authorization: Console: ################################################################### openssl s_client -connect mailserver.com:993 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. a login "username" "password" closed ################################################################### Here are the logs of this moment: ################################################################### Mar 24 13:48:46 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=10662, TLS Mar 24 13:48:46 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 24 13:48:46 imap(username): Info: Connection closed in=0 out=303 ################################################################### So just after sucussful login with correct username / password the connection is closed. From tss at iki.fi Sat Mar 24 15:16:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:16:38 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DC3F6.70306@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: On 24.3.2012, at 14.54, Charles Marcus wrote: > On 2012-03-24 8:08 AM, Timo Sirainen wrote: >> You can do full backups from a filesystem snapshot, which works >> "well enough" (might leave some unused attachments lying around in >> some rare cases, but that can also happen if Dovecot crashes/dies). > > But the problem isn't with backups, but with restores, right? Ah, right. Then it gets tricky. >> The other possibility is to already use dsync (doveadm backup) to do >> full backups. With the redesigned dsync you would be able to do >> incremental backups also. In any case the solution involves >> de-SISing mails for backup. > > So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Some backup systems can do internal deduplication. > I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. > > Am I correct that enabling SIS as it is currently implemented would break this backup tool? I'm not sure. Are you running rsnapshot on live filesystem or on a snapshot? On live filesystem there would be race conditions. > I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... This should be possible, just point the namespace to such snapshot. You may need to point CONTROL dir to some temporary directory and index dir as well to either temp or to memory. From tss at iki.fi Sat Mar 24 15:17:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:17:28 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6DC677.1000100@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> Message-ID: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> On 24.3.2012, at 15.04, Michael Neubert wrote: > openssl s_client -connect mailserver.com:993 > * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. > a login "username" "password" > closed And what happens without SSL? e.g. telnet localhost 143 From mcbdovecot at robuust.nl Sat Mar 24 15:21:51 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Sat, 24 Mar 2012 14:21:51 +0100 (CET) Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: On Fri, 23 Mar 2012, Jeff Gustafson wrote: > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per second. Depending on the harddisks used, that would be quite a challenge. If you use rsync to only transfer the files that changed (based on file modification time) you may or may not miss files that have changed but still have the same time stamp. I assume you didn't use the --checksum parameter to rsync, right? dsync does so much more than simply copy some files... -- Maarten From post at michael-neubert.de Sat Mar 24 18:00:13 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:00:13 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> Message-ID: <4F6DEF8D.7090309@michael-neubert.de> > And what happens without SSL? e.g. telnet localhost 143 Without SSL it is no problem: ############################################## telnet imap-server 143 Trying xxx.xxx.xxx.xxx... Connected to imap-server. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. a login "username" "password" a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE QUOTA] Logged in b select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded \*)] Flags permitted [...] ############################################## From tss at iki.fi Sat Mar 24 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 18:02:59 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". So what exactly is this version? dovecot --version? From post at michael-neubert.de Sat Mar 24 18:09:43 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:09:43 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> Message-ID: <4F6DF1C7.1020306@michael-neubert.de> >> I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using >> Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > So what exactly is this version? dovecot --version? At the moment the version is "2.1.3-0~auto+6" from rename-it.nl. dovecot -n: # 2.1.3 (4ae85f573c93): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.4 ocfs2 From jackie.craig.sparks at live.com Sat Mar 24 22:34:49 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 16:34:49 -0400 Subject: [Dovecot] dovecot and cloudfile systems Message-ID: I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. I cant lock files, This is accessed using the fuse library. I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. From tss at iki.fi Sat Mar 24 22:43:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 22:43:34 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: Message-ID: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> On 24.3.2012, at 22.34, jackie sparks wrote: > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > I cant lock files, > > This is accessed using the fuse library. > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From jackie.craig.sparks at live.com Sat Mar 24 23:15:58 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 17:15:58 -0400 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. > From: tss at iki.fi > Date: Sat, 24 Mar 2012 22:43:34 +0200 > To: jackie.craig.sparks at live.com > CC: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot and cloudfile systems > > > On 24.3.2012, at 22.34, jackie sparks wrote: > > > > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > > I cant lock files, > > > > This is accessed using the fuse library. > > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. > > http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From tss at iki.fi Sat Mar 24 23:27:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 23:27:47 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: <00827B46-E6B3-4DD8-A035-421A522F489F@iki.fi> What you're trying to do is quite unlikely to work with any IMAP server / cloud filesystem combination. And if it does work, the performance will most likely be horrible. Of course, if it does work with any kind of a combination I'm interested in knowing about it. On 24.3.2012, at 23.15, jackie sparks wrote: > > This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. > It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= > This e-mail (including attachments) is covered by the Electronic > Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is > confidential, and is intended solely for the use of the individuals or > entities to whom it is addressed. If you are not the intended > recipient or the person responsible for delivering the e-mail to the > intended recipient, be advised that you have received this e-mail in > error and that any use, dissemination, forwarding, printing, or > copying of this e-mail and any file attachments is strictly > prohibited. If you have received this e-mail in error, please > immediately notify me by email at jackie.craig.sparks at live.com. You must destroy > the original transmission and its contents. > > >> From: tss at iki.fi >> Date: Sat, 24 Mar 2012 22:43:34 +0200 >> To: jackie.craig.sparks at live.com >> CC: dovecot at dovecot.org >> Subject: Re: [Dovecot] dovecot and cloudfile systems >> >> >> On 24.3.2012, at 22.34, jackie sparks wrote: >> >>> >>> I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. >>> I cant lock files, >>> >>> This is accessed using the fuse library. >>> I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. >>> I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. >> >> http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. > From post at michael-neubert.de Sun Mar 25 01:00:27 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sun, 25 Mar 2012 00:00:27 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6E520B.4060303@michael-neubert.de> I just did some more tests with different binaries. The problem occurs since: Dovecot 2.1.3-0~auto+5 dovecot --version 2.1.3 (f30437ed63dc) Dovecot 2.1.3-0~auto+4 works fine dovecot --version 2.1.3 (ff5c341f8838) So my title is wrong. The problem only affects people "who like to live on the edge" of 2.1.3 release ;) The stable Dovecot 2.1.3 release http://dovecot.org/list/dovecot-news/2012-March/000219.html is not affected. From noel.butler at ausics.net Sun Mar 25 05:48:36 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:48:36 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332643716.4515.23.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > > > No-one is suggesting putting all the individual conf files in /etc, only > > for existence of dovecot.conf itself. > > So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Not at all, I'm suggesting that in search for dovecot.conf file only, the search path be preferenced by @sysconfdir@/dovecot.conf such as --sysconfdir=/etc it looks for /etc/dovecot.conf, if not found, the config file location search continues on to look for /etc/dovecot/dovecot.conf I might be wrong, there might only be a handful of people annoyed by this change, but as more and more using custom builds test out moving from 1.2 to 2.x, well, more and more might be caught out, wouldn't it be better to, as you said previously, " avoid emails". As I'm sure you got better things to do than read any tripe I post :) Anyway I have made my suggestion, nothing more I think I can say on the matter that I haven't said already, so I'll leave it there, if it's implemented, great, if not, well, it's not... Cheers -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From noel.butler at ausics.net Sun Mar 25 05:56:06 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:56:06 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332644166.4515.30.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > grrr meant to comment on this too, umm since a default custom build doesnt install any config files, this would only become a confusion if one were using say an RPM package, and then decided to custom install, but IIRC, RPM renames the old config anyway, least it used to in some packages, dont know about .deb stuff though, it lacks a lot of intelligence so probably not (/me starts flamewars) Now I've said my bit.. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From stsiol at yahoo.co.uk Sun Mar 25 10:24:56 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 08:24:56 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot Message-ID: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Hello all, I want to ask about something I never did before. I have a dovecot/XMail/LAMP/Horde installation on a CentOS 5.5 32-bit system with two domains : domainA and domainB All the users used to have their mailboxes on domainA. However the personell dept decided that 90% of the users will have to have their mailboxes set to domainB and the other 10% will stay at domainA. So, I need to migrate those mailboxes from domainA to domainB. The only thing is I haven't done that ever. The directory structure is : "/var/MailRoot/domains/domainX/Username/Maildir/" and under there the usual suspects : cur (directory) dovecot.index.cache (file) dovecot-keywords (file) dovecot-uidvalidity (file) ????????? new (directory) tmp (directory) dovecot.index (file) dovecot.index.log (file) dovecot-uidlist (file) dovecot-uidvalidity.4cc055c6 (file) subscriptions (file) Using dovecot v1.2.15 Any help would be appreciated. Iam supposed to do this today !! :-) Thank you all, s. ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis? From jtam.home at gmail.com Sun Mar 25 10:46:25 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sun, 25 Mar 2012 00:46:25 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value Message-ID: Subject: Different user messages clustered around the same date.saved value After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." to iterate through all user trash folders and expunge old messages. However, I noticed a strange thing: querying what would have been deleted doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d showed many date.saved values are clustered around the same timestamp, even among different user's Trash mailbox. One user's trash mailbox having the same date.saved is explained by a user deleting a lot of message at one time, but I can't explain why many different users would have messages with the same (or closeby) date.saved value. For example, the output of the above query on my system showed the 10s window /2012-03-05 18:08:0[0-9]/ matched 7658 messages among 22 different user Trash mailboxes, which is statistically unlikely. I did't see anything special in the dovecot logs at this time to explain this. What would cause this? Joseph Tam From gedalya at gedalya.net Sun Mar 25 11:23:50 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 25 Mar 2012 04:23:50 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Message-ID: <4F6ED616.5090504@gedalya.net> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: > The directory structure is : > > > "/var/MailRoot/domains/domainX/Username/Maildir/" You can probably just: 1. Do something to prevent the user from logging in, and any deliveries from happening, e.g. delete the user. 2. Kick any existing connections. 3. Just move the Username directory from domainA to domainB. 4. Create the new user in the new domain Test the procedure first. Don't let a client log in to a mailbox and see something he's not supposed to see, like an empty mailbox. It can cause the client to drop its local cache and possibly other data. From lists at wildgooses.com Sun Mar 25 14:16:55 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:16:55 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6EFEA7.3030406@wildgooses.com> On 24/03/2012 13:21, Maarten Bezemer wrote: > > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > >> That didn't seem to make much of a difference. On a 3.1GB backup it >> shaved off 5 seconds. dsync's time was over 6 minutes with or without >> the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. >> It seems to me that dsync *should* be able to be just as fast, >> but it >> currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB > per second. Depending on the harddisks used, that would be quite a > challenge. rsync is only going to transfer files it believes has changed, so the transfer bandwidth will likely be lower > If you use rsync to only transfer the files that changed (based on > file modification time) you may or may not miss files that have > changed but still have the same time stamp. I assume you didn't use > the --checksum parameter to rsync, right? Dovecot is not very resiliant to files changing under it, but without the filename changing. I have no idea if it's supposed to work at all, but you might at least expect to see problems if you start doing this? > dsync does so much more than simply copy some files... Quite probably, but I don't think your expose above illustrates this? Regards Ed W From lists at wildgooses.com Sun Mar 25 14:41:35 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:41:35 +0100 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C8281.10906@hardwarefreak.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> Message-ID: <4F6F046F.1070003@wildgooses.com> On 23/03/2012 14:02, Stan Hoeppner wrote: > On 3/23/2012 6:41 AM, Radim Kolar wrote: >> Can somebody provide maildrop syntax for using deliver-lda as final >> delivery program during sorting mail in user mailfilter? >> >> i mean replacement for "to" statement >> >> if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) >> { >> to $MAIL/.dovecot/ >> } > Dovecot's local delivery agent uses the Sieve language: > http://wiki.dovecot.org/LDA/Sieve > > The syntax is quite different than maildrop or procmail. > I think that's why he asked the question? I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? In answer to the OP: read the maildropex man pages, but you have several options, eg: to "| someprogram" or: xfilter someprogram `someprogram` However, almost certainly I think you want the top option? Good luck Ed W From fumiyas at osstech.jp Sun Mar 25 15:17:36 2012 From: fumiyas at osstech.jp (SATOH Fumiyasu) Date: Sun, 25 Mar 2012 21:17:36 +0900 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <87zkb4j2n3.wl%fumiyas@osstech.jp> At Fri, 23 Mar 2012 12:53:16 +0200, Timo Sirainen wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > OpenLDAP's (and smbldap-tools's) configure script has --with-subdir option. This option defaults to "/openldap". (/smbldap-tools) $ grep subdir configure.in build/top.mk configure.in:dnl --with-subdir configure.in:ldap_subdir="/openldap" configure.in:AC_ARG_WITH(subdir, configure.in:[ --with-subdir=DIR change default subdirectory used for installs], configure.in: no) ldap_subdir="" configure.in: ldap_subdir="$withval" configure.in: ldap_subdir="/$withval" configure.in:AC_SUBST(ldap_subdir)dnl build/top.mk:ldap_subdir = @ldap_subdir@ build/top.mk:datadir = @datadir@$(ldap_subdir) build/top.mk:moduledir = @libexecdir@$(ldap_subdir) build/top.mk:sysconfdir = @sysconfdir@$(ldap_subdir) -- -- Name: SATOH Fumiyasu (fumiyas @ osstech co jp) -- Business Home: http://www.OSSTech.co.jp/ -- Personal Home: https://github.com/fumiyas/ From dovecot at vosslamber.nl Sun Mar 25 15:53:16 2012 From: dovecot at vosslamber.nl (Luuk@dovecot) Date: Sun, 25 Mar 2012 14:53:16 +0200 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6ED616.5090504@gedalya.net> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> Message-ID: <4F6F153C.2020005@vosslamber.nl> On 25-03-2012 10:23, Gedalya wrote: > On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >> The directory structure is : >> >> >> "/var/MailRoot/domains/domainX/Username/Maildir/" > > You can probably just: > 1. Do something to prevent the user from logging in, and any deliveries > from happening, e.g. delete the user. > 2. Kick any existing connections. > 3. Just move the Username directory from domainA to domainB. > 4. Create the new user in the new domain > > Test the procedure first. > > Don't let a client log in to a mailbox and see something he's not > supposed to see, like an empty mailbox. It can cause the client to drop > its local cache and possibly other data. > > i would also setup a forward from domainA to domainB for all the users that have moved, at least until most people who do send email know the 'old'address has changed. From stsiol at yahoo.co.uk Sun Mar 25 16:04:33 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 14:04:33 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <1332680673.52988.YahooMailNeo@web132203.mail.ird.yahoo.com> >On 25-03-2012 10:23, Gedalya wrote: >> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >>> The directory structure is : >>> >>> >>> "/var/MailRoot/domains/domainX/Username/Maildir/" >>? >> You can probably just: >> 1. Do something to prevent the user from logging in, and any deliveries >> from happening, e.g. delete the user. >> 2. Kick any existing connections. >> 3. Just move the Username directory from domainA to domainB. >> 4. Create the new user in the new domain >>? >> Test the procedure first. >>? >> Don't let a client log in to a mailbox and see something he's not >> supposed to see, like an empty mailbox. It can cause the client to drop >> its local cache and possibly other data. >>? >>? > >i would also setup a forward from domainA to domainB for all the users >that have moved, at least until most people who do send email know the >'old'address has changed. Hi chaps and thank you for your replies, Sorry forgot to mention. Most of the users use thunderbird. And yes, thank you for the forwarding issue of old e-mail accounts to the new ones. However, that is not my problem. Let me suggest something . . . : What if : 1. I setup the new accounts 2. Users log on normally from thunderbird 3. While they are using thunderbird, I get rid of their old e-mail address, create the new one (in thunderbird) 4. I make sure that the newly created address syncs with dovecot Wouldn't that be enough to migrate "on-the-fly" so to speak their existing directory tree structure and related messages to their new e-mail account ? Any ideas ? Cheers, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From CMarcus at Media-Brokers.com Sun Mar 25 18:01:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:01:39 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <4F6F3353.6000700@Media-Brokers.com> On 2012-03-25 8:53 AM, Luuk at dovecot wrote: > i would also setup a forward from domainA to domainB for all the users > that have moved, at least until most people who do send email know the > 'old'address has changed. I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 25 18:12:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:12:58 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: <4F6F35FA.6050207@Media-Brokers.com> On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: > >> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>> You can do full backups from a filesystem snapshot, which works >>> "well enough" (might leave some unused attachments lying around in >>> some rare cases, but that can also happen if Dovecot crashes/dies). >> >> But the problem isn't with backups, but with restores, right? > > Ah, right. Then it gets tricky. Yeah, I seem to remember it was a comment like that that scared me about enabling it... Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) >> So, this would make the backup storage requirements larger - maybe >> dramatically larger for sites that have a lot of large >> attachments? > Some backup systems can do internal deduplication. Hmmm... and actually, rsnapshot (which uses rsync) does just that, which is *why* each additional snapshot only requires a small fraction of additional disk space (compared to the first main/full snapshot). >> Am I correct that enabling SIS as it is currently implemented would >> break this backup tool? > I'm not sure. Are you running rsnapshot on live filesystem or on a > snapshot? On live filesystem there would be race conditions. I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] but the rsnapshot guys assured me this will and does not cause any real problems, other than those files don't get backed up. I am however looking forward to migrating this to a VM so I can do snapshot for backups to get consistent point-in-time backups. >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. This is great news! I'm looking forward to getting this all working. -- Best regards, Charles From stsiol at yahoo.co.uk Sun Mar 25 18:40:08 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 16:40:08 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F3353.6000700@Media-Brokers.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> <4F6F3353.6000700@Media-Brokers.com> Message-ID: <1332690008.99877.YahooMailNeo@web132204.mail.ird.yahoo.com> Thanks Charles, s. ? ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis >________________________________ > From: Charles Marcus >To: dovecot at dovecot.org >Sent: Sunday, 25 March 2012, 17:01 >Subject: Re: [Dovecot] migrating mailboxes on dovecot > >On 2012-03-25 8:53 AM, Luuk at dovecot wrote: >> i would also setup a forward from domainA to domainB for all the users >> that have moved, at least until most people who do send email know the >> 'old'address has changed. > >I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. > >What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. > >-- >Best regards, > >Charles > > > From hsn at filez.com Mon Mar 26 12:25:47 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 26 Mar 2012 11:25:47 +0200 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6F046F.1070003@wildgooses.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> <4F6F046F.1070003@wildgooses.com> Message-ID: <4F70361B.1070304@filez.com> I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? yes > In answer to the OP: read the maildropex man pages, but you have > several options, eg: > Yes found that dovecot-lda -m will do it nicely. echo "mail message Test " | /usr/local/libexec/dovecot/dovecot-lda -m dovecot just user agents are not able to display message with 0 headers. That confused me. From jeetuindian at gmail.com Mon Mar 26 12:51:00 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Mon, 26 Mar 2012 15:21:00 +0530 Subject: [Dovecot] dovecot.log warning Message-ID: Hi Guys, Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data i.e home directory and mail data from previous server which was on dovecot 1.2.8 to new one. Every thing is working fine. Mails are going and coming. But in dovecot.log file I an getting like imap(user at example.com):Warning: fscking index file /home/ example.com/user/mail/.imap/VISA/dovecot.index Can any one specify why its coming and how can I fix it ? -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From Attila.Sipos at netcall.com Mon Mar 26 16:28:31 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 14:28:31 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied Message-ID: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Hi, When I issue an IMAP copy command using the wrong UID, the server gives an "OK No messages copied" response. This seems like the wrong response to me. If the UID doesn't exist, then it should respond with a "No" response - maybe something like "NO - copy error: bad UID" I believe "OK No messages copied" would only be a suitable response if the email with the supplied UID had already been known to be copied successfully. I am using dovecot 1.2.9 - has this been fixed in newer versions of dovecot? Regards Attila Attila Sipos Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From bob at db.org Mon Mar 26 16:29:08 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Mon, 26 Mar 2012 15:29:08 +0200 Subject: [Dovecot] fts-solr not indexing body content Message-ID: This could easily be me missing something, but I can't seem to get the fts-solr plugin to index message bodies. Tcpdump shows me the following being sent from Dovecot to Solr as a messages is indexed: 6549fde08816e80d6b4f26650000b5f0b4b2user6549/fde08816e80d6b4f26650000b5f0b4b2/user Return-path: ... As you can see, the -node contains just an empty line. The above is just a snippet to illustrate the problem. A full dump can be found at the following URL: http://db.org/temp/solr.xml.txt Full text search is configures such: plugin { fts = solr fts_solr = break-imap-search url=http://127.0.0.1:8080/solr/ } So; am I missing something, or is this a Dovecot problem? dovecot.conf: http://db.org/temp/dovecot.conf Regards, B?rd Johannessen From tss at iki.fi Mon Mar 26 17:45:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:45:45 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332773145.26095.121.camel@innu> On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server gives > an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response - > maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. From tss at iki.fi Mon Mar 26 17:47:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:47:15 +0300 Subject: [Dovecot] dovecot.log warning In-Reply-To: References: Message-ID: <1332773235.26095.122.camel@innu> On Mon, 2012-03-26 at 15:21 +0530, Jitendra Bhaskar wrote: > Hi Guys, > > Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data > i.e home directory and mail data from previous server which was on dovecot > 1.2.8 to new one. Every thing is working fine. Mails are going and coming. > But in dovecot.log file I an getting like > > imap(user at example.com):Warning: fscking index file /home/ > example.com/user/mail/.imap/VISA/dovecot.index > > Can any one specify why its coming and how can I fix it ? This warning should not exist alone. Isn't there anything else logged? From tomislav.mihalicek at gmail.com Mon Mar 26 17:48:13 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 07:48:13 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap Message-ID: <33544743.post@talk.nabble.com> Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't create namespace 'Share/' for user miha-share at example.com: userdb didn't return a home directory, but location used it (%h): maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u Where is the problem, the same config worked with Dovecot 1.2.x -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544743.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Mon Mar 26 18:01:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:01:18 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <1332774078.26095.124.camel@innu> On Sun, 2012-03-25 at 00:46 -0700, Joseph Tam wrote: > Subject: Different user messages clustered around the same date.saved value > > After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." > to iterate through all user trash folders and expunge old messages. > > However, I noticed a strange thing: querying what would have been deleted > > doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d > > showed many date.saved values are clustered around the same > timestamp, even among different user's Trash mailbox. One user's trash > mailbox having the same date.saved is explained by a user deleting a > lot of message at one time, but I can't explain why many different users > would have messages with the same (or closeby) date.saved value. Which mailbox format? With Maildir the date.saved is taken from dovecot.index.cache file, and in some cases that might get dropped. If it does, then it fallbacks to using the file's ctime. From tss at iki.fi Mon Mar 26 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:02:59 +0300 Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <1332774179.26095.125.camel@innu> On Mon, 2012-03-26 at 07:48 -0700, Tomislav Mihalicek wrote: > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x I'm guessing it didn't work properly with v1.2. Anyway, these would help giving suggestions: 1. dovecot -n output 2. Logs with auth_debug=yes and mail_debug=yes enabled From tss at iki.fi Mon Mar 26 18:06:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:06:58 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: Message-ID: <1332774418.26095.126.camel@innu> On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > This could easily be me missing something, but I can't seem to get the > fts-solr plugin to index message bodies. What Dovecot version? From Attila.Sipos at netcall.com Mon Mar 26 18:13:29 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:13:29 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332773145.26095.121.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 15:46 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server > gives an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response > - maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:18:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:18:46 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332774418.26095.126.camel@innu> References: <1332774418.26095.126.camel@innu> Message-ID: <1332775126.26095.127.camel@innu> On Mon, 2012-03-26 at 18:06 +0300, Timo Sirainen wrote: > On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > > This could easily be me missing something, but I can't seem to get the > > fts-solr plugin to index message bodies. > > What Dovecot version? Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This should fix it: http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 From tss at iki.fi Mon Mar 26 18:22:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:22:59 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332775379.26095.130.camel@innu> It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response if > > the email with the supplied UID had already been known to be copied > > successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From lukas.mueller at newmedia.ch Mon Mar 26 18:25:54 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Mon, 26 Mar 2012 15:25:54 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: Thanks for the quick answer. I realised, that the error didn't occur since quite a while, opposed to what our client suggested. Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. Is it possible that those could have improved the situation? For now I don't have a way of reproducing the problem, so I will have to wait for an error to happen. Until then I will consider the problem as "Solved until happens again ;-) ", since the last error occured a while back (as mentioned above). >> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. >> My suspicion/speculation what happens is the following: >> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >> Somehow this leads to problems with Locks on NFS, which leads to the crash. >Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). I had a quick look a tour NFS (NetApp), but didn't find anything useful. In case the problem persists, I will check with the coworker responsible for NetApp. I will check what sort of locking is used by postfix, since I'm not sure if postfix and dovecot are configured to use the same mechanisms. If not, I think it could be part of the Problem. >> I have no idea how to solve this problem and any help is greatly appreciated. >The only way to fully fix this is: http://wiki2.dovecot.org/Director Unfortunately that is not an option right now, but I will keep it in mind. Thanks again. From Attila.Sipos at netcall.com Mon Mar 26 18:30:24 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:30:24 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332775379.26095.130.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> OK, clearly I am not experienced enough in IMAP to argue. I am sure the imap-protocol people will tell me to get lost! Thanks for your time. If you could possibly tell me how to know if an IMAP "UID COPY" is successful, I would appreciate it. Basically I'm moving a message from one folder to another. I thought I could issue a COPY command, check for success, then delete the email from the source folder. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:23 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" > > response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response > > if the email with the supplied UID had already been known to be > > copied successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : > 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 > 1BB > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:38:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:38:26 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332776306.26095.139.camel@innu> On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. From Attila.Sipos at netcall.com Mon Mar 26 18:51:19 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:51:19 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332776306.26095.139.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> thanks. I find it odd that the 2nd copy attempt returns OK. I would've thought the "expungeissued" reason would still stand. For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:38 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 19:08:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 19:08:44 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332778124.26095.141.camel@innu> Note how Dovecot sent the client EXPUNGE notification. Prior to that client knew that the message existed. After that client knows that the message no longer exists. It was only during the COPY command that client didn't know that the message had already been expunged. On Mon, 2012-03-26 at 16:51 +0100, Attila Sipos wrote: > thanks. > I find it odd that the 2nd copy attempt returns OK. > I would've thought the "expungeissued" reason would still stand. > > For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? > > Regards > Attila > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 16:38 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > > Thanks for your time. If you could possibly tell me how to know if an > > IMAP "UID COPY" is successful, I would appreciate it. > > Basically I'm moving a message from one folder to another. I thought > > I could issue a COPY command, check for success, then delete the email > > from the source folder. > > What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. > > If the client sees that some UID exists, but another session deletes it, the COPY will fail: > > a fetch 1 uid > * 1 FETCH (UID 820) > a OK Fetch completed. > b uid copy 820 Trash > * 1 EXPUNGE > b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. > c uid copy 820 Trash > c OK No messages copied. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From ncjeffgus at zimage.com Mon Mar 26 22:11:40 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:11:40 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6D65DC.7030304@tlinx.org> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> <4F6D65DC.7030304@tlinx.org> Message-ID: <1332789100.28702.7.camel@sally> On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: > Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of > the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can > use large buffers (~16MB gives best results on my local Gbit network), > but it > misses all those pesky acls and extended attrs, not to mention file > perms...*sigh* Compare that to the I/O done 4k at a time by many older > utils... cp -ax: real 0m3.088s user 0m0.034s sys 0m3.054s rsync -axHAX real 0m15.850s user 0m19.314s sys 0m8.816s dsync's time was over six minutes. Each time I cleared out the destination folder. dsync is doing something that is taking much, much, much longer to do. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:25:28 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:25:28 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <1332789928.28702.16.camel@sally> On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > > > That didn't seem to make much of a difference. On a 3.1GB backup it > > shaved off 5 seconds. dsync's time was over 6 minutes with or without > > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > > It seems to me that dsync *should* be able to be just as fast, but it > > currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per > second. Depending on the harddisks used, that would be quite a challenge. > If you use rsync to only transfer the files that changed (based on file > modification time) you may or may not miss files that have changed but > still have the same time stamp. I assume you didn't use the --checksum > parameter to rsync, right? The destination directory was empty. I was doing a full backup. > dsync does so much more than simply copy some files... I realize that. I am hoping that the extra data that dsync has available to it would improve the speed of syncing backups. My baseline testing of simply backing up a mailbox to an empty directory shows that dsync is takes way too long to backup a single mailbox. I have over a terabyte of data to backup. I'm currently using rsync and it must traverse tens of thousands of files and check the time information. It works, but I was hoping dsync would be a better solution. dsync should be able to sync faster, by gulping in the index information for each mailbox. I haven't even moved to the point of sync'ing since the baseline test of simply exporting a mailbox is so slow. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:34:50 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:34:50 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <1332790490.28702.23.camel@sally> On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: > > I personally think that Dovecot could gain much more if the amount of > work going into fixing or improving dsync would go into making Dovecot > to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences > compared to the "low latency" and consistency problem free local file > systems, but its fruits are also sweeter for the long term. :) Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. I think a solid, fast dsync implementation would be very useful for a large installation. ...Jeff From tomislav.mihalicek at gmail.com Tue Mar 27 00:08:36 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 14:08:36 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <33544762.post@talk.nabble.com> When i put service=lib-storage to users in ldap everything works. Is this a bug? cartman dovecot: auth: Debug: master in: USER 1 user at example.net service=lib-storage Tomislav Mihalicek wrote: > > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x > -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544762.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Tue Mar 27 00:14:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 27 Mar 2012 00:14:17 +0300 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > > Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Fixed: http://hg.dovecot.org/dovecot-2.1/rev/339b1337aab0 From andrei.michescu at miau.ca Tue Mar 27 01:14:22 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Mon, 26 Mar 2012 18:14:22 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Hello Timo, Thank you very much for planning a redesign of the dsycn and for opening this discussion. As I can see from the replies that came until now everybody misses the main point of IMAP: IMAP has been designed to work as a disconnected, high-latency data store. To make this more clear: once and IMAP client finishes the synchronization with the server, both have client and server have a consistent state of the mailbox. After this both the "client" and the "server" act like master for their own local copy (on the "server" new emails get created etc, on the "client" existing emails get changed (flags) and moved, and new emails appear (sent items)). So the protocol is designed, originally, to handle the master-master replication. And as this it make sense a deployment global-wide, where servers work independently and from time to time they "merge" the changes. This being said and acknowledged here are my 2 cents: I think that the current '1 brain / 2 workers' seems to be the correct model. The "the client" connects to the "server" and pushes the local changes and after retrieves the updated/new items from the "server". "The brain" considers first server as the "local storage" and the second server as "server storage". For the split design, "come to the same conclusion of the state" is very race-condition prone. As long as the algorithm is kept as you described it in the original document then the backups should really be incremental (because you only do the changes since last sync). As the most changes are "metadata-only" the sync can be pretty fast by merging indexes. Thank you, Andrei > In case anyone is interested in reading (and maybe helping!) with a dsync > redesign that's intended to fix all of its current problems, here are some > possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising > as an algorithm class problem :) If anyone has thoughts on how to solve > it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but > those should be much simpler. > > > !DSPAM:4f6cea4c260302917022693! > > From abruce at tumnus.co.nz Tue Mar 27 03:57:04 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Tue, 27 Mar 2012 13:57:04 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage Message-ID: Hi there, We're setting up a Dovecot virtual email setup - we've got everything working perfect with LDAP logins authenticating against AD and so forth, but we're having issues with retrieving the maxStorage value from AD (this is a pre-setup field in AD that we'd like to use to set per user quotas). In our LDAP lookup, we have the maxStorage entry listed under user_attrs for the quota (user_attrs = maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see it trying to get the entry, but it fails with: Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=maxStorage Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no fields returned by the server At this point, we then see the default quota applied. If we change the name of the field from maxStorage to instanceType we see the value show up in the logs and passed through to the quota system and applied successfully: Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=instanceType Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: instanceType(quota_rule=*:storage=%$M)=*:storage=4M Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 username at site quota_rule=*:storage=4M Which seems a bit weird. If we use ldapsearch and pass it the same search string and look for the field maxStorage, we clearly see the field and the value being returned. The result looks the same if we also lookup instanceType. We're using Dovecot 2.0.9. Does anyone have any idea as to why we can't use this field? Thanks, Andrew From jtam.home at gmail.com Tue Mar 27 04:16:24 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 26 Mar 2012 18:16:24 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >> However, I noticed a strange thing: querying what would have been >> deleted >> >> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >> >> showed many date.saved values are clustered around the same timestamp, >> even among different user's Trash mailbox. >> ... >> I can't explain why many different users would have messages with the >> same (or closeby) date.saved value. > > Which mailbox format? With Maildir the date.saved is taken from > dovecot.index.cache file, and in some cases that might get dropped. If > it does, then it fallbacks to using the file's ctime. mbox. A further look into this reveals that the clustered date.saved values are the earliest values for every mailbox in the system. This timestamp is close to the time I was testing "doveadm ... -A", so the likely explanation is that I accidentally deleted/updated these values using some variation of doveadm, even though I remember confining my testing to query/search/fetch. This appears to be a case of PEBKAC. These "wrong" values shouldn't cause problems with expunge queries since they err on the side of safety. Thanks for the insight though. Joseph Tam From koshikov at gmail.com Tue Mar 27 09:14:25 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Tue, 27 Mar 2012 09:14:25 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: Message-ID: <20120327091425.73963576@jimbo> On Tue, 27 Mar 2012 13:57:04 +1300 Bruce, Andrew wrote: > Hi there, > > We're setting up a Dovecot virtual email setup - we've got everything > working perfect with LDAP logins authenticating against AD and so > forth, but we're having issues with retrieving the maxStorage value > from AD (this is a pre-setup field in AD that we'd like to use to set > per user quotas). > > In our LDAP lookup, we have the maxStorage entry listed under > user_attrs for the quota (user_attrs = > maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > it trying to get the entry, but it fails with: > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=maxStorage > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > fields returned by the server > > At this point, we then see the default quota applied. > Try to change your quota rule to be like: maxStorage=quota_rule=*:bytes=%$ ^^^^^^^^^ And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > > If we change the name of the field from maxStorage to instanceType we > see the value show up in the logs and passed through to the quota > system and applied successfully: > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=instanceType > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > instanceType(quota_rule=*:storage=%$M)=*:storage=4M > Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 > username at site quota_rule=*:storage=4M > > > Which seems a bit weird. > > If we use ldapsearch and pass it the same search string and look for > the field maxStorage, we clearly see the field and the value being > returned. The result looks the same if we also lookup instanceType. > > We're using Dovecot 2.0.9. > > Does anyone have any idea as to why we can't use this field? > > Thanks, > > Andrew From luca.palazzo at unict.it Tue Mar 27 09:57:32 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 27 Mar 2012 08:57:32 +0200 Subject: [Dovecot] 2.1.2 Corrupted squat uidlist Message-ID: <4F7164DC.7010706@unict.it> Hi Timo and All, after upgrading to 2.1.2 i'm getting a lot of these messages: Error: Corrupted squat uidlist file XXXXXX wrong indexid I did not have them before. Ideas? Luca From bob at db.org Tue Mar 27 10:12:42 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Tue, 27 Mar 2012 09:12:42 +0200 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332775126.26095.127.camel@innu> References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: 2012/3/26 Timo Sirainen : > Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This > should fix it: > > http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 Nope; exactly same result; body field contains just the empty line. -- B?rd Johannessen From nmilas at noa.gr Tue Mar 27 11:13:35 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 27 Mar 2012 11:13:35 +0300 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C5741.3000408@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> <4F6C5741.3000408@univ-evry.fr> Message-ID: <4F7176AF.6000607@noa.gr> On 23/3/2012 12:58 ??, Alain DEFRANCE wrote: > so if i understand correctly i can mix the 2 quota_rule ? > the one who came from ldap user_attrs (quota_rule=*:bytes=%$) > and the other which from quota_rule2 = Trash:storage=+3%% Actually, in user_attrs you define the applicable ldap attributes and associated info. Rules are specified in the plugin {} section, but quota values may be overridden by ldap attribute values (but not for Trash). > in your case you add 3% quota more for Trash ? > Am i write ? Yes. Nick From janfrode at tanso.net Tue Mar 27 12:47:10 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 27 Mar 2012 11:47:10 +0200 Subject: [Dovecot] doveadm purge on clusterfs Message-ID: <20120327094710.GA10878@dibs.tanso.net> Since doveadm service proxying apparently doesn't work with dovecot v2.0, we need to find a way to safely run doveadm purge on the host the user is logged into. Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? -jf From pw at wk-serv.de Tue Mar 27 13:11:59 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Tue, 27 Mar 2012 12:11:59 +0200 Subject: [Dovecot] Merge mails from two mail_locations Message-ID: <4F71926F.30500@wk-serv.de> Hi guys, recently I had some trouble with my ocfs2 cluster and it unmounted itself from /var/mail. Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. Regards Patrick From jacek at hapay.pl Tue Mar 27 14:20:14 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:20:14 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time Message-ID: <4F71A26E.5030400@hapay.pl> Hi all, I want to start new server with Postfix (I still have qmail ) and I think I have a problem with authentication in dovecot - it takes 3 seconds. Is this normal time? My configuration: Usernames: from MySQL Passwords: from Active Directory dovecot -n # 1.1.20: /etc/dovecot.conf # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 log_path: /var/log/dovecot.log protocols: pop3 pop3s imap imaps listen: * ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem ssl_key_file: /etc/pki/tls/private/iRedMail.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_uid: 500 mail_gid: 500 mail_location: maildir:/xxx/%u:INDEX=/xxx/%u mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota zlib mail_plugins(imap): quota imap_quota zlib mail_plugins(pop3): quota zlib mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 lda: postmaster_address: postmaster at domain.com mail_plugins: cmusieve quota log_path: /var/log/sieve.log auth default: mechanisms: plain login default_realm: infor.pl username_format: %Lu debug: yes debug_passwords: yes passdb: driver: pam passdb: driver: sql args: /etc/dovecot-ldap.conf userdb: driver: sql args: /etc/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/dovecot-auth mode: 438 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 438 user: vmail group: vmail plugin: quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 quota: maildir quota_rule: *:storage=300M quota_rule2: *:messages=0 expire: Trash 7 Trash/* 7 Junk 30 expire_dict: proxy::expire auth_socket_path: /var/run/dovecot/auth-master sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve dict: expire: db:/xxx/expire.db grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf driver = mysql default_pass_scheme = CRYPT connect = host=localhost dbname=xxx user=xxx password=xxx #password_query = SELECT password FROM mailbox WHERE username='%u' AND active='1' user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' postfix: postfix-2.5.9-5.ired dovecot: dovecot-1.1.20-1_98.el5 mysql: mysql-server-5.0.77-4.el5_6.6 This is not a TCP connection problem, because i have results from tcpdump and wireshark. There is a information that Active directory is answering with password in 0,2 second. Regards Jacek From jacek at hapay.pl Tue Mar 27 14:39:32 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:39:32 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time [SOLVED] In-Reply-To: <4F71A26E.5030400@hapay.pl> References: <4F71A26E.5030400@hapay.pl> Message-ID: <4F71A6F4.7030203@hapay.pl> W dniu 27.03.2012 13:20, Jacek Kowalski pisze: > Hi all, > > I want to start new server with Postfix (I still have qmail ) and I > think I have a problem with authentication in dovecot - it takes 3 > seconds. Is this normal time? > > My configuration: > Usernames: from MySQL > Passwords: from Active Directory > > > dovecot -n > > # 1.1.20: /etc/dovecot.conf > # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 > log_path: /var/log/dovecot.log > protocols: pop3 pop3s imap imaps > listen: * > ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem > ssl_key_file: /etc/pki/tls/private/iRedMail.key > login_dir: /var/run/dovecot/login > login_executable(default): /usr/libexec/dovecot/imap-login > login_executable(imap): /usr/libexec/dovecot/imap-login > login_executable(pop3): /usr/libexec/dovecot/pop3-login > mail_uid: 500 > mail_gid: 500 > mail_location: maildir:/xxx/%u:INDEX=/xxx/%u > mail_executable(default): /usr/libexec/dovecot/imap > mail_executable(imap): /usr/libexec/dovecot/imap > mail_executable(pop3): /usr/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota zlib > mail_plugins(imap): quota imap_quota zlib > mail_plugins(pop3): quota zlib > mail_plugin_dir(default): /usr/lib/dovecot/imap > mail_plugin_dir(imap): /usr/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 > lda: > postmaster_address: postmaster at domain.com > mail_plugins: cmusieve quota > log_path: /var/log/sieve.log > auth default: > mechanisms: plain login > default_realm: infor.pl > username_format: %Lu > debug: yes > debug_passwords: yes > passdb: > driver: pam > passdb: > driver: sql > args: /etc/dovecot-ldap.conf > userdb: > driver: sql > args: /etc/dovecot-mysql.conf > socket: > type: listen > client: > path: /var/spool/postfix/dovecot-auth > mode: 438 > user: postfix > group: postfix > master: > path: /var/run/dovecot/auth-master > mode: 438 > user: vmail > group: vmail > plugin: > quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 > quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 > quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 > quota: maildir > quota_rule: *:storage=300M > quota_rule2: *:messages=0 > expire: Trash 7 Trash/* 7 Junk 30 > expire_dict: proxy::expire > auth_socket_path: /var/run/dovecot/auth-master > sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve > dict: > expire: db:/xxx/expire.db > > > > grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf > > driver = mysql > default_pass_scheme = CRYPT > connect = host=localhost dbname=xxx user=xxx password=xxx > #password_query = SELECT password FROM mailbox WHERE username='%u' AND > active='1' > user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, > '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule > FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' > > > postfix: postfix-2.5.9-5.ired > dovecot: dovecot-1.1.20-1_98.el5 > mysql: mysql-server-5.0.77-4.el5_6.6 > > > > This is not a TCP connection problem, because i have results from > tcpdump and wireshark. There is a information that Active directory is > answering with password in 0,2 second. > > > Regards > > Jacek Ok. Problem Solved. It was "passdb pam" problem. Regards Jacek From campbell at cnpapers.com Tue Mar 27 17:40:11 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 27 Mar 2012 10:40:11 -0400 Subject: [Dovecot] Namespace, prefix questions Message-ID: <4F71D14B.2010301@cnpapers.com> We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) disable_plaintext_auth = no listen = * mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mbox_write_locks = fcntl namespace { hidden = yes inbox = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = separator = / type = private } namespace { hidden = yes list = no location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = "#mbox/" separator = / type = private } namespace { hidden = yes list = no location = prefix = mail/ separator = / type = private } namespace { hidden = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = ~/mail/ separator = / type = private } namespace { hidden = yes list = no location = prefix = ~%u/mail/ separator = / type = private } passdb { driver = pam } protocols = pop3 imap ssl_cert = Message-ID: <87obrixcyp.fsf@algae.riseup.net> Timo Sirainen writes: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: thank you for opening this discussion about dsync! besides the problems I've encountered with dsync, there are a couple things that I think would be great to build into the new vision of the protocol. One would be the ability to perform *intelligent* incremental/rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it would be more intelligent if this were baked into the backup process. Secondly, being able to filter out mailboxes could result in much more efficient syncing. Now there is the capability to operate on only specific mailboxes, but this doesn't scale well when I am trying to backup thousands of users and I want to omit the Spam and Trash folders from the sync. I would have to get a mailbox list of each user, and then iterate over each mailbox for each user, skipping the Spam and Trash folders, forking a new 'dsync backup' for each of their mailboxes, for each user. Lastly, there isn't a good method for restoring backups. I can reverse the backup process, onto the user's "live" mailbox, but that brings the user into an undesirable state (eg. their mailbox state one day ago). Better would be if their backup could be restored in such a way that the user can resolve the missing pieces manually, as they know best. thanks again for your work on this, from my position dovecot is an amazing piece of software, the only part that seems to have some issues is dsync and I applaud the effort to redesign to fix things! micah From me at benschumacher.com Tue Mar 27 19:49:50 2012 From: me at benschumacher.com (Ben Schumacher) Date: Tue, 27 Mar 2012 10:49:50 -0600 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: > the zlib_save question reminds me of a wish: > I think it's not possible to set zlib_save parameter per namespace (or even > mailbox). Per namespace would be something for the wish list to get rid of > the cron job method to compress archival mailboxes. > And maybe an option to add a "Z" flag to compressed maildir message files > as recommended in the wiki regarding compress crob job. +1 on this request. I have a slightly different use case -- I have both an dbox and Maildir. Incoming email goes to Maildir, but I archive off to dbox (using Thunderbird). After I archive my emails, compression seems like a reasonable choice. Any idea if this feature will be available at some point? Thanks, Ben From lists at wiesinger.com Tue Mar 27 20:28:56 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:28:56 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F8D8.6040700@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From gerhard at wiesinger.com Tue Mar 27 20:27:59 2012 From: gerhard at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:27:59 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F89F.9060903@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From CMarcus at Media-Brokers.com Tue Mar 27 22:34:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 27 Mar 2012 15:34:40 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <87obrixcyp.fsf@algae.riseup.net> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <87obrixcyp.fsf@algae.riseup.net> Message-ID: <4F721650.4030901@Media-Brokers.com> On 2012-03-27 11:47 AM, Micah Anderson wrote: > One would be the ability to perform *intelligent* incremental / > rotated backups. I can do this now by running a dsync backup > operation and then doing manual hardlinking or moving of the backup > directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it > would be more intelligent if this were baked into the backup process. There are already numerous tools that do this flawlessly - I've been using rsnapshot (which uses rsync) for this for years. I don't know if Timo should be spending his time reinventing the wheel. I'm much more interested in dsync working flawlessly to keep one or more secondary servers in sync, and leave backups to backup software. > Lastly, there isn't a good method for restoring backups. I can reverse > the backup process, onto the user's "live" mailbox, but that brings the > user into an undesirable state (eg. their mailbox state one day > ago). Better would be if their backup could be restored in such a way > that the user can resolve the missing pieces manually, as they know > best. Again, best left to the backup software I think? Although, one interesting piece that I am hopeful I'll be able to implement soon (with Timo's professional help) is the ability to easily and automatically map my rsnapshot snapshots directory to a read-only 'Backups' namespace that automatically shows the snapshots by date and time as they are produced. This way users could 'go back in time' anytime they wanted without having to call me... :) > thanks again for your work on this, from my position dovecot is an > amazing piece of software, the only part that seems to have some issues > is dsync and I applaud the effort to redesign to fix things! Ditto all of that! :) -- Best regards, Charles From stan at hardwarefreak.com Tue Mar 27 23:09:44 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 27 Mar 2012 15:09:44 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332790490.28702.23.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> Message-ID: <4F721E88.8020309@hardwarefreak.com> On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > Do you have any suggestions for a distributed replicated filesystem > that works well with dovecot? I've looked into glusterfs, but the > latency is way too high for lots of small files. They claim this problem > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > see how any of the distributed filesystems would help me. I've also > tried out ZFS, but it appears to have issues with metadata look ups with > directories that have tens or hundreds of thousands of files in them. > For me, the best filesystem is straight up ext4 running on locally > attached storage. > I think a solid, fast dsync implementation would be very useful for a > large installation. It sounds like you're in need of a more robust and capable storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or incremental snapshot capability. Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands. It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput by a factor of 2-50+ depending on thread count, and decreases IOPS and MB/s hitting the storage by about the same factor, depending on thread count. Before this patch XFS sucked at the write portion of the maildir workload due to the extremely high IOPS and MB/s hitting just the log journal, not including the actual file writes. It's parallel maildir read performance was better than any other, but the write was so bad it bogged down the storage producing high latency for everything. With the delaylog patch, XFS now trounces every filesystem at medium to high parallelism levels. Delaylog was introduced in mid 2009, included in 2.6.35 as experimental, and is the default in 2.6.39 and later. If you're a Red Hat or CentOS user it's included in 6.2. This one patch, which was 5+ years in development, dramatically changed the character of XFS with this class of metadata intensive parallel workloads. Many people with such a workload who ran from XFS in the past, as if it were the Fukushima reactor, are now adopting it in droves. What a difference a few hundred lines of very creative code can make... -- Stan From abruce at tumnus.co.nz Tue Mar 27 23:39:37 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 09:39:37 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120327091425.73963576@jimbo> References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:36, Bruce, Andrew wrote: > On 27 March 2012 19:14, Nikita Koshikov wrote: >> On Tue, 27 Mar 2012 13:57:04 +1300 >> Bruce, Andrew wrote: >> >> Hi there, >> >> We're setting up a Dovecot virtual email setup - we've got everything >> working perfect with LDAP logins authenticating against AD and so >> forth, but we're having issues with retrieving the maxStorage value >> from AD (this is a pre-setup field in AD that we'd like to use to set >> per user quotas). >> >> In our LDAP lookup, we have the maxStorage entry listed under >> user_attrs for the quota (user_attrs = >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >> it trying to get the entry, but it fails with: >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=maxStorage >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >> fields returned by the server >> >> At this point, we then see the default quota applied. >> > Try to change your quota rule to be like: > maxStorage=quota_rule=*:bytes=%$ > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >> >> If we change the name of the field from maxStorage to instanceType we >> see the value show up in the logs and passed through to the quota >> system and applied successfully: >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=instanceType >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >> username at site ? ?quota_rule=*:storage=4M >> >> >> Which seems a bit weird. >> >> If we use ldapsearch and pass it the same search string and look for >> the field maxStorage, we clearly see the field and the value being >> returned. ?The result looks the same if we also lookup instanceType. >> >> We're using Dovecot 2.0.9. >> >> Does anyone have any idea as to why we can't use this field? >> >> Thanks, >> >> Andrew Tried your suggestion Nikita, no joy unfortunately. ?It still looks like the value never gets returned from the LDAP server to Dovecot. It definitely has something in the field (equivalent of 10GB, but in bytes as suggested) and I changed the user_attrs also, but still get the same "no fields returned by the server" error message. Modifying the user_attrs to lookup from a different field (instanceType) definitely works. What exact version are you using - perhaps it's a problem with our copy of 2.0.9. Thanks, Andrew From ncjeffgus at zimage.com Tue Mar 27 23:57:41 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Tue, 27 Mar 2012 13:57:41 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F721E88.8020309@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> Message-ID: <1332881861.29480.8.camel@sally> On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: > On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > > > Do you have any suggestions for a distributed replicated filesystem > > that works well with dovecot? I've looked into glusterfs, but the > > latency is way too high for lots of small files. They claim this problem > > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > > see how any of the distributed filesystems would help me. I've also > > tried out ZFS, but it appears to have issues with metadata look ups with > > directories that have tens or hundreds of thousands of files in them. > > For me, the best filesystem is straight up ext4 running on locally > > attached storage. > > It sounds like you're in need of a more robust and capable > storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or > incremental snapshot capability. We do have a FC system that another department is using. The company dropped quite a bit of cash on it for a specific purpose. Our department does not have access it to. People are somewhat afraid of iSCSI around here because they believe it will add too much latency to the overall IO performance. They're a big believer in locally attached disks. Less features, but very good performance. We thought ZFS would provide us with a nice snapshot and backup system (with zfs send). We never got that far once we discovered that ZFS doesn't work very well in this context. Running rsync on it gave us terrible performance. > Also, you speak of a very large maildir store, with hundreds of > thousands of directories, obviously many millions of files, of 1TB total > size. Thus I would assume you have many thousands of users, if not 10s > of thousands. > > It's a bit hard to believe you're not running XFS on your storage, given > your level of parallelism. You'd get much better performance using XFS > vs EXT4. Especially with kernel 2.6.39 or later which includes the > delayed logging patch. This patch increases metadata write throughput > by a factor of 2-50+ depending on thread count, and decreases IOPS and > MB/s hitting the storage by about the same factor, depending on thread > count. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. ...Jeff From abruce at tumnus.co.nz Wed Mar 28 00:06:55 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 10:06:55 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:39, Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: >> On 27 March 2012 19:14, Nikita Koshikov wrote: >>> On Tue, 27 Mar 2012 13:57:04 +1300 >>> Bruce, Andrew wrote: >>> >>> Hi there, >>> >>> We're setting up a Dovecot virtual email setup - we've got everything >>> working perfect with LDAP logins authenticating against AD and so >>> forth, but we're having issues with retrieving the maxStorage value >>> from AD (this is a pre-setup field in AD that we'd like to use to set >>> per user quotas). >>> >>> In our LDAP lookup, we have the maxStorage entry listed under >>> user_attrs for the quota (user_attrs = >>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>> it trying to get the entry, but it fails with: >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=maxStorage >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>> fields returned by the server >>> >>> At this point, we then see the default quota applied. >>> >> Try to change your quota rule to be like: >> maxStorage=quota_rule=*:bytes=%$ >> ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ >> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >> >> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>> >>> If we change the name of the field from maxStorage to instanceType we >>> see the value show up in the logs and passed through to the quota >>> system and applied successfully: >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=instanceType >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >>> username at site ? ?quota_rule=*:storage=4M >>> >>> >>> Which seems a bit weird. >>> >>> If we use ldapsearch and pass it the same search string and look for >>> the field maxStorage, we clearly see the field and the value being >>> returned. ?The result looks the same if we also lookup instanceType. >>> >>> We're using Dovecot 2.0.9. >>> >>> Does anyone have any idea as to why we can't use this field? >>> >>> Thanks, >>> >>> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Further investigation shows that there are a few other fields that we can't retrieve in Dovecot, but can using the same search string and lookup user with ldapsearch. maxStorage is obviously one, but I tried a couple of other fields of varying types: mobile - Octet String and logonCount - Integer. Doesn't seem to be the type that restricts the search, just some fields won't return. From sorr at rightnow.com Wed Mar 28 00:38:19 2012 From: sorr at rightnow.com (Orr, Steve) Date: Tue, 27 Mar 2012 21:38:19 +0000 Subject: [Dovecot] Using getmail with sieve Message-ID: <1AA32A754D17E9478500E421F4099F9D1D13A5E8@IS-BOZ-MB02.corp.rightnow.com> I'm trying to setup a personal "mailmover" where I use getmail to retrieve remote IMAP server mail and load it into my local Dovecot then filter all email on the Dovecot server side with sieve. (I'm using Dovecot v. 2.0.9 with IMAP/Maildir.) 1) From the docs I gather that Dovecot sieve will not work as mail is loaded by getmail and I need to refilter the email after it has been loaded with getmail, right? 2) In http://wiki2.dovecot.org/HowTo/RefilterMail it says, "This HOWTO helps you create a folder for mail that needs refiltering..." But contrary to the doc I don't see any actual instructions about creating said folder. The HOWTO assumes mail already exists in a folder called "REFILTER' but doesn't say how this folder was populated. "How to?" 3) As a test I manually created the "REFILTER" folder and put my inbox mail in it (with Thunderbird) then following the HOWTO I ran my getmail script and my sieve script performed as expected. 4) How do I automate this? I tried a new getmail script to run against the inbox and tag new email so a subsequent getmail script could move it into the REFILTER folder for later sieve processing. This seems like way too much work. The HOWTO doc seems incomplete or presumes other knowledge, especially for an IMAP server and email sieve nubie. TIA, D. B. ---------------------------------------------- $ dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-131.0.15.el6.i686 i686 Red Hat Enterprise Linux Server release 6.1 (Santiago) log_path = /var/log/dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = test at host1 protocols = imap lmtp service lmtp { user = test } ssl_cert = I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that is optimized to convert mailboxes without the fancy locking that dsync needs to do. Does have (or seen) a tool that could do this? We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process. ...Jeff From stonegate at stonegate.homeip.net Wed Mar 28 01:24:59 2012 From: stonegate at stonegate.homeip.net (stonegate) Date: Tue, 27 Mar 2012 15:24:59 -0700 (PDT) Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook Message-ID: <33544803.post@talk.nabble.com> Hi, i use dovecot 2.1.3 on a gentoo system. Before i installed my new imap server box, everything worked. Now with Dovecot it does not. Problem: When i receive a new email, it does not appear in my Outlook unless i have the IMAP Inbox Folder open (highlighted selection). Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize it unless i click on the inbox folder. Before that problem occured on my old system ( i think it was dovecot as well ) the inbox folder refreshed automatically and i instantly knew when i had new emails. It was kind of like with my exchange account. Does anyone have a clue what might be wrong ? Since i did not re-install outlook or something it cannot be a client thing since no settings changed on the client side. Maybe i have to do something with dovecots config file? I think it should keep the connection to the server open or something. I?d appreciate any kind of help. Thanks alot Stoney Dovecot is capable of the following things: telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. Thats my dovecot.conf: # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-gentoo-r2 x86_64 Gentoo Base System release 2.0.3 auth_mechanisms = plain login listen = * disable_plaintext_auth = no mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } #passdb { # args = * # driver = pam #} #passdb { # args = /etc/dovecot/dovecot-sql.conf.ext # driver = sql #} passdb { driver = shadow } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = References: <1332888019.29480.17.camel@sally> Message-ID: <4F727ECE.4050305@r.paypc.com> On 3/27/2012 3:40 PM, Jeff Gustafson wrote: > I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. Is it slower than doing an IMAP APPEND over an authenticated dovecot connection? I've used a simple PERL script based on Mail::IMAPClient and Mail::Box to import 180,000+ mailboxes into dovecot's mdbox at fairly high speed, and all it does is IMAP APPENDs. (I had to shard the mailboxes because these PERL based tools exhaust RAM when run with mailboxes larger than about 600MB). On my development VM test box (32 bit Slack 13.37, 2G/2G split kernel, no RAID, Q6600 with only two cores allocated to the VM) and 8GB of DDR2 RAM does Emails=180,044 real 237m28.485s (12.5 emails/second) user 94m50.425s sys 10m09.389s 21,984,824 /mail/home I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox "transcoding" tool, since none appear to exist. To keep it simple, I/O to/from "remote" mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's worth of RAM (if one of the directions involves a remote connection), and so far when processing MIX, Maildir, and Mbox files, it's extremely fast. Adding support for [sm]dbox wouldn't appear to be problematic. At the moment, it supports everything Panda's c-client supports plus Maildir/Maildir++ (including Panda's "MIX"). Write support for Maildir's extremely UNDER-tested so far, as I've mainly used it to import Maildir hives. I've experimented with Maildir as a format, and while the one email to a file model seems like a sensible idea, it seems to simply transfer stress from one part of the system to another, mainly filesystems, and not many of those are really up for handling that many files in one directory very efficiently. None of my users have mailboxes with fewer than 100K emails in them, some have more than a million. =R= From koshikov at gmail.com Wed Mar 28 09:25:34 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Wed, 28 Mar 2012 09:25:34 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: <20120328092534.5690fa40@jimbo> On Wed, 28 Mar 2012 09:39:37 +1300 Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: > > On 27 March 2012 19:14, Nikita Koshikov wrote: > >> On Tue, 27 Mar 2012 13:57:04 +1300 > >> Bruce, Andrew wrote: > >> > >> Hi there, > >> > >> We're setting up a Dovecot virtual email setup - we've got everything > >> working perfect with LDAP logins authenticating against AD and so > >> forth, but we're having issues with retrieving the maxStorage value > >> from AD (this is a pre-setup field in AD that we'd like to use to set > >> per user quotas). > >> > >> In our LDAP lookup, we have the maxStorage entry listed under > >> user_attrs for the quota (user_attrs = > >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > >> it trying to get the entry, but it fails with: > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=maxStorage > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > >> fields returned by the server > >> > >> At this point, we then see the default quota applied. > >> > > Try to change your quota rule to be like: > > maxStorage=quota_rule=*:bytes=%$ > > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > >> > >> If we change the name of the field from maxStorage to instanceType we > >> see the value show up in the logs and passed through to the quota > >> system and applied successfully: > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=instanceType > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M > >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 > >> username at site ? ?quota_rule=*:storage=4M > >> > >> > >> Which seems a bit weird. > >> > >> If we use ldapsearch and pass it the same search string and look for > >> the field maxStorage, we clearly see the field and the value being > >> returned. ?The result looks the same if we also lookup instanceType. > >> > >> We're using Dovecot 2.0.9. > >> > >> Does anyone have any idea as to why we can't use this field? > >> > >> Thanks, > >> > >> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Show your full dovecot-ldap.conf file, also what port do you using ? maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) And show exact result of ldapsearch tool, binding under user from dovecot-ldap.conf + debug for this user when it trying to login and 'doveadm -D quota get -u $user' for this one. Also ensure that your search query returns only 1 result. We are using dovecot 2.0.19 now, but all versions of dovecot 2.0 branch was there in the past. I'm updating server since version 2.0.1 - no problem found. From janfrode at tanso.net Wed Mar 28 10:24:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 28 Mar 2012 09:24:07 +0200 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <1332888019.29480.17.camel@sally> References: <1332888019.29480.17.camel@sally> Message-ID: On Wed, Mar 28, 2012 at 12:40 AM, Jeff Gustafson wrote: > ? ? ? ?I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. I wonder if time could be saved with a > program that is optimized to convert mailboxes without the fancy locking > that dsync needs to do. Does have (or seen) a tool that could do this? Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir->mdbox convertion yesterday, using the attached script. I only converted a little over 10000 easy accounts (accounts with simple folder names, as I expect to run into problems once we start hitting accounts with trailing dot or broken latin1/utf8 characters in the folder names). I might agree it wasn't quick, but that really doesn't matter as the only downtime for the user is that he's potentially kicked out during the userdb update. -jf > ? ? ? ?We're hoping that converting away from Maildir will help us speed up > the backup processes by reducing the number of files to process. > -------------- next part -------------- A non-text attachment was scrubbed... Name: migrer-til-mdbox.sh Type: application/x-sh Size: 2131 bytes Desc: not available URL: From tomislav.mihalicek at gmail.com Wed Mar 28 11:57:07 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Wed, 28 Mar 2012 01:57:07 -0700 (PDT) Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage Message-ID: <33544816.post@talk.nabble.com> Hi Could someone explain what this strings mean in dovecot 2.1.3 debug log? Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 test1 at example.net service=lib-storage Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 test2 at example.net service=lib-storage -- View this message in context: http://old.nabble.com/Shared-mailboxes-with-dovecot-problem-service%3Dlib-storage-tp33544816p33544816.html Sent from the Dovecot mailing list archive at Nabble.com. From mafonso at hangas.net Wed Mar 28 13:13:17 2012 From: mafonso at hangas.net (Hangas) Date: Wed, 28 Mar 2012 10:13:17 +0000 (UTC) Subject: [Dovecot] dbox vs. mdbox References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: Timo Sirainen iki.fi> writes: > > 4. Are there real-world benchmarks showing measurable differences between > >maildir, sdbox mdbox? > > Not that I'm aware of. So far everyone I've tried to ask have replaced their > whole mail system and their storage, so the before/after numbers can't be > compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. The environment will be virtually the same as this is running in a virtualized environment. I'm keeping the same storage and storage network, same host hardware and about the same VM specs. Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information. My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly From campbell at cnpapers.com Wed Mar 28 17:10:32 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 28 Mar 2012 10:10:32 -0400 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: <4F731BD8.8050307@cnpapers.com> On 3/27/2012 10:40 AM, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The > client allows me to specify the root folder, but not a prefix or > namespace. I'm still struggling with some users on our new server that > have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix > definitions on the imap server? > > On some clients, like Thunderbird, I have the option of specifying > namespace OR prefix. How do these differ? I thought that the prefix > was the "name" of the namespace. > > It appears that I have to delete and re-create the account on these OE > 6 clients to make the list of folders show properly. Does that sound > right? > > This all came about because one of these OE 6 users was not able to > use their imap folders (server errors). Turns out it was one of the > users that had their folders directly under ~. So I moved them to > ~/mail, created a .subscriptions file from their .mailboxlist file and > tried everything in the world to get the folders to list properly. > Only after specifying the root folder as ~/mail after recreating the > account and restarting OE did it show properly and the folders > remained listed. My default config has this setup as the > "mail_location" parm, but blanks as the root folder don't seem to work > in this situation. I'm also wondering where I specify the "list", > "hidden" and other parms that are usually set in namespace blocks. > > dovecot -n > # 2.0.9: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) > disable_plaintext_auth = no > listen = * > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > driver = pam > } > protocols = pop3 imap > ssl_cert = ssl_key = userdb { > driver = passwd > } > protocol pop3 { > pop3_uidl_format = %08Xu%08Xv > } > > > Thanks > > steve campbell > > > > Thanks > > steve campbell > > After googling a bit, it seems that all 3 can come into play in the same or different meanings. Seems that prefix and namespace mean the same thing. Root folder can mean the same as above, but can also stand alone as an individual pointer to a personal folder that differs from from what the imap server uses. It's still not clear to me, but at least I'm getting an idea of what may or may not work. Still not sure why the null or blank prefixed namespace doesn't take precedence when nothing is set in the client. steve From kiwi at oav.net Wed Mar 28 18:50:54 2012 From: kiwi at oav.net (Xavier Beaudouin) Date: Wed, 28 Mar 2012 17:50:54 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. Message-ID: <4F73335E.2070800@oav.net> Hi there, I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. One thing used by some of powerusers are archiving mail automatically with autocreated folders based on year + month. Is there any good way to make that with sieve... One example require "fileinto"; if address :is ["From", "To"] "dovecot at dovecot.org" { fileinto "INBOX.mls.%Y.%m.dovecot"; } This will fill any mails into INBOX.mls.2012.03.dovecot uppon receiving... I don't know if some sieve guru can tell me how to do that... ? Kind regards, Xavier From stephan at rename-it.nl Wed Mar 28 19:04:48 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:04:48 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F7336A0.3070202@rename-it.nl> Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? require ["variables","date","fileinto","mailbox"]; # Extract date info if currentdate :matches "year" "*" { set "year" "${1}"; } if currentdate :matches "month" "*" { set "month" "${1}"; } # Archive Dovecot mailing list items by year and month. # Create folder when it does not exist. if header :is "list-id" "dovecot.dovecot.org" { fileinto :create "INBOX.mls.${year}.${month}.dovecot"; } The above also uses a more reliable way to detect the Dovecot mailinglist. Regards, Stephan From stan at hardwarefreak.com Wed Mar 28 19:07:59 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 28 Mar 2012 11:07:59 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332881861.29480.8.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> Message-ID: <4F73375F.3070200@hardwarefreak.com> On 3/27/2012 3:57 PM, Jeff Gustafson wrote: > We do have a FC system that another department is using. The company > dropped quite a bit of cash on it for a specific purpose. Our department > does not have access it to. People are somewhat afraid of iSCSI around > here because they believe it will add too much latency to the overall IO > performance. They're a big believer in locally attached disks. Less > features, but very good performance. If you use a software iSCSI initiator with standard GbE ports, block IO latency can become a problem, but basically in only 3 scenarios: 1. Slow CPUs or not enough CPUs/cores. This is unlikely to be a problem in 2012, given the throughput of today's multi-core CPUs. Low CPU throughput hasn't generally been the cause of software iSCSI initiator latency problems since pre-2007/8 with most applications. I'm sure some science/sim apps that tax both CPU and IO may have still had issues. Those would be prime candidates for iSCSI HBAs. 2. An old OS kernel that doesn't thread IP stack, SCSI encapsulation, and/or hardware interrupt processing amongst all cores. Recent Linux kernels do this rather well, especially with MSI-X enabled, older ones not so well. I don't know about FreeBSD, Solaris, AIX, HP-UX, Windows, etc. 3. System under sufficiently high CPU load to slow IP stack and iSCSI encapsulation processing, and or interrupt handling. Again, with today's multi-core fast CPUs this probably isn't going to be an issue, especially given that POP/IMAP are IO latency bound, not CPU bound. Most people running Dovecot today are going to have plenty of idle CPU cycles to perform the additional iSCSI initiator and TCP stack processing without introducing undue block IO latency effects. As always, YMMV. The simply path is to acquire your iSCSI SAN array and use software initiators on client hosts. In the unlikely event you do run into block IO latency issues, you simply drop an iSCSI HBA into each host suffering the latency. They run ~$700-900 USD each for single port models, and they eliminate block IO latency completely, which is one reason they cost so much. They have an onboard RISC chip and memory doing the TCP and SCSI encapsulation processing. They also give you the ability to boot diskless servers from LUNs on the SAN array. This is very popular with blade server systems, and I've done this many times myself, albeit with fibre channel HBAs/SANs, not iSCSI. Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. > We thought ZFS would provide us with a nice snapshot and backup system > (with zfs send). We never got that far once we discovered that ZFS > doesn't work very well in this context. Running rsync on it gave us > terrible performance. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. >> Also, you speak of a very large maildir store, with hundreds of >> thousands of directories, obviously many millions of files, of 1TB total >> size. Thus I would assume you have many thousands of users, if not 10s >> of thousands. >> >> It's a bit hard to believe you're not running XFS on your storage, given >> your level of parallelism. You'd get much better performance using XFS >> vs EXT4. Especially with kernel 2.6.39 or later which includes the >> delayed logging patch. This patch increases metadata write throughput >> by a factor of 2-50+ depending on thread count, and decreases IOPS and >> MB/s hitting the storage by about the same factor, depending on thread >> count. > > I've relatively new here, but I'll ask around about XFS and see if > anyone had tested it in the development environment. If they'd tested it properly, and relatively recently, I would think they'd have already replaced EXT4 on your Dovecot server. Unless others factors prevented such a migration. Or unless I've misunderstood the size of your maildir workload. -- Stan From gfinch at ldmltd.ca Wed Mar 28 19:08:07 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:08:07 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F733767.60003@ldmltd.ca> On 2012-03-28 8:50 AM, Xavier Beaudouin wrote: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? > > Kind regards, > > Xavier I don't remember where I found out how to do this, but the following is what I use: require ["fileinto", "imap4flags", "date", "variables"]; if currentdate :matches "month" "*" { set "month" "${1}"; } if currentdate :matches "year" "*" { set "year" "${1}"; } fileinto :flags "\\seen" "${year}-${month}"; Deliver/lmtp is set to allow creation of folders. -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From gfinch at ldmltd.ca Wed Mar 28 19:26:25 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:26:25 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F7336A0.3070202@rename-it.nl> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> Message-ID: <4F733BB1.5060804@ldmltd.ca> On 2012-03-28 9:04 AM, Stephan Bosch wrote: > Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: >> Hi there, >> >> I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. >> >> One thing used by some of powerusers are archiving mail automatically >> with autocreated folders based on year + month. >> >> Is there any good way to make that with sieve... >> >> One example >> >> require "fileinto"; >> >> if address :is ["From", "To"] "dovecot at dovecot.org" { >> fileinto "INBOX.mls.%Y.%m.dovecot"; >> } >> >> This will fill any mails into INBOX.mls.2012.03.dovecot uppon >> receiving... >> >> I don't know if some sieve guru can tell me how to do that... ? > > require ["variables","date","fileinto","mailbox"]; > > # Extract date info > if currentdate :matches "year" "*" { set "year" "${1}"; } > if currentdate :matches "month" "*" { set "month" "${1}"; } > > # Archive Dovecot mailing list items by year and month. > # Create folder when it does not exist. > if header :is "list-id" "dovecot.dovecot.org" { > fileinto :create "INBOX.mls.${year}.${month}.dovecot"; > } > > > The above also uses a more reliable way to detect the Dovecot > mailinglist. > > > Regards, > > Stephan Stephan, Is the "mailbox" extension the one that lets "fileinto" use ":create"? I've had a hard time trying to find a useful sieve reference that I can understand. Thank you, -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From stephan at rename-it.nl Wed Mar 28 19:31:31 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:31:31 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F733BB1.5060804@ldmltd.ca> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> <4F733BB1.5060804@ldmltd.ca> Message-ID: <4F733CE3.4050101@rename-it.nl> Op 3/28/2012 6:26 PM, Gregory Finch schreef: > On 2012-03-28 9:04 AM, Stephan Bosch wrote: >> >> require ["variables","date","fileinto","mailbox"]; >> >> # Extract date info >> if currentdate :matches "year" "*" { set "year" "${1}"; } >> if currentdate :matches "month" "*" { set "month" "${1}"; } >> >> # Archive Dovecot mailing list items by year and month. >> # Create folder when it does not exist. >> if header :is "list-id" "dovecot.dovecot.org" { >> fileinto :create "INBOX.mls.${year}.${month}.dovecot"; >> } >> >> >> The above also uses a more reliable way to detect the Dovecot >> mailinglist. > Stephan, > > Is the "mailbox" extension the one that lets "fileinto" use ":create"? > I've had a hard time trying to find a useful sieve reference that I can > understand. > > Thank you, Yes: http://tools.ietf.org/html/rfc5490#section-3.2 You can find links to specifications of the various Sieve extensions implemented for Pigeonhole here: http://pigeonhole.dovecot.org Regards, Stephan. From ncjeffgus at zimage.com Wed Mar 28 23:54:01 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:54:01 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F73375F.3070200@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> Message-ID: <1332968041.26122.19.camel@sally> On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > Locally attached/internal/JBOD storage typically offers the best > application performance per dollar spent, until you get to things like > backup scenarios, where off node network throughput is very low, and > your backup software may suffer performance deficiencies, as is the > issue titling this thread. Shipping full or incremental file backups > across ethernet is extremely inefficient, especially with very large > filesystems. This is where SAN arrays with snapshot capability come in > really handy. I'm a new employee at the company. I was a bit surprised they were not using iSCSI. They claim they just can't risk the extra latency. I believe that you are right. It seems to me that offloading snapshots and backups to an iSCSI SAN would improve things. The problem is that this company has been burned on storage solutions more than once and they are a little skeptical that a product can scale to what they need. There are some SAN vendor names that are a four letter word here. So far, their newest FC SAN is performing well. I think having more, small, iSCSI boxes would be a good solution. One problem I've seen with smaller iSCSI products is that feature sets like snapshotting are not the best implementation. It works, but doing any sort of automation can be painful. > The snap takes place wholly within the array and is very fast, without > the problems you see with host based snapshots such as with Linux LVM, > where you must first freeze the filesystem, wait for the snapshot to > complete, which could be a very long time with a 1TB FS. While this > occurs your clients must wait or timeout while trying to access > mailboxes. With a SAN array snapshot system this isn't an issue as the > snap is transparent to hosts with little or no performance degradation > during the snap. Two relatively inexpensive units that have such > snapshot capability are: How does this work? I've always had Linux create a snapshot. Would the SAN doing a snapshot without any OS buy-in cause the filesystem to be saved in an inconsistent state? I know that ext4 is pretty good at logging, but still, wouldn't this be a problem? > > http://www.equallogic.com/products/default.aspx?id=10613 > > http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html > > The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be > had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or > more host/network connection ports when equipped with dual controllers. > There are many other vendors with similar models/capabilities. I > mention these simply because Dell/HP are very popular and many OPs are > already familiar with their servers and other products. I will take a look. I might have some convincing to do. > There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, > Linux FUSE. Which were you using? If the last, that would fully > explain the suck. There is one more that I had never used before coming on board here: ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My understanding is that they were using it on the backup machines with the front end dovecot machines using ext4. I'm told the metadata issue is a ZFS thing and they have the same problem on Solaris/Nexenta. > > I've relatively new here, but I'll ask around about XFS and see if > > anyone had tested it in the development environment. > > If they'd tested it properly, and relatively recently, I would think > they'd have already replaced EXT4 on your Dovecot server. Unless others > factors prevented such a migration. Or unless I've misunderstood the > size of your maildir workload. I don't know the entire history of things. I think they really wanted to use ZFS for everything and then fell back to ext4 because it performed well enough in the cluster. Performance becomes an issue with backups using rsync. Rsync is faster than Dovecot's native dsync by a very large margin. I know that dsync is doing more than rsync, but still, seconds compared to over five minutes? That is a significant difference. The problem is that rsync can't get a perfect backup. ...Jeff From ncjeffgus at zimage.com Wed Mar 28 23:58:38 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:58:38 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: References: <1332888019.29480.17.camel@sally> Message-ID: <1332968318.26122.22.camel@sally> On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: > Why is it a problem that dsync takes a long time, when it can be done > without downtime for the users? > > I just started our maildir->mdbox convertion yesterday, using the > attached script. I only converted a little over 10000 easy accounts > (accounts with simple folder names, as I expect to run into problems > once we start hitting accounts with trailing dot or broken latin1/utf8 > characters in the folder names). I might agree it wasn't quick, but > that really doesn't matter as the only downtime for the user is that > he's potentially kicked out during the userdb update. I looked over your script. I plan on doing some trial runs with it. I think the trick where you re-run the sync and then boot the user off the connection should work pretty well. I hadn't totally fleshed out the scripting on the conversion since there is a lot more I need to do with the database and configuration files first. It appears I can use your script as a starting point for our configuration. ...Jeff > > > -jf > > > We're hoping that converting away from Maildir will help us speed up > > the backup processes by reducing the number of files to process. > > From ncjeffgus at zimage.com Thu Mar 29 00:01:19 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 14:01:19 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <4F727ECE.4050305@r.paypc.com> References: <1332888019.29480.17.camel@sally> <4F727ECE.4050305@r.paypc.com> Message-ID: <1332968479.26122.24.camel@sally> On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: > I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox > "transcoding" tool, since none appear to exist. To keep it simple, I/O > to/from "remote" mailbox (connections) are not pipelined. It won't > require more than MAXEMAILSIZE's worth of RAM (if one of the directions > involves a remote connection), and so far when processing MIX, Maildir, > and Mbox files, it's extremely fast. This sounds interesting. If it could so [sm]dbox, it would be very, very useful to large installations. ...Jeff From tss at iki.fi Thu Mar 29 01:30:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:30:34 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Message-ID: On 27.3.2012, at 1.14, Michescu Andrei wrote: > This being said and acknowledged here are my 2 cents: > > I think that the current '1 brain / 2 workers' seems to be the correct > model. The "the client" connects to the "server" and pushes the local > changes and after retrieves the updated/new items from the "server". "The > brain" considers first server as the "local storage" and the second server > as "server storage". This design makes it too easy to design it in a way that adds extra roundtrips = extra latency. It also kind of hides other problems as well. For example now dsync can way too easily just fail if something unexpected happens during dsync (e.g. mailbox gets renamed/deleted). And there are of course some bugs that I don't really understand why some people are seeing them at all. > For the split design, "come to the same conclusion of the state" is very > race-condition prone. It's race-condition prone with the brain design as well. dsync can't just lock the mailbox during its sync, since the sync can take a long time. With a "brainless" design it's clear from the beginning that there are race conditions and they need to be dealt with. From tss at iki.fi Thu Mar 29 01:43:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:43:07 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: On 23.3.2012, at 23.25, Timo Sirainen wrote: > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. Step #3 was more difficult than I first realized. I spent last two days figuring out a way to make it work, and looks like I finally did. I didn't update the document yet, but I wrote a test program: http://dovecot.org/tmp/test-dsync.c Step #2 should be easy enough. Step #4 I think I'll forget about and just implement a per-mailbox dsync lock. The main reason I wanted to get rid of locks was because a per-user lock can't work with shared mailboxes. But a per-mailbox lock is okay enough. Note that #3 allows the two dsyncs to run in parallel and send duplicate changes, just not modifying the same mailbox at the same time (which would duplicate mails due to two transactions adding the same mails). From tss at iki.fi Thu Mar 29 01:52:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:52:38 +0300 Subject: [Dovecot] Merge mails from two mail_locations In-Reply-To: <4F71926F.30500@wk-serv.de> References: <4F71926F.30500@wk-serv.de> Message-ID: On 27.3.2012, at 13.11, Patrick Westenberg wrote: > recently I had some trouble with my ocfs2 cluster and it unmounted > itself from /var/mail. > > Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. > > Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. You can use "doveadm import" to copy mails from one location to another. From tss at iki.fi Thu Mar 29 01:59:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:59:15 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: <8C621D75-BC36-4C85-B80A-28473279DFAB@iki.fi> On 27.3.2012, at 10.12, B?rd Johannessen wrote: > 2012/3/26 Timo Sirainen : >> Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This >> should fix it: >> >> http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 > > Nope; exactly same result; body field contains just the empty line. Always? It worked in my tests, and I don't see anything else wrong in the code.. From tss at iki.fi Thu Mar 29 02:12:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:12:13 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: On 22.3.2012, at 23.25, Jeff Gustafson wrote: > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > mdbox:/home/bu/testuser > > real 1m9.519s > user 1m7.592s > sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox From tss at iki.fi Thu Mar 29 02:25:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:25:57 +0300 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: On 27.3.2012, at 17.40, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? Like you noticed, these are pretty much the same things, since most clients don't understand about namespaces. And Dovecot doesn't "select" a namespace for clients. They mostly affect mailbox listing.. Like when a client asks Dovecot to list mailboxes under foo/, then Dovecot checks if a foo/ namespace exists. > On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. I have no idea how Thunderbird handles them differently. > It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? Not really. > This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. .. > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } All of your namespaces have hidden=yes. There should be (at least) one with hidden=no (this prefix= namespace in your case). I guess I'll need to add a check to have Dovecot fail if there aren't any. Also it's not necessary to duplicate the "location" setting, since it defaults to the global mail_location. > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } This namespace is unlikely to be useful to you, unless you actually have some IMAP client configured to use #mbox/ prefix. > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } This shouldn't have list=yes .. I'm not sure what even happens with it. I guess if client lists all mailboxes from root you'll get a mailbox named "~", which has a "mail" child, which has all of your mailboxes duplicated. From tss at iki.fi Thu Mar 29 02:30:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:30:51 +0300 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On 28.3.2012, at 13.13, Hangas wrote: > Timo Sirainen iki.fi> writes: > >>> 4. Are there real-world benchmarks showing measurable differences between >>> maildir, sdbox mdbox? >> >> Not that I'm aware of. So far everyone I've tried to ask have replaced their >> whole mail system and their storage, so the before/after numbers can't be >> compared. I'm very interested in knowing myself too. > > I think I can give my contribution here. I'm planning to migrate from dovecot > 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at > the time, but now I'm trying to decide the mailbox format for a 2.x > fresh install. .. > My ideia is to install a fresh server and replicate the production maildir on it > to build a test "source disk" that I'll use then to experiment the conversions > to sdbox and mdbox. > I then plan to test the performance of the dbox formats, but I can include > Maildir measurements just for the record. > > I'm open to suggestions on how to test this properly The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much. From tss at iki.fi Thu Mar 29 02:32:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:32:01 +0300 Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage In-Reply-To: <33544816.post@talk.nabble.com> References: <33544816.post@talk.nabble.com> Message-ID: On 28.3.2012, at 11.57, Tomislav Mihalicek wrote: > Could someone explain what this strings mean in dovecot 2.1.3 debug log? > > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 > test1 at example.net service=lib-storage > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 > test2 at example.net service=lib-storage Dovecot is asking a user's home directory via userdb lookup. Looks like your userdb isn't returning a home directory. There should be an error message about it? From tss at iki.fi Thu Mar 29 02:34:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:34:54 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28.3.2012, at 0.06, Bruce, Andrew wrote: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server .. > Further investigation shows that there are a few other fields that we > can't retrieve in Dovecot, Looks to me like you can't retrieve any fields from LDAP, possibly because the dn user doesn't have access to the information or some other reason. From tss at iki.fi Thu Mar 29 02:37:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:37:03 +0300 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: <28C4EF51-12FB-4B11-A3FB-54949CAF0444@iki.fi> On 27.3.2012, at 19.49, Ben Schumacher wrote: > On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: >> the zlib_save question reminds me of a wish: >> I think it's not possible to set zlib_save parameter per namespace (or even >> mailbox). Per namespace would be something for the wish list to get rid of >> the cron job method to compress archival mailboxes. >> And maybe an option to add a "Z" flag to compressed maildir message files >> as recommended in the wiki regarding compress crob job. > > +1 on this request. I have a slightly different use case -- I have > both an dbox and Maildir. Incoming email goes to Maildir, but I > archive off to dbox (using Thunderbird). After I archive my emails, > compression seems like a reasonable choice. > > Any idea if this feature will be available at some point? For v2.2 I'm hoping to have per-namespace mail settings, although I'm not sure if it actually gets implemented - so many other things to do as well. Then (among other things) you could do: namespace foo { plugin { zlib_save = gz } } From tss at iki.fi Thu Mar 29 02:42:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:42:50 +0300 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <4F71F8D8.6040700@wiesinger.com> References: <4F71F8D8.6040700@wiesinger.com> Message-ID: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. doveconf -n output? > 2.0.x: with Prefix ~/Mail > A0003 LIST "" ~/Mail/sent > A0003 OK List completed.: Works with my v2.0.19 config: x list "" ~/Mail/sent * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" x OK List completed. From tss at iki.fi Thu Mar 29 02:46:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:46:17 +0300 Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook In-Reply-To: <33544803.post@talk.nabble.com> References: <33544803.post@talk.nabble.com> Message-ID: <009AB5A8-303D-45F9-B1CF-33E26D859B0C@iki.fi> On 28.3.2012, at 1.24, stonegate wrote: > Problem: When i receive a new email, it does not appear in my Outlook unless > i have the IMAP Inbox Folder open (highlighted selection). > > Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize > it unless i click on the inbox folder. > > Before that problem occured on my old system ( i think it was dovecot as > well ) the inbox folder refreshed automatically and i instantly knew when i > had new emails. It was kind of like with my exchange account. > > Does anyone have a clue what might be wrong ? Since i did not re-install > outlook or something it cannot be a client thing since no settings changed > on the client side. > > Maybe i have to do something with dovecots config file? I think it should > keep the connection to the server open or something. I?d appreciate any kind > of help. I'm not aware of any Dovecot change or any Dovecot setting that could change this behavior in Outlook. It's the client's choice how it monitors the mailboxes. From tss at iki.fi Thu Mar 29 02:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:48:53 +0300 Subject: [Dovecot] doveadm purge on clusterfs In-Reply-To: <20120327094710.GA10878@dibs.tanso.net> References: <20120327094710.GA10878@dibs.tanso.net> Message-ID: On 27.3.2012, at 12.47, Jan-Frode Myklebust wrote: > Would it be OK to run purge in the pop/imap postlogin scripts? We > already do a conditional: > > test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today > then > touch /var/log/activemailaccounts/imap/$USER > fi > > so adding a: > > doveadm purge -u $USER > > in this section would make it run once every day the users that log in. > Does that sound like an OK solution? Yeah, should work fine. Or you should make it run in background so user's login won't slow down because of the purging. I'm not sure if simply adding & at the end works (or if the process dies after login is finished, or what happens if it runs over 30 secs which is when normally post-login script is killed by SIGALRM). From tss at iki.fi Thu Mar 29 02:49:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:49:56 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <712362F5-EDFF-4BFA-B932-1C6E19855F7D@iki.fi> On 27.3.2012, at 4.16, Joseph Tam wrote: >>> However, I noticed a strange thing: querying what would have been >>> deleted >>> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >>> showed many date.saved values are clustered around the same timestamp, >>> even among different user's Trash mailbox. >>> ... >>> I can't explain why many different users would have messages with the >>> same (or closeby) date.saved value. >> Which mailbox format? With Maildir the date.saved is taken from >> dovecot.index.cache file, and in some cases that might get dropped. If >> it does, then it fallbacks to using the file's ctime. > > mbox. Ah, with mbox there isn't any usable fallback for date.saved. If it's not in dovecot.index.cache, the current time is used. > These "wrong" values shouldn't cause problems with expunge queries since > they err on the side of safety. Right. From ncjeffgus at zimage.com Thu Mar 29 02:51:49 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 16:51:49 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> Message-ID: <1332978709.26122.29.camel@sally> On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: > On 22.3.2012, at 23.25, Jeff Gustafson wrote: > > > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > > mdbox:/home/bu/testuser > > > > real 1m9.519s > > user 1m7.592s > > sys 0m1.126s > > Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: > > - mbox:/tmp/mbox > - mbox:/tmp/mbox:INDEX=MEMORY > - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. ...Jeff From tss at iki.fi Thu Mar 29 02:53:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:53:35 +0300 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: <2DF0AEB9-7410-473C-8404-1D270711A89B@iki.fi> On 26.3.2012, at 18.25, M?ller Lukas wrote: > Thanks for the quick answer. > > I realised, that the error didn't occur since quite a while, opposed to what our client suggested. > Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. > > Is it possible that those could have improved the situation? Unlikely. >>> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) > .. > >>> My suspicion/speculation what happens is the following: >>> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >>> Somehow this leads to problems with Locks on NFS, which leads to the crash. > >> Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > > I had a quick look a tour NFS (NetApp), but didn't find anything useful. > In case the problem persists, I will check with the coworker responsible for NetApp. I guess you just had very bad luck. NetApp has a good NFS implementation and normally works fine with Dovecot. From tss at iki.fi Thu Mar 29 02:57:22 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:57:22 +0300 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6F35FA.6050207@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> Message-ID: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> On 25.3.2012, at 18.12, Charles Marcus wrote: > On 2012-03-24 9:16 AM, Timo Sirainen wrote: >> On 24.3.2012, at 14.54, Charles Marcus wrote: >> >>> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>>> You can do full backups from a filesystem snapshot, which works >>>> "well enough" (might leave some unused attachments lying around in >>>> some rare cases, but that can also happen if Dovecot crashes/dies). >>> >>> But the problem isn't with backups, but with restores, right? >> >> Ah, right. Then it gets tricky. > > Yeah, I seem to remember it was a comment like that that scared me about enabling it... > > Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) It's easy to restore a full backup. And it's easy to restore specific users if you have the full backup easily accessible (just run doveadm import with proper settings pointing to backup). What's difficult is if you just want to restore a specific user from the backup and can't easily do random access to all files. Then you'll first need to restore the user's dbox files and then somehow figure out which attachments to restore from the SIS directory. >>> Am I correct that enabling SIS as it is currently implemented would >>> break this backup tool? > >> I'm not sure. Are you running rsnapshot on live filesystem or on a >> snapshot? On live filesystem there would be race conditions. > > I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: > > file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," > rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] I'd guess that with rsnapshot + Maildir you can get duplicate Maildir files if the rsnapshot is accessing a large maildir at the same time as user is changing a message flag. Dovecot usually notices these duplicates and logs a warning about them. From tss at iki.fi Thu Mar 29 03:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 03:06:53 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332978709.26122.29.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> Message-ID: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> On 29.3.2012, at 2.51, Jeff Gustafson wrote: >> Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: >> >> - mbox:/tmp/mbox >> - mbox:/tmp/mbox:INDEX=MEMORY >> - sdbox:/tmp/sdbox > > My tests show that maildir to mdbox or sdbox backup/conversions take > about the same length in time. I noticed maybe a second or two > difference between mdbox and sdbox). On a 3.1GB mailbox either one took > about 6 minutes. Rsync, on the other hand, took less than a minute. I > will re-run the tests with a maildir to maildir backup and see how long > it takes. Try also with INDEX=MEMORY, since the problem may be related to updating the indexes. Another way to test if the problem is dsync or Dovecot's generic mail saving code is to run: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all Or if it's the mail reading code: time doveadm fetch -u user at domain text all > /dev/null From ncjeffgus at zimage.com Thu Mar 29 03:48:06 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 17:48:06 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> Message-ID: <1332982086.26122.34.camel@sally> On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/user at domain.com/Maildir "" all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test > Or if it's the mail reading code: > > time doveadm fetch -u user at domain text all > /dev/null This ran quicker than a full dsync. Only 40s for 3.1GB. rsync still beat it clocking in at 16s. I ran the fetch command twice figuring the files would get cached by the OS. ...Jeff From tss at iki.fi Thu Mar 29 04:07:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 04:07:51 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332982086.26122.34.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> Message-ID: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> On 29.3.2012, at 3.48, Jeff Gustafson wrote: > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > This tried to write to /root for some reason and failed (dovecot > 2.1.3): > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > maildir:/home/users/user at domain.com/Maildir "" all > doveadm(root): Error: chdir(/root/) failed: Permission denied > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > in group 0(root), dir owned by 0:0 mode=0550) > doveadm(root): Error: chdir(/root) failed: Permission denied > doveadm(root): Error: Can't find namespace for mailbox Trash > doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. From ncjeffgus at zimage.com Thu Mar 29 05:07:13 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 19:07:13 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> Message-ID: <1332986833.26122.36.camel@sally> On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: > On 29.3.2012, at 3.48, Jeff Gustafson wrote: > > > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > > > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > > > This tried to write to /root for some reason and failed (dovecot > > 2.1.3): > > > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > > maildir:/home/users/user at domain.com/Maildir "" all > > doveadm(root): Error: chdir(/root/) failed: Permission denied > > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > > in group 0(root), dir owned by 0:0 mode=0550) > > doveadm(root): Error: chdir(/root) failed: Permission denied > > doveadm(root): Error: Can't find namespace for mailbox Trash > > doveadm(root): Error: Can't find namespace for mailbox test > > > Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash doveadm(user at domain.com): Error: Can't find namespace for mailbox test ...Jeff From jtam.home at gmail.com Thu Mar 29 05:41:16 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 28 Mar 2012 19:41:16 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >>> Which mailbox format? With Maildir the date.saved is taken from >>> dovecot.index.cache file, and in some cases that might get dropped. If >>> it does, then it fallbacks to using the file's ctime. >> >> mbox. > > Ah, with mbox there isn't any usable fallback for date.saved. If it's > not in dovecot.index.cache, the current time is used. I'm a little confused as to why it needed a fallback. In other words, why wasn't date.saved put into the index as soon as the IMAP operation copied it into "Trash"? If this data isn't set at that time, when does it get instantiated? When I actually ask for it? Joseph Tam From tss at iki.fi Thu Mar 29 07:04:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 07:04:26 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332986833.26122.36.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> Message-ID: <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> On 29.3.2012, at 5.07, Jeff Gustafson wrote: > That fixed some errors, but it still is having some sort of trouble > with that command: > > # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all > doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash > doveadm(user at domain.com): Error: Can't find namespace for mailbox test Oh, you don't have prefix="" namespace? If you have e.g. prefix="INBOX." namespace then use: time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all From lists at wiesinger.com Thu Mar 29 08:25:17 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Thu, 29 Mar 2012 07:25:17 +0200 (CEST) Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> References: <4F71F8D8.6040700@wiesinger.com> <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> Message-ID: On Thu, 29 Mar 2012, Timo Sirainen wrote: > On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > >> After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. > > doveconf -n output? > >> 2.0.x: with Prefix ~/Mail >> A0003 LIST "" ~/Mail/sent >> A0003 OK List completed.: > > Works with my v2.0.19 config: > > x list "" ~/Mail/sent > * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" > x OK List completed. # 2.0.19: /etc/dovecot/dovecot.conf # OS: cutted for security reasons listen = * mail_full_filesystem_access = yes mail_location = mbox:~:INBOX=/var/mail/%u mbox_lazy_writes = no mbox_write_locks = fcntl passdb { driver = pam } protocols = imap service auth { unix_listener /var/run/dovecot-auth-master { group = users mode = 0660 } user = root } service imap { executable = imap postlogin } service postlogin { executable = script-login -d rawlog } ssl_cert = References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> Message-ID: <405020e5dfb341332e535e905ff183c3@alpha.zimage.com> On Thu, 29 Mar 2012 07:04:26 +0300, Timo Sirainen wrote: > On 29.3.2012, at 5.07, Jeff Gustafson wrote: > >> That fixed some errors, but it still is having some sort of trouble >> with that command: >> >> # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u >> user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> Trash >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> test > > Oh, you don't have prefix="" namespace? If you have e.g. > prefix="INBOX." namespace then use: > > time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all Oh! I should have known that was the problem. This was very, very fast. This test is maildir to maildir: # time doveadm -o mail=maildir:/home/bu/test import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m0.412s user 0m0.036s sys 0m0.088s But it was just as slow to import into mdbox: # time doveadm -o mail=mdbox:/home/bu/test2 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 7m12.738s user 6m46.161s sys 0m7.046s mbox... still pretty fast: # time doveadm -o mail=mbox:/home/bu/test3 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m58.534s user 0m52.264s sys 0m5.762s sdbox seems a little on the slow side too: # time doveadm -o mail=sdbox:/home/bu/test4 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 6m11.616s user 6m6.924s sys 0m4.579s Does information help? It seems that [sm]dbox is on the slow side for the purpose of doing backups. ...Jeff From fabio.ferrari at unimore.it Thu Mar 29 11:15:32 2012 From: fabio.ferrari at unimore.it (FABIO FERRARI) Date: Thu, 29 Mar 2012 10:15:32 +0200 (CEST) Subject: [Dovecot] Problem about dovecot Panic Message-ID: Good morning, we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some mail services in it (postfix and dovecot 2). The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). >From last week we have this dovecot problem: suddenly dovecot doesn't accept any new connections, the dovecot.log file reports lines like these Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ libdovecot.so.0 [0x36ea4362e8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) [0x36ea440b7f] -> /usr/li b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ imap [0x408179] Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 killed with signal 6 (core dumps disabled) Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command startup failed, throttling Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument and the kern.log file reports Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 can you help us? thanks in advance Fabio Ferrari From javierdemiguel at us.es Thu Mar 29 11:18:36 2012 From: javierdemiguel at us.es (=?UTF-8?Q?Javier_Miguel_Rodr=C3=ADguez?=) Date: Thu, 29 Mar 2012 10:18:36 +0200 Subject: [Dovecot] Problem about dovecot Panic In-Reply-To: References: Message-ID: <5ae4435a57981464edec3590216c7b41@us.es> We had the same problem. Reboot with an older kernel (2.6.18-274.17.1.el5 works for us). It is known bug of RHEL, see this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=681578 Regards Javier On Thu, 29 Mar 2012 10:15:32 +0200 (CEST), FABIO FERRARI wrote: > Good morning, > we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some > mail services in it (postfix and dovecot 2). > > The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). > > From last week we have this dovecot problem: suddenly dovecot doesn't > accept any new connections, the dovecot.log file reports lines like these > > Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ > libdovecot.so.0 [0x36ea4362e8] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) > [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) > [0x36ea440b7f] -> /usr/li > b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) > [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> > /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ > imap [0x408179] > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 > killed with signal 6 (core dumps disabled) > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command > startup failed, throttling > Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > > and the kern.log file reports > > Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 > Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 > > can you help us? > > thanks in advance > > Fabio Ferrari From c at roessner-network-solutions.com Thu Mar 29 13:24:18 2012 From: c at roessner-network-solutions.com (=?iso-8859-1?Q?Christian_R=F6=DFner?=) Date: Thu, 29 Mar 2012 12:24:18 +0200 Subject: [Dovecot] File/folder permission issues in 2.1.3 Message-ID: <8B296F70-22B8-487B-AD7A-47BEB8C84F62@roessner-network-solutions.com> Hi, I figured out that Dovecot does not honer secondary groups with auth/auth-worker (??), if doing LDAP/TLS stuff. I had to use file system acls to add the user "vmail" to /etc/ssl/private and to the corresponding key file: doveconf -n # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-40-generic-pae i686 Ubuntu 10.04.4 LTS auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes hostname = mail.roessner-net.de lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_access_groups = vmail mail_gid = vmail mail_location = mdbox:~/mdbox mail_plugins = autocreate quota acl fts fts_solr zlib mail_log notify mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = mdbox:%%h/mdbox prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox junkmail { special_use = \Junk } prefix = separator = / type = private } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/var/mail/virtual/shared-mailboxes.db autocreate = Trash autocreate2 = Sent autocreate3 = Drafts autocreate4 = junkmail autosubscribe = Trash autosubscribe2 = Sent autosubscribe3 = Drafts autosubscribe4 = junkmail fts = solr fts_solr = break-imap-search url=http://localhost:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:User quota::file:%h/mdbox/dovecot-quota quota_rule = *:storage=300M:messages=20000 quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = gz zlib_save_level = 6 } protocols = imap pop3 lmtp sieve service auth-worker { unix_listener auth-worker { user = vmail } user = vmail } service auth { unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = ::1 port = 24 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_ca = From mafonso at hangas.net Thu Mar 29 14:16:35 2012 From: mafonso at hangas.net (Miguel Afonso) Date: Thu, 29 Mar 2012 12:16:35 +0100 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen wrote: > > The main problem is that it's difficult to do any "real world" tests with > IMAP, especially when users are using many different kinds of IMAP clients. > So I'm very interested in hearing some numbers (and disk IO graphs for a > few weeks would be great) before your migration and after your migration, > but the numbers for your tests might not mean all that much. I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days. I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards. From me at junc.org Thu Mar 29 14:57:39 2012 From: me at junc.org (Benny Pedersen) Date: Thu, 29 Mar 2012 13:57:39 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <47266fb4a9b1a50c72ab892ac67d9744@junc.org> Den 2012-03-28 17:50, Xavier Beaudouin skrev: > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > is this valid sieve ? > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... not all sieve have date support, and imho no one have macro supported > I don't know if some sieve guru can tell me how to do that... ? why not keep lmtp ? :) http://sieve.info From stan at hardwarefreak.com Thu Mar 29 15:24:05 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 29 Mar 2012 07:24:05 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332968041.26122.19.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> Message-ID: <4F745465.1030304@hardwarefreak.com> On 3/28/2012 3:54 PM, Jeff Gustafson wrote: > On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > >> Locally attached/internal/JBOD storage typically offers the best >> application performance per dollar spent, until you get to things like >> backup scenarios, where off node network throughput is very low, and >> your backup software may suffer performance deficiencies, as is the >> issue titling this thread. Shipping full or incremental file backups >> across ethernet is extremely inefficient, especially with very large >> filesystems. This is where SAN arrays with snapshot capability come in >> really handy. > > I'm a new employee at the company. I was a bit surprised they were not > using iSCSI. They claim they just can't risk the extra latency. I The tiny amount of extra latency using a software initiator is a non argument for a mail server workload, unless the server is undersized for the workload--high CPU load and low memory constantly. As I said, in that case you drop in an iSCSI HBA and eliminate any possibility of block latency. > believe that you are right. It seems to me that offloading snapshots and > backups to an iSCSI SAN would improve things. If you get the right unit you won't understand how you ever lived without it. The snaps complete transparently, and the data is on the snap LUN within a few minutes, depending on the priority you give to internal operations, snaps/rebuilds/etc, vs external IO requests. Depending on model > The problem is that this > company has been burned on storage solutions more than once and they are > a little skeptical that a product can scale to what they need. There are More than once? More than once?? Hmm... > some SAN vendor names that are a four letter word here. So far, their > newest FC SAN is performing well. Interesting. Care to name them (off list)? > I think having more, small, iSCSI boxes would be a good solution. One > problem I've seen with smaller iSCSI products is that feature sets like > snapshotting are not the best implementation. It works, but doing any > sort of automation can be painful. As is most often the case, you get what you pay for. >> The snap takes place wholly within the array and is very fast, without >> the problems you see with host based snapshots such as with Linux LVM, >> where you must first freeze the filesystem, wait for the snapshot to >> complete, which could be a very long time with a 1TB FS. While this >> occurs your clients must wait or timeout while trying to access >> mailboxes. With a SAN array snapshot system this isn't an issue as the >> snap is transparent to hosts with little or no performance degradation >> during the snap. Two relatively inexpensive units that have such >> snapshot capability are: > > How does this work? I've always had Linux create a snapshot. Would the > SAN doing a snapshot without any OS buy-in cause the filesystem to be > saved in an inconsistent state? I know that ext4 is pretty good at > logging, but still, wouldn't this be a problem? Instead of using "SAN" as a generic term for a "box", which it is not, please use the terms "SAN" for "storage area network", "SAN array" or "SAN controller" when talking about a box with or without disks that performs the block IO shipping and other storage functions, "SAN switch" for a fiber channel switch, or ethernet switch dedicated to the SAN infrastructure. The acronym "SAN" is an umbrella covering many different types of hardware and network topologies. It drives me nuts when people call a fiber channel or iSCSI disk array a "SAN". These can be part of a SAN, but are not themselves, a SAN. If they are direct connected to a single host they are simple disk arrays, and the word "SAN" isn't relevant. Only uneducated people, or those who simply don't care to be technically correct, call a single intelligent disk box a "SAN". Ok, end rant on "SAN". Read this primer from Dell: http://files.accord.com.au/EQL/Docs/CB109_Snapshot_Basic.pdf The snapshots occur entirely at the controller/disk level inside the box. This is true of all SAN units that offer snap ability. No host OS involvement at all in the snap. As I previously said, It's transparent. Snaps are filesystem independent, and are point-in-time, or PIT copies of one LUN to another. Read up on "LUN" if you're not familiar with the term. Everything in SAN storage is based on LUNs. Now, as the document above will tell you, array based snapshots may or may not be a total backup solution for your environment. You need to educate yourself and see if this technology is a feature that fits your file backup and disaster avoidance and recovery needs. >> http://www.equallogic.com/products/default.aspx?id=10613 >> >> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html >> >> The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be >> had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or >> more host/network connection ports when equipped with dual controllers. >> There are many other vendors with similar models/capabilities. I >> mention these simply because Dell/HP are very popular and many OPs are >> already familiar with their servers and other products. > > I will take a look. I might have some convincing to do. SAN array features/performance are an easy sell. Price not so much. Each fully loaded ~24 drive SAN array is going to run you between $15k-30k USD depending on the vendor and how many spindles you need for IOPS, disk size for total storage, snap/replication features you need, expandability, etc. >> There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, >> Linux FUSE. Which were you using? If the last, that would fully >> explain the suck. > > There is one more that I had never used before coming on board here: > ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My It's a "roll your own" patch set not in mainline and not supported by any Linux distro/vendor, AFAIK. Which is why I didn't include it. > understanding is that they were using it on the backup machines with the > front end dovecot machines using ext4. I'm told the metadata issue is a > ZFS thing and they have the same problem on Solaris/Nexenta. I've never used ZFS, and don't plan to, so I can't really comment on this. That and I have no technical details of the problem. >>> I've relatively new here, but I'll ask around about XFS and see if >>> anyone had tested it in the development environment. >> >> If they'd tested it properly, and relatively recently, I would think >> they'd have already replaced EXT4 on your Dovecot server. Unless others >> factors prevented such a migration. Or unless I've misunderstood the >> size of your maildir workload. > > I don't know the entire history of things. I think they really wanted > to use ZFS for everything and then fell back to ext4 because it > performed well enough in the cluster. Performance becomes an issue with > backups using rsync. Rsync is faster than Dovecot's native dsync by a > very large margin. I know that dsync is doing more than rsync, but > still, seconds compared to over five minutes? That is a significant > difference. The problem is that rsync can't get a perfect backup. This happens with a lot of "fan boys". There was so much hype surrounding ZFS that even many logically thinking people were frothing at the mouth waiting to get their hands on it. Then, as with many/most things in the tech world, the goods didn't live up to the hype. XFS has been around since 1994, has never had hype surrounding it, has simply been steadily, substantially improved over time. It has been since day 1 the highest performance filesystem with parallel workloads, and finally overcame its last barrier preventing it from being suitable for just about any workload: metadata write performance. Which makes it faster than any FS with the maildir workload when sufficient parallelism/concurrency is present. Meaning servers with a few thousand active users will benefit. Those with 7 users won't. -- Stan From CMarcus at Media-Brokers.com Thu Mar 29 16:56:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 29 Mar 2012 09:56:39 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> Message-ID: <4F746A17.8040500@Media-Brokers.com> On 2012-03-28 7:57 PM, Timo Sirainen wrote: > It's easy to restore a full backup. And it's easy to restore specific > users if you have the full backup easily accessible (just run doveadm > import with proper settings pointing to backup). What's difficult is > if you just want to restore a specific user from the backup and can't > easily do random access to all files. Then you'll first need to > restore the user's dbox files and then somehow figure out which > attachments to restore from the SIS directory. Well, I think I'm not going to worry about this, since you recently said: On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. If we really can get these snapshots to automatically show up under a 'Backups' namespace, with each users folders under each snapshot showing by date, so they can easily 'go back in time' and retrieve anything they want from them, that totally eliminates any need for me to do individual restores... :) > I'd guess that with rsnapshot + Maildir you can get duplicate Maildir > files if the rsnapshot is accessing a large maildir at the same time > as user is changing a message flag. Dovecot usually notices these > duplicates and logs a warning about them. This won't be a problem wither, because our new system will be performing filesystem snapshots for rsnapshot to use as a source. Thanks again! -- Best regards, Charles From dovecot at r.paypc.com Thu Mar 29 21:13:58 2012 From: dovecot at r.paypc.com (Robin) Date: Thu, 29 Mar 2012 11:13:58 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F745465.1030304@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> <4F745465.1030304@hardwarefreak.com> Message-ID: <4F74A666.4000705@r.paypc.com> On 3/29/2012 5:24 AM, Stan Hoeppner wrote: > This happens with a lot of "fan boys". There was so much hype > surrounding ZFS that even many logically thinking people were frothing > at the mouth waiting to get their hands on it. Then, as with many/most > things in the tech world, the goods didn't live up to the hype. The problem with zfs especially is that there are so many different implementations, with only the commercial Sun, er, Oracle paid Solaris having ALL of the promised features and the bug-fixes to make them safely usable. For those users, with very large RAM-backed Sun, er, Oracle, hardware, it probably works well. FreeBSD and even the last versions of OpenSolaris lack fixes for some wickedly nasty box-bricking bugs in de-dup, as well as many of the "sexy" features in zpool that had people flocking to it in the first place. The bug database that used to be on the OpenSolaris portal by Sun's gone dark, but you may have some luck through archive.org. I know when I tried it out for myself using the "Community Edition" of Solaris, I did feel annoyed by the bait-and-switch, and the RAM requirements to run de-dupe with merely adequate performance were staggering if I wanted to have plenty of spare block cache left over for improving performance overall. Sun left some of the FOSS operating systems a poison pill with its CDDL licence, which is the main reason why the implementations of zfs on Linux are immature and is being "re-implemented" with US DOE sponsorship, ostensibly in a GNU compatible licence. zfs reminds me a great deal of TIFF - lots of great ideas in the "White Paper", but an elusive (or very very costly) white elephant to acquire. "Rapidly changing", "bleeding edge", and "hot & new" are not descriptors for filesystems I want to trust more than a token amount of data to. =R= From abruce at tumnus.co.nz Thu Mar 29 22:05:24 2012 From: abruce at tumnus.co.nz (Andrew Bruce) Date: Fri, 30 Mar 2012 08:05:24 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120328092534.5690fa40@jimbo> References: <20120327091425.73963576@jimbo> <20120328092534.5690fa40@jimbo> Message-ID: <4F74B274.8000600@tumnus.co.nz> On 28/03/2012 19:25, Nikita Koshikov wrote: > On Wed, 28 Mar 2012 09:39:37 +1300 > Bruce, Andrew wrote: > >> On 28 March 2012 09:36, Bruce, Andrew wrote: >>> On 27 March 2012 19:14, Nikita Koshikov wrote: >>>> On Tue, 27 Mar 2012 13:57:04 +1300 >>>> Bruce, Andrew wrote: >>>> >>>> Hi there, >>>> >>>> We're setting up a Dovecot virtual email setup - we've got everything >>>> working perfect with LDAP logins authenticating against AD and so >>>> forth, but we're having issues with retrieving the maxStorage value >>>> from AD (this is a pre-setup field in AD that we'd like to use to set >>>> per user quotas). >>>> >>>> In our LDAP lookup, we have the maxStorage entry listed under >>>> user_attrs for the quota (user_attrs = >>>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>>> it trying to get the entry, but it fails with: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=maxStorage >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server >>>> >>>> At this point, we then see the default quota applied. >>>> >>> Try to change your quota rule to be like: >>> maxStorage=quota_rule=*:bytes=%$ >>> ^^^^^^^^^ >>> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >>> >>> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>>> If we change the name of the field from maxStorage to instanceType we >>>> see the value show up in the logs and passed through to the quota >>>> system and applied successfully: >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=instanceType >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>>> Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 >>>> username at site quota_rule=*:storage=4M >>>> >>>> >>>> Which seems a bit weird. >>>> >>>> If we use ldapsearch and pass it the same search string and look for >>>> the field maxStorage, we clearly see the field and the value being >>>> returned. The result looks the same if we also lookup instanceType. >>>> >>>> We're using Dovecot 2.0.9. >>>> >>>> Does anyone have any idea as to why we can't use this field? >>>> >>>> Thanks, >>>> >>>> Andrew >> Tried your suggestion Nikita, no joy unfortunately. It still looks >> like the value never gets returned from the LDAP server to Dovecot. >> It definitely has something in the field (equivalent of 10GB, but in >> bytes as suggested) and I changed the user_attrs also, but still get >> the same "no fields returned by the server" error message. >> >> Modifying the user_attrs to lookup from a different field >> (instanceType) definitely works. >> >> What exact version are you using - perhaps it's a problem with our >> copy of 2.0.9. >> >> Thanks, >> >> Andrew > maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) > Dead on - it was a restriction of ldap port 3268 - as soon as we pointed ldapsearch at the same port, we got the same result - some of the fields were missing. It all makes perfect sense and I wish I noticed that earlier. Now need to work out why Dovecot can get the fields and username back from ldap on port 389, but it can't do the auth through it like it could with 3268. Thanks Nikita for your help. Andrew From tss at iki.fi Fri Mar 30 03:07:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 03:07:40 +0300 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6B5CB9.9080204@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: On 22.3.2012, at 19.09, Martin Schitter wrote: > Am 2012-03-22 03:46, schrieb Gedalya: >>> >>> doveadm sync/backup via impac puts the same message all over the place... >> >> Thanks Martin, I've set up a test platform to investigate this further >> but I've been short on time... > > after some debugging a few more remarks about this problem: > > the bug only appears on recursive folder hierarchies. > if you specity option "-m INBOX" everything works fine. > > for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( > > has anyone a clue how to fix this problem in the source code? http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. From tss at iki.fi Fri Mar 30 05:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 05:27:09 +0300 Subject: [Dovecot] Dovecot migration from any IMAP/POP3 server Message-ID: With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: http://wiki2.dovecot.org/Migration/Dsync The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. From neuronetv at gmail.com Fri Mar 30 00:33:57 2012 From: neuronetv at gmail.com (neuronetv) Date: Thu, 29 Mar 2012 14:33:57 -0700 (PDT) Subject: [Dovecot] newbie: keep getting same emails in mail client Message-ID: <33544893.post@talk.nabble.com> dovecot-2.0.9-2.el6_1.1.i686 I've just set up dovecot in centos 6.2 (server install) and finally got it working (kind of). I set up a unix user (not a virtual user) sent a test email to this user but in my mail client I keep getting this test email over and over again. I don't think the fault is with the email client because other emails work fine and never duplicate and I've tweaked the account settings too, so it must be something I've done wrong in the dovecot setup. Here is my dovecot.conf file: # Dovecot configuration file protocols = pop3 imap disable_plaintext_auth = no mail_location = mbox:~/mail:INBOX=/var/spool/mail/unix-username ssl_cert = References: Message-ID: <4F7521D7.2080808@gedalya.net> On 3/29/2012 10:27 PM, Timo Sirainen wrote: > With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: > > http://wiki2.dovecot.org/Migration/Dsync > > The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. > Bravo!! From joe at tao.org.uk Fri Mar 30 14:29:39 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 12:29:39 +0100 Subject: [Dovecot] How do I test if the anti spam plugin is working? Message-ID: <3779AD95-CA9E-484B-8B63-039F50B0426E@tao.org.uk> I've configured the dspam anti spam plugin, but it doesn't appear to be doing anything when I move mail between mailboxes. Can anyone help me determine what's going on? The plugin appears to be loading; at least if I don't define all the required configuration options I get a complaint in the log file. But beyond that I don't see any activity. Here's the config: babel# dovecot --version 2.1.3 The dovecot anti spam plugin config is: protocol imap { mail_plugins = $mail_plugins antispam } and plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = dspam antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_trash_pattern_ignorecase = trash;Deleted * antispam_spam_pattern = SPAM antispam_dspam_binary = /usr/local/bin/dspam antispam_dspam_args = --deliver=;--user;%n at _%d } I don't even appear to be seeing any log entries from the plugin. I've moving an email from my main mailbox into a mailbox called 'SPAM', which is how I thought that it was supposed to be triggered. Cheers, Joe From pw at wk-serv.de Fri Mar 30 14:31:58 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 13:31:58 +0200 Subject: [Dovecot] Hints for a NFS-Setup Message-ID: <4F7599AE.9080300@wk-serv.de> Hi everyone, as I have often trouble with OCFS2 I want to switch to NFS but I'm not sure how to rebuild my cluster with regard to locking and indexing problems. By now my I have a four server configuration (there are another 2 servers for outgoing mail but they can be ignored): MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 As far as I understood I will get poor performance if I'd just switch from OCFS2 to NFS (while keeping this configuration) with 4 hosts accessing the NFS-share and the index files on it and it is recommended to assign users to a specific host (http://wiki2.dovecot.org/NFS). I'm uncertain what's the meaning of "user" in this context. Is it an IMAP-User or every incoming mail? An IMAP-User assigned to a specific IMAP-Server is ok for me and I could store and profit from local index files. However, I want my incoming mailservers to be equally receiving mails. Both should accept mails for every mailbox but in this case I won't have local indexes. I would appreciate any hints. Patrick From busseniu at in.tum.de Fri Mar 30 14:37:44 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Fri, 30 Mar 2012 13:37:44 +0200 Subject: [Dovecot] Dovecot allows creation of folders outside of a user's directory Message-ID: <4F759B08.1060603@in.tum.de> Hi, in our dovecot 2.0 setup with shared folders, users can make dovecot create directories outside their mail directory. Which is a bit scary imho. The following command: . create inbox.shared.abc123 or even . create "inbox.shared.strange &ANY- characters" -- even though it will fail with a "permission denied" error -- will create a directory like "/mail/users/strange &ANY- characters". That directory will only contain a subdirectory "Maildir" and therein dovecot-acl-list. I think basically the reason for this behaviour is that Dovecot checks whether the directory has enough ACLs for the user to access it, and auto-creates the directory in the process. Is there way to avoid this auto-creation - or maybe a way to make Dovecot check whether the directory name is an existing username? Here's a config to reproduce this: # 2.0.19: /usr/local/dovecot/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-35-server x86_64 Ubuntu 10.04.4 LTS auth_username_format = %Ln disable_plaintext_auth = no mail_gid = vmail mail_home = /mail/users/%u mail_location = maildir:~/Maildir mail_plugins = " acl" mail_uid = vmail maildir_very_dirty_syncs = yes namespace default { inbox = yes location = prefix = INBOX. separator = . type = private } namespace sharedns { inbox = no list = children location = maildir:/mail/users/%%u/Maildir prefix = INBOX.shared.%%u. separator = . subscriptions = no type = shared } passdb { args = /usr/local/dovecot/etc/dovecot/users driver = passwd-file } plugin { acl = vfile:/usr/local/dovecot/etc/dovecot/global-acls:cache_secs=300 acl_shared_dict = file:/mail/vmail/shared-mailboxes.db } service auth { unix_listener auth-userdb { group = vmail mode = 0660 } } ssl_cert = Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From nick at mobilia.it Fri Mar 30 15:38:01 2012 From: nick at mobilia.it (Nick Warr) Date: Fri, 30 Mar 2012 14:38:01 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F7599AE.9080300@wk-serv.de> References: <4F7599AE.9080300@wk-serv.de> Message-ID: <4F75A929.9090902@mobilia.it> Il 30/03/2012 13.31, Patrick Westenberg ha scritto: > Hi everyone, > > as I have often trouble with OCFS2 I want to switch to NFS but > I'm not sure how to rebuild my cluster with regard to locking > and indexing problems. > > By now my I have a four server configuration (there are another 2 > servers for outgoing mail but they can be ignored): > > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 > IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 > > As far as I understood I will get poor performance if I'd just switch > from OCFS2 to NFS (while keeping this configuration) with 4 hosts > accessing the NFS-share and the index files on it and it is recommended > to assign users to a specific host (http://wiki2.dovecot.org/NFS). > > I'm uncertain what's the meaning of "user" in this context. Is it an > IMAP-User or every incoming mail? > > An IMAP-User assigned to a specific IMAP-Server is ok for me and I > could store and profit from local index files. However, I want my > incoming mailservers to be equally receiving mails. Both should accept > mails for every mailbox but in this case I won't have local indexes. > > I would appreciate any hints. > > Patrick If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). SMTP wouldn't have to be balanced in the same way, you could just use round robin in that case.. I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. From andy at xecu.net Fri Mar 30 16:03:19 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:03:19 -0400 (EDT) Subject: [Dovecot] Multiple instances In-Reply-To: References: Message-ID: <20120330083210.L21999@shell.xecu.net> Sorry to respond to an old post, but I've just recently begun implementing multiple instances to facilitate our director proxies running along with our normal dovecot config on the same servers in the cluster. This is a VERY useful feature Timo, it may need just a little refinement. On Mon, 6 Feb 2012, Timo Sirainen wrote: > # doveadm instance remove proxy Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but when I do this, while it dissapears from doveadm, it still responds to pop/imap requests, and the process continues to run. Is remove supposed to be different than say, "stop"? > It would be possible to add commands to start/shutdown some/all > instances using doveadm, but is it all that useful? I'd guess people > would have their own init.d scripts anyway doing that. Eh, in a FreeBSD port-build environment, I have to hack something in place in the rc script that gets installed, and then make sure to duplicate it every time I upgrade dovecot...not ideal. So, if dovecot had some sort of mechanism in the main config file to alert it of the additional instances to start and their config files, that would be nice. Or, if you were to add a "instance_enable" switch in the config files and then have dovecot scan the /usr/local/etc/dovecot directory for appropriate config files to automatically parse. I dunno, it doesn't feel right to push the startup of the additional instances outside of dovecot. For example, consider postfix's master.cf file. BTW somebody needs to poke the dovecot port maintaner, he still has 2.0.18, I had to manually update the port to get 2.1.3. > Anything else that could be useful related to this? Yes...we should probably be able to start instances back up as well. Other then that, looks good. Definitely a great feature. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From andy at xecu.net Fri Mar 30 16:25:11 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:25:11 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides Message-ID: <20120330091204.B22325@shell.xecu.net> I've recently set up a director proxy environment on my test servers, with the intention of deploying on our cluster soon. One thing I found confusing in the proxying documentation [1] was the first bit about their being two ways to do the authentication...either you have the proxy forward the auth to the real server for authentication, or you have the proxy authenticate it and then login to the real server with a master password. Well, we use /bin/checkpassword authentication which hooks into a variety of subsytems for various specific customer needs, and sometimes we need to know the username AND password of the user in order to determine their home directory information. So, using a master password (which requires the back-end server not getting the user password) is out. However, when we have the front-end server do a static director proxy, the problem is that authentication failures are logged on the back-end server with a source IP of the proxy, and no authentication failure with the client IP address is logged on the proxy. So, fail2ban (which is a MUST these days, at least for us) will not be able to properly filter out the brute force attackers. My solution was an alternative: I authenticate with our /bin/checkpassword on the proxy, which authenticates the user and only at that point returns the proxy=y nopassword=y switch to proxy the connection and forward the authentication. As a result, we get logs on the proxy for failed attempts, and the full username and password is supplied to the back-end server for proper processing. Food for thought in case anybody else is implementing this. Thanks, Andy [1] http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From ms at mur.at Fri Mar 30 17:15:28 2012 From: ms at mur.at (Martin Schitter) Date: Fri, 30 Mar 2012 16:15:28 +0200 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: <4F75C000.2010201@mur.at> Am 2012-03-30 02:07, schrieb Timo Sirainen: >> has anyone a clue how to fix this problem in the source code? > > http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. thanks! -- now it works! :) From tss at iki.fi Fri Mar 30 17:30:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:30:12 +0300 Subject: [Dovecot] Multiple instances In-Reply-To: <20120330083210.L21999@shell.xecu.net> References: <20120330083210.L21999@shell.xecu.net> Message-ID: <1E4998FE-26F5-4F2C-A1DB-D4EF244A15A5@iki.fi> On 30.3.2012, at 16.03, Andy Dills wrote: > On Mon, 6 Feb 2012, Timo Sirainen wrote: > >> # doveadm instance remove proxy > > Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but > when I do this, while it dissapears from doveadm, it still responds to > pop/imap requests, and the process continues to run. > > Is remove supposed to be different than say, "stop"? Yes, the "remove" is meant to simply remove already stopped instances, e.g. some test instances. You can stop instances with "doveadm -i proxy stop". Dunno if there should be another "doveadm instance stop proxy" alias for that?.. >> It would be possible to add commands to start/shutdown some/all >> instances using doveadm, but is it all that useful? I'd guess people >> would have their own init.d scripts anyway doing that. > > Eh, in a FreeBSD port-build environment, I have to hack something in place > in the rc script that gets installed, and then make sure to duplicate it > every time I upgrade dovecot...not ideal. > > So, if dovecot had some sort of mechanism in the main config file to alert > it of the additional instances to start and their config files, that would > be nice. > > Or, if you were to add a "instance_enable" switch in the config files and > then have dovecot scan the /usr/local/etc/dovecot directory for > appropriate config files to automatically parse. Hmm. Perhaps a "doveadm instance auto yes|no" command to set which instances are started up automatically when Dovecot starts up. From tss at iki.fi Fri Mar 30 17:35:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:35:33 +0300 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: On 30.3.2012, at 15.38, Nick Warr wrote: >> As far as I understood I will get poor performance if I'd just switch >> from OCFS2 to NFS (while keeping this configuration) with 4 hosts >> accessing the NFS-share and the index files on it and it is recommended >> to assign users to a specific host (http://wiki2.dovecot.org/NFS). >> >> I'm uncertain what's the meaning of "user" in this context. Is it an >> IMAP-User or every incoming mail? IMAP, POP3, LDA so everything. > If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). IP stickiness isn't enough if user uses more than one IMAP client, which is pretty common nowadays. And doesn't help at all with LDA. > I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. Anything else except Dovecot director will cause corruption with NFS. Several really large sites already use director in production. From tss at iki.fi Fri Mar 30 17:39:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:39:34 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330091204.B22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> Message-ID: On 30.3.2012, at 16.25, Andy Dills wrote: > However, when we have the front-end server do a static director proxy, the > problem is that authentication failures are logged on the back-end server > with a source IP of the proxy, and no authentication failure with the > client IP address is logged on the proxy. So, fail2ban (which is a MUST > these days, at least for us) will not be able to properly filter out the > brute force attackers. This is a simple fix (and something you should do anyway): Add the proxy's IP/netmask to login_trusted_networks setting in the remote server. For this to work with POP3 you need v2.1.2+. > My solution was an alternative: I authenticate with our /bin/checkpassword > on the proxy, which authenticates the user and only at that point returns > the proxy=y nopassword=y switch to proxy the connection and forward the > authentication. Hm. Doesn't it do that even without nopassword=y? From andy at xecu.net Fri Mar 30 17:51:00 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 10:51:00 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: References: <20120330091204.B22325@shell.xecu.net> Message-ID: <20120330104543.N22325@shell.xecu.net> On Fri, 30 Mar 2012, Timo Sirainen wrote: > On 30.3.2012, at 16.25, Andy Dills wrote: > > > However, when we have the front-end server do a static director proxy, the > > problem is that authentication failures are logged on the back-end server > > with a source IP of the proxy, and no authentication failure with the > > client IP address is logged on the proxy. So, fail2ban (which is a MUST > > these days, at least for us) will not be able to properly filter out the > > brute force attackers. > > This is a simple fix (and something you should do anyway): Add the > proxy's IP/netmask to login_trusted_networks setting in the remote > server. For this to work with POP3 you need v2.1.2+. Well, the problem isn't that my proxies would be banned; the problem is I have no way of seeing the remote IP of the failed authentication so I can ban the people who should be banned. > > My solution was an alternative: I authenticate with our /bin/checkpassword > > on the proxy, which authenticates the user and only at that point returns > > the proxy=y nopassword=y switch to proxy the connection and forward the > > authentication. > > Hm. Doesn't it do that even without nopassword=y? Perhaps...I was going by the docs which seemed to suggest that nopassword=y was how you get the proxy to forward the users authentication credentials to the back-end server. I had been trying a lot of different things, and it was only when I realized I needed to not do a static passdb on the proxy, but instead do a full authentication so that the auth failure would be logged on the proxy for fail2ban, that things began to work the way I needed. It seems obvious in retrospect, but for whatever reason the way the docs were written made me feel like having the full authentication happen on both the proxy and the backend wasn't possible. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From tss at iki.fi Fri Mar 30 18:03:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 18:03:00 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330104543.N22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> <20120330104543.N22325@shell.xecu.net> Message-ID: <0E6CE6EC-8682-4DF3-B983-DA0906B41BF2@iki.fi> On 30.3.2012, at 17.51, Andy Dills wrote: > On Fri, 30 Mar 2012, Timo Sirainen wrote: > >> On 30.3.2012, at 16.25, Andy Dills wrote: >> >>> However, when we have the front-end server do a static director proxy, the >>> problem is that authentication failures are logged on the back-end server >>> with a source IP of the proxy, and no authentication failure with the >>> client IP address is logged on the proxy. So, fail2ban (which is a MUST >>> these days, at least for us) will not be able to properly filter out the >>> brute force attackers. >> >> This is a simple fix (and something you should do anyway): Add the >> proxy's IP/netmask to login_trusted_networks setting in the remote >> server. For this to work with POP3 you need v2.1.2+. > > Well, the problem isn't that my proxies would be banned; the problem is I > have no way of seeing the remote IP of the failed authentication so I can > ban the people who should be banned. This is what the setting changes. The remote IP will be seen by the backends. > It seems obvious in retrospect, but for whatever reason the way the docs > were written made me feel like having the full authentication happen on > both the proxy and the backend wasn't possible. Oh. This is a pretty common configuration. I guess the docs could be clarified. From pw at wk-serv.de Fri Mar 30 22:37:05 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 21:37:05 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: <4F760B61.3090209@wk-serv.de> Nick Warr schrieb: > I think some of the new Dovecot (director?) software is user aware, but > I don't know if it's quite ready for production. Yes, with director it should be something like that: MTA --(lmtp)--\ /--(lmtp)--> backend1 --\ -- director -- -- NFS MTA --(lmtp)--/ \--(lmtp)--> backend2 --/ IMAP-User --> frontend1 --\ /--(imap)--> backend1 --\ -- director -- -- NFS IMAP-User --> frontend2 --/ \--(imap)--> backend2 --/ So what happens if user1 at example.tld receives a mail? - The director decides to connect to backend1 which in turn stores the mail on the NFS share and the index file locally? - Then, user1 at example.tld connects to one of the frontends. Does the director know that, earlier, this user received a mail and proxies him to backend1 too? From joe at tao.org.uk Fri Mar 30 22:44:08 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:44:08 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? Message-ID: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> I'm excited to hear that dovecot now supports imap as a mailbox type. I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. But, I can't find much documentation on imapc, other than it exists. Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? Thanks :), Joe From tss at iki.fi Fri Mar 30 22:52:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:52:18 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > I'm excited to hear that dovecot now supports imap as a mailbox type. > > I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. > > But, I can't find much documentation on imapc, other than it exists. > > Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. From joe at tao.org.uk Fri Mar 30 22:55:27 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:55:27 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> On 30 Mar 2012, at 20:52, Timo Sirainen wrote: > On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > >> I'm excited to hear that dovecot now supports imap as a mailbox type. >> >> I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. >> >> But, I can't find much documentation on imapc, other than it exists. >> >> Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? > > One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. > > http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? Joe From tss at iki.fi Fri Mar 30 22:59:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:59:47 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> Message-ID: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: >> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >> >> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >> > > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. From joe at tao.org.uk Fri Mar 30 23:02:46 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:02:46 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: On 30 Mar 2012, at 20:59, Timo Sirainen wrote: > On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: > >>> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >>> >>> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >>> >> >> But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? > > It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? Joe From tss at iki.fi Fri Mar 30 23:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:08:07 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: >> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. > > I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. From joe at tao.org.uk Fri Mar 30 23:26:24 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:26:24 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> Message-ID: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> On 30 Mar 2012, at 21:08, Timo Sirainen wrote: > On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: > >>> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. >> >> I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? > > I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. Do you think that this will reliably do the trick? Joe -------------- next part -------------- A non-text attachment was scrubbed... Name: imapc.patch Type: application/octet-stream Size: 2299 bytes Desc: not available URL: From tss at iki.fi Fri Mar 30 23:28:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:28:53 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> Message-ID: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: >> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. > > Do you think that this will reliably do the trick? > > If it works at all, I guess it should work reliably. :) From joe at tao.org.uk Fri Mar 30 23:42:13 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:42:13 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> Message-ID: <695D3A65-CFBF-4DC3-9DAC-E0C299ED0E6D@tao.org.uk> On 30 Mar 2012, at 21:28, Timo Sirainen wrote: > On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: > >>> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. >> >> Do you think that this will reliably do the trick? >> >> > > If it works at all, I guess it should work reliably. :) Hmm. Didn't appear to: % dsync -Dv -u joe at local.com -o imapc_host=mail.remoteimap.com -o imapc_port=143 -o imapc_username=joe@ remoteimap.com -o imapc_password='somepass' mirror imapc: > & /tmp/output produced the following output. What do you think? Joe. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output2.txt URL: From ka at pacific.net Sat Mar 31 00:30:05 2012 From: ka at pacific.net (Ken A) Date: Fri, 30 Mar 2012 16:30:05 -0500 Subject: [Dovecot] Problem managing mbox In-Reply-To: <1332165600.26095.75.camel@innu> References: <4F64E923.5060401@gmail.com> <1332165600.26095.75.camel@innu> Message-ID: <4F7625DD.1050109@pacific.net> I have seen this error msg too. Deleting the index files 'fixes' the error messages in the log. The indexes seem to get corrupted sometimes, when the client is logged doing: copy dest=Trash, delete, expunge Ken On 3/19/2012 9:00 AM, Timo Sirainen wrote: > On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: >> Hello, >> >> I have a problem with dovecot. seems that do not erase mail that mail >> client request to be erased. > > Are you sure the clients have actually issued the EXPUNGE command, > rather than simply marked the mail with \Deleted flag? > >> And I have this errors: >>> Error: Next message unexpectedly corrupted in mbox file >> Info: > > mbox code isn't perfect, but if this doesn't happen often it shouldn't > matter much. doveconf -n output might have been helpful in giving more > suggestions. > > > -- Ken Anderson Pacific Internet - http://www.pacific.net Latest Pacific.Net Status - http://twitter.com/pacnetstatus From dm-list-email-dovecot at scs.stanford.edu Sat Mar 31 18:38:48 2012 From: dm-list-email-dovecot at scs.stanford.edu (dm-list-email-dovecot at scs.stanford.edu) Date: Sat, 31 Mar 2012 08:38:48 -0700 Subject: [Dovecot] dovecot 2.1 breaks FTS + pre-auth? Message-ID: <878vigok53.wl@ta.scs.stanford.edu> Hi. I use dovecot in the simplest possible way, as an IMAP server in pre-auth mode over ssh or just locally over a unix-domain socket (e.g., with offlineimap, which runs much faster using dovecot for the local message store). Ideally I would like to avoid running any extra daemons or setting up anything as root. Until recently, this has worked fine by just setting the CONFIG_FILE environment variable to something in my home directory. Here is my configuration: $ export CONFIG_FILE=$HOME/etc/dovecot.conf $ dovecot -n # 2.1.3: /home/dm/etc/dovecot.conf # OS: Linux 3.2.13-1-ARCH x86_64 mail_location = maildir:~/Mail/inbox mail_plugins = " fts fts_squat" plugin { fts = squat fts_squat = partial=4 full=10 } doveconf: Error: ssl enabled, but ssl_cert not set doveconf: Fatal: Error in configuration file /home/dm/etc/dovecot.conf: ssl enabled, but ssl_cert not set Full text search used to work just fine with this configuration, and still does on a machine I have running dovecot 2.0.13. However, on the machine with 2.1, I get errors about /var/run/dovecot/index not existing. $ printf "a select INBOX\nb search text xyzzy\nc logout\n" \ | /usr/lib/dovecot/imap * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE] Logged in as dm imap(dm): Error: net_connect_unix(/var/run/dovecot/indexer) failed: No such file or directory ... Needless to say, no dovecot.index.search or dovecot.index.search.uids file is created after this error. While I can't write /var/run/dovecot, this is not a permission issue. For example, adding base_dir=/home/dm (my home directory) to the configuration file yields the same error for /home/dm/indexer. I'm guessing something has changed where imap requires an indexer daemon and doesn't launch it in pre-auth mode any more, but I can't find anything about this in the documentation. In short, if anyone can tell me how to use FTS in conjunction with pre-auth mode or point me to a working example, I would appreciate it. From ghilt at shadowprojects.org Sat Mar 31 19:55:47 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 18:55:47 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 Message-ID: <4F773713.8030904@shadowprojects.org> Hello, Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. dovecot-antispam package is compiled for dovecot 2.0.15. So, i'm trying to compile a new dovecot antispam plugin. I followed dovecot wiki but i'm running into this error : Successfully compiled dspam.c (plugin). mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying_via_save' mailbox.c: In function 'antispam_save_finish': mailbox.c:174:12: error: 'struct mail_save_context' has no member named 'copying_via_save' Failed to compile mailbox.c (plugin)! Any hint please ? Regards, -- Guillaume Hilt From e-frog at gmx.de Sat Mar 31 20:01:53 2012 From: e-frog at gmx.de (e-frog) Date: Sat, 31 Mar 2012 19:01:53 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773713.8030904@shadowprojects.org> References: <4F773713.8030904@shadowprojects.org> Message-ID: <4F773881.10907@gmx.de> On 31.03.2012 18:55, wrote Guillaume Hilt: > Hello, > > Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. > dovecot-antispam package is compiled for dovecot 2.0.15. > > So, i'm trying to compile a new dovecot antispam plugin. > I followed dovecot wiki but i'm running into this error : > Successfully compiled dspam.c (plugin). > mailbox.c: In function 'antispam_save_begin': > mailbox.c:138:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > mailbox.c: In function 'antispam_save_finish': > mailbox.c:174:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > Failed to compile mailbox.c (plugin)! > > Any hint please ? > > Regards, > Revert this patch and it should work: http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 From ghilt at shadowprojects.org Sat Mar 31 20:13:55 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 19:13:55 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773881.10907@gmx.de> References: <4F773713.8030904@shadowprojects.org> <4F773881.10907@gmx.de> Message-ID: <4F773B53.5070004@shadowprojects.org> Thanks, compilation works fine now. Guillaume Hilt Le 31/03/2012 19:01, e-frog a ?crit : > On 31.03.2012 18:55, wrote Guillaume Hilt: >> Hello, >> >> Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. >> dovecot-antispam package is compiled for dovecot 2.0.15. >> >> So, i'm trying to compile a new dovecot antispam plugin. >> I followed dovecot wiki but i'm running into this error : >> Successfully compiled dspam.c (plugin). >> mailbox.c: In function 'antispam_save_begin': >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> mailbox.c: In function 'antispam_save_finish': >> mailbox.c:174:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> Failed to compile mailbox.c (plugin)! >> >> Any hint please ? >> >> Regards, >> > > Revert this patch and it should work: > > http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 > > > > From mcbdovecot at robuust.nl Thu Mar 1 00:37:24 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Wed, 29 Feb 2012 23:37:24 +0100 (CET) Subject: [Dovecot] remove messages once downloaded In-Reply-To: References: <4F4C8DB2.9060302@mobilia.it> <6477DAB9-6840-410D-875D-5885A7E9962D@leuxner.net> Message-ID: On Tue, 28 Feb 2012, Thomas Leuxner wrote: > Am 28.02.2012 um 09:46 schrieb Thomas Leuxner: > >> Assuming this is the task you could also use: >> >> $ doveadm expunge -u jane.doe at example.org before 2012-02-01 unseen >> >> Play around with the scope a bit before you actually run it: >> >> $ doveadm search -u jane.doe at example.org before 2012-02-01 unseen | wc -l > > Of course this was supposed to read 'seen' :/ Maybe so, maybe not. I've seen people with over 2GB of unread mail in their inbox alone. Subscribed to dozens of mailing lists, and just not reading 95% of all incoming email. Don't know how pop handles 'leave on server' with regard to mail storage. Will messages loose their 'unseen' state when doing something like a TOP msgnumber maxint or something similar? Will messages loose their 'unseen' state when an RSET is issued after downloading? If you don't want people to keep a lot of mail in popboxes, quota and sending out warning messages would be the better way to handle this, I think. Applying 'magic' to customer's email might not be seen as a very polite action. Just my 2 cents.. -- Maarten From dbenfell at gmail.com Thu Mar 1 02:54:40 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 16:54:40 -0800 Subject: [Dovecot] need simpler directions on user authentication Message-ID: <20120301005440.GA5290@-e> Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From dbenfell at gmail.com Thu Mar 1 03:40:56 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 17:40:56 -0800 Subject: [Dovecot] need simpler instructions for user authentication Message-ID: Hi all, My situation is this: I have postfix running on a Linode under Arch Linux. A recent upgrade to dovecot broke the configuration. Now I get "Couldn't drop privileges: User is missing UID (see mail_uid setting)." I do not want virtual users. I do not particularly want SQL or LDAP. I do have multiple domains. I do use Maildir. I already have SSL certificates for the server which I did manage to get dovecot to quit complaining about. The documentation is driving me nuts. It keeps telling me about stuff I think I don't need. And I can't tell what I actually *do* need. All I want is for system users, including myself, to be able to get their mail. How do I do this? Thank you! David Benfell dbenfell at gmail.com From stan at hardwarefreak.com Thu Mar 1 04:15:27 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 20:15:27 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4BB559.6050405@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> Message-ID: <4F4EDBBF.40004@hardwarefreak.com> On 2/27/2012 10:54 AM, Charles Marcus wrote: > These two locations will be connected via a private Gb ethernet > connection, and each location will have its own internet connection (I > think - still waiting on some numbers to present to the owner to see > what he wants to do in that regard, but that will be my recommendation), > so bandwidth for replication won't be an issue. Say you're a boutique mail services provider or some such. In your own datacenter you have a Dovecot server w/64 processors, 512GB RAM, and 4 dual port 8Gb fiber channel cards. It's connected via 8 redundant fiber channel links to 4 SAN array units, each housing 120 x15k SAS drives, 480 drives total, ~140,000 random IOPs. This gear eats 36U of a 40U rack, and about $400,000 USD out of your wallet. In the remaining 4U at the top of the rack you have a router, with two GbE links connected to the server, and an OC-12 SONET fiber link (~$15k-20k USD/month) to a national ISP backbone. Not many years ago OC-12s comprised the backbone links of the net. OC-48s handle that today. Today OC-12s are most often used to link midsized ISPs to national ISPs, act as the internal backbone of midsized ISPs, and link large ISPs' remote facilities to the backbone. Q: How many concurrent IMAP clients could you serve with this setup before hitting a bottleneck at any point in the architecture? What is the first bottleneck you'd run into? The correct answer to this question, and the subsequent discussion that will surely take place, may open your eyes a bit, and prompt you to rethink some of your assumptions that went into the architectural decisions you've presented here. -- Stan From david at davidrenstrom.com Thu Mar 1 04:53:31 2012 From: david at davidrenstrom.com (=?us-ascii?Q?David_Renstrom?=) Date: Thu, 1 Mar 2012 03:53:31 +0100 Subject: [Dovecot] Permission problems Message-ID: <34D750D1EC41408BA8AAA495077048CD@daviddator> Hi, I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and Postfixadmin on Fedora. Almost everything works correctly now. The only thing bugging me right now is that I always have to change the access privileges of the directories in /var/run/dovecot/ manually to 777 to make it work. As you understand, I'm not particularly happy about this as it doesn't feel secure at all. And I also have to do this every time I reboot since Dovecot resets the old access priviliges (600). As you see I've got a vmail user who belongs to the mail group. Almost all directories under /var/run/dovecot belong to user root and group root. They all have the permissions rw--------. Below is some output from the log file dovecot.log: Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Mar 01 03:31:17 master: Error: service(auth): command startup failed, throttling Mar 01 03:31:46 imap-login: Info: Disconnected (auth failed, 1 attempts): method=PLAIN, rip=94.254.99.6, lip=91.123.204.174 What is wrong here and how can I correct it? Thanks in advance! /David R. From stan at hardwarefreak.com Thu Mar 1 06:16:47 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 29 Feb 2012 22:16:47 -0600 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: <4F4EF82F.6090306@hardwarefreak.com> On 2/29/2012 8:53 PM, David Renstrom wrote: > Hi, > > I've set up a Postfix and Dovecot (only IMAP) combination using MySQL and > Postfixadmin on Fedora. Almost everything works correctly now. The only > thing bugging me right now is that I always have to change the access > privileges of the directories in /var/run/dovecot/ manually to 777 to make > it work. As you understand, I'm not particularly happy about this as it > doesn't feel secure at all. And I also have to do this every time I reboot > since Dovecot resets the old access priviliges (600). > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. Is this a Fedora Dovecot RPM, or did you use source? Do you have selinux/apparmor enabled? Do you like shellfish? -- Stan From tss at iki.fi Thu Mar 1 06:30:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:30:19 +0200 Subject: [Dovecot] need simpler instructions for user authentication In-Reply-To: References: Message-ID: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> On 1.3.2012, at 3.40, David Benfell wrote: > My situation is this: I have postfix running on a Linode under Arch Linux. > A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." You're missing userdb. > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: userdb { driver = passwd } From tss at iki.fi Thu Mar 1 06:34:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 06:34:19 +0200 Subject: [Dovecot] Permission problems In-Reply-To: <34D750D1EC41408BA8AAA495077048CD@daviddator> References: <34D750D1EC41408BA8AAA495077048CD@daviddator> Message-ID: On 1.3.2012, at 4.53, David Renstrom wrote: > As you see I've got a vmail user who belongs to the mail group. Almost all > directories under /var/run/dovecot belong to user root and group root. They > all have the permissions rw--------. That's how they should be mostly. > Below is some output from the log file dovecot.log: > > Mar 01 03:31:17 auth: Fatal: net_connect_unix(auth-worker) in directory > /run/dovecot failed: Permission denied (euid=101(vmail) egid=12(mail) > missing +r perm: /run/dovecot/auth-worker, dir owned by 0:97 mode=0755) Why is your auth process running as vmail? Show your dovecot -n output From dbenfell at gmail.com Thu Mar 1 07:53:37 2012 From: dbenfell at gmail.com (David Benfell) Date: Wed, 29 Feb 2012 21:53:37 -0800 Subject: [Dovecot] SOLVED - Re: need simpler instructions for user authentication In-Reply-To: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> References: <7D2D3C20-F585-4E6E-9CBD-7602E2E57D10@iki.fi> Message-ID: <20120301055337.GC25866@-e> On Thu, Mar 01, 2012 at 06:30:19AM +0200, Timo Sirainen wrote: > > You didn't say what Dovecot version, and dovecot -n output would have been useful, but here's my guess: Add to dovecot.conf: Sorry. The Arch Linux package is dovecot-2.1.1-3. > > userdb { > driver = passwd > } It *looks* like this worked. Thank you *very* much! David Benfell benfell at parts-unknown.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From joshua at hybrid.pl Thu Mar 1 10:21:50 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Thu, 1 Mar 2012 09:21:50 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? Message-ID: Hi all, While reading the "Multiple locations, 2 servers - planning questions..." thread, it just hit me that I was planning to do something that seems to be a bit hazardous... I am setting up an high-availability server set, which consists of two (or more) servers with common disk space, all set behind redundant hardware load balancers. At first, there will be two servers and disk space will be kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what creates kind of networked RAID1 storage space). This will be done mostly to keep WWW service available in case when one of servers fails. However, if we have everything redundant, why not have the same with SMTP and POP3/IMAP? But - won't anything fail if two (or more) dovecots are accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If it is possible, that would be a good solution for the mentioned thread. If not - where are possible dangers, and are there any plans to make it possible in the future? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From jtam.home at gmail.com Thu Mar 1 10:44:58 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Thu, 1 Mar 2012 00:44:58 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid I would like to run various doveadm commands that involves all (mail) users like doveadm expunge -A mailbox Trash savedbefore 30d but any doveadm command that uses "-A" to iterate through all users will stop processing at the first account with UIDname mapping for utilities like ls, chown, etc. There are various workaround like iterating manually, or setting first_valid_uid to zero, or even reordering users to put all system accounts at the end, but is there a better way to do this? Joseph Tam From CMarcus at Media-Brokers.com Thu Mar 1 13:43:47 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 01 Mar 2012 06:43:47 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4EDBBF.40004@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> Message-ID: <4F4F60F3.5050508@Media-Brokers.com> On 2012-02-29 9:15 PM, Stan Hoeppner wrote: > Q: How many concurrent IMAP clients could you serve with this setup > before hitting a bottleneck at any point in the architecture? No idea how to calculate it... > What is the first bottleneck you'd run into? Unless this is a trick question, the OC-12 link (since it is only 644Mb), and the next bottleneck would be the 2 GbE server connections to the router (are these bonded? if so, what mode?... > The correct answer to this question, and the subsequent discussion that > will surely take place, may open your eyes a bit, and prompt you to > rethink some of your assumptions that went into the architectural > decisions you've presented here. Since the vast majority of our connections will be *local*, I'm unconcerned about the internet connect speeds (one office has a 100/10Mb Cable (Comcast Business Class) connection, the other will have a 100/100Mb fiber/ethernet connection). My main priority is that the user experience at each physical location be optimal, which is why I'm more focused on making sure each offices users are connected to only the local server for all services (file/print/mail). I also neglected to mention how each server would be physically connected to the network, which I guess I should have done, since I'm fairly sure that will be the bottleneck I should mostly be concerned about... My choices are, as I see it, single GbE connections, or add some multiport GbE cards (these Dells support up to 3 PCIe cards) and bond some ports together for each VM. 10GbE is simply not in our price range (and I don't think we need it anyway), although I did stumble on these while googling and am waiting on pricing, since they claim to be 'much cheaper': http://www.mellanox.com/ethernet/ Since neither the multi-port GbE cards or decent switches that have reliable support for bonding/teaming are really not that expensive (especially when comparing to 10GbE solutions), I don't really see any reason *not* to do this (at a minimum I'd get redundancy if one of the ports on the server failed), but I'm also not sure which mode would be best - round-robin or IEEE 802.3ad dynamic link aggregation? Obviously, I don't have the experience or expertise to answer these questions myself (never analyzed IMAP traffic to have an idea of the bandwidth each user uses, and probably wouldn't trust my efforts if I made the attempt). Hopefully, there are some people here who have a rough idea, which is why I brought this question up here. Oh - and I am/will be working with a local I.T. services company to help with the design and implementation (since obviously I don't have the experience to do this myself), and will be asking them these same questions, I just like to usually know the general answers to questions like this ahead of time, so that I know if the guys I'm hiring know what they are doing and are giving me the best options for my budget. Thanks for your thoughts... -- Best regards, Charles From mark at ecs.vuw.ac.nz Thu Mar 1 13:49:11 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:49:11 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> Message-ID: <4F4F6237.1060100@ecs.vuw.ac.nz> On 02/29/12 23:41, Timo Sirainen wrote: >> Oh, yes, this is a bug in Dovecot.. > > Hmm. Or it looked like a bug, since it replied only with "+", so I thought all auth mechanisms would have such a bug, but no.. So I'm not really sure why it's not sending more data. I don't have a Kerberos setup to test this with. v2.1's GSSAPI code is anyway identical to v2.0's. With auth debugging on a successful connection gives: Mar 2 00:33:34 bats dovecot: auth: Debug: auth client connected (pid=1584) Mar 2 00:33:34 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49116 Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 YIGaBgkqhkiG9xIBAgICAG+BijCBh6ADAgEFoQMCAQ+iezB5oAMCARKicgRwXldfEmBHqH3DiVbw7aXtx54iBNjo1Rv/KxBSK5G3TmYFm3YskYN/23EiaOQ0Tdyi4bc4jhv5cFWMpH/xM89wAFJVW8Ue27/fmCasfDWXE+i4TKA3UCm78Wy8YyiNVae8X341LspBk86R1Zl5MNRMvA== Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): Negotiated security layer Mar 2 00:33:34 bats dovecot: auth: Debug: client out: CONT 1 BQQF/wAMAAwAAAAA47846FHFUOykdXinGYvMKwH///8= Mar 2 00:33:34 bats dovecot: auth: Debug: client in: CONT Mar 2 00:33:34 bats dovecot: auth: Debug: client out: OK 1 user=mark and the failing kmail gives Mar 2 00:38:08 bats dovecot: auth: Debug: auth client connected (pid=2720) Mar 2 00:38:08 bats dovecot: auth: Debug: client in: AUTH 1 GSSAPI service=imap lip=130.195.5.13 rip=130.195.5.88 lport=143 rport=49118 resp= Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(?,130.195.5.88): Using all keytab entries Mar 2 00:38:08 bats dovecot: auth: Debug: gssapi(mark at ECS.VUW.AC.NZ,130.195.5.88): security context state completed. Mar 2 00:38:08 bats dovecot: auth: Debug: client out: CONT 1 Mar 2 00:38:08 bats dovecot: auth: Debug: client in: CONT Mar 2 00:38:08 bats dovecot: auth: GSSAPI(mark at ECS.VUW.AC.NZ,130.195.5.88): Invalid base64 data in continued response Mar 2 00:38:08 bats dovecot: auth: Debug: client out: FAIL 1 reason=Invalid base64 data in continued response so what bit of the code should I be looking at to see what happens between the "security context state completed" and the "client out"? cheers mark From tss at iki.fi Thu Mar 1 13:52:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 13:52:34 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6237.1060100@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> Message-ID: <1330602754.2081.34.camel@innu> On Fri, 2012-03-02 at 00:49 +1300, Mark Davies wrote: > so what bit of the code should I be looking at to see what happens > between the "security context state completed" and the "client out"? All of the code is in mech-gssapi.c Are these working vs. non-working Dovecots in same or different servers? From joseba.torre at ehu.es Thu Mar 1 13:52:55 2012 From: joseba.torre at ehu.es (Joseba Torre) Date: Thu, 01 Mar 2012 12:52:55 +0100 Subject: [Dovecot] need simpler directions on user authentication In-Reply-To: <20120301005440.GA5290@-e> References: <20120301005440.GA5290@-e> Message-ID: <4F4F6317.2090605@ehu.es> El 01/03/12 01:54, David Benfell escribi?: > Hi all, > > My situation is this: I have postfix running on a Linode under Arch > Linux. A recent upgrade to dovecot broke the configuration. Now I get > "Couldn't drop privileges: User is missing UID (see mail_uid setting)." > > I do not want virtual users. I do not particularly want SQL or LDAP. I > do have multiple domains. I do use Maildir. I already have SSL > certificates for the server which I did manage to get dovecot to quit > complaining about. > > The documentation is driving me nuts. It keeps telling me about stuff I > think I don't need. And I can't tell what I actually *do* need. All I > want is for system users, including myself, to be able to get their mail. > > How do I do this? > > Thank you! > David Benfell > dbenfell at gmail.com If you give us some information maybe we could help you. At least, we need the output of doveadm config -n and the relevant log lines. From mark at ecs.vuw.ac.nz Thu Mar 1 13:57:56 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Fri, 02 Mar 2012 00:57:56 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330602754.2081.34.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> Message-ID: <4F4F6444.1050107@ecs.vuw.ac.nz> On 03/02/12 00:52, Timo Sirainen wrote: >> so what bit of the code should I be looking at to see what happens >> between the "security context state completed" and the "client out"? > > All of the code is in mech-gssapi.c Yes, I'm just trying to work out the flow of the calls in and out of there. > Are these working vs. non-working Dovecots in same or different servers? All the working and non working connections are against a single dovecot instance, just using different clients. cheers mark From tss at iki.fi Thu Mar 1 14:04:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 01 Mar 2012 14:04:30 +0200 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <4F4F6444.1050107@ecs.vuw.ac.nz> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> Message-ID: <1330603470.2081.37.camel@innu> On Fri, 2012-03-02 at 00:57 +1300, Mark Davies wrote: > > On 03/02/12 00:52, Timo Sirainen wrote: > >> so what bit of the code should I be looking at to see what happens > >> between the "security context state completed" and the "client out"? > > > > All of the code is in mech-gssapi.c > > Yes, I'm just trying to work out the flow of the calls in and out of there. The problem is that that mech_gssapi_sec_context() calls gss_accept_sec_context(), which is supposed to return some output in output_token, but it doesn't. So I don't think there's anything in Dovecot code that is helpful in debugging this. You'd have to look into the GSSAPI/Kerbereros libraries. > > Are these working vs. non-working Dovecots in same or different servers? > > All the working and non working connections are against a single dovecot > instance, just using different clients. Oh. So GSSAPI in general is working, just not with kmail. I think if you downgraded to Dovecot v2.0 in your current system it would fail as well. The difference between your previously working system and currently working system is the GSSAPI/Kerberos libraries. From campbell at cnpapers.com Thu Mar 1 16:16:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 01 Mar 2012 09:16:39 -0500 Subject: [Dovecot] Multiple namespaces seems to be used at the same time Message-ID: <4F4F84C7.1060502@cnpapers.com> I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Thanks for any help. The learning curve is getting less steep, but it's still a ways off to the top of the hill. steve campbell From xchris89x at googlemail.com Thu Mar 1 17:08:28 2012 From: xchris89x at googlemail.com (Chris) Date: Thu, 1 Mar 2012 16:08:28 +0100 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: 2012/2/28 Ralf Hildebrandt : > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. Is that fixed? -- Chris From tss at iki.fi Thu Mar 1 18:15:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 18:15:05 +0200 Subject: [Dovecot] 2.1.1: subscriptions: Removing invalid entry In-Reply-To: <20120228184619.GA3250@charite.de> References: <20120228184619.GA3250@charite.de> Message-ID: On 28.2.2012, at 20.46, Ralf Hildebrandt wrote: > Admittedly, it's not a valid folder name and dovecot is right in > removing it: > > Feb 28 17:50:05 postamt dovecot: imap(kbaessle): Warning: Subscriptions file /home/k/b/kbaessle/Maildir/subscriptions: Removing invalid entry: Entw?rfe > > ...but when looking at the subscription file AFTER the removal, that file > contained an empty line. I don't think that's correct. I can't reproduce this. From slusarz at curecanti.org Thu Mar 1 20:03:46 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 11:03:46 -0700 Subject: [Dovecot] LIST-STATUS issue Message-ID: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: >> Timestamp: Thu, 01 Mar 2012 10:55:47 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. 3 seconds later... >> Timestamp: Thu, 01 Mar 2012 10:55:50 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: * OK [XPROXYREUSE] IMAP connection reused by squirrelmail-imap_proxy S: 1 OK User logged in C: 2 EXAMINE IN.dovecot (QRESYNC (1254166849 25800 18484,20260,20393,20705,20860:20868)) S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft impflag1 impflag2 impflag0 Junk seen NonJunk) S: * OK [PERMANENTFLAGS ()] Read-only mailbox. S: * 13 EXISTS S: * 0 RECENT S: * OK [UNSEEN 5] First unseen. S: * OK [UIDVALIDITY 1254166849] UIDs valid S: * OK [UIDNEXT 20869] Predicted next UID S: * OK [HIGHESTMODSEQ 25800] Highest S: 2 OK [READ-ONLY] Select completed. [...] C: 7 SEARCH RETURN (COUNT) UNSEEN S: * ESEARCH (TAG "7") COUNT 8 S: 7 OK Search completed (0.000 secs). I can confirm that I did not receive 8 dovecot list messages in 3 seconds. :) Once the mailbox is accessed, I am seeing correct counts in LIST-STATUS return again: >> Timestamp: Thu, 01 Mar 2012 10:57:28 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 8) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. I am using imapproxy for these connections, so all access are happening in the same Dovecot access. michael From tss at iki.fi Thu Mar 1 20:22:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 1 Mar 2012 20:22:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: On 1.3.2012, at 20.03, Michael M Slusarz wrote: > Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... > > UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: Only with LIST-STATUS, or also STATUS itself? And with what kind of config (doveconf -n)? Can you reproduce this? LIST-STATUS should use the exact same code as STATUS. STATUS should use pretty much the same code as SELECT. mailbox_list_index=yes makes a difference, but other than that I can't really think of a reason. From slusarz at curecanti.org Thu Mar 1 21:17:41 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Thu, 01 Mar 2012 12:17:41 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> Message-ID: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 1.3.2012, at 20.03, Michael M Slusarz wrote: > >> Looked at the hg commits since 2.1.1, but didn't see anything that >> was exactly on-point to this... >> >> UNSEEN counts seem to be broken when done in a LIST-STATUS call. I >> see this, for example: > > Only with LIST-STATUS, or also STATUS itself? And with what kind of > config (doveconf -n)? Can you reproduce this? I've been seeing this intermittently the last few days. When I notice it, I try to reproduce and never can. Then it eventually comes back, as in the first time I access the next morning. Maybe its some kind of cache/timeout issue. (e.g. - I can't reproduce right now) > LIST-STATUS should use the exact same code as STATUS. STATUS should > use pretty much the same code as SELECT. mailbox_list_index=yes > makes a difference, but other than that I can't really think of a > reason. # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-1-ARCH x86_64 auth_mechanisms = plain login namespace { inbox = yes location = maildir:~/Maildir prefix = separator = . type = private } namespace { location = maildir:~/mailtest/mboxtesting2 prefix = "#shared." separator = . type = shared } namespace { location = mbox:~/mailtest/mail prefix = "#public2." separator = . type = public } passdb { args = dovecot driver = pam } plugin { acl = vfile } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } ssl_cert = I'm running imapproxy as shown at http://wiki2.dovecot.org/HowTo/ImapcProxy. In fact, that's my config in the wiki. 8-) It's been working, but has performance issues when the Exchange server that's hard-coded as imapc_host=xxx.xxx.xxx.xxx doesn't happen to be the user's home exchange server. I'd like to point dovecot at the correct Exchange server based on an LDAP query, and in fact, have an LDAP search that works: DC=example,DC=com (&(objectCategory=person)(objectClass=user)(!(userAccountcontrol:1.2.840.113556.1.4.803:=2))(sAMAccountName=username)) With the exchange server being returned in the msExchHomeServerName property as: /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Can anybody toss me a clue? Once I get it working, I'll update the wiki. Thanks! Terry From dbenfell at gmail.com Fri Mar 2 03:23:41 2012 From: dbenfell at gmail.com (David Benfell) Date: Thu, 1 Mar 2012 17:23:41 -0800 Subject: [Dovecot] Thanks, was Re: need simpler instructions for user authentication Message-ID: Thanks to everyone who responded to my request. And I apologize that this message isn't properly threaded. I'm in trouble whenever I have to send mail from my gmail account because I automatically pull it down into the same place from which I send my regular mail (from benfell at parts-unknown.org ) and I haven't configured that to deal with the gmail account. That makes it difficult to reply properly to mail on this list which I have subscribed to with the gmail account so that I'd actually receive it even when I'm having problems with my main mail system. Timo Sirainen responded that I needed a userdb and offered that I should probably add the following to dovecot.conf : userdb { driver = passwd } As near as I can tell--and it's been over a day now--that solved the problem. Thanks! From stan at hardwarefreak.com Fri Mar 2 03:38:13 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 01 Mar 2012 19:38:13 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F502485.9070503@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > On 2012-02-29 9:15 PM, Stan Hoeppner wrote: >> Q: How many concurrent IMAP clients could you serve with this setup >> before hitting a bottleneck at any point in the architecture? > > No idea how to calculate it... The correct answer is approximately 120,000 concurrent users, based on an assumed average of ~3MB-5MB of ram consumed in all processes for each user. >> What is the first bottleneck you'd run into? > > Unless this is a trick question, the OC-12 link (since it is only > 644Mb), and the next bottleneck would be the 2 GbE server connections to > the router (are these bonded? if so, what mode?... It was a bit of a trick question, with a somewhat elaborate setup, designed to shift your focus/thinking. Apparently I failed in my effort here. The correct answer is that RAM will be the first bottleneck. Then disk IOPS, finally followed by the OC-12 assuming we beef up the others. > Since the vast majority of our connections will be *local*, I'm > unconcerned about the internet connect speeds (one office has a 100/10Mb > Cable (Comcast Business Class) connection, the other will have a > 100/100Mb fiber/ethernet connection). You didn't grasp why I used the OC-12 in my example. It had nothing to do with LAN/WAN, local or remote, but the total users/traffic a 600Mb/s link can carry. > My main priority is that the user experience at each physical location > be optimal, which is why I'm more focused on making sure each offices > users are connected to only the local server for all services > (file/print/mail). A single MAN (Metropolitan Area Network) 1000BASE-LX link, good for 5km, likely what you will have, is more than sufficient to carry the 2nd office site traffic while keeping all of your servers/etc where they are now. > My choices are, as I see it, single GbE connections, or add some > multiport GbE cards (these Dells support up to 3 PCIe cards) and bond > some ports together for each VM. 10GbE is simply not in our price range > (and I don't think we need it anyway), although I did stumble on these > while googling and am waiting on pricing, since they claim to be 'much > cheaper': With specs like that you must be supporting 100,000 users. ;) > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Your company/employer has less than 250 users IIRC. Is this right? You're a media company that works with files much larger than the average company. Is that correct? Let's cut to the chase shall we? Your 1000BASE-LX MAN link has an after link overhead bandwidth of just over 100MB/s full duplex. To put this into real world perspective, you can copy a single 4.7GB DVD in 47 seconds, or 1 in each direction in the same time, 2 total, 9.4GB total. You can copy 20 full DVDs over this link, 10 in each direction, in less than 8 minutes. Add heavy IMAP traffic for 500 concurrent users and it's still less than 10 minutes and the IMAP users won't have a clue if the switch VLAN QOS is setup correctly. You see GbE as mundane, slow, because it has been ubiquitous for some time, being a freebie on both servers and desktops. This is why I used the OC-12 example at $15K/month, hoping you'd start to grasp that cost has little direct relationship to performance. GbE is "free" now because the cost of the silicon to drive a 1000MHz signal over 300 meters of copper wire is no longer higher than for 100BASE-T. Here's another comparison. All internet backbone links are OC-48 at 2.5Gb/s. It takes only 2.5 GbE links to equal a backbone link. Backbone links carry the traffic of *millions* of users, all applications, all data stream types. And that's *only* 250MB/s. So, the point is, a single 1000BASE-LX MAN link is far more than plenty to carry all of the traffic you'll throw at it, and quite a bit more, with some minor QOS configuration. Consider how much money, time, and duplication of services and servers you are going to save now that you realize you need nothing other than the 1000BASE-LX MAN link, and closet switches at the second office site? Get yourself a qualified network architect. Pay for a full network traffic analysis. He'll attach sniffers at multiple points in your network to gather traffic/error/etc data. Then you'll discuss the new office, which employees/types with move there, and you'll be able to know almost precisely the average and peak bandwidth needs over the MAN link. He'll very likely tell you the same thing I have, that a single gigabit MAN link is plenty. If you hire him to do the work, he'll program the proper QOS setup to match the traffic patterns gleaned from the sniffers. -- Stan From sdavies at sdc.com.au Fri Mar 2 04:14:04 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Fri, 2 Mar 2012 12:44:04 +1030 Subject: [Dovecot] Log sybnch error Message-ID: <201203021244.05034.sdavies@sdc.com.au> My mail log has many entries like: Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: Extension header update points outside header size Version 2.1.1 dovecot -n gives: # 2.1.1: /usr/etc/dovecot/dovecot.conf # OS: Linux 2.6.33.5-desktop-2mnb i686 Mandriva Linux 2010.2 mail_access_groups = mail mail_privileged_group = mail passdb { driver = pam } protocols = pop3 imap service anvil { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = mode = 0600 user = } unix_listener anvil { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = root vsz_limit = 18446744073709551615 B } service config { chroot = client_limit = 0 drop_priv_before_exec = no executable = config extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = config unix_listener config { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service dict { chroot = client_limit = 1 drop_priv_before_exec = no executable = dict extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dict { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service director { chroot = . client_limit = 0 drop_priv_before_exec = no executable = director extra_groups = fifo_listener login/proxy-notify { group = mode = 00 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener director-admin { group = mode = 0600 user = } unix_listener login/director { group = mode = 00 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service dns_client { chroot = client_limit = 1 drop_priv_before_exec = no executable = dns-client extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dns-client { group = mode = 0666 user = } unix_listener login/dns-client { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = port = 143 ssl = no } inet_listener imaps { address = port = 993 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service indexer-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = indexer-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 10 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service indexer { chroot = client_limit = 0 drop_priv_before_exec = no executable = indexer extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service ipc { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = ipc extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener ipc { group = mode = 0600 user = } unix_listener login/ipc-proxy { group = mode = 0600 user = $default_login_user } user = $default_internal_user vsz_limit = 18446744073709551615 B } service lmtp { chroot = client_limit = 1 drop_priv_before_exec = no executable = lmtp extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = lmtp service_count = 0 type = unix_listener lmtp { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service pop3-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = pop3-login extra_groups = group = idle_kill = 0 inet_listener pop3 { address = port = 110 ssl = no } inet_listener pop3s { address = port = 995 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = pop3 service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service pop3 { chroot = client_limit = 1 drop_priv_before_exec = no executable = pop3 extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = pop3 service_count = 1 type = unix_listener login/pop3 { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service ssl-params { chroot = client_limit = 0 drop_priv_before_exec = no executable = ssl-params extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = startup unix_listener login/ssl-params { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service stats { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = stats extra_groups = fifo_listener stats-mail { group = mode = 0600 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener stats { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } ssl_cert = References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> Message-ID: <4F503FFD.5040504@r.paypc.com> > My initial tests for CLucene were that it would take 30% of mailbox size > (compared to 50% for Xapian). But this was before I actually implemented > it to Dovecot.. I haven't really looked at how large the indexes > actually are. Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. =R= From tss at iki.fi Fri Mar 2 09:31:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:31:01 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F503FFD.5040504@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: On 2.3.2012, at 5.35, Robin wrote: > >> My initial tests for CLucene were that it would take 30% of mailbox size >> (compared to 50% for Xapian). But this was before I actually implemented >> it to Dovecot.. I haven't really looked at how large the indexes >> actually are. > > Did you ever make an fts_xapian plugin, Timo? I've looked into Xapian as an alternative to the solr codebase, mainly out of a dislike of java and its downstream technologies. No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. From dchenusa at yahoo.com Fri Mar 2 09:33:14 2012 From: dchenusa at yahoo.com (D Chen) Date: Thu, 1 Mar 2012 23:33:14 -0800 (PST) Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! Message-ID: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! Thx. From tss at iki.fi Fri Mar 2 09:36:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 09:36:40 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203021244.05034.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: On 2.3.2012, at 4.14, Stephen Davies wrote: > My mail log has many entries like: > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error > at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: > Extension header update points outside header size What filesystem is this? Are you using NFS or some other remote/shared filesystem? Do these messages repeat more than once for the same mailbox? Have you recently upgraded from Dovecot v1.x? From slusarz at curecanti.org Fri Mar 2 09:53:27 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 00:53:27 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> Message-ID: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Quoting Michael M Slusarz : > Quoting Timo Sirainen : > >> On 1.3.2012, at 20.03, Michael M Slusarz wrote: >> >>> Looked at the hg commits since 2.1.1, but didn't see anything that >>> was exactly on-point to this... >>> >>> UNSEEN counts seem to be broken when done in a LIST-STATUS call. >>> I see this, for example: >> >> Only with LIST-STATUS, or also STATUS itself? And with what kind of >> config (doveconf -n)? Can you reproduce this? > > I've been seeing this intermittently the last few days. When I > notice it, I try to reproduce and never can. Then it eventually > comes back, as in the first time I access the next morning. Maybe > its some kind of cache/timeout issue. (e.g. - I can't reproduce > right now) It happened again. Once it got around midnight local time I stopped seeing correct STATUS updates with the UNSEEN count. Running two MUAs: I opened a mailbox with new messages in one, and the next time the other MUA (webmail) polled it magically started seeing unseen messages again. And now if I change the seen flag in one MUA, it will continue to be updated (as normal) in the other MUA. michael From tss at iki.fi Fri Mar 2 10:27:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:27:16 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> Message-ID: <1330676836.2081.46.camel@innu> On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > It happened again. Once it got around midnight local time I stopped > seeing correct STATUS updates with the UNSEEN count. Running two > MUAs: I opened a mailbox with new messages in one, and the next time > the other MUA (webmail) polled it magically started seeing unseen > messages again. And now if I change the seen flag in one MUA, it will > continue to be updated (as normal) in the other MUA. Next time try talking IMAP protocol directly? First giving the same LIST STATUS command, and if it doesn't show the updated counts, try STATUS directly and finally EXAMINE. I don't see anything in the code that could explain why this could be happening. What filesystem are you using? From tss at iki.fi Fri Mar 2 10:45:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 02 Mar 2012 10:45:51 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: Message-ID: <1330677951.2081.49.camel@innu> On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: > I am setting up an high-availability server set, which consists of two (or > more) servers with common disk space, all set behind redundant hardware > load balancers. At first, there will be two servers and disk space will be > kept on NFS server or on both servers using DRBD+OCFS2 filesystem (what > creates kind of networked RAID1 storage space). This will be done mostly > to keep WWW service available in case when one of servers fails. > > However, if we have everything redundant, why not have the same with SMTP > and POP3/IMAP? But - won't anything fail if two (or more) dovecots are > accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? If both servers randomly access users' mails, with NFS you'll have some trouble, with OCFS2 probably less trouble. But in both cases you'll have better performance and no problems if you use Dovecot director in both servers (install both director and backend to both servers). http://wiki2.dovecot.org/Director From stan at hardwarefreak.com Fri Mar 2 11:17:21 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 02 Mar 2012 03:17:21 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F4F60F3.5050508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> Message-ID: <4F509021.2050202@hardwarefreak.com> On 3/1/2012 5:43 AM, Charles Marcus wrote: > Obviously, I don't have the experience or expertise to answer these > questions myself (never analyzed IMAP traffic to have an idea of the > bandwidth each user uses, and probably wouldn't trust my efforts if I > made the attempt). Hopefully, there are some people here who have a > rough idea, which is why I brought this question up here. Expanding on my previous statements, and hopefully answering some questions here, or at least getting in the ballpark, lets see what a single GbE link is capable of. Let's assume an average transfer size of SMTP/IMAP email including headers is roughly 4096 bytes, or 32768 bits. TCP over GbE after all framing and protocol overhead = 992,697,000 bits/sec maximum bandwidth with jumbo frames = 941,482,000 bits/sec max without jumbo frames We'll go without jumbo frames in our example. Every GbE interface on one router segment must support jumbo or you can't enable it. If you do, interfaces that don't do jumbo will have bad to horrible performance, or maybe not work at all. Many workstation NICs don't do jumbo frames as well as many commercial routers. Typical IMAP command payload is absolutely tiny, so we'll concentrate on response traffic. Theoretical steady state IMAP server to client 4KB message transfer rates: = 28,731 msgs/sec = 1,723,905 msgs/minute = 103,434,301 msgs/hour = 2,482,423,242 msgs/day General file transfer bandwidth, 5MB JPG: = 22 files/sec = 1,346 files/minute = 80,808 files/hour = 1,939,393 files/day General file transfer bandwidth, 100MB TIFF: = 1 files/sec = 67 files/minute = 4,040 files/hour = 96,969 files/day General file transfer bandwidth, 500MB video file: = 1 files 4.5 seconds = 10 files 44.6 seconds = 100 files 7.4 minutes As you can see, a single GbE interface has serious capacity and will probably easily carry your inter-site traffic without needing duplicate servers at the second site. You mentioned putting multiple GbE interfaces on your servers. Very, very few servers *need* 900+ Mb/s of bandwidth, however having two links is good for redundancy. So I'd not worry about the aggregation performance, only the proper and seamless failover functionality. I obviously haven't seen your workflows Charles, but I recall you do a lot of media work. By 'you' I mean Media Brokers. So obviously your users will be hitting the network harder than average office workers. I'm taking that into account. My gut instinct, based on experience and the match, is that a single GbE inter site MAN link will be plenty, without the need to duplicate server infrastructure. Again, have a qualified network architect sniff your current network traffic patterns, and discuss with you the anticipated user traffic at the 2nd site to determine your average and peak inter-site b/w needs. The average will absolutely be much less than 1Gb/s, but the peak may be well above 1Gb/s. You can still avoid the myriad problems/costs of server duplication without incurring significant additional link costs. There are a couple of options that should be available to you: 1. A second fiber pair and GbE link You might negotiate a burst contract. You pay a flat monthly rate for a base bit rate of X and pay extra for bursts. Burst contract availability will depend on the provider's network topology. If at any point they're aggregating multiple customer's traffic on a single trunk fiber pair a burst contract should be available. Burst contract allow them to oversubscribe their trunks, just as ISPs and broadband providers do. Your network architect should be able to assist you in figuring out what you'd want for your base and peak bit rates for such a contract. Why pay for 1000Mb/s from 8pm to 6am if you're only using 20Kb/s? 2. Add a second GbE link on a different transceiver wavelength using a prism on each end to transmit both links on one fiber pair. This is typically cheaper when the provider has limited fiber runs in a given area or to a given building. You may or may not be able to save money with a burst contract in this scenario. Talk to your provider and find out what your options are. Wait until your architect has finished your network analysis before speaking to the provider. Treat this link as a traditional WAN link. Do NOT treat it as simply another switch segment. Put an IP router on each side of the GbE MAN link and create a separate IP subnet for hosts and devices in the new office. By doing this you keep broadcast traffic from traversing the link. This includes things like ARP discovery, DHCP, NTP broadcast, and most importantly: broadcast traffic from disk imaging software. If you don't make this an IP routed link, network disk imaging traffic will traverse the MAN link just as it traverses your entire switched LAN. This could be anywhere from 25-80MB/s (200-640Mb/s) of broadcast traffic. You obviously don't want this clogging the link. You *might* be able to eliminate broadcast traffic using special VLAN configurations on sufficiently advanced layer2-7 "switch routers", but it's cheaper and fool proof when done with standard IP routers. Again, chat with your architect. With this being a routed connection, and broadcast traffic being eliminated, any services that rely on broadcast traffic will need to be duplicated or tweaked accordingly. You will need a DHCP server in the new office. The router should be able to serve DHCP, unless you're currently serving some custom scope it can't handle. If you rely on broadcast for WINS, or have any other Microsoft services that rely on broadcast, you will need to address those. If you currently use NTP broadcast for time updates you'll need another NTP server in the new office. Again, the router should be able to broadcast NTP updates. The solutions to these things have been around forever, so I'm not going to go into all of them, but you need to be aware. You'll need to discuss these things with your network architect or a qualified Microsoft consultant. If you run no MS servers and don't use broadcast, then no need to worry about. And hooray for you, no MS! :) This may be of interest given the topic. At a previous $dayjob a few years back, we ran the traffic of about 580 desktops/wireless laptops through a single GbE uplink into an 11 blade server farm backed by a small fiber channel SAN. Blade-blade IP traffic was through a dedicated 14x6 port GbE switch module, so things like vmotion, backups, etc worked at full boogie. But the uplink from the switch module in the BladeCenter to the Cisco 5000 core switch was a single copper GbE uplink. All user traffic flowed over this link. We never had performance issues. We'd configured QOS to keep the IP phones happy but that's about it for traffic shaping. Before I left I jacked in a 2nd GbE uplink for redundancy and configured Cisco's link aggregation protocol. We didn't notice a performance difference. I could have aggregated 6 GbE uplinks. One did the job, two gave resiliency, more would have just wasted ports on the core switch. Hope you find this educational/informational/useful Charles, and maybe others. -- Stan From dovecot at r.paypc.com Fri Mar 2 11:51:53 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 01:51:53 -0800 Subject: [Dovecot] fts size In-Reply-To: References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> Message-ID: <4F509839.8010801@r.paypc.com> > No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. That sounds like a great deal to me! I'm glad you're still interested enough in it. =R= From Ralf.Hildebrandt at charite.de Fri Mar 2 12:25:01 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:25:01 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from Message-ID: <20120302102501.GZ11180@charite.de> Hi! Starting with 2.1.1 we suddely encounter quite a lot of these messages: Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) While this has (assumedly) been working with 2.0.18. Another issue with this: This fixes ONE file, and throws an error. Repeatedly accessing this folder fixes more files, until at some point all files were fixed. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From mailing at securitylabs.it Fri Mar 2 12:30:04 2012 From: mailing at securitylabs.it (mailing at securitylabs.it) Date: Fri, 02 Mar 2012 11:30:04 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <4F50A12C.4080304@securitylabs.it> Il 02/03/2012 11:25, Ralf Hildebrandt ha scritto: > Hi! > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > Mar 2 11:09:28 postamt dovecot: imap(username): Error: read(/home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S) failed: Input/output error (uid=69) > > While this has (assumedly) been working with 2.0.18. Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. > > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, apparently it happens only on servers with qmail, not on servers with exim or dovecot as lda: Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: Broken physical size for mail UID 40669 Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) From tss at iki.fi Fri Mar 2 12:32:08 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:32:08 +0200 Subject: [Dovecot] fts size In-Reply-To: <4F509839.8010801@r.paypc.com> References: <20120228140233.GA18412@dibs.tanso.net> <1330439130.2081.8.camel@innu> <4F503FFD.5040504@r.paypc.com> <4F509839.8010801@r.paypc.com> Message-ID: On 2.3.2012, at 11.51, Robin wrote: >> No, but I can help you with any questions if you want to try implementing it, and even finish it if you get at least the basic index/search functionality working. You can use v2.1's fts-lucene as a start. > > That sounds like a great deal to me! I'm glad you're still interested enough in it. Having more choices is always good. :) From Ralf.Hildebrandt at charite.de Fri Mar 2 12:34:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:34:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <4F50A12C.4080304@securitylabs.it> References: <20120302102501.GZ11180@charite.de> <4F50A12C.4080304@securitylabs.it> Message-ID: <20120302103445.GC11180@charite.de> > Hello, same problem here after upgrading from 2.0.18 to 2.1.0, > apparently it happens only on servers with qmail, not on servers with > exim or dovecot as lda: I'm using the dovecot LDA, but then it's not clear if the messages affected are REALLY old and thus might predate the use of the dovecot LDA... > > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Cached message size smaller than expected (59998 < 60150) > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Maildir filename > has wrong S value, renamed the file from /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2, > to /home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=60150:2, > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: Corrupted index > cache file /home/vpopmail/domains/2/root/Maildir/dovecot.index.cache: > Broken physical size for mail UID 40669 > Mar 2 10:18:45 mercurio dovecot: pop3(user): Error: read(/home/vpopmail/domains/2/root/Maildir/cur/1330679783.8428.mercurio,S=59998:2,) failed: Input/output error (uid=40669) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Fri Mar 2 12:37:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 12:37:06 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302102501.GZ11180@charite.de> References: <20120302102501.GZ11180@charite.de> Message-ID: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S .. > While this has (assumedly) been working with 2.0.18. Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. > Another issue > with this: This fixes ONE file, and throws an error. Repeatedly > accessing this folder fixes more files, until at some point all files > were fixed. Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. From Ralf.Hildebrandt at charite.de Fri Mar 2 12:43:33 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 2 Mar 2012 11:43:33 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> Message-ID: <20120302104333.GD11180@charite.de> * Timo Sirainen : > On 2.3.2012, at 12.25, Ralf Hildebrandt wrote: > > > Starting with 2.1.1 we suddely encounter quite a lot of these messages: > > > > Mar 2 11:09:28 postamt dovecot: imap(username): Error: Maildir filename has wrong S value, renamed the file from /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S to /home/a/i/username/Maildir/.A*Teens.Eink&AOQ-ufe, Spenden etc/cur/1323207735.M64829P19819.postamt.charite.de,S=5137:2,S > .. > > While this has (assumedly) been working with 2.0.18. > > Dovecot v2.0 didn't detect these problems, and might have truncated some mails in some situations. COuld be! > > Another issue > > with this: This fixes ONE file, and throws an error. Repeatedly > > accessing this folder fixes more files, until at some point all files > > were fixed. > > Right, because after it notices a problem it disconnects the client since it can't really do anything else. Running doveadm fetch for all the mails should fix all of them. Ah yes, good idea Mar 2 11:39:39 postamt dovecot: imap-login: Login: user=, method=PLAIN, rip=141.42.206.38, lip=141.42.206.36, mpid=28959, secured Mar 2 11:39:41 postamt dovecot: imap(user): Error: Cached message size smaller than expected (168202 < 170440) Mar 2 11:39:41 postamt dovecot: imap(user): Error: Maildir filename has wrong S value, renamed the file from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ Mar 2 11:39:41 postamt dovecot: imap(user): Error: Corrupted index cache file /home/g/z/user/Maildir/.Partys/dovecot.index.cache: Broken physical size for mail UID 81 Mar 2 11:39:41 postamt dovecot: imap(user): Error: read(/home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ) failed: Input/output error (uid=81) Mar 2 11:39:41 postamt dovecot: imap(user): Disconnected: Internal error occurred. Refer to server log for more information. [2012-03-02 11:39:41] in=735 out=5258 Look at that renaming operation: It simply reused the same name: from /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ to /home/g/z/user/Maildir/.Partys/cur/1289296464.M845813P3466.postamt.charite.de,S=168202:2,SZ > Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. In the case above that mail was gzipped twice :( -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From CMarcus at Media-Brokers.com Fri Mar 2 14:40:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 02 Mar 2012 07:40:54 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F4D009F.7000107@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> Message-ID: <4F50BFD6.5010808@Media-Brokers.com> On 2012-02-28 11:28 AM, Charles Marcus wrote: > On 2012-02-28 11:05 AM, kfx wrote: >> Ok I feel ashame... it was a third party init scrip who was the >> problem :( >> >> Sorry for the noise and thank you for dovecot > > So... you're saying that Thunderbird now correctly uses server side search? Please respond... I need to know whether or not I need to pursue this, since we use Thunderbird in house and will be switching soon to dovecot... Thanks, -- Best regards, Charles From slusarz at curecanti.org Fri Mar 2 20:27:42 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 11:27:42 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <1330676836.2081.46.camel@innu> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> Message-ID: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Quoting Timo Sirainen : > On Fri, 2012-03-02 at 00:53 -0700, Michael M Slusarz wrote: > >> It happened again. Once it got around midnight local time I stopped >> seeing correct STATUS updates with the UNSEEN count. Running two >> MUAs: I opened a mailbox with new messages in one, and the next time >> the other MUA (webmail) polled it magically started seeing unseen >> messages again. And now if I change the seen flag in one MUA, it will >> continue to be updated (as normal) in the other MUA. > > Next time try talking IMAP protocol directly? First giving the same LIST > STATUS command, and if it doesn't show the updated counts, try STATUS > directly and finally EXAMINE. Weird. In the MUA I was seeing this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. Then I did this: slusarz at bigworm % dovecot * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.dovecot" * STATUS "IN.dovecot" (UNSEEN 7) * LIST () "." "INBOX" * STATUS "INBOX" (UNSEEN 1) 1 OK List completed. Sure enough, I went back to the MUA and now see this: C: 4 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 7) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 1) S: 4 OK List completed. The only difference... in the MUA I am enabling QRESYNC. This is what the previous commands look like (before reaching the LIST-STATUS command): >> Timestamp: Fri, 02 Mar 2012 11:17:56 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: 1 OK User logged in C: 2 ENABLE QRESYNC S: * ENABLED QRESYNC S: 2 OK Enabled. C: 3 STATUS IN.horde.cvs (MESSAGES UIDNEXT UIDVALIDITY HIGHESTMODSEQ) S: * STATUS "IN.horde.cvs" (MESSAGES 11 UIDNEXT 16767 UIDVALIDITY 1239077891 HIGHESTMODSEQ 31409) S: 3 OK Status completed. Next time I see this issue, I will try to go back and issue ENABLE QRESYNC from the command line. > I don't see anything in the code that could explain why this could be > happening. What filesystem are you using? Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. michael From tss at iki.fi Fri Mar 2 20:36:41 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 2 Mar 2012 20:36:41 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: On 2.3.2012, at 20.27, Michael M Slusarz wrote: >> I don't see anything in the code that could explain why this could be >> happening. What filesystem are you using? > > Nothing fancy: jfs on Archlinux. I've been using it for years - it's not something I have recently changed. jfs used to have a problem with not updating directory's mtime when link()ing files to it, which caused Dovecot not to notice new mails. This got fixed a few years ago though. But it is a bit suspicious that this is happening with jfs.. From dovecot at r.paypc.com Sat Mar 3 00:13:18 2012 From: dovecot at r.paypc.com (Robin) Date: Fri, 02 Mar 2012 14:13:18 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F50BFD6.5010808@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> Message-ID: <4F5145FE.3070301@r.paypc.com> On 3/2/2012 4:40 AM, Charles Marcus wrote: > Please respond... I need to know whether or not I need to pursue this, > since we use Thunderbird in house and will be switching soon to dovecot... This mailing list is for dovecot, not Thunderbird support. The lack of replies to Thunderbird usage questions no doubt reflects this. I would look at the GUI interface and/or "manual" for Thunderbird to find the answer to that question. I suspect there is a check-box or configuration item that's been right in front of you all along that you've not thought twice about. =R= From mcguire at neurotica.com Sat Mar 3 00:18:12 2012 From: mcguire at neurotica.com (Dave McGuire) Date: Fri, 02 Mar 2012 17:18:12 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F514724.7070001@neurotica.com> On 03/02/2012 05:13 PM, Robin wrote: > On 3/2/2012 4:40 AM, Charles Marcus wrote: >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. Please forgive me for jumping in, but I believe this is very much on-topic. It isn't a matter of "Thunderbird support", it's a matter of Dovecot interoperability. Please DO keep stuff like this on-list. -Dave -- Dave McGuire, AK4HZ New Kensington, PA From sdavies at sdc.com.au Sat Mar 3 01:45:02 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Sat, 3 Mar 2012 10:15:02 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: References: <201203021244.05034.sdavies@sdc.com.au> Message-ID: <201203031015.02716.sdavies@sdc.com.au> No NFS. The file system is local. Yes. There are multiple copies of the message for multiple mailboxes for each of at least two users. Yes. Did recently upgrade from 1.2.15. Cheers and thanks, Stephen On Fri, 2 Mar 2012 06:06:40 PM Timo Sirainen wrote: > On 2.3.2012, at 4.14, Stephen Davies wrote: > > My mail log has many entries like: > > > > Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization > > error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS > > Emails/dovecot.index: Extension header update points outside header size > > What filesystem is this? Are you using NFS or some other remote/shared > filesystem? > > Do these messages repeat more than once for the same mailbox? Have you > recently upgraded from Dovecot v1.x? -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From slusarz at curecanti.org Sat Mar 3 02:48:09 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 17:48:09 -0700 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data Message-ID: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> I'm seeing this: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) It should be: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) since the RFC 2822 quoting characters must be removed. michael From slusarz at curecanti.org Sat Mar 3 03:48:23 2012 From: slusarz at curecanti.org (Michael M Slusarz) Date: Fri, 02 Mar 2012 18:48:23 -0700 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> Message-ID: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Quoting Timo Sirainen : > On 2.3.2012, at 20.27, Michael M Slusarz wrote: > >>> I don't see anything in the code that could explain why this could be >>> happening. What filesystem are you using? >> >> Nothing fancy: jfs on Archlinux. I've been using it for years - >> it's not something I have recently changed. > > jfs used to have a problem with not updating directory's mtime when > link()ing files to it, which caused Dovecot not to notice new mails. > This got fixed a few years ago though. But it is a bit suspicious > that this is happening with jfs.. I can now verify that QRESYNC is triggering this behavior. * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SEARCH=FUZZY SPECIAL-USE ACL RIGHTS=texk] Logged in as slusarz 1 ENABLE QRESYNC * ENABLED QRESYNC 1 OK Enabled. 2 LIST "" (IN.horde.dev) RETURN (STATUS (UNSEEN)) * LIST () "." "IN.horde.dev" * STATUS "IN.horde.dev" (UNSEEN 0) 2 OK List completed. 3 EXAMINE IN.horde.dev * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded NonJunk impflag0 impflag1) * OK [PERMANENTFLAGS ()] Read-only mailbox. * 3 EXISTS * 1 RECENT * OK [UNSEEN 3] First unseen. * OK [UIDVALIDITY 1255685339] UIDs valid * OK [UIDNEXT 2805] Predicted next UID * OK [HIGHESTMODSEQ 8266] Highest 3 OK [READ-ONLY] Select completed. (The unseen message arrived about 7 minutes before I issued these commands, so it didn't sneak in between command #2 and #3). michael From busseniu at in.tum.de Sat Mar 3 14:33:27 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Sat, 03 Mar 2012 13:33:27 +0100 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F4DF07A.7020408@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> Message-ID: <4F520F97.5030002@in.tum.de> On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: > when the private namespace has "prefix = INBOX." and you use doveadm > fetch to search for "mailbox INBOX", then it prints every message twice: Apparently the bug has been introduced with this changeset: changeset: 14112:f5353573d3a0 user: Timo Sirainen date: Sun Feb 12 02:50:49 2012 +0200 summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. http://hg.dovecot.org/dovecot-2.1/rev/f5353573d3a0 Cheers, Christoph From netwiz at crc.id.au Sat Mar 3 14:07:44 2012 From: netwiz at crc.id.au (Steven Haigh) Date: Sat, 03 Mar 2012 23:07:44 +1100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <20110311215739.GD13492@state-of-mind.de> References: <20110311215739.GD13492@state-of-mind.de> Message-ID: <4F520990.2000903@crc.id.au> Hi all, I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... -- Steven Haigh Email: netwiz at crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 >Timo, > >would you consider adding support for "IMAP LIST Extension for >Special-Use >Mailboxes" any time near >in the >future? > >I would really love to get rid of all those folders created by all >those >different mail clients just because they can't agree to use the same >folder >for special purpose. > >Obviously clients need to support it too. Having Dovecot support it >certainly >would make them adopt the standard sooner. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4952 bytes Desc: S/MIME Cryptographic Signature URL: From CMarcus at Media-Brokers.com Sat Mar 3 14:52:10 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 07:52:10 -0500 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5213FA.30700@Media-Brokers.com> On 2012-03-02 2:33 AM, D Chen wrote: > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start > successfully with lots of errors i.e. "dovecot: doveconf: Warning: > ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on > Ubuntu 11.10, by following the Postfix installation and configuration > instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section > on page 190, after run "sudo apt-get install dovecot-common", it > requires to edit the section of "auth default" and the "socket > listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my > /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth > default" "socket listen" ! Distro specific questions are usually much better asked on the distro support lists... -- Best regards, Charles From eliezer at ec.hadorhabaac.com Sat Mar 3 15:03:40 2012 From: eliezer at ec.hadorhabaac.com (Eliezer Croitoru) Date: Sat, 03 Mar 2012 15:03:40 +0200 Subject: [Dovecot] Desperately need help ! a default dovecot.conf and/or Ubuntu 11.10 postfix/dovecot server configuration issue! In-Reply-To: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> References: <1330673594.23753.YahooMailNeo@web161603.mail.bf1.yahoo.com> Message-ID: <4F5216AC.3030602@ec.hadorhabaac.com> On 02/03/2012 09:33, D Chen wrote: use the command dovecot -n to get dovecot settings output and we can try to help you a bit. Regards, Eliezer > When ungraded from Ubuntu 11.04 to 11.10, dovecot can't start successfully with lots of errors i.e. > "dovecot: doveconf: Warning: ... 'imaps' protocol is no longer necessary, remove it"... > > At any rate, I want to setup a postfix(MTA)/dovecot(MDA) servers on Ubuntu 11.10, by following the Postfix installation and configuration instruction in Ubuntu Serverguide, in "1.4 Configuring SASL" section on page 190, after run "sudo apt-get install dovecot-common", it requires to edit the section of "auth default" and the "socket listen" option...,in the /etc/dovecot/dovecot.conf file, BUT my /etc/dovecot/dovecot.conf (only about 4k byes) CAN'T find the "auth default" "socket listen" ! > > I also checked into the /usr/share/doc/dovecot-common/dovecot/example-config, there is a > dovecot.conf, it's also about 4k size, and there is no such "auth default" or "socket listen" words can be found ! where is the default dovecot.conf file I can get a copy ? > > BTW, there is the dovecot.conf.ucf file (what is this for?) which's about 50k and has the "auth default" and "socket listen" words there ! what is supposed the size for the /etc/dovecot/dovecot.conf ? i'm confused! > > Thx. From arnaud.abelard at univ-nantes.fr Sat Mar 3 18:06:17 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Sat, 03 Mar 2012 17:06:17 +0100 Subject: [Dovecot] keywords/flags questions Message-ID: <4F524179.2040407@univ-nantes.fr> Hello, I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? For example: . STORE 1:1 flags &AOA-_refaire * 1 FETCH (FLAGS (&aoa-_refaire)) This makes retrieving the keyword properly impossible. What did I miss? My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? Thanks in advance, Arnaud -- Arnaud Ab?lard jabber: arnaud.abelard at univ-nantes.fr / twitter: ArnY Administrateur Syst?me DSI Universit? de Nantes - From public-mail at alekciy.ru Sat Mar 3 18:51:28 2012 From: public-mail at alekciy.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KHRg9C90LTRg9C60L7Qsg==?=) Date: Sat, 3 Mar 2012 20:51:28 +0400 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: 3 ????? 2012??. 20:06 ???????????? Arnaud Ab?lard ???????: > But docevot save keywords in lowercase Yes. http://www.dovecot.org/list/dovecot/2011-April/058493.html From tss at iki.fi Sat Mar 3 19:05:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 3 Mar 2012 19:05:13 +0200 Subject: [Dovecot] keywords/flags questions In-Reply-To: <4F524179.2040407@univ-nantes.fr> References: <4F524179.2040407@univ-nantes.fr> Message-ID: <5CA4B56F-D26E-492B-9B4D-9BD8E6EAD018@iki.fi> On 3.3.2012, at 18.06, Arnaud Ab?lard wrote: > I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. > > First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between & and - chars. UTF-7 encoding is case sensitive, &AOA- isn't the same chars as &aoa-. But docevot save keywords in lowercase or am I mistaken? > > For example: > . STORE 1:1 flags &AOA-_refaire > * 1 FETCH (FLAGS (&aoa-_refaire)) > > This makes retrieving the keyword properly impossible. What did I miss? Sorry, doesn't work like that. This was recently discussed in imap-protocol mailing list though, but nothing has come of it. > My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? For now yes. I've been thinking about doing something about this for last 8 years, but it has never become a real problem so I haven't bothered. From trashcan at odo.in-berlin.de Sat Mar 3 20:03:42 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:03:42 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <1330346709.11500.324.camel@innu> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: Hi -- On 27.02.2012, at 13:45, Timo Sirainen wrote: > On Thu, 2012-02-23 at 20:55 +0100, Michael Grimm wrote: >> My working 2.0.18 syntax threw the following error: >> >> vmail> dsync -v -f -u test ssh vmail at remote-host.tld dsync -v -f -u test >> doveadm(vmail): Fatal: Unknown print formatter: -u >> dsync-local(test): Error: read() from worker server failed: EOF > > You left out "mirror" from that command, but after adding it the latest > hg version works. I did use 'mirror', I just forgot to paste it. JFTR: vmail> dovecot --version 20120303 (1002733ca266+) vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test dsync-local(test): Error: remote: dsync: illegal option -- f doveadm dsync-server [-u |-A] [-S ] dsync-local(test): Error: read() from worker server failed: EOF If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. >> Now I switched to the recommended new syntax as stated in http://wiki2.dovecot.org/Upgrading/2.1: >> >> vmail> doveadm sync -v -f -u test ssh vmail at remote-host.tld doveadm sync -v -f -u test >> doveadm: illegal option -- v >> doveadm sync [-u |-A] [-S ] [-fR] [-m ] > > The -v parameter is in wrong place now, needs to be "doveadm -v sync". Ah, yes. That was my mistake, sorry. >> After some trial by error I finally found a working syntax: >> >> vmail> doveadm sync -u test -f ssh vmail at remote-host.tld doveadm dsync-server -u test > > Oh, hmm. I hadn't thought about this problem, it shouldn't have been > necessary to give the dsync-server parameter. But I guess there's not a > whole lot of other possibilities to do this. Hmm. > > BTW. I think you can do this simply: > > doveadm sync -f -u test test at remote-host.tld vmail> doveadm sync -f -u test ssh vmail at remote-host.tld dsync-local(test): Error: remote: dsync-server: Command not found. dsync-local(test): Error: read() from worker server failed: EOF Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that I'm the only one reporting that. How could I help to debug this issue? Regards, Michael From tlx at leuxner.net Sat Mar 3 20:10:35 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Sat, 3 Mar 2012 19:10:35 +0100 Subject: [Dovecot] RFE: IMAP LIST Extension for Special-Use Mailboxes In-Reply-To: <4F520990.2000903@crc.id.au> References: <20110311215739.GD13492@state-of-mind.de> <4F520990.2000903@crc.id.au> Message-ID: <83D77B81-EC49-4755-A866-E30B41E8B246@leuxner.net> Am 03.03.2012 um 13:07 schrieb Steven Haigh: > I'm just wondering if anyone knows if this got implemented? I've been looking at doing this for quite some time... Yes it was. It has been discussed extensively: http://www.dovecot.org/list/dovecot-news/2012-February/000213.html http://www.dovecot.org/list/dovecot/2011-December/062327.html Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From trashcan at odo.in-berlin.de Sat Mar 3 20:12:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 19:12:21 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <8D5B4989-571D-4F5D-927E-65198CECFADD@odo.in-berlin.de> Hi -- On 03.03.2012, at 19:03, Michael Grimm wrote: > Thus I can't test user suggestion. s/user/your/ Sorry, Michael From anyaddress at gmx.net Sat Mar 3 22:08:48 2012 From: anyaddress at gmx.net (Tom Fernandes) Date: Sat, 3 Mar 2012 21:08:48 +0100 Subject: [Dovecot] directly addressable public folders issues Message-ID: <201203032108.49489.anyaddress@gmx.net> Hi, I would like to have an address info at example.com whose mails are stored in a public folder. I also want certain users to be able to create sieve-filter-rules and subfolders. From what I understand it's a good idea to have a separate location for home and for mail_location. For my normal accounts I have: home = /var/vmail/ mail_location = ~/Maildir Is there a way to have the same for public folders? This are my current settings: namespace public { separator = / prefix = public/ location = maildir:/var/vmail/public subscriptions = no } user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The LDAP-homeDirectory-attribute for info at example.com is "public/.info" Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which is good. The MUA seems to read to read from /var/vmail/public/.info though. How can I make the MUA read from /var/vmail/public/.info/Maildir instead? And in case this works - is it possible to have only certain users modify the sieve-rule for this public folder or do I definitely need a passdb-entry for info at example.com? Modifying scripts via commandline is not an option... I'm using dovecot 1.2. regards, Tom Fernandes From CMarcus at Media-Brokers.com Sat Mar 3 22:14:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:14:55 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5145FE.3070301@r.paypc.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> Message-ID: <4F527BBF.3060607@Media-Brokers.com> On 2012-03-02 5:13 PM, Robin wrote: > This mailing list is for dovecot, not Thunderbird support. The lack of > replies to Thunderbird usage questions no doubt reflects this. What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? -- Best regards, Charles From CMarcus at Media-Brokers.com Sat Mar 3 22:20:24 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 03 Mar 2012 15:20:24 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F509021.2050202@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> Message-ID: <4F527D08.6070508@Media-Brokers.com> Thanks very much for taking the time for your detailed reply, Stan, but I'll need more time to study it... On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > My gut instinct, based on experience and the match, is that a single GbE > inter site MAN link will be plenty, without the need to duplicate server > infrastructure. I just wanted to point out one thing - I have two primary goals - yes, one is to maximize performance, but the other is accomplish a level of *redundancy*... Also - I already have the servers (I have 3 Poweredge 2970's available to me, only one of which is currently being used)... So, the only extra expenses involved will be relatively minor hardware expenses (multi-port Gb NICs), and some consulting services for making sure I implement the VM environment (including the routing) correctly. So, honestly, we'd be incurring most of these expenses anyway, even if we didn't set up redundant servers, so I figure why not get redundancy too (now is the time to get the boss to pay for it)... -- Best regards, Charles From trashcan at odo.in-berlin.de Sat Mar 3 22:27:11 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sat, 3 Mar 2012 21:27:11 +0100 Subject: [Dovecot] Dovecot clustering with dsync-based replication In-Reply-To: <1330437834.2081.2.camel@innu> References: <1330437834.2081.2.camel@innu> Message-ID: Hi -- On 28.02.2012, at 15:03, Timo Sirainen wrote: > This document describes a design for a dsync-replicated Dovecot cluster. Whow! That's more than interesting, that's a real bummer ;-) At least for my setup of redundant mail servers. Looking forward to test it, Michael From piotr-l at netexpert.pl Sat Mar 3 22:41:24 2012 From: piotr-l at netexpert.pl (Piotr NetExpert) Date: Sat, 03 Mar 2012 21:41:24 +0100 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F5281F4.1070503@netexpert.pl> > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Take into account costs of administering a more complex environment too. -- pozdrawiam Piotr Szafarczyk http://www.netexpert.pl From bradley.giesbrecht at gmail.com Sat Mar 3 23:16:31 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sat, 3 Mar 2012 13:16:31 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F527BBF.3060607@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> Message-ID: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > On 2012-03-02 5:13 PM, Robin wrote: >> This mailing list is for dovecot, not Thunderbird support. The lack of >> replies to Thunderbird usage questions no doubt reflects this. > > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? Show dovecot misbehaving. On Feb 28, 2012, at 6:57 AM, Timo Sirainen wrote: > On Tue, 2012-02-28 at 15:47 +0100, kfx wrote: >>> Did you enable the 'Run search on server' option in the Advanced Search >>> window? Doing this *should* result in Thunderbird using dovecots indexes >>> server side. >>> >> >> Yes I did. >> >> Some more info: >> >> by telnet'ing directly and issuing: >> c search text pattern >> * SEARCH 1208 >> c OK Search completed (0.003 secs). > > So, Solr in Dovecot works perfectly. > >> But the same search in thunderbird return "No matches found" :( > > Thunderbird problem, nothing you can do about it from Dovecot's side. Regards, Bradley Giesbrecht From stan at hardwarefreak.com Sun Mar 4 02:51:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 03 Mar 2012 18:51:39 -0600 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F527D08.6070508@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F509021.2050202@hardwarefreak.com> <4F527D08.6070508@Media-Brokers.com> Message-ID: <4F52BC9B.9000005@hardwarefreak.com> On 3/3/2012 2:20 PM, Charles Marcus wrote: > Thanks very much for taking the time for your detailed reply, Stan, but > I'll need more time to study it... > > On 2012-03-02 4:17 AM, Stan Hoeppner wrote: > >> My gut instinct, based on experience and the match, is that a single GbE >> inter site MAN link will be plenty, without the need to duplicate server >> infrastructure. > > I just wanted to point out one thing - I have two primary goals - yes, > one is to maximize performance, but the other is accomplish a level of > *redundancy*... What type of redundancy are you looking for? I.e. is one reason for duplicating servers at site #2 to avoid disruption in the event the MAN link fails? Do you currently have redundant GbE links to each closet switch stack in site #1, and also redundant switches in the datacenter? I.e. do you skip a beat if a core or closet switch fails? If you do not currently have, nor plan to create such network redundancy internally at site #1, then why build application redundancy with the single goal of mitigating failure of a single network link? Do you have reason to believe there is a higher probability of failure of the MAN link than any other single link in the current network? > Also - I already have the servers (I have 3 Poweredge 2970's available > to me, only one of which is currently being used)... > > So, the only extra expenses involved will be relatively minor hardware > expenses (multi-port Gb NICs), and some consulting services for making > sure I implement the VM environment (including the routing) correctly. Again, you don't need multi-port GbE NICs or bonding for performance--a single GbE link is all each server needs. Your switches should be able to demonstrate that, without even needing a sniffer, assuming they're decent managed units. If you're after link redundancy, use two single port NICs per server, or one mobo mounted port and once single port NIC. Most dual port NICs duplicate the PHYs but not the ethernet chip nor power circuits, etc. Thus, when a dual port NIC fails you usually loose both ports. > So, honestly, we'd be incurring most of these expenses anyway, even if > we didn't set up redundant servers, so I figure why not get redundancy > too (now is the time to get the boss to pay for it)... Don't forget power backup at site #2. Probably not a huge cost in the overall scheme of things, but it's still another $5000 or so. In summary, my advice is: 1. One 1000Mb MAN link is plenty of bandwidth for all users at site #2 including running internet traffic through site #1, saving the cost of an internet pipe at site #2 2. If truly concerned about link failure, get a backup 100Mb/s link, or get two GbE links with a burst contract, depending on price 3. Keep your servers in one place. If you actually desire application level redundancy (IMAP, SMB/CIFS, etc) unrelated to a network link failure, then do your clustering etc "within the rack". It will be much easier to manage and troubleshoot this than two datacenters w/ all kinds of replication etc between them 4. If site #1 is not already link redundant, it makes little sense to make a big redundancy push to cover a possible single network link failure, regardless of which link 5. Building a 2nd datacenter and using the MAN link for data replication gives you no performance advantage, and may actually increase overall utilization, vs using the link as a regular trunk 6. *Setup QOS appropriately to maintain low latency of IMAP and other priority data, giving a back seat to SMB/CIFS/FTP/HTTP and other bulk transfer protocols* With proper QOS the single GbE MAN link will simply scream for everyone, regardless of saturation level -- Stan From dchenusa at yahoo.com Sun Mar 4 08:25:12 2012 From: dchenusa at yahoo.com (D Chen) Date: Sat, 3 Mar 2012 22:25:12 -0800 (PST) Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... Message-ID: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! ?? admin at server:/etc/dovecot$ doveconf -n ? ? # 2.0.13: /etc/dovecot/dovecot.conf ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } ? ? doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } ? ? # OS: Linux 3.0.0-16-server x86_64 Ubuntu 11.10? ? ? log_timestamp = "%Y-%m-%d %H:%M:%S " ? ? mail_location = maildir:~/Maildir ? ? mail_privileged_group = mail ? ? managesieve_notify_capability = mailto ? ? managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ? ? passdb { ? ? ? driver = pam - Ignored: ? ? } ? ? passdb { ? ? ? driver = pam ? ? } ? ? plugin { ? ? ? sieve = ~/.dovecot.sieve ? ? ? sieve_dir = ~/sieve ? ? } ? ? protocols = imap pop3 sieve ? ? service auth { ? ? ? unix_listener /var/spool/postfix/private/auth-client { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? unix_listener /var/spool/postfix/private/dovecot-auth { ? ? ? ? group = postfix ? ? ? ? mode = 0660 ? ? ? ? user = postfix ? ? ? } ? ? ? user = root ? ? } ? ? ssl_cert = was automatically rejected:%n%r ? ? } From tss at iki.fi Sun Mar 4 12:44:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 04 Mar 2012 12:44:46 +0200 Subject: [Dovecot] dsync replication available for testing Message-ID: <4F53479E.40703@iki.fi> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: - public namespace isn't replicated at all - shared namespace is replicated, but not private mail flags - I've only tested SSH replication setup now, not director replication setup (and director setup is still missing many things) - SSH replication setup uses aggregator process, which isn't really necessary and can probably be avoided in future Below is a configuration for virtual user setup. System user configuration works pretty much the same, except doveadm/ssh is run as root. Try first that dsync works successfully with ssh in host1: doveadm sync -u user at domain remote:vmail at host2.example.com and also in host2: doveadm sync -u user at domain remote:vmail at host1.example.com ------ mail_plugins = $mail_plugins notify replication service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } From tss at iki.fi Sun Mar 4 13:05:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:05:26 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: On 3.3.2012, at 20.03, Michael Grimm wrote: > vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test > dsync-local(test): Error: remote: dsync: illegal option -- f > doveadm dsync-server [-u |-A] [-S ] > dsync-local(test): Error: read() from worker server failed: EOF > > If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 >> doveadm sync -f -u test test at remote-host.tld > > vmail> doveadm sync -f -u test ssh vmail at remote-host.tld > dsync-local(test): Error: remote: dsync-server: Command not found. > dsync-local(test): Error: read() from worker server failed: EOF Remove the "ssh" parameter from the middle and change vmail@ to test@ > Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. Thus I > can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's a good idea. In the latest hg version the preferred way is: doveadm sync -f -u test remote:vmail at host It automatically adds the remote -u test. > Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). > The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that > I'm the only one reporting that. How could I help to debug this issue? Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or other scripts that "rm" mails? From tss at iki.fi Sun Mar 4 13:13:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:13:18 +0200 Subject: [Dovecot] directly addressable public folders issues In-Reply-To: <201203032108.49489.anyaddress@gmx.net> References: <201203032108.49489.anyaddress@gmx.net> Message-ID: <7E27E7D3-DCE8-4FF9-9689-24815D2895CB@iki.fi> On 3.3.2012, at 22.08, Tom Fernandes wrote: > I would like to have an address info at example.com whose mails are stored in a > public folder. .. > From what I understand it's a good idea to have a separate location for home and > for mail_location. > > For my normal accounts I have: > home = /var/vmail/ > mail_location = ~/Maildir mail_location = maildir:~/Maildir to unnecessary avoid autodetection. > Is there a way to have the same for public folders? This are my current > settings: > > namespace public { > separator = / > prefix = public/ > location = maildir:/var/vmail/public > subscriptions = no > } That's ok. > user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The "mail" isn't necessary here, since it's already globally set. > The LDAP-homeDirectory-attribute for info at example.com is "public/.info" That's not going to work too well. > Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which > is good. The MUA seems to read to read from /var/vmail/public/.info though. > > How can I make the MUA read from /var/vmail/public/.info/Maildir instead? You can't. > And in case this works - is it possible to have only certain users modify the > sieve-rule for this public folder or do I definitely need a passdb-entry for > info at example.com? > Modifying scripts via commandline is not an option... The way I did it was to make info@, sales@ and others aliases to "company" user, which is a rather regular user (except can't actually log in). For this "company" user I've a Sieve script that puts the mails into the proper mailbox, e.g. fileinto "public/info". From tss at iki.fi Sun Mar 4 13:21:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:21:07 +0200 Subject: [Dovecot] LIST-STATUS issue In-Reply-To: <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> References: <20120301110346.Horde.NsnvYIF5lbhPT7oCJZrifkA@bigworm.curecanti.org> <20120301121741.Horde.07vsT4F5lbhPT8tVZRESfkA@bigworm.curecanti.org> <20120302005327.Horde.GoZME4F5lbhPUHx3vdAVz6A@bigworm.curecanti.org> <1330676836.2081.46.camel@innu> <20120302112742.Horde.IaqqGYF5lbhPUREeyM7RX5A@bigworm.curecanti.org> <20120302184823.Horde.ZUgDUYF5lbhPUXhn1QTyMgA@bigworm.curecanti.org> Message-ID: <75EBCBDD-B30A-401F-A6D9-517C03B1873B@iki.fi> On 3.3.2012, at 3.48, Michael M Slusarz wrote: > I can now verify that QRESYNC is triggering this behavior. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/8cbc130c2b72 http://hg.dovecot.org/dovecot-2.1/rev/31ae11fe18b2 From trashcan at odo.in-berlin.de Sun Mar 4 13:31:45 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:31:45 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> Message-ID: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:05, Timo Sirainen wrote: > On 3.3.2012, at 20.03, Michael Grimm wrote: >> vmail> dsync -v -f -u test mirror ssh vmail at remote-host.tld dsync -v -f -u test >> dsync-local(test): Error: remote: dsync: illegal option -- f >> doveadm dsync-server [-u |-A] [-S ] >> dsync-local(test): Error: read() from worker server failed: EOF >> >> If I do omit the remote '-f' the old syntax is being accepted. Thanks for fixing that. > > Right, the remote -f parameter doesn't do anything. But it's anyway now allowed: > http://hg.dovecot.org/dovecot-2.1/rev/9c6eeeb810c0 Ok, that means it has always been ignored in 2.0.x ;-) >>> doveadm sync -f -u test test at remote-host.tld >> >> vmail> doveadm sync -f -u test ssh vmail at remote-host.tld >> dsync-local(test): Error: remote: dsync-server: Command not found. >> dsync-local(test): Error: read() from worker server failed: EOF > > Remove the "ssh" parameter from the middle and change vmail@ to test@ That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: ssh -p 1234 vmail at remote-host.tld Sorry, I should have mentioned that before. In your other mail about 'dsync replication' you refer to a config option: #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} Would that allow for ssh options to be set? >> Users are virtual ones at both mail servers, and vmail is the only system user to run ssh. >> Thus I can't test user suggestion. But it's ok to add a 'doveadm dsync-server -u test'. > > In this syntax the test@ means Dovecot user, not system user. Although I'm not sure if that's > a good idea. In the latest hg version the preferred way is: > > doveadm sync -f -u test remote:vmail at host See above regarding ssh options. >> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >> I'm the only one reporting that. How could I help to debug this issue? > > Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or > other scripts that "rm" mails? No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. But no scripts or such are used to remove mail. I'm running mdbox only. Thanks and regards, Michael From trashcan at odo.in-berlin.de Sun Mar 4 13:34:03 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:34:03 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Hi -- On 04.03.2012, at 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Great news. I would love to test it, if I will be able to run this on a test account, only. All other users should become synced the "old way" for the time being. Would that be possible with the current implementation? Regards, Michael From tss at iki.fi Sun Mar 4 13:35:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:35:04 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> Message-ID: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> On 4.3.2012, at 13.31, Michael Grimm wrote: > That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: > > ssh -p 1234 vmail at remote-host.tld > > Sorry, I should have mentioned that before. > > In your other mail about 'dsync replication' you refer to a config option: > #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > Would that allow for ssh options to be set? Yes. >> doveadm sync -f -u test remote:vmail at host > > See above regarding ssh options. So this works by changing the dsync_remote_cmd. >>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>> I'm the only one reporting that. How could I help to debug this issue? >> >> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >> other scripts that "rm" mails? > > No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. > But no scripts or such are used to remove mail. I'm running mdbox only. By "undeletable" do you mean you have mails that always come back after expunging them? I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they don't contain any sensitive information. From tss at iki.fi Sun Mar 4 13:38:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:38:14 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> Message-ID: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> On 4.3.2012, at 13.34, Michael Grimm wrote: > On 04.03.2012, at 11:44, Timo Sirainen wrote: > >> In dovecot-2.1 hg you can now test dsync-based replication. > > Great news. I would love to test it, if I will be able to run this on a test > account, only. All other users should become synced the "old way" for the time > being. > > Would that be possible with the current implementation? 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) 2) You can enable replication plugin only for one user by changing mail_plugins setting via userdb extra fields. Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So you could even ignore 1) and just let it sync everyone at startup. From trashcan at odo.in-berlin.de Sun Mar 4 13:41:47 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 12:41:47 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> Message-ID: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:35, Timo Sirainen wrote: > On 4.3.2012, at 13.31, Michael Grimm wrote: >> That doesn't work in my ssh setup, because I'm using a different ssh port and thus have to run: >> >> ssh -p 1234 vmail at remote-host.tld >> >> Sorry, I should have mentioned that before. >> >> In your other mail about 'dsync replication' you refer to a config option: >> #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} >> >> Would that allow for ssh options to be set? > > Yes. Good news. >>>> Now, I will stick to 2.1.x because syncing is done without loss of mails (after 10 days of testing). >>>> The only inconvenience remaining is reappearing of deleted and "undeletable" mail. But it seems that >>>> I'm the only one reporting that. How could I help to debug this issue? >>> >>> Is anything else besides Dovecot modifying the mailboxes? Especially deleting mails? No cronjobs or >>> other scripts that "rm" mails? >> >> No. Only dovecot is allowed to deliver mail (lmtp). Sieve's 'copy:' and 'fileinto' are used as well. >> But no scripts or such are used to remove mail. I'm running mdbox only. > > By "undeletable" do you mean you have mails that always come back after expunging them? Yes. Deleting by the client will return them after the next dsync run. > I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they > don't contain any sensitive information. From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Thanks and regards, Michael From tss at iki.fi Sun Mar 4 13:54:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 13:54:34 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> Message-ID: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> On 4.3.2012, at 13.41, Michael Grimm wrote: >> By "undeletable" do you mean you have mails that always come back after expunging them? > > Yes. Deleting by the client will return them after the next dsync run. > >> I'd like to get dovecot.index and dovecot.index.log files from those mailboxes from both servers, they >> don't contain any sensitive information. > > From all mailboxes? I can do that if you wish. But that will need some time (tomorrow). Just one mailbox where that consistently happens is enough: 1. Expunge the mail 2. Get a copy of the dbox-Mails/dovecot.index, dbox-Mails/dovecot.index.log and dbox-Mails/dovecot.index.log.2 from both servers 3. Run dsync, and verify that the message is undeleted 4. Get another copy of the indexes from both servers From tss at iki.fi Sun Mar 4 14:14:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:14:23 +0200 Subject: [Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data In-Reply-To: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> References: <20120302174809.Horde.A41wKYF5lbhPUWpJQHqSHZA@bigworm.curecanti.org> Message-ID: <90B35FA4-651C-40CA-8149-8FE7E3E09E50@iki.fi> On 3.3.2012, at 2.48, Michael M Slusarz wrote: > I'm seeing this: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} > XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > It should be: > > 1 UID FETCH 31734 (ENVELOPE) > * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} > XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001 at scheduler>")) > > since the RFC 2822 quoting characters must be removed. Oops. This has been buggy forever. Added the fix to all Dovecot hg trees. From tss at iki.fi Sun Mar 4 14:33:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:33:35 +0200 Subject: [Dovecot] doveadm fetch prints duplicate results in 2.1 In-Reply-To: <4F520F97.5030002@in.tum.de> References: <4F4DF07A.7020408@in.tum.de> <4F520F97.5030002@in.tum.de> Message-ID: <8AAD13E5-CE14-44BF-9CD9-DDB984B0BF31@iki.fi> On 3.3.2012, at 14.33, Christoph Bu?enius wrote: > On 02/29/2012 10:31 AM, Christoph Bu?enius wrote: >> when the private namespace has "prefix = INBOX." and you use doveadm >> fetch to search for "mailbox INBOX", then it prints every message twice: > > Apparently the bug has been introduced with this changeset: > > changeset: 14112:f5353573d3a0 > user: Timo Sirainen > date: Sun Feb 12 02:50:49 2012 +0200 > summary: lib-storage: Added MAILBOX_LIST_ITER_LIST_PREFIXES flag. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/bbe6b6c2ee99 From tss at iki.fi Sun Mar 4 14:35:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:35:18 +0200 Subject: [Dovecot] Log sybnch error In-Reply-To: <201203031015.02716.sdavies@sdc.com.au> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> Message-ID: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> On 3.3.2012, at 1.45, Stephen Davies wrote: > No NFS. The file system is local. > > Yes. There are multiple copies of the message for multiple mailboxes for each > of at least two users. But does the error keep repeating for the same mailbox? It's supposed to fix itself automatically after logging the error once. > Yes. Did recently upgrade from 1.2.15. I think in earlier versions mbox used somewhat different index file structures and now Dovecot logs some errors about them. Anyway, one sure way to fix this is to just delete all the .imap/ directories. From tss at iki.fi Sun Mar 4 14:36:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:36:23 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> Message-ID: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> On 4.3.2012, at 8.25, D Chen wrote: > Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf So, doveconf -n > dovecot-new.conf mv dovecot-new.conf /etc/dovecot/dovecot.conf That should do it. From tss at iki.fi Sun Mar 4 14:41:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:41:20 +0200 Subject: [Dovecot] Multiple namespaces seems to be used at the same time In-Reply-To: <4F4F84C7.1060502@cnpapers.com> References: <4F4F84C7.1060502@cnpapers.com> Message-ID: <42CE6C31-D246-4AE1-9E79-3DD457E20E39@iki.fi> On 1.3.2012, at 16.16, Steve Campbell wrote: > I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. > > On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. > > The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Difficult to say without knowing 1) doveconf -n output and 2) .subscriptions file contents. > Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Yes, assuming you have: mail_location = mbox:~/mail (with maybe the :INBOX=/var/mail/%u) Also the .subscriptions needs to be in ~/mail/ then. From tss at iki.fi Sun Mar 4 14:45:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:45:48 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> Message-ID: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> On 2.3.2012, at 0.35, Terry Carmen wrote: > With the exchange server being returned in the msExchHomeServerName property as: > > /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername > > I believe this should somehow end up in the userdb section, which currently contains "driver = prefetch", but can't seem to figure out specifically what should be there. .. > The only important part is "cn=exchangeservername", which is the machine name and would need to be prepended to example.com to get the fqdn. Do all of the values have the same prefix? Then I guess you can do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%49.100$.example.com If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: pass_attrs = ..., \ msExchHomeServerName=userdb_imapc_host=%-10$.example.com There's no otherwise nice way to parse this string. From tss at iki.fi Sun Mar 4 14:47:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:47:34 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <28E8C0DA-B388-42F0-B39E-B08CA7960D09@iki.fi> On 1.3.2012, at 10.44, Joseph Tam wrote: > I would like to run various doveadm commands that involves all (mail) users like > > doveadm expunge -A mailbox Trash savedbefore 30d > > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users > > However, these accounts are system accounts (locked password, no shell) > and are in userdb to provide UID<->name mapping for utilities like ls, > chown, etc. What userdb are you using? userdb passwd should already skip users that aren't in the valid range. And what Dovecot version are you using? From tss at iki.fi Sun Mar 4 14:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:48:53 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f From tss at iki.fi Sun Mar 4 14:51:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 14:51:03 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On 1.3.2012, at 10.44, Joseph Tam wrote: > but any doveadm command that uses "-A" to iterate through all users will > stop processing at the first account with UID > doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop > privileges: Mail access for users with GID 5551 not permitted > (see first_valid_gid in config file, gid from userdb lookup). > doveadm(sysdaemon): Error: User init failed > doveadm: Error: Failed to iterate through some users And one more thing: Does it really even stop there? Looking at the code it's supposed to log an error and continue to next user. Note that it says "Failed to iterate through SOME users". From tss at iki.fi Sun Mar 4 15:32:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:32:15 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: <20120224012247.GA6512@krell.zikzak.de> <1330342560.11500.308.camel@innu.invalid> Message-ID: On 29.2.2012, at 3.03, Andreas M. Kirchwitz wrote: > Timo Sirainen wrote: > >>> There seems to be a new dependency in some modules (eg, lib-storage, >>> libdovecot-lda, libdovecot-ssl) on OpenSSL. In Dovecot 2.0, those >>> modules didn't require OpenSSL, but 2.1 does. >>> >>> For the linking process the path to the OpenSSL library isn't >>> specified properly (SSL_LIBS). Dovecot fails to build if OpenSSL >>> is in a non-standard path. (Haven't checked if SSL_CFLAGS isn't >>> properly used as well.) >> >> Maybe http://hg.dovecot.org/dovecot-2.1/rev/c07415305d9e fixes >> everything? > > That's the way to go. Makes things better, but I've found three more > dependencies. This patch is against the daily snapshot 20120228. > (Sorry for the changes to Makefile.in which you won't need. But this way > I don't have to rebuild Makefile.in from Makefile.am when compiling. ;-) The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 From tss at iki.fi Sun Mar 4 15:36:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 15:36:59 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> On 28.2.2012, at 19.45, Steve Platt wrote: > Most of this is working but I'm stuck on how to convert users' mail folders > from the existing setup to the new one. I'm using the convert plugin but of > course the problem is that the plugin executes as the "vmail" user and cannot > access the existing mail folders that belong to the users: and I'd be worried > if it could, of course! Convert plugin also has some other problems. > I have the idea that I should be able to run some command (as a privileged > user) on the mail server and have it do the conversion for me, changing the > ownership/permissions on the way. > > Can convert-tool do this? Possibly, but I remember it had some problems. The best solution would be to use Dovecot v2.0's dsync. Also you can use one of the scripts in http://wiki2.dovecot.org/Migration/MailFormat such as mb2md. > I'd prefer to go with the automatic (plugin) conversion if I can bodge the > ownership issues somehow. Failing that, some tool or script may be the next > best answer. You could set mail_drop_priv_before_exec=yes, mail_access_groups=vmail and chgrp vmail, chmod g+rw the old mailboxes. From tss at iki.fi Sun Mar 4 16:10:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:10:28 +0200 Subject: [Dovecot] [PATCH] Pop3 order in courier migration script In-Reply-To: <4F4B2F62.1020204@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> Message-ID: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> On 27.2.2012, at 9.23, Christoph Bu?enius wrote: > I found a problem in the courier conversion script (courier-dovecot-migrate.pl). In some cases, it does not correctly preserve the order of POP3 UIDLs. Thanks, updated. BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. From tss at iki.fi Sun Mar 4 16:23:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:23:39 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120302104333.GD11180@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: >> Alternatively you can just tell Dovecot not to care about it: maildir_broken_filename_sizes=yes. Although you probably can't do that if you have compressed mails. > > In the case above that mail was gzipped twice :( Yes, looks like Dovecot can't correctly fix the wrong S size for gzipped mails. I don't know if I should bother fixing it, especially since in your case the doubly-gzipped mails will look corrupted to user.. From terry at cnysupport.com Sun Mar 4 16:48:17 2012 From: terry at cnysupport.com (Terry Carmen) Date: Sun, 04 Mar 2012 09:48:17 -0500 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> Message-ID: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> ----- Message from Timo Sirainen ---------    Date: Sun, 4 Mar 2012 14:45:48 +0200    From: Timo Sirainen Subject: Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location      To: Terry Carmen      Cc: dovecot at dovecot.org > On 2.3.2012, at 0.35, Terry Carmen wrote: >> With the exchange server being returned in the msExchHomeServerName >> property as: >> >> /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername >> >> I believe this should somehow end up in the userdb section, >> which currently contains "driver = prefetch", but can't seem to >> figure out specifically what should be there. > .. > The only important part is "cn=exchangeservername", which is > the machine name and would need to be prepended to example.com to > get the fqdn. > Do all of the values have the same prefix? Then I guess you can do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%49.100$.example.com > > If the prefix differs, but all of the exchange server names have > the same length, for example 10, you can also do: > > pass_attrs = ..., \ > msExchHomeServerName=userdb_imapc_host=%-10$.example.com > There's no otherwise nice way to parse this string. If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). homeservers.txt: user1:exch1.example.com user2:exch1.example.com user3:exch1.example.com user4:exch2.example.com Is it possible to do a lookup in a text file to get this? Terry From tss at iki.fi Sun Mar 4 16:58:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 4 Mar 2012 16:58:59 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> Message-ID: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> On 4.3.2012, at 16.48, Terry Carmen wrote: >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >> >> If the prefix differs, but all of the exchange server names have the same length, for example 10, you can also do: >> >> pass_attrs = ..., \ >> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >> There's no otherwise nice way to parse this string. > > > If by prefix, you mean the "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, they're different. OK, so if the prefix or suffix isn't always the same length you can't do the above. > I could export the data to a text file as username:homeexchangeserver (or whatever other format is needed). > > homeservers.txt: > user1:exch1.example.com > user2:exch1.example.com > user3:exch1.example.com > user4:exch2.example.com > > Is it possible to do a lookup in a text file to get this? If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile Example line: user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com Note that you can't then return any userdb fields from passdb ldap lookup. From CMarcus at Media-Brokers.com Sun Mar 4 17:27:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:27:40 -0500 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <4F5389EC.1040503@Media-Brokers.com> On 2012-03-04 7:36 AM, Timo Sirainen wrote: > So, doveconf -n> dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. One suggestion... since 'doveconf -n' is sort of a clone of 'postconf -n', maybe it would be a good idea to clone the postfix way for upgrading the configuration file as well? Postfix does it as: postfix upgrade-configuration Man page details for options are here: http://www.postfix.org/postfix.1.html I hate to keep suggesting that you 'copy' anyone or anything, but if you *are*, I think postfix is one of the ones you'd want to emulate... ;) Just a thought... it would be more intuitive for those of us who use postfix with dovecot (and I think there are a lot)... -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 4 17:57:45 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 10:57:45 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> Message-ID: <4F5390F9.4000301@Media-Brokers.com> On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: > On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >> On 2012-03-02 5:13 PM, Robin wrote: >>> This mailing list is for dovecot, not Thunderbird support. The lack of >>> replies to Thunderbird usage questions no doubt reflects this. >> What precisely about a possible bug with *any* IMAP client when >> using dovecot+fts makes you think that this is not on topic for the >> dovecot list? > Show dovecot misbehaving. The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:21:13 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:21:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? > > The bottom line, though, until it can be determined that it *is* a Thunderbird bug, we won't know if it is a dovecot bug or not, will we? well as I wrote in the mini-tutorial, if you use roundcube search, you will see that it uses solr. So from my point of view it would be a Thunderbird thing. -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From CMarcus at Media-Brokers.com Sun Mar 4 18:42:04 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 11:42:04 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> Message-ID: <4F539B5C.3030003@Media-Brokers.com> On 2012-03-04 11:21 AM, Christian Roessner wrote: >> The OP showed where *something* was misbehaving - maybe you should >> read an entire thread before jumping in? >> >> The bottom line, though, until it can be determined that it *is* a >> Thunderbird bug, we won't know if it is a dovecot bug or not, will >> we? > well as I wrote in the mini-tutorial, if you use roundcube search, ? First post from you in this thread, much less a reference to some mini-tutorial you wrote > you will see that it uses solr. So from my point of view it would be > a Thunderbird thing. As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... Looks like he isn't interested in replying (or has unsubbed from the list), so looks like there is no point in pursuing this at this point. I'll just have to wait until we get switched over to dovecot, and see if we have any problems with fts... -- Best regards, Charles From c at roessner-network-solutions.com Sun Mar 4 18:59:42 2012 From: c at roessner-network-solutions.com (Christian Roessner) Date: Sun, 4 Mar 2012 17:59:42 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F539B5C.3030003@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> Message-ID: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> >>> > >> well as I wrote in the mini-tutorial, if you use roundcube search, > > ? First post from you in this thread, much less a reference to some mini-tutorial you wrote This: http://www.roessner-network-solutions.com/2012/02/19/full-text-search-with-solr-and-dovecot-on-ubuntu-10-04/ is from my blog ;) > >> you will see that it uses solr. So from my point of view it would be >> a Thunderbird thing. > > As I said, I would like confirmation *from the OP* about his last comment that his problem with Thunderbird was actually fixed by fixing whatever 'third party init script who was the problem'... > ok -Christian --- Roessner-Network-Solutions Bachelor of Science Informatik Nahrungsberg 81, 35390 Gie?en F: +49 641 5879091, M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com From bradley.giesbrecht at gmail.com Sun Mar 4 19:39:37 2012 From: bradley.giesbrecht at gmail.com (Bradley Giesbrecht) Date: Sun, 4 Mar 2012 09:39:37 -0800 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F5390F9.4000301@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: On Mar 4, 2012, at 7:57 AM, Charles Marcus wrote: > On 2012-03-03 4:16 PM, Bradley Giesbrecht wrote: >> On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: >>> On 2012-03-02 5:13 PM, Robin wrote: >>>> This mailing list is for dovecot, not Thunderbird support. The lack of >>>> replies to Thunderbird usage questions no doubt reflects this. > >>> What precisely about a possible bug with *any* IMAP client when >>> using dovecot+fts makes you think that this is not on topic for the >>> dovecot list? > >> Show dovecot misbehaving. > > The OP showed where *something* was misbehaving - maybe you should read an entire thread before jumping in? I have been reading this thread from the beginning. You asked the question: On Mar 3, 2012, at 12:14 PM, Charles Marcus wrote: > What precisely about a possible bug with *any* IMAP client when using dovecot+fts makes you think that this is not on topic for the dovecot list? It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. At what point should this issue be taken to a Thunderbird support venue? I will butt out now. From CMarcus at Media-Brokers.com Sun Mar 4 20:03:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 04 Mar 2012 13:03:14 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <76FCDC9A-9574-4D3D-918E-9ED15BB330F8@roessner-network-solutions.com> <4F539B5C.3030003@Media-Brokers.com> <28197240-C50C-4BE0-91F7-465B14137085@roessner-network-solutions.com> Message-ID: <4F53AE62.3000005@Media-Brokers.com> On 2012-03-04 11:59 AM, Christian Roessner wrote: >>> you will see that it uses solr. So from my point of view it would be >>> a Thunderbird thing. >> As I said, I would like confirmation *from the OP* about his last >> comment that his problem with Thunderbird was actually fixed by fixing >> whatever 'third party init script who was the problem'... > ok One other thing I neglected to mention - I'll probably use fts+lucene, since it appears to be simpler (is a plugin and only requires clucene as a dependency)... Next is to push for full support in Thunderbird for a per account config option to simply run all searches on the server Should only be enabled for an account the server for which support fts search indexes): https://bugzilla.mozilla.org/show_bug.cgi?id=564168 -- Best regards, Charles From trashcan at odo.in-berlin.de Sun Mar 4 23:39:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Sun, 4 Mar 2012 22:39:22 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Hi -- On 04.03.2012, at 12:38, Timo Sirainen wrote: > On 4.3.2012, at 13.34, Michael Grimm wrote: >> On 04.03.2012, at 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. >> >> Great news. I would love to test it, if I will be able to run this on a test >> account, only. All other users should become synced the "old way" for the time >> being. >> >> Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration > to return only one test user for replicator that avoids it. (You may be able to > do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > 2) You can enable replication plugin only for one user by changing mail_plugins > setting via userdb extra fields. > > Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So > you could even ignore 1) and just let it sync everyone at startup. Does that mean that the new functionality (queue) does only run dsync replication the usual way whenever new mail arrives? That's at least what I read in your code committed today (but I'm not that good in reading code I do have to confess). If you could approve my assumption, I'm willing to give it a try to all users. Regards, Michael From stan at hardwarefreak.com Mon Mar 5 00:29:32 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 16:29:32 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> Message-ID: <4F53ECCC.7060302@hardwarefreak.com> On 3/4/2012 11:39 AM, Bradley Giesbrecht wrote: > It has been demonstrated that dovecot+fts is working properly and that this not a dovecot issue. Most software contains workarounds to bugs/misfeatures in other vendors' programs. Dovecot already has many: NFS: mmap_disable = no mail_nfs_index = no Workarounds for various client bugs: delay-newmail: netscape-eoh: tb-extra-mailbox-sep: To state that a problem in other software that interacts with Dovecot is not worth discussing seems a bit naive, or arrogant, or both. Given how long it takes, never in some cases, for Mozilla to fix IMAP related problems in TBird, you can't blame the OP for looking in other directions for a solution. Note the bug I filed 2+ years on broken IMAP custom header search: https://bugzilla.mozilla.org/show_bug.cgi?id=546925 2 years later and it's not even been assigned to a dev... -- Stan From p at state-of-mind.de Mon Mar 5 00:47:13 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Sun, 04 Mar 2012 23:47:13 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53ECCC.7060302@hardwarefreak.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> Message-ID: <4F53F0F1.9010002@state-of-mind.de> On 04.03.2012 23:29, Stan Hoeppner wrote: > not worth discussing seems a bit naive, or arrogant, or both. Given how > long it takes, never in some cases, for Mozilla to fix IMAP related > problems in TBird, you can't blame the OP for looking in other > directions for a solution. Note the bug I filed 2+ years on broken IMAP > custom header search: > > https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > > 2 years later and it's not even been assigned to a dev... We started buying features/fixes. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5372 bytes Desc: S/MIME Cryptographic Signature URL: From stan at hardwarefreak.com Mon Mar 5 01:07:36 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 04 Mar 2012 17:07:36 -0600 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F0F1.9010002@state-of-mind.de> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> Message-ID: <4F53F5B8.8070105@hardwarefreak.com> On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how >> long it takes, never in some cases, for Mozilla to fix IMAP related >> problems in TBird, you can't blame the OP for looking in other >> directions for a solution. Note the bug I filed 2+ years on broken IMAP >> custom header search: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 >> >> 2 years later and it's not even been assigned to a dev... > > We started buying features/fixes. Does Mozilla have a page listing such services and prices, err, required/expected donation amounts? -- Stan From sdavies at sdc.com.au Mon Mar 5 01:18:40 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Mon, 5 Mar 2012 09:48:40 +1030 Subject: [Dovecot] Log sybnch error In-Reply-To: <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> References: <201203021244.05034.sdavies@sdc.com.au> <201203031015.02716.sdavies@sdc.com.au> <4141EB43-EA6C-49AF-839A-A7C3F43E2E81@iki.fi> Message-ID: <201203050948.40819.sdavies@sdc.com.au> Sorry. I wasn't clear. The message did repeat for the same mailbox. eg Mar 5 09:41:40 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42304 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:41:44 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42392 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42480 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Drafts/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Templates/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=12964 for /home/scldad/Mail/Mail/.imap/Sent/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=42568 for /home/scldad/Mail/Mail/.imap/Trash/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Junk/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=11864 for /home/scldad/Mail/Mail/.imap/Outbox/dovecot.index: Extension header update points outside header size Mar 5 09:42:58 server dovecot: imap(scldad): Error: Log synchronization error at seq=2,offset=60240 for /home/scldad/Mail/Mail/.imap/storage1/dovecot.index: Extension header update points outside header size I have deleted the .imap directories and the message seems to have disappeared. Cheers and thanks, Stephen On Sun, 4 Mar 2012 11:05:18 PM Timo Sirainen wrote: > On 3.3.2012, at 1.45, Stephen Davies wrote: > > No NFS. The file system is local. > > > > Yes. There are multiple copies of the message for multiple mailboxes for > > each of at least two users. > > But does the error keep repeating for the same mailbox? It's supposed to > fix itself automatically after logging the error once. > > > Yes. Did recently upgrade from 1.2.15. > > I think in earlier versions mbox used somewhat different index file > structures and now Dovecot logs some errors about them. > > Anyway, one sure way to fix this is to just delete all the .imap/ > directories. -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From amk at spamfence.net Mon Mar 5 02:32:17 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Mon, 5 Mar 2012 00:32:17 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: Message-ID: Hello Timo! Timo Sirainen wrote: > The SSL libraries shouldn't be linked when imapc isn't built. Also it probably wouldn't have built with you if you used configure --without-shared-libs. I did a bit large change that hopefully fixes everything: > http://hg.dovecot.org/dovecot-2.1/rev/e540404debb7 Thanks for this patch. I've applied it to the dovecot-20120303 nightly snapshot. The good news is, compilation works fine. The bad news is, the libraries and binaries don't work because they don't find the custom SSL libraries. Greetings, Andreas =============================================================================== $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install =============================================================================== $ ldd src/*/.libs/*.so src/auth/.libs/libauthdb_imap.so: linux-gate.so.1 => (0x0013a000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0091d000) librt.so.1 => /lib/librt.so.1 (0x003c7000) libc.so.6 => /lib/libc.so.6 (0x00c6a000) libdl.so.2 => /lib/libdl.so.2 (0x009a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00491000) /lib/ld-linux.so.2 (0x007a4000) src/lib-dovecot/.libs/libdovecot.so: linux-gate.so.1 => (0x0053c000) libdl.so.2 => /lib/libdl.so.2 (0x0056d000) librt.so.1 => /lib/librt.so.1 (0x00925000) libc.so.6 => /lib/libc.so.6 (0x00626000) /lib/ld-linux.so.2 (0x00a61000) libpthread.so.0 => /lib/libpthread.so.0 (0x003ec000) src/lib-lda/.libs/libdovecot-lda.so: linux-gate.so.1 => (0x00b75000) libdovecot-storage.so.0 => /usr/local/src/dovecot-20120303/src/lib-storage/.libs/libdovecot-storage.so.0 (0x00c9a000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0062d000) librt.so.1 => /lib/librt.so.1 (0x00b3d000) libc.so.6 => /lib/libc.so.6 (0x00110000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x002f1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ab1000) /lib/ld-linux.so.2 (0x00f23000) src/lib-sql/.libs/libdovecot-sql.so: linux-gate.so.1 => (0x006d3000) libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x0096c000) libdl.so.2 => /lib/libdl.so.2 (0x0078c000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x00119000) /lib/ld-linux.so.2 (0x00731000) libpthread.so.0 => /lib/libpthread.so.0 (0x00569000) src/lib-ssl-iostream/.libs/libdovecot-ssl.so: linux-gate.so.1 => (0x00ea1000) libdl.so.2 => /lib/libdl.so.2 (0x00b31000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00110000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00dcf000) librt.so.1 => /lib/librt.so.1 (0x00fa5000) libc.so.6 => /lib/libc.so.6 (0x002d3000) /lib/ld-linux.so.2 (0x002b4000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d3c000) src/lib-storage/.libs/libdovecot-storage.so: linux-gate.so.1 => (0x002ee000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00395000) libdl.so.2 => /lib/libdl.so.2 (0x00958000) librt.so.1 => /lib/librt.so.1 (0x00333000) libc.so.6 => /lib/libc.so.6 (0x00d45000) /lib/ld-linux.so.2 (0x008e9000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f06000) src/login-common/.libs/libdovecot-login.so: linux-gate.so.1 => (0x00d66000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/src/dovecot-20120303/src/lib-dovecot/.libs/libdovecot.so.0 (0x00c82000) librt.so.1 => /lib/librt.so.1 (0x00f64000) libc.so.6 => /lib/libc.so.6 (0x00110000) libdl.so.2 => /lib/libdl.so.2 (0x00b26000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /lib/ld-linux.so.2 (0x00520000) =============================================================================== $ ldd /usr/local/Dovecot-20120303/lib/dovecot/*.so /usr/local/Dovecot-20120303/*bin/* /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so: linux-gate.so.1 => (0x00230000) librt.so.1 => /lib/librt.so.1 (0x00b69000) libc.so.6 => /lib/libc.so.6 (0x00231000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a82000) /lib/ld-linux.so.2 (0x007eb000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_imap_acl_plugin.so: linux-gate.so.1 => (0x004fb000) lib01_acl_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib01_acl_plugin.so (0x00c6f000) librt.so.1 => /lib/librt.so.1 (0x0061d000) libc.so.6 => /lib/libc.so.6 (0x001dd000) libpthread.so.0 => /lib/libpthread.so.0 (0x00fae000) /lib/ld-linux.so.2 (0x00b89000) /usr/local/Dovecot-20120303/lib/dovecot/lib02_lazy_expunge_plugin.so: linux-gate.so.1 => (0x00e5b000) librt.so.1 => /lib/librt.so.1 (0x00847000) libc.so.6 => /lib/libc.so.6 (0x00110000) libpthread.so.0 => /lib/libpthread.so.0 (0x0039d000) /lib/ld-linux.so.2 (0x00a4e000) /usr/local/Dovecot-20120303/lib/dovecot/lib05_snarf_plugin.so: linux-gate.so.1 => (0x001e2000) librt.so.1 => /lib/librt.so.1 (0x00441000) libc.so.6 => /lib/libc.so.6 (0x00220000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cfa000) /lib/ld-linux.so.2 (0x00acd000) /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so: linux-gate.so.1 => (0x00fa9000) librt.so.1 => /lib/librt.so.1 (0x00bb4000) libc.so.6 => /lib/libc.so.6 (0x00d5e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00be7000) /lib/ld-linux.so.2 (0x0055c000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_imap_quota_plugin.so: linux-gate.so.1 => (0x00426000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x004c9000) librt.so.1 => /lib/librt.so.1 (0x00e35000) libc.so.6 => /lib/libc.so.6 (0x009b5000) libpthread.so.0 => /lib/libpthread.so.0 (0x00146000) /lib/ld-linux.so.2 (0x00507000) /usr/local/Dovecot-20120303/lib/dovecot/lib11_trash_plugin.so: linux-gate.so.1 => (0x00baf000) lib10_quota_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib10_quota_plugin.so (0x00d8b000) librt.so.1 => /lib/librt.so.1 (0x0041e000) libc.so.6 => /lib/libc.so.6 (0x00bb6000) libpthread.so.0 => /lib/libpthread.so.0 (0x0088a000) /lib/ld-linux.so.2 (0x00a6d000) /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00f77000) libc.so.6 => /lib/libc.so.6 (0x00146000) libpthread.so.0 => /lib/libpthread.so.0 (0x00df9000) /lib/ld-linux.so.2 (0x004dd000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_autocreate_plugin.so: linux-gate.so.1 => (0x005d2000) librt.so.1 => /lib/librt.so.1 (0x007ed000) libc.so.6 => /lib/libc.so.6 (0x00262000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b48000) /lib/ld-linux.so.2 (0x00243000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_expire_plugin.so: linux-gate.so.1 => (0x00110000) librt.so.1 => /lib/librt.so.1 (0x0099e000) libc.so.6 => /lib/libc.so.6 (0x0013c000) libpthread.so.0 => /lib/libpthread.so.0 (0x00a93000) /lib/ld-linux.so.2 (0x00460000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so: linux-gate.so.1 => (0x002c1000) librt.so.1 => /lib/librt.so.1 (0x00ef2000) libc.so.6 => /lib/libc.so.6 (0x00c3a000) libpthread.so.0 => /lib/libpthread.so.0 (0x0028d000) /lib/ld-linux.so.2 (0x003f5000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_listescape_plugin.so: linux-gate.so.1 => (0x0027c000) librt.so.1 => /lib/librt.so.1 (0x007cf000) libc.so.6 => /lib/libc.so.6 (0x00427000) libpthread.so.0 => /lib/libpthread.so.0 (0x00110000) /lib/ld-linux.so.2 (0x00d25000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_mail_log_plugin.so: linux-gate.so.1 => (0x00cff000) lib15_notify_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib15_notify_plugin.so (0x00a2b000) librt.so.1 => /lib/librt.so.1 (0x00152000) libc.so.6 => /lib/libc.so.6 (0x00355000) libpthread.so.0 => /lib/libpthread.so.0 (0x00680000) /lib/ld-linux.so.2 (0x00c16000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_virtual_plugin.so: linux-gate.so.1 => (0x0074d000) librt.so.1 => /lib/librt.so.1 (0x008d2000) libc.so.6 => /lib/libc.so.6 (0x004c2000) libpthread.so.0 => /lib/libpthread.so.0 (0x009c5000) /lib/ld-linux.so.2 (0x0038e000) /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so: linux-gate.so.1 => (0x004ab000) libz.so.1 => /lib/libz.so.1 (0x0091c000) libbz2.so.1 => /lib/libbz2.so.1 (0x00c4d000) librt.so.1 => /lib/librt.so.1 (0x00b30000) libc.so.6 => /lib/libc.so.6 (0x0075b000) libpthread.so.0 => /lib/libpthread.so.0 (0x00611000) /lib/ld-linux.so.2 (0x00bfd000) /usr/local/Dovecot-20120303/lib/dovecot/lib21_fts_squat_plugin.so: linux-gate.so.1 => (0x00df4000) lib20_fts_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_fts_plugin.so (0x00ea8000) librt.so.1 => /lib/librt.so.1 (0x0021d000) libc.so.6 => /lib/libc.so.6 (0x00bd1000) libpthread.so.0 => /lib/libpthread.so.0 (0x00d63000) /lib/ld-linux.so.2 (0x00b83000) /usr/local/Dovecot-20120303/lib/dovecot/lib30_imap_zlib_plugin.so: linux-gate.so.1 => (0x0059f000) lib20_zlib_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib20_zlib_plugin.so (0x00652000) librt.so.1 => /lib/librt.so.1 (0x00f2d000) libc.so.6 => /lib/libc.so.6 (0x00735000) libz.so.1 => /lib/libz.so.1 (0x00110000) libbz2.so.1 => /lib/libbz2.so.1 (0x00125000) libpthread.so.0 => /lib/libpthread.so.0 (0x00136000) /lib/ld-linux.so.2 (0x001f9000) /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so: linux-gate.so.1 => (0x004c0000) librt.so.1 => /lib/librt.so.1 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x006b2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00992000) /lib/ld-linux.so.2 (0x00d92000) /usr/local/Dovecot-20120303/lib/dovecot/lib95_imap_stats_plugin.so: linux-gate.so.1 => (0x0076b000) lib90_stats_plugin.so => /usr/local/Dovecot-20120303/lib/dovecot/lib90_stats_plugin.so (0x00b90000) librt.so.1 => /lib/librt.so.1 (0x00215000) libc.so.6 => /lib/libc.so.6 (0x0021e000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /lib/ld-linux.so.2 (0x00e48000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-lda.so: linux-gate.so.1 => (0x00d82000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x009bb000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x00542000) libc.so.6 => /lib/libc.so.6 (0x002f6000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x006a6000) libpthread.so.0 => /lib/libpthread.so.0 (0x00189000) /lib/ld-linux.so.2 (0x002d7000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-login.so: linux-gate.so.1 => (0x00f96000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00e56000) librt.so.1 => /lib/librt.so.1 (0x00371000) libc.so.6 => /lib/libc.so.6 (0x00168000) libdl.so.2 => /lib/libdl.so.2 (0x00623000) libpthread.so.0 => /lib/libpthread.so.0 (0x00ccd000) /lib/ld-linux.so.2 (0x0074f000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so: linux-gate.so.1 => (0x00bf3000) libdl.so.2 => /lib/libdl.so.2 (0x00a54000) librt.so.1 => /lib/librt.so.1 (0x00ad8000) libc.so.6 => /lib/libc.so.6 (0x00e63000) /lib/ld-linux.so.2 (0x00b1c000) libpthread.so.0 => /lib/libpthread.so.0 (0x005ac000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-sql.so: linux-gate.so.1 => (0x008b7000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00909000) libdl.so.2 => /lib/libdl.so.2 (0x005f5000) librt.so.1 => /lib/librt.so.1 (0x008bc000) libc.so.6 => /lib/libc.so.6 (0x00675000) /lib/ld-linux.so.2 (0x004bc000) libpthread.so.0 => /lib/libpthread.so.0 (0x00184000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-ssl.so: linux-gate.so.1 => (0x00ef2000) libdl.so.2 => /lib/libdl.so.2 (0x0033f000) libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00a3d000) libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x0034e000) librt.so.1 => /lib/librt.so.1 (0x002ea000) libc.so.6 => /lib/libc.so.6 (0x00110000) /lib/ld-linux.so.2 (0x007d4000) libpthread.so.0 => /lib/libpthread.so.0 (0x0029a000) /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so: linux-gate.so.1 => (0x0089e000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00be9000) libdl.so.2 => /lib/libdl.so.2 (0x00852000) librt.so.1 => /lib/librt.so.1 (0x001aa000) libc.so.6 => /lib/libc.so.6 (0x00442000) /lib/ld-linux.so.2 (0x00b8a000) libpthread.so.0 => /lib/libpthread.so.0 (0x00b0e000) /usr/local/Dovecot-20120303/bin/doveadm: linux-gate.so.1 => (0x00c6a000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00110000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00acf000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x0066e000) libc.so.6 => /lib/libc.so.6 (0x00247000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x001f4000) librt.so.1 => /lib/librt.so.1 (0x00ab5000) libfreebl3.so => /lib/libfreebl3.so (0x003d1000) /lib/ld-linux.so.2 (0x00228000) libpthread.so.0 => /lib/libpthread.so.0 (0x00cc3000) /usr/local/Dovecot-20120303/bin/doveconf: linux-gate.so.1 => (0x00830000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x0032a000) libc.so.6 => /lib/libc.so.6 (0x00876000) libdl.so.2 => /lib/libdl.so.2 (0x00110000) librt.so.1 => /lib/librt.so.1 (0x002b6000) /lib/ld-linux.so.2 (0x007d2000) libpthread.so.0 => /lib/libpthread.so.0 (0x00634000) /usr/local/Dovecot-20120303/bin/dsync: linux-gate.so.1 => (0x00c8b000) libdovecot-storage.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot-storage.so.0 (0x00257000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00662000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00d15000) libc.so.6 => /lib/libc.so.6 (0x00dd1000) libcrypto.so.1.0.0 => not found libssl.so.1.0.0 => not found libdl.so.2 => /lib/libdl.so.2 (0x009b3000) librt.so.1 => /lib/librt.so.1 (0x0037f000) libfreebl3.so => /lib/libfreebl3.so (0x00110000) /lib/ld-linux.so.2 (0x00a8e000) libpthread.so.0 => /lib/libpthread.so.0 (0x001e9000) /usr/local/Dovecot-20120303/sbin/dovecot: linux-gate.so.1 => (0x00f00000) libcap.so.2 => /lib/libcap.so.2 (0x0037c000) libdovecot.so.0 => /usr/local/Dovecot-20120303/lib/dovecot/libdovecot.so.0 (0x00555000) libc.so.6 => /lib/libc.so.6 (0x0080b000) libattr.so.1 => /lib/libattr.so.1 (0x00ece000) libdl.so.2 => /lib/libdl.so.2 (0x00ea2000) librt.so.1 => /lib/librt.so.1 (0x00dce000) /lib/ld-linux.so.2 (0x00d49000) libpthread.so.0 => /lib/libpthread.so.0 (0x00bb2000) =============================================================================== From isolecki at gmail.com Mon Mar 5 03:42:12 2012 From: isolecki at gmail.com (Ian Solecki) Date: Sun, 4 Mar 2012 20:42:12 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine Message-ID: Hello, hoping someone here might be able to help me or at least point me in the right direction. My company recently (last week) moved to a new dedicated server for website and email hosting. It is a fairly run-of-the-mill Linux machine running cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail server. When we first set up the server, I was able to create a mailbox, access it via any desktop mail client, webmail, AND by setting it up as a basic POP3 account on my BlackBerry. Mail was running fine to and from the BlackBerry, no problems. I deleted that account from my BlackBerry as it was a test account, and went to add my actual account but was unable to do so. Received the "Cannot log in. Verify your email address, user name and password. If the error persists, contact (my domain name)" message. I tried the test account that had been running successfully and sending/receiving emails not minutes earlier, and it would not set up either. I have since tried multiple accounts on multiple BlackBerry devices (different models) on multiple carriers on several of my different domains (all of which point to the same server, of course), to no avail. Yet, any of these accounts still work flawlessly in any desktop mail client (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these BlackBerrys work flawlessly with any other mail server. The username to log in to these mailboxes is not standard, it's mailboxname+ example.com if the email address is mailboxname at example.com and the mail server is mail.example.com. So, I know I have to access the "Advanced Settings" in BlackBerry email setup in order to put this username in. Still, no effect. So, there is something wrong with how mail SETUP works (not sending/receiving, though that may also not work, I have no way of knowing now) between RIM and my server, and it's something that has changed since the server was set up a week ago. My carrier(s) are clueless, my dedicated server provider (Lunarpages) is clueless. Can anyone help? From gedalya at gedalya.net Mon Mar 5 04:52:06 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 04 Mar 2012 21:52:06 -0500 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F542A56.6070104@gedalya.net> Do you have the dovecot logs? What do they say about connections coming from RIM? On 03/04/2012 08:42 PM, Ian Solecki wrote: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > From dchenusa at yahoo.com Mon Mar 5 05:57:16 2012 From: dchenusa at yahoo.com (Dennis Chen) Date: Sun, 4 Mar 2012 19:57:16 -0800 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> Message-ID: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Thanks Timo, I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? I also posted the similar question to ubuntuforums, but not much response. Hope you understand my point. It's very frustrated! Sent from my iPhone On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > On 4.3.2012, at 8.25, D Chen wrote: > >> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! > > v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: > >> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf > > So, doveconf -n > dovecot-new.conf > mv dovecot-new.conf /etc/dovecot/dovecot.conf > > That should do it. > From tss at iki.fi Mon Mar 5 08:41:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:41:36 +0200 Subject: [Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ... In-Reply-To: <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> References: <1330842312.50410.YahooMailNeo@web161605.mail.bf1.yahoo.com> <451532F5-3070-47B3-951E-0A91DFC77207@iki.fi> <76EB5093-B9CA-42F8-9679-C1E3F30CD3C4@yahoo.com> Message-ID: Dovecot v2.0 changed the settings a lot compared to v1.x. There are no longer auth default or socket listen sections. The doveconf -n generated dovecot.conf should contain all of the settings that you had in v1.x, converted for v2.0. So you should be able to use it directly without problems. If you want, you could look for Dovecot's example-config that probably comes with Ubuntu (in /usr/share/doc/dovecot*/ maybe?), copy those to /etc/dovecot/ and change the settings in there based on the generated dovecot.conf. In any case you shouldn't try to add those v1.x-specific things back there anymore, since they'll add back the "obsolete settings" warnings. On 5.3.2012, at 5.57, Dennis Chen wrote: > Thanks Timo, > > I believe I'm running dovecot 2.0.x when first installed Ubuntu server 11.04 then upgraded to 11.10. The primary reason I posted this question was not only the warning msg but also looking for the default dovecot.conf so that I can modify from the scratch, however, I couldn't find the "auth default" section or the "socket listen" option in the dovecot.conf; note that the "auth default" section and "socket listen" need to be modified according Ubuntu 11.10 serverguide for Dovecot SASL configuration. There is a dovecot.conf.ucf under /etc/dovecot which contain the "auth default" and "socket listen" stuff, I renamed it to dovecot.conf and modified the "auth default" section and "socket listen" option there accordingly. Note that the dovecot.conf.ucf file is about 50k while the new dovecot.conf generated from the "doveconf -n" is about 4k ! Now I'm confused of using of which dovecot.conf I should use (the one generated from the "doveconf -n" or the one renamed from the dovecot.conf.ucf" ?) > > Does your dovecot.conf contain the "auth default" section and the "socket listen" ? If not, should I complain to the Ubuntu serverguide ? > > I also posted the similar question to ubuntuforums, but not much response. > > Hope you understand my point. > > It's very frustrated! > > > Sent from my iPhone > > On Mar 4, 2012, at 4:36 AM, Timo Sirainen wrote: > >> On 4.3.2012, at 8.25, D Chen wrote: >> >>> Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! >> >> v2.0 has different configuration from v1.x, you need to migrate the configuration the way it says: >> >>> doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf >> >> So, doveconf -n > dovecot-new.conf >> mv dovecot-new.conf /etc/dovecot/dovecot.conf >> >> That should do it. >> > From tss at iki.fi Mon Mar 5 08:43:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 08:43:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <3B9E0D19-9833-4E61-9786-17CC0832B41E@odo.in-berlin.de> Message-ID: <86F8DB00-F1B0-4666-B3EC-B3EA25F87C0B@iki.fi> On 4.3.2012, at 23.39, Michael Grimm wrote: >> Anyway, replicator simply runs doveadm, so there's not much that can go wrong. So >> you could even ignore 1) and just let it sync everyone at startup. > > Does that mean that the new functionality (queue) does only run dsync replication > the usual way whenever new mail arrives? That's at least what I read in your code > committed today (but I'm not that good in reading code I do have to confess). > > If you could approve my assumption, I'm willing to give it a try to all users. Yes, the replicator simply runs "doveadm sync -u user at domain -d" (and sometimes with -f). The -d gets the default location from mail_replica setting. From robert at schetterer.org Mon Mar 5 08:48:01 2012 From: robert at schetterer.org (Robert Schetterer) Date: Mon, 05 Mar 2012 07:48:01 +0100 Subject: [Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine In-Reply-To: References: Message-ID: <4F5461A1.7000100@schetterer.org> Am 05.03.2012 02:42, schrieb Ian Solecki: > Hello, hoping someone here might be able to help me or at least point me in > the right direction. > > My company recently (last week) moved to a new dedicated server for website > and email hosting. It is a fairly run-of-the-mill Linux machine running > cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail > server. > > When we first set up the server, I was able to create a mailbox, access it > via any desktop mail client, webmail, AND by setting it up as a basic POP3 > account on my BlackBerry. Mail was running fine to and from the BlackBerry, > no problems. > > I deleted that account from my BlackBerry as it was a test account, and > went to add my actual account but was unable to do so. Received the "Cannot > log in. Verify your email address, user name and password. If the error > persists, contact (my domain name)" message. I tried the test account that > had been running successfully and sending/receiving emails not minutes > earlier, and it would not set up either. > > I have since tried multiple accounts on multiple BlackBerry devices > (different models) on multiple carriers on several of my different domains > (all of which point to the same server, of course), to no avail. > > Yet, any of these accounts still work flawlessly in any desktop mail client > (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these > BlackBerrys work flawlessly with any other mail server. > > The username to log in to these mailboxes is not standard, it's mailboxname+ > example.com if the email address is mailboxname at example.com and the mail > server is mail.example.com. So, I know I have to access the "Advanced > Settings" in BlackBerry email setup in order to put this username in. > Still, no effect. > > So, there is something wrong with how mail SETUP works (not > sending/receiving, though that may also not work, I have no way of knowing > now) between RIM and my server, and it's something that has changed since > the server was set up a week ago. > > My carrier(s) are clueless, my dedicated server provider (Lunarpages) is > clueless. Can anyone help? > this is the dovecot mail list, not support for blackberry, we cant help unless you have dovecot configs and dediacted logs to your problem -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From p at state-of-mind.de Mon Mar 5 08:53:39 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Mon, 5 Mar 2012 07:53:39 +0100 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F53F5B8.8070105@hardwarefreak.com> References: <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F5145FE.3070301@r.paypc.com> <4F527BBF.3060607@Media-Brokers.com> <65364CD3-67CC-4FDD-A7E5-D483AF046FC2@gmail.com> <4F5390F9.4000301@Media-Brokers.com> <4F53ECCC.7060302@hardwarefreak.com> <4F53F0F1.9010002@state-of-mind.de> <4F53F5B8.8070105@hardwarefreak.com> Message-ID: <20120305065339.GC5094@state-of-mind.de> Stan, * Stan Hoeppner : > On 3/4/2012 4:47 PM, Patrick Ben Koetter wrote: > > > On 04.03.2012 23:29, Stan Hoeppner wrote: > > > > > > > >> not worth discussing seems a bit naive, or arrogant, or both. Given how > >> long it takes, never in some cases, for Mozilla to fix IMAP related > >> problems in TBird, you can't blame the OP for looking in other > >> directions for a solution. Note the bug I filed 2+ years on broken IMAP > >> custom header search: > >> > >> https://bugzilla.mozilla.org/show_bug.cgi?id=546925 > >> > >> 2 years later and it's not even been assigned to a dev... > > > > We started buying features/fixes. > > Does Mozilla have a page listing such services and prices, err, > required/expected donation amounts? to my knowledge they don't have a page listing services and prices. Recently they discussed pros and cons of crowd sourcing, but without much progress. I can get you in contact with one of the TB programmers, who implemented features for us, if you want to. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3603 bytes Desc: not available URL: From bra at fsn.hu Mon Mar 5 09:25:34 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 08:25:34 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <4F546A6E.6020400@fsn.hu> Hi, On 03/04/12 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. Everything > isn't finished yet, but it appears to work and I've enabled it for my > @dovecot.fi mails. Some issues: > > - public namespace isn't replicated at all > - shared namespace is replicated, but not private mail flags > - I've only tested SSH replication setup now, not director > replication setup (and director setup is still missing many things) > - SSH replication setup uses aggregator process, which isn't really > necessary and can probably be avoided in future Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. (if I understand things correctly) Thanks for working on this. From janfrode at tanso.net Mon Mar 5 10:35:25 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 09:35:25 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> Message-ID: <20120305083525.GA20889@dibs.tanso.net> On Sun, Mar 04, 2012 at 01:38:14PM +0200, Timo Sirainen wrote: > > > > Great news. I would love to test it, if I will be able to run this on a test > > account, only. All other users should become synced the "old way" for the time > > being. > > > > Would that be possible with the current implementation? > > 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) IMHO it would be great if it didn't sync all users. We probably av have hundreds of thousands of inactive users that we would like to sync at a later point. Also when we provision users that's just an entry in a LDAP-directory without any files or directories. So dovecot shouldn't create any directories for these before they've received mail or logged in. So, ideally (for us), dovecot should keep a log over which accounts are active (has received or checked mail), and only sync users that has been active for the last $timeperiode on startup. -jf From ccourvoisier70 at yahoo.com Mon Mar 5 11:51:19 2012 From: ccourvoisier70 at yahoo.com (Charles C) Date: Mon, 5 Mar 2012 09:51:19 +0000 (GMT) Subject: [Dovecot] 1.0beta to latest and greatest? Message-ID: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> dear dovecot users, This is my first post and with a rather embarassing question. To soften the question up I just want to say I've been very pleased with Dovecot thus far and its low maintenance requirements - perhaps to pleased! I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Thanks for your input. Charles From tss at iki.fi Mon Mar 5 11:56:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 11:56:30 +0200 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> Message-ID: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> On 4.3.2012, at 13.54, Timo Sirainen wrote: > On 4.3.2012, at 13.41, Michael Grimm wrote: > >>> By "undeletable" do you mean you have mails that always come back after expunging them? >> >> Yes. Deleting by the client will return them after the next dsync run. Luckily this just started happening to me as well. After some debugging I found and fixed the problem: http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 From tss at iki.fi Mon Mar 5 12:08:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:08:35 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F546A6E.6020400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: On 5.3.2012, at 9.25, Attila Nagy wrote: > On 03/04/12 11:44, Timo Sirainen wrote: >> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >> > Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. Sure the idea is to improve the performance :) There are two ways: 1) Use longer running SSH sessions which dsync more than one user at a time. 2) Use TCP connections instead of SSH. > It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. From tss at iki.fi Mon Mar 5 12:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:41:30 +0200 Subject: [Dovecot] 1.0beta to latest and greatest? In-Reply-To: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> References: <1330941079.48402.YahooMailNeo@web29406.mail.ird.yahoo.com> Message-ID: On 5.3.2012, at 11.51, Charles C wrote: > I am running an ancient version of Dovecot, version 1.0.beta9. Do I risk corrupting indices etc by upgrading in one go to 2.1.2? Just delete the indexes and you don't have to worry about problems related to them. The config file is different though and you'll probably have to spend some time converting it. (I'm assuming you're using mbox/maildir, not the broken dbox implementation.) From tss at iki.fi Mon Mar 5 12:45:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 12:45:26 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305083525.GA20889@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> Message-ID: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> On 5.3.2012, at 10.35, Jan-Frode Myklebust wrote: >> 1) Replicator syncs all users at startup. If you can change your userdb iteration to return only one test user for replicator that avoids it. (You may be able to do protocol replicator { userdb {..} } and protocol !replicator { .. }) > > IMHO it would be great if it didn't sync all users. We probably av have > hundreds of thousands of inactive users that we would like to sync at a > later point. Also when we provision users that's just an entry in a > LDAP-directory without any files or directories. So dovecot shouldn't > create any directories for these before they've received mail or logged in. > > So, ideally (for us), dovecot should keep a log over which accounts are > active (has received or checked mail), and only sync users that has been > active for the last $timeperiode on startup. Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. From CMarcus at Media-Brokers.com Mon Mar 5 13:13:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:13:07 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F547CAB.2030005@gmail.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <4F4CFB37.2080505@gmail.com> <4F4D009F.7000107@Media-Brokers.com> <4F50BFD6.5010808@Media-Brokers.com> <4F547CAB.2030005@gmail.com> Message-ID: <4F549FC3.5030602@Media-Brokers.com> On 2012-03-05 3:43 AM, kadafax at gmail.com wrote: > Le 02/03/12 13:40, Charles Marcus a ?crit : >> On 2012-02-28 11:28 AM, Charles Marcus wrote: >>> On 2012-02-28 11:05 AM, kfx wrote: >>>> Ok I feel ashame... it was a third party init scrip who was the >>>> problem :( >>> So... you're saying that Thunderbird now correctly uses server side >>> search? >> Please respond... I need to know whether or not I need to pursue this, >> since we use Thunderbird in house and will be switching soon to >> dovecot... > Yes, for me thunderbird correctly use server side search. Just got confirmation from the OP that his problem with Thunderbird indeed is now gone, so it was a 3rd party init script causing his problems... One less thing to worry about... -- Best regards, Charles From CMarcus at Media-Brokers.com Mon Mar 5 13:24:28 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 06:24:28 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <1330441042.2081.24.camel@innu> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> Message-ID: <4F54A26C.4040005@Media-Brokers.com> On 2012-02-28 9:57 AM, Timo Sirainen wrote: > So, Solr in Dovecot works perfectly. Timo, a follow-up on this... Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? I don't have a dovecot test server set up yet, but even if I did, I wouldn't really know what to look for or how to test this myself... Thanks, -- Best regards, Charles From mark at ecs.vuw.ac.nz Mon Mar 5 13:26:22 2012 From: mark at ecs.vuw.ac.nz (Mark Davies) Date: Tue, 06 Mar 2012 00:26:22 +1300 Subject: [Dovecot] GSSAPI auth failing for kmail In-Reply-To: <1330603470.2081.37.camel@innu> References: <4F4AD9AC.5000300@ecs.vuw.ac.nz> <1330338730.11500.306.camel@innu> <4F4B5FFF.9090201@ecs.vuw.ac.nz> <4F4DFAD8.8040002@ecs.vuw.ac.nz> <4F4F6237.1060100@ecs.vuw.ac.nz> <1330602754.2081.34.camel@innu> <4F4F6444.1050107@ecs.vuw.ac.nz> <1330603470.2081.37.camel@innu> Message-ID: <4F54A2DE.5000500@ecs.vuw.ac.nz> On 03/02/12 01:04, Timo Sirainen wrote: > The difference between your previously working system and currently > working system is the GSSAPI/Kerberos libraries. Just to close this thread off, seems that the bug was in the cyrus-sasl libraries that kmail uses. Reverting from the 2.1.25 version that the latest was trying to use to 2.1.23 that was on the older systems got it working again. When I get a moment I'll try and work out what specifically changed. cheers mark From bra at fsn.hu Mon Mar 5 14:15:39 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 13:15:39 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> Message-ID: <4F54AE6B.4060400@fsn.hu> On 03/05/12 11:08, Timo Sirainen wrote: > On 5.3.2012, at 9.25, Attila Nagy wrote: > >> On 03/04/12 11:44, Timo Sirainen wrote: >>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>> >> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. > Sure the idea is to improve the performance :) There are two ways: > > 1) Use longer running SSH sessions which dsync more than one user at a time. > > 2) Use TCP connections instead of SSH. Don't forget about connection pooling to get concurrency. :) BTW, despite being somewhat harder to implement, I personally like native connections better. > >> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. > The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. > It depends. For a moderately loaded server I get this: # time ssh root at be02 "echo 1" 1 0.000u 0.009s 0:00.30 0.0% 0+0k 0+0io 0pf+0w ICMP echo RTT is 0.878 ms. So the ssh connection adds ~29 ms overhead to each sync request. Yes, dsync seems to need some optimizations too. :) I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. The method was similar to yours: - an external library wrote modified user ids to a file - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) The runs were longer and longer... BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... From janfrode at tanso.net Mon Mar 5 14:35:08 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 5 Mar 2012 13:35:08 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> Message-ID: <20120305123508.GA22845@dibs.tanso.net> On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: > > > > So, ideally (for us), dovecot should keep a log over which accounts are > > active (has received or checked mail), and only sync users that has been > > active for the last $timeperiode on startup. > > Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. > .. or we could keep touching /activemailaccounts/$address in post-login scripts, and run "doveadm sync" for any user updated the last $timeperiode and avoid the need for SQL-userdatabase. But we still don't have a last-login update on lmtp delivery... or has this changed? -jf From tss at iki.fi Mon Mar 5 14:48:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 14:48:40 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54AE6B.4060400@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: On 5.3.2012, at 14.15, Attila Nagy wrote: >>> On 03/04/12 11:44, Timo Sirainen wrote: >>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>> >>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >> Sure the idea is to improve the performance :) There are two ways: >> >> 1) Use longer running SSH sessions which dsync more than one user at a time. >> >> 2) Use TCP connections instead of SSH. > Don't forget about connection pooling to get concurrency. :) There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. > BTW, despite being somewhat harder to implement, I personally like native connections better. Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. >>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >> > It depends. For a moderately loaded server I get this: > # time ssh root at be02 "echo 1" I meant doveadm/dsync costs, ssh startup is rather slow. > Yes, dsync seems to need some optimizations too. :) > I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. > The method was similar to yours: > - an external library wrote modified user ids to a file > - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) > > The runs were longer and longer... dsync doesn't currently take enough advantage of modseqs and send only the changed data. > BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... Definitely doesn't help. From apm at one.com Mon Mar 5 15:01:54 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 14:01:54 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST Message-ID: <4F54B942.9070005@one.com> Hi, I noticed a difference between courier and dovecot, and I'm not sure which of them is wrong wrt. RFC3501 - if any. I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: INBOX INBOX.Trash INBOX.INBOX.folder INBOX.INBOX.folder.a INBOX.INBOX.folder.b The INBOX.INBOX folder does not exist on disk and is not subscribed. Courier responds to: . list "" "*" with * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" But dovecot does not list that folder using "*". However, if you issue: . list "" "INBOX.%" Dovecot answers: * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". I know the recommended client way is to use "%", but I'm still curious about which is the correct behaviour. /Peter From bra at fsn.hu Mon Mar 5 15:11:06 2012 From: bra at fsn.hu (Attila Nagy) Date: Mon, 05 Mar 2012 14:11:06 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> Message-ID: <4F54BB6A.20702@fsn.hu> On 03/05/12 13:48, Timo Sirainen wrote: > On 5.3.2012, at 14.15, Attila Nagy wrote: > >>>> On 03/04/12 11:44, Timo Sirainen wrote: >>>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything isn't finished yet, but it appears to work and I've enabled it for my @dovecot.fi mails. Some issues: >>>>> >>>> Do you plan to make it more performant in the future? I mean calling doveadm (and ssh) for every change -even when they are aggregated- seems to be very resource intensive, it won't keep up on a machine with a lot of modifications happening every seconds. >>> Sure the idea is to improve the performance :) There are two ways: >>> >>> 1) Use longer running SSH sessions which dsync more than one user at a time. >>> >>> 2) Use TCP connections instead of SSH. >> Don't forget about connection pooling to get concurrency. :) > There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. Good to hear. > >> BTW, despite being somewhat harder to implement, I personally like native connections better. > Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. Yes. For large installations there may be some backend channel already (SSL tunnels, IPSec etc), so it seems to be OK. > >>>> It would be good to have constantly running daemons on both sides to eliminate the high startup/teardown costs. >>> The process startup/teardown costs are pretty low. I'll need to improve dsync's performance at some point though. Actually I pretty much redesigned the whole dsync already, but I'll probably leave that to v2.2. The current design can still be improved. >>> >> It depends. For a moderately loaded server I get this: >> # time ssh root at be02 "echo 1" > I meant doveadm/dsync costs, ssh startup is rather slow. I see. Running from network makes this worse slightly. Long running processes with long running connections rule. :) > >> Yes, dsync seems to need some optimizations too. :) >> I've tried previously on one pair of our servers with a higher level of concurrency (8-16 or so, I can't remember), and it couldn't keep up with the changes. >> The method was similar to yours: >> - an external library wrote modified user ids to a file >> - in an endless loop a script picked up those (moved the file) and started parallel dsyncs (on ssh) >> >> The runs were longer and longer... > dsync doesn't currently take enough advantage of modseqs and send only the changed data. Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. > >> BTW, we modify the maildirs externally, so this adds a lot of inefficiency here... > Definitely doesn't help. I know, we are working on this. :) From tss at iki.fi Mon Mar 5 16:37:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:37:44 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F54BB6A.20702@fsn.hu> References: <4F53479E.40703@iki.fi> <4F546A6E.6020400@fsn.hu> <4F54AE6B.4060400@fsn.hu> <4F54BB6A.20702@fsn.hu> Message-ID: <8E6FD156-E7F5-49BD-9C3A-1F012E600DD8@iki.fi> On 5.3.2012, at 15.11, Attila Nagy wrote: >> dsync doesn't currently take enough advantage of modseqs and send only the changed data. > Hm. What is your estimate about the performance capability of the current "best" replication scheme available in Dovecot? > I know it's hard to tell, because there are a lot of parameters, but do you think it's good for a real world environment with (10-1000*x :) thousands of users, and a lot of changes? The plan is to get it working with at least a few thousand users to several tens of thousands. > BTW, it would even better to have something scalable as Cassandra, so Dovecout wouldn't have to worry about replication and (read/write) scalability. Yes, that's also in my future plans, but it's a larger change. Also I don't think Cassandra (or any nosql?) still supports application-level merging of data after split brain. From tss at iki.fi Mon Mar 5 16:45:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:45:55 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54B942.9070005@one.com> References: <4F54B942.9070005@one.com> Message-ID: On 5.3.2012, at 15.01, Peter Mogensen wrote: > I have a Maildir which has been accessed by an Apple Mail client, so it got folders like: > > INBOX > INBOX.Trash > INBOX.INBOX.folder > INBOX.INBOX.folder.a > INBOX.INBOX.folder.b > > The INBOX.INBOX folder does not exist on disk and is not subscribed. > > Courier responds to: > . list "" "*" > with > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" I'm surprised Courier would return this. > But dovecot does not list that folder using "*". But it returns all of the mailboxes under INBOX.INBOX, right? > However, if you issue: > . list "" "INBOX.%" > > Dovecot answers: > * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. > This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Also if you used LAYOUT=fs in Dovecot, it would always show the \Noselect mailboxes because they happen to exist physically. From tss at iki.fi Mon Mar 5 16:48:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:48:11 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54A26C.4040005@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> Message-ID: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> On 5.3.2012, at 13.24, Charles Marcus wrote: > On 2012-02-28 9:57 AM, Timo Sirainen wrote: >> So, Solr in Dovecot works perfectly. > > Timo, a follow-up on this... > > Thunderbird has a 'Quickfilter Toolbar' with a little searchbox that applies a filter of the current folder message view pane (to show you only messages in the pane that meet the criteria specified). The default criteria that are selected are just 'Sender', 'Recipients' and 'Subject', but you can also select 'Body' (and a few others like 'Unread', 'Contain attachments', etc), but the 'Body' criteria is the one that would want/need to use the fts indexes, so... > > Can dovecots fts indexes be used in a case like this? In other words, will it 'just work'? Or will it *not* work? Or, would the client need to specify the folder in the commands it sends to limit the search and/or results to just the currently selected folder? Or is this even possible for such a simple/limited use filtering mechanism? The regular IMAP protocol supports searching only from the selected folder. But I guess this quickfilter search also searches from only the selected folder. So I don't see a problem. From tss at iki.fi Mon Mar 5 16:56:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 16:56:21 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <20120305123508.GA22845@dibs.tanso.net> References: <4F53479E.40703@iki.fi> <0D659114-70FE-4D22-827A-57741B20F642@odo.in-berlin.de> <6D8BC4F7-B606-4BD2-BDE8-A0140610445B@iki.fi> <20120305083525.GA20889@dibs.tanso.net> <5B67B2E7-D4A1-4EB3-990E-847D8C415A7D@iki.fi> <20120305123508.GA22845@dibs.tanso.net> Message-ID: On 5.3.2012, at 14.35, Jan-Frode Myklebust wrote: > On Mon, Mar 05, 2012 at 12:45:26PM +0200, Timo Sirainen wrote: >>> >>> So, ideally (for us), dovecot should keep a log over which accounts are >>> active (has received or checked mail), and only sync users that has been >>> active for the last $timeperiode on startup. >> >> Well, all of this could be done already, although not very automatically.. Whenever a new mail is delivered or user is logged in, the user's last-login timestamp in SQL could be updated. And replicator's userdb iterate_query could return only users whose last-login timestamp is new enough. The SQL userdb could be used only by replicator, everything else could keep using LDAP. >> > > .. or we could keep touching /activemailaccounts/$address in post-login > scripts, and run "doveadm sync" for any user updated the last $timeperiode > and avoid the need for SQL-userdatabase. But we still don't have a > last-login update on lmtp delivery... or has this changed? It would be pretty simple to write such a plugin that globally does it for all imap/pop3/lmtp. Here, works for v2.0 and v2.1: http://dovecot.org/patches/2.1/lastaccess-plugin.c From apm at one.com Mon Mar 5 16:56:52 2012 From: apm at one.com (Peter Mogensen) Date: Mon, 05 Mar 2012 15:56:52 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> Message-ID: <4F54D434.6090300@one.com> On 2012-03-05 15:45, Timo Sirainen wrote: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > I'm surprised Courier would return this. > >> But dovecot does not list that folder using "*". > > But it returns all of the mailboxes under INBOX.INBOX, right? Yes. And they exists on disk and are subscribed to. >> However, if you issue: >> . list "" "INBOX.%" >> >> Dovecot answers: >> * LIST (\Noselect \HasChildren) "." "INBOX.INBOX" > > Yes, because if it didn't the client wouldn't know that there are mailboxes under INBOX.INBOX. Seems reasonable. >> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". > > What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. /Peter From tss at iki.fi Mon Mar 5 17:08:02 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 17:08:02 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F54D434.6090300@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> Message-ID: <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> On 5.3.2012, at 16.56, Peter Mogensen wrote: >>> This makes some clients using "*" to get the folder list ignore the folderes below "INBOX.INBOX". >> >> What clients? I haven't heard of this being a problem before. I think Cyrus has similar behavior as Dovecot. > > Well... mostly perl scripts :) - which could probably be changed to use "%" for wildcard, but since they always need to get the entire folder tree it would result in more IMAP traffic. Couldn't the scripts be simply fixed to figure out that if foo.bar.baz is returned without foo or foo.bar, then just internally assume them being there as \noselect? From CMarcus at Media-Brokers.com Mon Mar 5 18:12:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 11:12:19 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> Message-ID: <4F54E5E3.2010509@Media-Brokers.com> On 2012-03-05 9:48 AM, Timo Sirainen wrote: > On 5.3.2012, at 13.24, Charles Marcus wrote: >> Thunderbird has a 'Quickfilter Toolbar' with a little searchbox >> that applies a filter of the current folder message view pane (to >> show you only messages in the pane that meet the criteria >> specified). The default criteria that are selected are just >> 'Sender', 'Recipients' and 'Subject', but you can also select >> 'Body' (and a few others like 'Unread', 'Contain attachments', >> etc), but the 'Body' criteria is the one that would want/need to >> use the fts indexes, so... >> >> Can dovecots fts indexes be used in a case like this? In other >> words, will it 'just work'? Or will it *not* work? Or, would the >> client need to specify the folder in the commands it sends to >> limit the search and/or results to just the currently selected >> folder? Or is this even possible for such a simple/limited use >> filtering mechanism? > The regular IMAP protocol supports searching only from the selected > folder. Interesting, thanks... so, just guessing, most likely Thunderbird simply iterates over each folder in an account when searching an entire account and 'Run search on server' is checked (only available in the Advanced Search window)... One last question then (couldn't find an answer on the wiki)... I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: https://bugzilla.mozilla.org/show_bug.cgi?id=541400 This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? This would pretty much neutralize/work around the above bug. > But I guess this quickfilter search also searches from only the > selected folder. So I don't see a problem. Correct, and thanks for the confirmation... -- Best regards, Charles From tss at iki.fi Mon Mar 5 19:01:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:01:45 +0200 Subject: [Dovecot] testing fts-solr? In-Reply-To: <4F54E5E3.2010509@Media-Brokers.com> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> Message-ID: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> On 5.3.2012, at 18.12, Charles Marcus wrote: > One last question then (couldn't find an answer on the wiki)... > > I'm currently planning on using fts/clucene, but I'm thinking I'd like the following to apply also to dovecots internal indexes too... > > What is the minimal number of characters that dovecots indexes are based on (I'm assuming that it doesn't index just individual characters)? 2+? 3+? Is this configurable? Lucene doesn't really work that way. It only searches full words. But some words are "stop words" that are ignored, such as "a" or "the" in the English language. So fts-lucene won't find anything when you search for "a", but other character searches will return all messages that contain it as a full word. (Or: That's how I think it works, too lazy to test it now.) > The reason I ask is, Thunderbird has an annoying behavior where it sends a new/separate query each time a character is typed, beginning with the very first character: > > https://bugzilla.mozilla.org/show_bug.cgi?id=541400 > > This causes massive performance degradation on my clients system that currently uses Courier-IMAP (this is the one we'll soon be converting to dovecot) when the users (usually accidentally (select the 'Body' criteria in the Quickfilter searchbox... I doubt this is going to be a problem with Dovecot+fts. The search results will be pretty much instantaneous, even if the search matches all of the messages. > Is there a way to tell dovecot to return an *empty* result unless/until a query contains X+ characters (where X is some configurable number, I'm thinking 3 or 4)? Nope. From CMarcus at Media-Brokers.com Mon Mar 5 19:11:50 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 05 Mar 2012 12:11:50 -0500 Subject: [Dovecot] testing fts-solr? In-Reply-To: <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> References: <1330338267.11500.299.camel@innu> <6B4DF036-BCC0-408D-9839-D7F9F27AA040@iki.fi> <4F4CD96D.5010706@gmail.com> <4F4CE361.3010705@Media-Brokers.com> <4F4CE919.8070505@gmail.com> <1330441042.2081.24.camel@innu> <4F54A26C.4040005@Media-Brokers.com> <40F44B3A-52D5-4285-B165-CF7A79151B8D@iki.fi> <4F54E5E3.2010509@Media-Brokers.com> <774036BE-3D95-4F3D-B058-545C765B8E6A@iki.fi> Message-ID: <4F54F3D6.9040508@Media-Brokers.com> On 2012-03-05 12:01 PM, Timo Sirainen wrote: > On 5.3.2012, at 18.12, Charles Marcus wrote: >> Thunderbird has an annoying behavior where it sends a new/separate >> query each time a character is typed, beginning with the very first >> character: >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=541400 >> >> This causes massive performance degradation on my clients system >> that currently uses Courier-IMAP (this is the one we'll soon be >> converting to dovecot) when the users (usually accidentally >> (select the 'Body' criteria in the Quickfilter searchbox... > I doubt this is going to be a problem with Dovecot+fts. The search > results will be pretty much instantaneous, even if the search matches > all of the messages. Cool... I guess I'll just wait and see then, and revisit this if we run into problems... Thanks Timo... -- Best regards, Charles From joshua at hybrid.pl Mon Mar 5 19:25:43 2012 From: joshua at hybrid.pl (Jacek Osiecki) Date: Mon, 5 Mar 2012 18:25:43 +0100 (CET) Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: <1330677951.2081.49.camel@innu> References: <1330677951.2081.49.camel@innu> Message-ID: On Fri, 2 Mar 2012, Timo Sirainen wrote: > On Thu, 2012-03-01 at 09:21 +0100, Jacek Osiecki wrote: >> However, if we have everything redundant, why not have the same with SMTP >> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > If both servers randomly access users' mails, with NFS you'll have some > trouble, with OCFS2 probably less trouble. But in both cases you'll have > better performance and no problems if you use Dovecot director in both > servers (install both director and backend to both servers). > http://wiki2.dovecot.org/Director Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Greetings, -- Jacek Osiecki joshua at ceti.pl GG:3828944 I don't want something I need. I want something I want. From tss at iki.fi Mon Mar 5 19:53:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 19:53:12 +0200 Subject: [Dovecot] Concurrent dovecot instances on same spool? In-Reply-To: References: <1330677951.2081.49.camel@innu> Message-ID: <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> On 5.3.2012, at 19.25, Jacek Osiecki wrote: >>> However, if we have everything redundant, why not have the same with SMTP >>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? > >> If both servers randomly access users' mails, with NFS you'll have some >> trouble, with OCFS2 probably less trouble. But in both cases you'll have >> better performance and no problems if you use Dovecot director in both >> servers (install both director and backend to both servers). >> http://wiki2.dovecot.org/Director > > Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... You can safely do that with director. Also the problem with NFS isn't locks, but caching. > Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). From sam at robots.org.uk Mon Mar 5 20:45:36 2012 From: sam at robots.org.uk (Sam Morris) Date: Mon, 05 Mar 2012 18:45:36 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users Message-ID: <1330973136.70967.33.camel@leela.office.red-redemption.com> The attached patch makes it possible for Kerberos principals to be associated with a password database entry by adding a new "k5principals" passdb setting. A client that successfully authenticates using GSSAPI will be able to log in as any user who has been associated with the client's Kerberos principal. This means that users can now use their Kerberos identities to access virtual mail accounts. The patch definitely needs review by someone familiar with Dovecot. It works for me on a small test installation using the passwd-file backend. Things that should probably be improved: 1. The list of authorized principals is stored in struct auth_request. I would prefer to store it in struct gssapi_auth_request, but auth-request.c does not know about structs that are specific to the different authentication plugins. This could be fixed in a more general way by adding a new function to struct mech_module to allow authplugins to read fields during passdb lookups. 2. The gssapi authplugin now does a credential lookup in order to trigger parsing of the k5principals setting. In order for this to work, auth_request_set_username is now called before mech_gssapi_userok. AFAICT the only impact of this is that messages logged by this function (and the functions it calls) will now use the name of the virtual account. 3. The credentials lookup triggers an info log message saying that credentials for GSSAPI were requested, "but we have only (e.g.) MD5-CRYPT". The authplugin doesn't actually want the credential, but I think that the only way the authplugin can trigger a passdb lookup is by requesting it. 4. The final part of the code in mech_gssapi_unwrap was moved to the callback that's triggered when the credentials lookup is complete. The code still needs access to the GSSAPI data, so the buffer pointer & length are now stored in struct gssapi_auth_request, making the inbuf parameter to the mech_gssapi_{sec_context,wrap,unwrap} functions superfluous. The parameters should be removed. 5. The k5principals list won't be processed on Solaris. The code added to the end of mech_gssapi_krb5_userok would have to be moved to a separate function and then be called from the Solaris code. 6. GCC tells me about assignment to incompatible pointer types in the code that iterates through gssapi_k5principals. I must be missing something. The patch is licensed under the MIT license. Please let me know what you think. -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_1.patch Type: text/x-patch Size: 6953 bytes Desc: not available URL: From tss at iki.fi Mon Mar 5 20:52:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 5 Mar 2012 20:52:19 +0200 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: <1330973136.70967.33.camel@leela.office.red-redemption.com> References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: On 5.3.2012, at 20.45, Sam Morris wrote: > 3. The credentials lookup triggers an info log message saying that > credentials for GSSAPI were requested, "but we have only (e.g.) > MD5-CRYPT". The authplugin doesn't actually want the credential, > but I think that the only way the authplugin can trigger a > passdb lookup is by requesting it. I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". From campbell at cnpapers.com Mon Mar 5 21:30:30 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Mon, 05 Mar 2012 14:30:30 -0500 Subject: [Dovecot] Shared mboxes Message-ID: <4F551456.102@cnpapers.com> I've been looking at some documentation on shared mail accounts. But I'm getting mixed thoughts on how this can or should be done. I use mbox for all my pop and imap folders since I've converted from a uw-imap server. The first thing that makes me wonder about setup is that I've been told to not use maildir and mbox on the same machine, although I'm not really sure why since it seems this would work OK, but anyway, I'm guessing I should stick with mbox for the shared accounts. Secondly, I'm sure I'd need a namespace to use which ever format, so there's private, public, and shared types. Most of the stuff I'm reading seems to suggest "public" as a type instead of "shared". So what's shared for anyway? I want to use this shared account so that email can be sent to this account, and be shared by only a few people, but I'm reading where locks and such don't work with mbox, so in my mind, how do you avoid corruption and why not just make a normal account and let people hack away at the data? I've not even got to the questions in my mind about how to set up the account, but figured if I could get the above straight, I might be able to fuddle my way through it. Help would be truly appreciated. steve campbell From tom at talpey.com Tue Mar 6 00:06:20 2012 From: tom at talpey.com (Tom Talpey) Date: Mon, 05 Mar 2012 17:06:20 -0500 Subject: [Dovecot] POP3C storage backend Message-ID: <4F5538DC.4060802@talpey.com> I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I don't see anything in the 2.1 doc directory or in the wiki. Can this be used to synchronize dovecot with external pop servers? Doing away with my current fetchmail and lmtp solution for this would be quite interesting. Thanks for any pointers to configuring and using this, if so... From stan at hardwarefreak.com Tue Mar 6 01:16:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Mon, 05 Mar 2012 17:16:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F551456.102@cnpapers.com> References: <4F551456.102@cnpapers.com> Message-ID: <4F55495B.10609@hardwarefreak.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. But I'm > getting mixed thoughts on how this can or should be done. > > I use mbox for all my pop and imap folders since I've converted from a > uw-imap server. The first thing that makes me wonder about setup is that > I've been told to not use maildir and mbox on the same machine, although > I'm not really sure why since it seems this would work OK, but anyway, > I'm guessing I should stick with mbox for the shared accounts. > > Secondly, I'm sure I'd need a namespace to use which ever format, so > there's private, public, and shared types. Most of the stuff I'm reading > seems to suggest "public" as a type instead of "shared". So what's > shared for anyway? > > I want to use this shared account so that email can be sent to this > account, and be shared by only a few people, but I'm reading where locks > and such don't work with mbox, so in my mind, how do you avoid > corruption and why not just make a normal account and let people hack > away at the data? > > I've not even got to the questions in my mind about how to set up the > account, but figured if I could get the above straight, I might be able > to fuddle my way through it. > > Help would be truly appreciated. Start here: http://wiki.dovecot.org/SharedMailboxes -- Stan From kgc at corp.sonic.net Tue Mar 6 03:33:32 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Mon, 5 Mar 2012 17:33:32 -0800 Subject: [Dovecot] Master Users Message-ID: <20120306013332.GE16881@corp.sonic.net> I have a setup where I need to use a Master User account to login on behalf of users normally authed via PAM. Is there any existing mechanism that will allow master users to be wired down to specific ip address rather than having these very magic user/pass combos be valid from any random host? It would be totally acceptable to be able to say that master logins were only valid from a specific list of hosts rather than wiring specific master users to specific hosts. -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtam.home at gmail.com Tue Mar 6 04:01:43 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 5 Mar 2012 18:01:43 -0800 (PST) Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: On Sun, 4 Mar 2012, Timo Sirainen writes: > > I would like to run various doveadm commands that involves all (mail) users like > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > but any doveadm command that uses "-A" to iterate through all users will > > stop processing at the first account with UID > What userdb are you using? userdb passwd should already skip users that > aren't in the valid range. And what Dovecot version are you using? passwd-file under dovecot 2.0.16. > And one more thing: Does it really even stop there? Looking at the code > it's supposed to log an error and continue to next user. Note that it says > "Failed to iterate through SOME users". The wording did not escape my notice, which is why I suspect it's not doing what it was designed to do. This is my test: # Command doveadm mailbox list -A # Start of password file sysdaemon:*:500:500:System daemon:/:/dev/null ... and the rest ... # dovecot.conf ... first_valid_uid = 10000 first_valid_gid = 10000 ... In this situation, doveadm will exit immediately with an UID error message. If I change the UID>10000, it will produce the analogous GID error message. If I satify both UID and GID constraints, it will fail on the next daemon entry. If I move the sysdaemon entry all the way to the bottom of the pasword file, I get user1 saved-messages user1 sent-mail user1 postponed-msgs user1 temp user1 temp/temp user1 INBOX user2 sent-mail user2 101 user2 345 user2 ckf ... all user's mailbox with UID>10000, then ... doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop privileges: Mail access for users with UID 500 not permitted (see first_valid_uid in config file, uid from userdb lookup). doveadm(sysdaemon): Error: User init failed doveadm: Error: Failed to iterate through some users > Oh, it says about first_valid_gid. Is sysdaemon's UID within valid range? Sorry for this error mismatch -- I cut&pasted the wrong test output; however, the problem I witnessed applies to both UID and GID (if either constraint is not met, user iteration terminates). > I also added this today: http://hg.dovecot.org/dovecot-2.1/rev/85a8d582d37f It looks like I'll be upgrading. Oh, I just spotted this in the ChangeLog -- maybe you are undoing this? (2010-10-21) * src/auth/auth-settings.c, src/auth/auth-settings.h, src/auth/userdb- passwd.c: auth: userdb passwd iteration now lists only users within first_valid_uid..last_valid_uid range. [745ef289b0ea] Joseph Tam From apm at one.com Tue Mar 6 09:17:32 2012 From: apm at one.com (Peter Mogensen) Date: Tue, 06 Mar 2012 08:17:32 +0100 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> Message-ID: <4F55BA0C.5090606@one.com> On 2012-03-05 16:36, Timo Sirainen wrote: >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > Neither behavior is wrong, just different. :) Ok... I were in doubt if I had missed something from the RFC. However... for testing, I tried to create "INBOX.INBOX" on dovecot. But then dovecot answers NO and complains that the folder already exists. Though it's still not on disk and dovecot still doesn't list it with "*". /Peter From frank at moltke28.B.Shuttle.DE Tue Mar 6 09:37:45 2012 From: frank at moltke28.B.Shuttle.DE (Frank Elsner) Date: Tue, 6 Mar 2012 08:37:45 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve Message-ID: (auto-added) Hello all, I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 installed and working. But I've problems to get the avelsieve plugin for squirrelmail working with dovecot. The "Message Filters" show up in "Options" of squirrelmail, but "Could not log on to timsieved daemon on your IMAP server ........." dovecot log shows: Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured Where to look for configuration error(s)? | root at seymour: /usr/local/dovecot/src<156> dovecot -n | # 2.1.1: /usr/local/dovecot/etc/dovecot/dovecot.conf | # OS: Linux 2.6.35.14-106.fc14.i686.PAE i686 Fedora release 14 (Laughlin) ext3 | default_vsz_limit = 512 M | disable_plaintext_auth = no | first_valid_uid = 200 | last_valid_uid = 65534 | listen = * | lmtp_save_to_detail_mailbox = yes | login_greeting = c64.shuttle.de - IMAPs Service (dovecot) ready. | login_log_format_elements = %u %r %c | mail_location = maildir:/var/spool/mail/%u | mail_log_prefix = "%Us(%u,%r): " | mail_plugin_dir = /usr/dovecot/lib/dovecot/ | mail_plugins = " notify quota" | managesieve_notify_capability = mailto | managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave | passdb { | args = dovecot | driver = pam | } | plugin { | mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append | mail_log_fields = uid box from subject msgid size flags | mail_log_group_events = yes | quota = maildir:User quota | quota_rule = *:storage=2G | quota_rule2 = Trash:storage=+100M | sieve = ~/.sieve | sieve_dir = ~/sieve | } | postmaster_address = postmaster at moltke28.b.shuttle.de | protocols = imap sieve | service anvil { | client_limit = 1027 | } | service auth { | unix_listener auth-client { | group = exim | mode = 0660 | user = exim | } | } | service imap-login { | inet_listener imap { | port = 143 | } | inet_listener imaps { | port = 993 | ssl = yes | } | process_limit = 512 | process_min_avail = 10 | } | service imap-postlogin { | executable = script-login /usr/dovecot/bin/imap-post-login | } | service imap { | executable = imap imap-postlogin | } | service lmtp { | inet_listener lmtp { | address = 0.0.0.0 | port = 24 | } | } | service managesieve-login { | inet_listener sieve { | port = 4190 | } | } | service pop3-login { | inet_listener pop3 { | port = 110 | } | inet_listener pop3s { | port = 995 | ssl = yes | } | } | service pop3 { | process_limit = 1024 | } | ssl_cert = References: <1330677951.2081.49.camel@innu> <1D9A5A83-5A7A-480B-A8BF-B33968C99ACE@iki.fi> Message-ID: <4F55C9DB.2070809@mobilia.it> Il 05/03/2012 18.53, Timo Sirainen ha scritto: > On 5.3.2012, at 19.25, Jacek Osiecki wrote: > >>>> However, if we have everything redundant, why not have the same with SMTP >>>> and POP3/IMAP? But - won't anything fail if two (or more) dovecots are >>>> accessing the same disk space, both for IMAP/POP3 and LDA/LMTP? >>> If both servers randomly access users' mails, with NFS you'll have some >>> trouble, with OCFS2 probably less trouble. But in both cases you'll have >>> better performance and no problems if you use Dovecot director in both >>> servers (install both director and backend to both servers). >>> http://wiki2.dovecot.org/Director >> Thanks, I'll probably give it a try. On the other hand, it would be nice to have a possibility to allow multiple dovecot instances to access mail spool (at cost of handling some extra file/directory locks) - a bit slower, but safe... > You can safely do that with director. > > Also the problem with NFS isn't locks, but caching. After reading a little bit, it seems that Director does the job of a decent load balancer, but in the middle instead of in front of your servers, I've limited problems with NFS by using "sticky" connections with long timeouts in my load balancer, unless they're disconnected for days, they'll always end up going through the same server for POP3/IMAP conections. Doesn't work great for the SMTP/LDA part though. > >> Another question: as I assume, when you wrote about troubles it was applying to IMAP. How about LMTP/LDA? Can anything bad happen when one mailbox is being filled by LMTP/LDA from more than one server)? > Yes, because they're still updating Dovecot index files. You could disable LMTP/LDA index updates, but I'm still not sure if it works 100% correctly (because dovecot-uidlist is appended to). > In the rare case it does happen, NFS locking and concurrent_connections set to one has seemed to reduce my problems to a minimum.. I like the Director idea though, since it's content aware it isn't organizing connections based on port/IP, but on the the actual users, especially if it does so with the LDA, it seems like an excellent solution to collisions (I guess they're called this) .. I wish it had been a reality when I was building my servers. From stephan at rename-it.nl Tue Mar 6 10:40:44 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 06 Mar 2012 09:40:44 +0100 Subject: [Dovecot] dovecot 2.1.1 + pigeonhole + avelsieve In-Reply-To: (auto-added) References: (auto-added) Message-ID: <4F55CD8C.20108@rename-it.nl> On 3/6/2012 8:37 AM, Frank Elsner wrote: > Hello all, > > I've squirrelmail-webmail-1.4.22, dovecot 2.1.1, dovecot-2.1-pigeonhole-0.3.0 > installed and working. But I've problems to get the avelsieve plugin for > squirrelmail working with dovecot. > > The "Message Filters" show up in "Options" of squirrelmail, but > "Could not log on to timsieved daemon on your IMAP server ........." > > dovecot log shows: > > Mar 6 00:00:47 seymour dovecot: managesieve-login: Disconnected: Too many invalid commands. (no auth attempts in 0 secs): 192.168.28.53, secured You should try to capture traffic between client and server with ngrep, e.g. sudo ngrep -d lo port 4190 However, I've noticed that avelsieve uses STARTTLS even on localhost, so if you want to see anything intelligible, you will have to turn that off temporarily. As far as I know, there is also a means to instruct managesieve-login to write its traffic somewhere (a login 'rawlog'), but I can't find where it is documented right now. > | protocol lmtp { > | mail_plugins = " notify quota quota" > | } > | protocol lda { > | mail_plugins = " notify quota quota" > | } > | protocol imap { > | imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags > | imap_logout_format = [%i/%o] > | mail_max_userip_connections = 0 > | mail_plugins = " notify quota mail_log quota imap_quota listescape" > | } Why do you have duplicate "quota" entries here? Also, "sieve" plugin is missing from lmtp and lda. Still, ManageSieve should accept connections with this config. Regards, Stephan. From dovecot at arvoreen.net Tue Mar 6 13:29:13 2012 From: dovecot at arvoreen.net (Pol Bettinger) Date: Tue, 06 Mar 2012 12:29:13 +0100 Subject: [Dovecot] LDAP auth_bind fails Message-ID: <4F55F509.4000507@arvoreen.net> Hello, I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. Dovecot version 2.1.1 (I started with 2.1.0 and hoped 2.1.1 would fix it) I tried to play around with the base, pass_attrs,pass_filter to no avail but didn't succeed. Looking at a wireshark trace i only saw 7 packets and it seemed to me dovecot did only an anonymous bind. any help would appreciated Sincerely Pol Bettinger output of mail.log: Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5#011service=imap#011secured#011lip=192.168.16.27#011rip=192.168.16.20#011lport=993#011rport=51838 Mar 6 12:16:34 Dell dovecot: auth: Debug: client out: CONT#0112#011PDQ1NjgyMjE3NjYyMDk3NjkuMTMzMTAzMjU5NEBEZWxsPg== Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: CONT Mar 6 12:16:34 Dell dovecot: auth: Debug: password(arvi at arvoreen.net,192.168.16.20): passdb doesn't support credential lookups Mar 6 12:16:36 Dell dovecot: auth: Debug: client out: FAIL#0112#011user=arvi at arvoreen.net output of dovecot -n: # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic i686 Ubuntu 11.10 ext4 auth_debug = yes auth_default_realm = arvoreen.net auth_mechanisms = plain digest-md5 cram-md5 auth_verbose = yes base_dir = /var/run/dovecot/ mail_location = maildir:/var/mail/%d/%1n/%n:INDEX=/var/indexes/%d/%1n/%n managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Drafts { auto = create special_use = \Drafts } mailbox Junk { auto = create special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = create special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap_pass.conf.ext driver = ldap } plugin { sieve = /var/sieve/%d/%1n/%n sieve_dir = /var/sieve/%d/%1n/%n } protocols = imap lmtp sieve service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = From tss at iki.fi Tue Mar 6 13:43:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 6 Mar 2012 13:43:26 +0200 Subject: [Dovecot] LDAP auth_bind fails In-Reply-To: <4F55F509.4000507@arvoreen.net> References: <4F55F509.4000507@arvoreen.net> Message-ID: On 6.3.2012, at 13.29, Pol Bettinger wrote: > I wanted to configure dovecot for using auth_bind but didn't succeed to me it seems like it does always an anonymous bind. .. > Mar 6 12:16:34 Dell dovecot: auth: Debug: client in: AUTH#0112#011CRAM-MD5 CRAM-MD5 can't work with auth_bind. http://wiki2.dovecot.org/Authentication/Mechanisms#Non-plaintext_authentication From jernej.porenta at arnes.si Tue Mar 6 15:28:50 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 6 Mar 2012 14:28:50 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) Message-ID: Heya, We are expiriencing issues with dovecot 2.1.1 on Linux with weird filenames in home directory of username. We are using mbox IMAP folders, with no special changes (mail_location = mbox:~/:INBOX=%h/.mailbox). Mar 6 13:37:17 machine dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 6 13:37:17 machine dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb79450] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb794a6] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot.so.0 [0x2ba41cb78963] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c87ebd5] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c88c12c] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_next+0x1b4) [0x2ba41c88c494] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0 [0x2ba41c885342] -> /opt/dovecot-2.1.1/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_next+0x234) [0x2ba41c885604] -> dovecot/imap [0x40b2d1] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410427] -> dovecot/imap [0x40f4cd] -> dovecot/imap [0x40f582] -> dovecot/imap(client_handle_input+0x3f) [0x40f6cf] -> dovecot/imap(client_input+0x62) [0x410052] -> /opt/dovecot Mar 6 13:37:17 machine dovecot: imap(username): Fatal: master: service(imap): child 20873 killed with signal 6 (core dumps disabled) The bug is reproducible by using home folder structure available from: http://bit.ly/x8pTXS AFAIK, the problem lies in processing the file list of home folder, which can contain filenames that do not have proper UTF-8 encoding of filenames, which causes dovecot to crash. On the other hand, UTF-8 filenames created on the system by hand (using touch), are not displayed in IMAP LIST command (sample is included in the folder structure; single letter file). Cheers, Jernej From campbell at cnpapers.com Tue Mar 6 16:28:55 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 09:28:55 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F55495B.10609@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> Message-ID: <4F561F27.5000102@cnpapers.com> On 3/5/2012 6:16 PM, Stan Hoeppner wrote: > On 3/5/2012 1:30 PM, Steve Campbell wrote: >> I've been looking at some documentation on shared mail accounts. But I'm >> getting mixed thoughts on how this can or should be done. >> >> I use mbox for all my pop and imap folders since I've converted from a >> uw-imap server. The first thing that makes me wonder about setup is that >> I've been told to not use maildir and mbox on the same machine, although >> I'm not really sure why since it seems this would work OK, but anyway, >> I'm guessing I should stick with mbox for the shared accounts. >> >> Secondly, I'm sure I'd need a namespace to use which ever format, so >> there's private, public, and shared types. Most of the stuff I'm reading >> seems to suggest "public" as a type instead of "shared". So what's >> shared for anyway? >> >> I want to use this shared account so that email can be sent to this >> account, and be shared by only a few people, but I'm reading where locks >> and such don't work with mbox, so in my mind, how do you avoid >> corruption and why not just make a normal account and let people hack >> away at the data? >> >> I've not even got to the questions in my mind about how to set up the >> account, but figured if I could get the above straight, I might be able >> to fuddle my way through it. >> >> Help would be truly appreciated. > Start here: > http://wiki.dovecot.org/SharedMailboxes That's where most of my questions originated, but thanks for the reply. (Sorry for the first response - I sent it to the poster, not the list). Maybe I'm misunderstanding concepts here and I'm trying to use something I don't need to use. I'm really new to dovecot, and as I learn all the ins and outs, I'm finding a lot of this doesn't seem to be "turning on any light bulbs" until after I've played with it a while. What I've done in the past with the old imap server is to create an account (unix account), so the smtp server puts the mbox (what is referred to as the INbox) in /var/spool/mail. Users who needed to "share" this mailbox would be give the account user name and the password for this account and would add an Imap account to their mail client. This would sometimes cause locking problems or client corruption due to email removals mostly. This is basically a normal, non-shared account. Now that I've moved to dovecot on a new, updated server, I'd like to use the facilities of dovecot for the truly shared accounts. I'm not sure if I need to create the account like before, but seems like I'd have to in order to get the smtp server to deliver new email to /var/spool/mail/%u. As I see it, I've got to create a namespace for shared accounts and configure this on the multiple-user's clients so that when they access the Inbox and imap files under /home/%u/mail, they don't butt heads, so they're some locking involved. I could use acls for this, but don't have to according to the documentation. I can grant permissions to each user that is included in the acl, and I can create dovecot "groups" to use as a basis for this permission. I'm hoping this is pretty much the way it's done, and I want to keep with mbox format for all files and folders. I'm also hoping that this is the way it's supposed to be used, but I get conflicting ideas about what the documentation is really telling me. Anyway, I'll play with this and see where I get. I've still not found out where to create these dovecot "groups" other than it seems to use a userdb file somewhere. Thanks for the help so far steve > From khoroshyy at gmail.com Tue Mar 6 18:58:55 2012 From: khoroshyy at gmail.com (Khoroshyy Petro) Date: Tue, 6 Mar 2012 17:58:55 +0100 Subject: [Dovecot] Dovecot saves mails in "wrong" folder. Message-ID: Hi all I have installed dovecot 1.2.15 and try to use it together with offlineimap and gnus. my problem is that it saves emails into /var/mail/petro instead of ~/Maildir Thanks. Petro. This is my .dovecot.conf default_mail_env = maildir:%h/Maildir And this is my .offlineimaprc [general] accounts = Gmail maxsyncaccounts = 1 [Account Gmail] localrepository = Local remoterepository = Remote [Repository Local] type = IMAP remotehost = localhost port = 143 remoteuser = petro [Repository Remote] type = IMAP remotehost = imap.gmail.com remoteuser = myname at gmail.com ssl = yes maxconnections = 1 realdelete = no folderfilter = lambda foldername: foldername in ['INBOX'] -- From kgc at corp.sonic.net Tue Mar 6 19:33:08 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Tue, 06 Mar 2012 09:33:08 -0800 Subject: [Dovecot] Master Users In-Reply-To: <20120306013332.GE16881@corp.sonic.net> References: <20120306013332.GE16881@corp.sonic.net> Message-ID: <4F564A54.9050400@corp.sonic.net> On 03/05/12 17:33, Kelsey Cummings wrote: > I have a setup where I need to use a Master User account to login on > behalf of users normally authed via PAM. Is there any existing mechanism > that will allow master users to be wired down to specific ip address rather Ah, found it. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/AllowNets -K From sam at robots.org.uk Tue Mar 6 20:12:01 2012 From: sam at robots.org.uk (Sam Morris) Date: Tue, 06 Mar 2012 18:12:01 +0000 Subject: [Dovecot] [PATCH] GSSAPI authorization and virtual users In-Reply-To: References: <1330973136.70967.33.camel@leela.office.red-redemption.com> Message-ID: <1331057521.84875.2.camel@leela.office.red-redemption.com> On Mon, 2012-03-05 at 20:52 +0200, Timo Sirainen wrote: > On 5.3.2012, at 20.45, Sam Morris wrote: > > > 3. The credentials lookup triggers an info log message saying that > > credentials for GSSAPI were requested, "but we have only (e.g.) > > MD5-CRYPT". The authplugin doesn't actually want the credential, > > but I think that the only way the authplugin can trigger a > > passdb lookup is by requesting it. > > I'll look at the rest more closely later, but this should be an easy fix: request "" instead of "GSSAPI". Thanks for pointing that out. Here's a newer version of the patch with that change. I also realised that the gss_buffer is not required in the code that runs once the passdb lookup is complete, so I removed the code that stashes it in struct gssapi_auth_request. Regards, -- Sam Morris -------------- next part -------------- A non-text attachment was scrubbed... Name: k5principals_2.patch Type: text/x-patch Size: 6020 bytes Desc: not available URL: From stan at hardwarefreak.com Tue Mar 6 22:17:22 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 06 Mar 2012 14:17:22 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F561F27.5000102@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> Message-ID: <4F5670D2.9090004@hardwarefreak.com> On 3/6/2012 8:28 AM, Steve Campbell wrote: >> http://wiki.dovecot.org/SharedMailboxes > That's where most of my questions originated, but thanks for the reply. Steve, all the information you need is behind that link. > Maybe I'm misunderstanding concepts here Very possibly. > What I've done in the past with the old imap server is to create an > account (unix account), so the smtp server puts the mbox (what is > referred to as the INbox) in /var/spool/mail. Users who needed to > "share" this mailbox would be give the account user name and the > password for this account and would add an Imap account to their mail > client. This would sometimes cause locking problems or client corruption > due to email removals mostly. This is basically a normal, non-shared > account. Locking problems with multiple users hitting mbox files is unavoidable. The same is true when a single user hits an mbox from multiple client devices simultaneously--PC, smart phone, tablet, etc. Which is why you do not want to use mbox file format for shared mailboxes, but maildir instead, because each email is a separate file. Please note, from the link I provided: ********************************************************************** Maildir: Per-user \Seen flag With Maildir a dovecot-shared file controls if the \Seen flags are shared or private. The file must be created separately inside each Maildir, although if the file already exists in the Maildir root it's automatically copied for newly created mailboxes. If dovecot-shared file doesn't exist in Maildir, the \Seen flags are shared. If it exists, the \Seen flag state is stored only in the user's index files. By making each user have their own private index files, you can make the \Seen flag private for the users. ********************************************************************** Simple concept above: each user of the shared mailbox sees "new" mail. One user accessing new mail and marking it as read doesn't mark that message as read for other shared users. You can not do this with mbox file format, only maildir. ********************************************************************** Maildir: Keyword sharing Make sure you don't try to use per-user CONTROL directory. Otherwise dovecot-keywords file doesn't get shared and keyword mapping breaks. Other mailbox formats Currently you can't have any per-user flags with other mailbox formats than Maildir. ********************************************************************** -- Stan From campbell at cnpapers.com Tue Mar 6 23:01:08 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 06 Mar 2012 16:01:08 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F5670D2.9090004@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> Message-ID: <4F567B14.3030908@cnpapers.com> On 3/6/2012 3:17 PM, Stan Hoeppner wrote: > On 3/6/2012 8:28 AM, Steve Campbell wrote: > >>> http://wiki.dovecot.org/SharedMailboxes >> That's where most of my questions originated, but thanks for the reply. > Steve, all the information you need is behind that link. I've gone over that set of links on that page a dozen times. Perhaps I'm trying to put a square peg in a round hole by using mbox, but they keep providing information on it, so I guess I was just pounding away. But then there's that "don't use maildir and mbox together". All of the accounts on this server are carry-overs from the UW-IMAP server, so perhaps I should have converted those to maildir. Seems as though it's OK when they don't apply to the same type namespace. > >> Maybe I'm misunderstanding concepts here > Very possibly. > >> What I've done in the past with the old imap server is to create an >> account (unix account), so the smtp server puts the mbox (what is >> referred to as the INbox) in /var/spool/mail. Users who needed to >> "share" this mailbox would be give the account user name and the >> password for this account and would add an Imap account to their mail >> client. This would sometimes cause locking problems or client corruption >> due to email removals mostly. This is basically a normal, non-shared >> account. > Locking problems with multiple users hitting mbox files is unavoidable. > The same is true when a single user hits an mbox from multiple client > devices simultaneously--PC, smart phone, tablet, etc. Which is why you > do not want to use mbox file format for shared mailboxes, but maildir > instead, because each email is a separate file. Please note, from the > link I provided: I've experienced that type of locked mailbox before on the old server. Users insist on accessing their email account as a pop account on their desktop with the "check for new mail every so many minutes" turned on and still keep their smartphones on while accessing it as an imap account so they can still download the files to their desktop when they return. > > ********************************************************************** > Maildir: Per-user \Seen flag > > With Maildir a dovecot-shared file controls if the \Seen flags are > shared or private. The file must be created separately inside each > Maildir, although if the file already exists in the Maildir root it's > automatically copied for newly created mailboxes. If dovecot-shared file > doesn't exist in Maildir, the \Seen flags are shared. If it exists, the > \Seen flag state is stored only in the user's index files. By making > each user have their own private index files, you can make the \Seen > flag private for the users. > ********************************************************************** > > > Simple concept above: each user of the shared mailbox sees "new" mail. > One user accessing new mail and marking it as read doesn't mark that > message as read for other shared users. You can not do this with mbox > file format, only maildir. > > > ********************************************************************** > Maildir: Keyword sharing > > Make sure you don't try to use per-user CONTROL directory. Otherwise > dovecot-keywords file doesn't get shared and keyword mapping breaks. > > Other mailbox formats > > Currently you can't have any per-user flags with other mailbox formats > than Maildir. > ********************************************************************** So just to clarify, is it OK to have a maildir account setup on this server for these shared/imap access only accounts along with the mbox accounts already on there? Thanks for the patience and help steve From sdavies at sdc.com.au Wed Mar 7 01:00:50 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:30:50 +1030 Subject: [Dovecot] Log sync errors (again) Message-ID: <201203070930.50847.sdavies@sdc.com.au> As suggested earlier, I deleted all .imap directories and the log sync errors stopped - for a while. They have now returned. It seems to happen for every mailbox that gets accessed. Dovecot version 2.1.1 with pidgeonhole 3.0.0 on Mandriva Linux. Could this interfere with sieve filters? Several users have filters but none of them seem to do anything. Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=38708 for /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index: Extension header update points outside header size Mar 7 09:25:51 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=41576 for /home/john/Mail/INBOX/.imap/Zerna/dovecot.index: Extension header update points outside header size Cheers and thanks, Stephen -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From sdavies at sdc.com.au Wed Mar 7 01:07:06 2012 From: sdavies at sdc.com.au (Stephen Davies) Date: Wed, 7 Mar 2012 09:37:06 +1030 Subject: [Dovecot] Fscking warnings Message-ID: <201203070937.06545.sdavies@sdc.com.au> Google tells me that these "should go away" but they don't. Seems to happen continuously while a user is viewing email. Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Archive/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Davies/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/FieldNET/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Invoices Out/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Lawrence and Hanson/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Call/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Logger Reset/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/River Murray/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Soil Moisture Alert/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Management Alarm/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Water Usage/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Weather Summaries/dovecot.index Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file /home/john/Mail/INBOX/.imap/Zerna/dovecot.index -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From jk at jkart.de Wed Mar 7 01:19:10 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:19:10 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? Message-ID: <4F569B6E.1080905@jkart.de> Hello, you knows, that http://xi.rename-it.nl is down? -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Die Oper ist eine h?bsche Unterhaltung, die noch besser w?re, wenn nicht dabei gesungen w?rde. (Claude Debussy) From stephan at rename-it.nl Wed Mar 7 01:33:09 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 07 Mar 2012 00:33:09 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569B6E.1080905@jkart.de> References: <4F569B6E.1080905@jkart.de> Message-ID: <4F569EB5.7030204@rename-it.nl> On 3/7/2012 12:19 AM, Jim Knuth wrote: > Hello, > > you knows, that http://xi.rename-it.nl is down? > Yep, and back. Regards, Stephan. From jk at jkart.de Wed Mar 7 01:36:42 2012 From: jk at jkart.de (Jim Knuth) Date: Wed, 07 Mar 2012 00:36:42 +0100 Subject: [Dovecot] http://xi.rename-it.nl down? In-Reply-To: <4F569EB5.7030204@rename-it.nl> References: <4F569B6E.1080905@jkart.de> <4F569EB5.7030204@rename-it.nl> Message-ID: <4F569F8A.1090200@jkart.de> am 07.03.12 00:33 schrieb Stephan Bosch : > On 3/7/2012 12:19 AM, Jim Knuth wrote: >> Hello, >> >> you knows, that http://xi.rename-it.nl is down? >> > > Yep, and back. > > Regards, > > Stephan. wow. Thank you -- Mit freundlichen Gr??en, with kind regards, Jim Knuth --------- Dilettanten erkennt man an der Plumpheit ihrer Komplimente. Der routinierte Verf?hrer riskiert Kritik. (Cath?rine Deneuve) From 24x7server at 24x7server.net Wed Mar 7 04:22:23 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Wed, 7 Mar 2012 07:52:23 +0530 (Asi) Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file Message-ID: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> hi i am using qmailtoaster with dovecot version 2 mailbox format is maildir i have a domain with around 5000 users which are distributed over 2 servers webmail (squirrelmail) runs using dovecot v2 is being used from server number one server number 2 had all the data stored in it and also has pop and smtp running from it. i am not using dovecot for pop as yet on the server with dovecot i get such errors in the log file access to data on server number 2 is via nfs on server number 1 i get errors as such Error: fcntl(read-lock) locking failed for file Input/output error squirrelmail gives error imap connection closed and i am not able to login so i set the parameters as such in the dovecot conf file and the error stopped mmap_disable=yes dotlock_use_excl = yes lock_method = dotlock can somebody please advise me if the above is correct ? or is it preferred to use fcntl with lockd (note that my mailbox is maildir format) thanks very much for your help rajesh From jd.beaubien at gmail.com Wed Mar 7 05:19:05 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Tue, 6 Mar 2012 22:19:05 -0500 Subject: [Dovecot] mdbox + gzip and rsync Message-ID: Hi, After reading the following paragraph from the dovecot doc, I've been wondering how it would affect rsync (when combined with gzip): "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation." How will the mailbox files (m.X) files be modified when I move or delete emails using mdbox+gzip. Will the resulting gzipped mdbox files be rsync-able or will they need a full re-upload? If I plan on using rsync for backups, am I better off not using the gzip feature (if i can spare the extra storage)??? Thanks, -JD From CMarcus at Media-Brokers.com Wed Mar 7 15:32:32 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 08:32:32 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? Message-ID: <4F576370.8040706@Media-Brokers.com> On 3/5/2012 1:30 PM, Steve Campbell wrote: > I've been looking at some documentation on shared mail accounts. > But I'm getting mixed thoughts on how this can or should be done. This brings up a question I have been meaning to ask. One thing I want to do on my new converted system is to implement shared mail. There will be two different scenarios - users sharing 'folders', which looks to be fairly simple using virtual ACL files - but for the other scenario, I'm not sure about a specific requirement we will have... I want to give multiple people shared access to some actual accounts with all of the special use folders, with the following requirements: 1. They can all read/reply to new messages as they come in, 2. They use shared \seen, \replied and \forwarded flags, so once someone else has read/dealt with a message, the others see that, 3. When they reply to/forward a message, the Sent message gets saved to that accounts 'Sent' folder, 4. They can *move* messages to other folders in that account (ie, 'file' them), and last (this is the tricky part) 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. These emails deal with financial transactions (AP and AR issues) and Faxes, thus the requirement to not be able to delete them. Can this be accomplished with the current state of things? Or would this require some coding? If the latter, could it be done as a plug-in, or would it require changes to the core code? Thanks, -- Best regards, Charles From lists at wildgooses.com Wed Mar 7 18:39:13 2012 From: lists at wildgooses.com (Ed W) Date: Wed, 07 Mar 2012 16:39:13 +0000 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F578F31.3000303@wildgooses.com> > I want to give multiple people shared access to some actual accounts > with all of the special use folders, with the following requirements: I have done this (unsatisfactorarily) by making it a normal mail account with normal login credentials. Add it like any other mail account. It then satisfies all your requirements, although: behind a nat, on thunderbird and with condstore, I sometimes see read/unread get out of sync... Believed to be a thunderbird bug, but unsure. Easy to resync > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) Interested to hear proper answers... Ed W From wgillespie at es2eng.com Wed Mar 7 20:04:44 2012 From: wgillespie at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 11:04:44 -0700 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <4F57A33C.3050808@es2eng.com> On 3/7/2012 6:32 AM, Charles Marcus wrote: > 5. No one other than a designated user or users (Master User(s)? Users > in a specified Group?) can delete any messages in this account, in any > of the folders. If you are using ACLs, just don't give them the delete permission? But I guess now that I am thinking about it as I write, you did want them to be able to move the messages (which is really a copy + delete). So... maybe not. From e-frog at gmx.de Wed Mar 7 20:17:36 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 07 Mar 2012 19:17:36 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command Message-ID: <4F57A640.1030202@gmx.de> Hello Timo, There seems to be something broken in v2.1 latest hg version: # 2.1.1 (94de7605f50f) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL * OK Namespace completed. Please note that the "OK Namespace completed." is send untagged. It worked on below version: # 2.1.1 (315f0d8cc2b2) 1 namespace * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL 1 OK Namespace completed. Thanks, e-frog From CMarcus at Media-Brokers.com Wed Mar 7 21:03:30 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 07 Mar 2012 14:03:30 -0500 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F57A33C.3050808@es2eng.com> References: <4F576370.8040706@Media-Brokers.com> <4F57A33C.3050808@es2eng.com> Message-ID: <4F57B102.80400@Media-Brokers.com> On 2012-03-07 1:04 PM, Willie Gillespie wrote: > On 3/7/2012 6:32 AM, Charles Marcus wrote: >> 5. No one other than a designated user or users (Master User(s)? Users >> in a specified Group?) can delete any messages in this account, in any >> of the folders. > If you are using ACLs, just don't give them the delete permission? But I > guess now that I am thinking about it as I write, you did want them to > be able to move the messages (which is really a copy + delete). > > So... maybe not. Right... although my understanding is that dovecot does indeed use mv (at least on linux) to do moves when using maildir, so maybe there is a way... I'll wait and see what Timo says about this... no hurry, as I'm still in the design stage, this is just how I'd *like* it to work, but if it won't/can't, I'll figure something else out. Thanks for the replies so far... -- Best regards, Charles From micah at riseup.net Wed Mar 7 21:43:49 2012 From: micah at riseup.net (Micah Anderson) Date: Wed, 07 Mar 2012 14:43:49 -0500 Subject: [Dovecot] dot named folders Message-ID: <87aa3s2o3u.fsf@algae.riseup.net> When a user makes a folder called 'x.y' it actually creates a folder called 'x' with a folder called 'y' inside, rather than a folder called 'x.y'. I'm guessing this has to do with an internal folder separator namespace configuration, but I'm a bit confused by how this works. I'm using 2.0.15 with mdbox and this is what I have configured for my namespaces: namespace { separator = . prefix = inbox = yes } namespace { separator = . prefix = INBOX. inbox = no hidden = yes list = no } I migrated from courier maildirs, so perhaps I no longer need some of these now that the conversion is finished? thanks for any suggestions, I've got my head mixed up on this issue, micah -- From wgillespie+dovecot at es2eng.com Wed Mar 7 22:41:25 2012 From: wgillespie+dovecot at es2eng.com (Willie Gillespie) Date: Wed, 07 Mar 2012 13:41:25 -0700 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <4F57C7F5.4030803@es2eng.com> On 03/07/2012 12:43 PM, Micah Anderson wrote: > > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. Correct. Similar to how in Linux, I could create a folder mkdir test1/test2 It will create test2 inside of test1. The difference being that IMAP doesn't necessarily need the parent mailbox to exist, where Linux would throw an error if test1/ didn't exist first. So basically, as far as I know, you can't have a folder with a "." in the name with the namespaces you have set up. From stan at hardwarefreak.com Wed Mar 7 22:47:43 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 14:47:43 -0600 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F567B14.3030908@cnpapers.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> Message-ID: <4F57C96F.7090602@hardwarefreak.com> On 3/6/2012 3:01 PM, Steve Campbell wrote: > I've experienced that type of locked mailbox before on the old server. > Users insist on accessing their email account as a pop account on their > desktop with the "check for new mail every so many minutes" turned on > and still keep their smartphones on while accessing it as an imap > account so they can still download the files to their desktop when they > return. Using IMAP on the phone and POP on the PC doesn't make any sense. Is there a (valid) reason why these people insist on this phone/IMAP and PC/POP setup? This seems seriously counter intuitive/productive. > So just to clarify, is it OK to have a maildir account setup on this > server for these shared/imap access only accounts along with the mbox > accounts already on there? Yes. With Dovecot it is possible to specify mail_location on a per user basis: http://wiki.dovecot.org/MailLocation You can even do a split mailbox type setup per user using multiple namespaces, for example specifying that INBOX use mbox with all other mail being stored in maildir format: http://wiki.dovecot.org/Namespaces > Thanks for the patience and help Sure thing. -- Stan From stan at hardwarefreak.com Wed Mar 7 23:03:35 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 07 Mar 2012 15:03:35 -0600 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: <4F57CD27.3000207@hardwarefreak.com> On 3/6/2012 5:07 PM, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. Is this thread what "Google tells you"? http://dovecot.org/list/dovecot/2010-October/053909.html Timo is the creator of Dovecot, if you didn't know. So you can take his words for gospel. Also note his last statement in that thread: "The next time you could do it with dsync to avoid these kind of problems." It would seem you omitted a very important detail from your problem report, which is that you recently performed a migration. Please don't omit such critical details in future requests for help. Provide as much relevant detail as possible. This speeds the process up for everyone, and avoids guesswork on our part. -- Stan From M.Roos at roosit.eu Thu Mar 8 01:26:55 2012 From: M.Roos at roosit.eu (Marc) Date: Thu, 8 Mar 2012 00:26:55 +0100 Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap Message-ID: Anybody also experiencing that imap processes are kept running/open by mac osx mail.app, so eventually users are getting to the mail_max_userip_connections limit? Outlook / other clients seem to run fine. Thanks, Marc From jd.beaubien at gmail.com Thu Mar 8 03:30:26 2012 From: jd.beaubien at gmail.com (Jean-Daniel Beaubien) Date: Wed, 7 Mar 2012 20:30:26 -0500 Subject: [Dovecot] Single instance storage Message-ID: I have read most of the doc on the dovecot website, and couldn't find any info on the single instance storage feature, so I'm posting my questions here. - Are these 3 parameters the only one necessary for single instance storage? I cannot find any doc on this feature on the website; is there anything specific I need to know about them? (the last one isn't exactly self-explanatory). - mail_attachment_dir = /srv/vmail/attachments - mail_attachment_hash = %{sha256} - mail_cache_min_mail_count = 2 - Is this feature ready for production? Thanks, -JD From schut at sarvision.nl Thu Mar 8 11:56:35 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 10:56:35 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. Message-ID: Hi, I'm currently migrating our old (colocated) mail server (running a [terribly outdated, I know] dovecot 1.1.11) to a new VPS (virtual private server). The old server was running gentoo linux (which is mainly the culprit of the old dovecot version: gentoo was too much trouble to keep updating); the new server will run debian (stable: 6). Debian currently has dovecot 1.2.15 in its repositories; not that much newer... I read in the docs about the auto-generated-from-hg debian dovecot packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what version to use... OK, 2.2 is development, which leaves the choice to: 1.2.15; 2.0.x, or 2.1.x. I would appreciate any consideration or thoughts on what version to choose. On a related note, there is the possibility to switch from maildir to dbox. I did not really find much pros or cons, except from performance and standards-compliance (ability to use e.g. mutt on the server itself). Any thoughts? About the server: we're just a small company. Think about 15 accounts, normal mail traffic, sometimes relatively large attachments (20mb+). Some accounts have many folders; some accounts are very large (5Gb+). Storage is on ext3, raid10. Performance has never been an issue; reliability and ease of maintenance is more important. Thanks, Vincent Schut. From trashcan at odo.in-berlin.de Thu Mar 8 11:59:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 10:59:37 +0100 Subject: [Dovecot] Failing: doveadm sync <--remote host--> dsync mirror In-Reply-To: <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> References: <0F2AC8D9-E7D0-455F-BB2A-ACC6AA32422F@odo.in-berlin.de> <973D6AE7-4330-4589-970D-F94CA12A6C91@iki.fi> <09FCAE83-5985-49B8-9445-B99157571418@odo.in-berlin.de> <3CD953C4-BCCB-4137-BA9F-6BEE5C2081FA@iki.fi> <8EB87965-B01A-4E4C-A45F-49F94200749E@iki.fi> <1330346709.11500.324.camel@innu> <6D07024B-94F1-4AAD-AF82-21A1F3F7A5DA@odo.in-berlin.de> <3689D904-0238-4FE6-B084-5DF72C8D1CB3@iki.fi> <6EDC01EE-2903-4BBB-A99B-8363BF141428@odo.in-berlin.de> <194C58B2-5189-41EA-9A24-F4D0461B0657@iki.fi> <478FA0EE-7CED-428C-B181-5BDB42A77609@iki.fi> Message-ID: HI -- On 05.03.2012 10:56, Timo Sirainen wrote: > On 4.3.2012, at 13.54, Timo Sirainen wrote: >> On 4.3.2012, at 13.41, Michael Grimm wrote: >>>> By "undeletable" do you mean you have mails that always come back >>>> after expunging them? >>> >>> Yes. Deleting by the client will return them after the next dsync >>> run. > > Luckily this just started happening to me as well. After some > debugging I found and fixed the problem: > > http://hg.dovecot.org/dovecot-2.1/rev/f549cd60fec9 I can confirm, that you fixed that issue successfully. Thanks and regards, Michael From trashcan at odo.in-berlin.de Thu Mar 8 12:26:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 11:26:56 +0100 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <4F53479E.40703@iki.fi> References: <4F53479E.40703@iki.fi> Message-ID: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Hi -- On 04.03.2012 11:44, Timo Sirainen wrote: > In dovecot-2.1 hg you can now test dsync-based replication. > Everything isn't finished yet, but it appears to work and I've > enabled > it for my @dovecot.fi mails. I did give it a try starting some days ago, and I can confirm that you are right, dsync replication can be used, but there are some issues, see below. Let me start with replicator's configuration ... > Below is a configuration for virtual user setup. [...] > service doveadm { > # if you're using a single virtual user, set this to > # start ssh as vmail (not root) > user = vmail > } ... that led to the following complaints at start-up: | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up | dovecot: doveadm: Error: Error reading configuration: net_connect_unix(/var/run/dovecot/config) failed: Permission denied | [...] | (repeatedly, presumably for the number of users in userdb?) Therefore, I modified dsync_remote_cmd ... > dsync_remote_cmd = ssh -p 1234 -l vmail %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} ... and used an empty 'service doveadm { }' instead. That worked, but I would love to run doveadm as vmail user (security), though. How should I do that without running into the error messages above? Now some observations regarding replicator: 1) I see a lot of error messages whenever replicator is in action like (although everything is being synced correctly): | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: save: box=INBOX, uid=27, msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld (admin), flags=() | mail dovecot: dsync-local(test): Error: remote: dsync-remote(test): Info: flag_change: box=TEST, uid=27568, msgid=<20120307144810.6360A74F013 at example.tld>, size=435, from=test at example.tld, flags=(\Seen) JFTR: I do have mail_log plugin activated. Some testing results: 1) I ran a test by sending locally produced mails every other minute on both servers simultaneously. That test ran for ~5 hours. All mails became synced correctly, and no losses were observable, but some duplicates. 2) I did send 100 small test mails from a distant server to my mailservers (mx1 and mx2): a) replicator and dsync deactivated: received 100 distinct mails (57 at mx1, 43 at mx2). b) now, replicator active: 172 mails (100 distinct, a lot of duplicates (up to 8 incarnations of the very same mail). Ok, 2b) is a rather 'mailbomb-like' scenario, but it worries me a bit: One of my users is receiving mails from a mailing list that sends individual mails batch-wise ... 3) replicator active: 1000 mails sent ended in 4523 mails at every server. Well, that was a mailbomb :-) 4) replicator active: 100 (and even 1000) locally produced mails at one server only: all 100 (and 1000 mails) became synced, prefectly well, without duplicates. 5) replicator active: 100 locally produced mails at both servers simultaneously: 341 mails, thus a lot of multiple incarnations. (This test differed from 1) because all mails were sent in one batch.) Final note to these tests: It doesn't matter whether sieve with redirecting, or sieve with redirecting and copying, or no sieve at all has been involved. It seems to me, that whenever a larger number of mails arrive on both servers simultaneously, the replicator gets into trouble [1]. I am unsure if one can expect that a replicator should deal with such stress, though. Or? R?sum?: The overall performance of replicator is very good from my point of view for my conditions (handful users, average workload of roughly 1000 mails a day). Thank you for replicator and regards, Michael [1] JFTR: I did similar tests in the past with dsync running from cron every other minute with similar results. From tss at iki.fi Thu Mar 8 13:35:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 08 Mar 2012 13:35:34 +0200 Subject: [Dovecot] dsync replication available for testing In-Reply-To: <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> References: <4F53479E.40703@iki.fi> <47470ffe12f36af0b969ccd89bc1962a@mx1.enfer-du-nord.net> Message-ID: <1331206534.2081.101.camel@innu> On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > Let me start with replicator's configuration ... > > > Below is a configuration for virtual user setup. > [...] > > service doveadm { > > # if you're using a single virtual user, set this to > > # start ssh as vmail (not root) > > user = vmail > > } > > ... that led to the following complaints at start-up: > > | dovecot: master: Dovecot v2.1.1 (d66568d34e40) starting up > | dovecot: doveadm: Error: Error reading configuration: > net_connect_unix(/var/run/dovecot/config) failed: Permission denied > | [...] > | (repeatedly, presumably for the number of users in userdb?) You can do for example: service config { unix_listener config { user = vmail } } > Now some observations regarding replicator: > > 1) I see a lot of error messages whenever replicator is in action > like (although everything is being synced correctly): > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: save: box=INBOX, uid=27, > msgid=<3V2JfH5Kv4z7Ft at example.tld>, size=547, from=test at example.tld > (admin), flags=() > > | mail dovecot: dsync-local(test): Error: remote: > dsync-remote(test): Info: flag_change: box=TEST, uid=27568, > msgid=<20120307144810.6360A74F013 at example.tld>, size=435, > from=test at example.tld, flags=(\Seen) > > JFTR: I do have mail_log plugin activated. Hmm. Right. I guess all the logging should go to the log files instead of via the ssh pipe. Of course that would also require that dsync has write access to your log files. > It seems to me, that whenever a larger number of mails arrive on both > servers simultaneously, > the replicator gets into trouble [1]. I am unsure if one can expect > that a replicator should > deal with such stress, though. Or? Were these mails delivered via LMTP or dovecot-lda? The locks should prevent duplicates I think, so there's something still going wrong. From Leo.Baltus at omroep.nl Thu Mar 8 13:56:41 2012 From: Leo.Baltus at omroep.nl (Leo Baltus) Date: Thu, 8 Mar 2012 12:56:41 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <4F459344.5020407@rename-it.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> Message-ID: <20120308115641.GB5700@omroep.nl> Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: > On 2/22/2012 12:15 AM, Adam Szpakowski wrote: > >On 22.02.2012 00:09, Timo Sirainen wrote: > >>Well, it would be possible to build a doveadm script that > >>deletes the duplicates after delivery, but currently there's no > >>implementation to avoid delivering duplicate Message-IDs in the > >>first place. > >> > >>I don't really like such a Message-ID-based deduplication > >>feature enabled by default, but something like this could be > >>nice: > >> > >>fileinto :copy :x-deduplicate "boss"; > >> > >>Anyway, probably not going to be implemented anytime soon. > >Maybe there is a way to use a procmail with something like this: > > > >:0 Wh: msgid.lock > >| formail -D 8192 .msgid.cache > > > >But is there a safe way to use it together with sieve? Using > >Pigeonhole Sieve Pipe Plugin? > > > > There are a few options: > > * You can use Procmail as primary delivery agent and invoke > dovecot-lda/sieve from within Procmail once Procmail has determined > that it is not a duplicate. > > * Invoke procmail from Sieve using the pipe extension (i.e. the > other way around). This has the disadvantage that Procmail will > have to take care of final delivery, meaning the Dovecot indexes are > not updated. > > * For Pigeonhole v0.3 there is the possibility to "filter" the > message through Procmail using the sieve_extprograms plugin, but I > haven't actually tested something like that. > > * I've just created an alternative that implements something similar > to the Procmail code you posted above, but from within Sieve itself. > It is a custom language extension called vnd.dovecot.duplicate and > it adds the "duplicate" test. This test keeps track of which > Message-IDs it has seen before in earlier deliveries and yields a > true result if the message was seen before, e.g.: > > require "vnd.dovecot.duplicate"; > > if duplicate { > discard; > } > > Read the specification for details ("name" argument is not yet implemented): > > http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate/raw-file/4b1dbda4d3fc/doc/rfc/spec-bosch-sieve-duplicate.txt > > The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate > > This plugin is only a few hours old, experimental, and largely > untested, so test it thoroughly before considering to use this. Read > the INSTALL file for compile and installation instructions. > > Comments are welcome. > I did some very basic testing and it seems to work fine. The example in spec-bosch-sieve-duplicate.txt however says: if duplicate { fileinto :create "Trash/Duplicate"; } This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' So this leads to: failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name I am not sure if this a bug or not, I suppose you know the rfc's better than I do, is the sieve language supposed to be agnostic of the internals of the storage-engine (dovecot)? -- Leo Baltus, internetbeheerder /\ NPO ICT Internet Services /NPO/\ Sumatralaan 45, 1217 GP Hilversum, Filmcentrum, west \ /\/ beheer at omroep.nl, 035-6773555 \/ From CMarcus at Media-Brokers.com Thu Mar 8 14:03:05 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 07:03:05 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F589FF9.7080608@Media-Brokers.com> On 2012-03-08 4:56 AM, Vincent Schut wrote: > The old server was running gentoo linux (which is mainly the culprit of > the old dovecot version: gentoo was too much trouble to keep updating); Please stop with the FUD... I've been running gentoo for 8+ years, and it is a *breeze* to keep updated, *especially* long term (since it is a 'rolling release' type of distro)... Yes, it actually does require some minimum amount of attention from the admin, like, say, once per week or once per month updates - buy so should *any* system... and yes, it does require a little more willingness to learn and 'get your hands dirty' (especially for the installation), but it is well worth it. Oh - and Portage rocks... :) -- Best regards, Charles From stephan at rename-it.nl Thu Mar 8 14:05:37 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 08 Mar 2012 13:05:37 +0100 Subject: [Dovecot] duplicates with multiple To/CC and sieve redirect copy In-Reply-To: <20120308115641.GB5700@omroep.nl> References: <4F441ED8.20908@3a.pl> <673D2924-344E-4E9E-9BBC-9AF4E92C5BE2@iki.fi> <4F44227F.9030502@3a.pl> <1287D4B6-BF86-4A96-9963-8029CADDBB13@iki.fi> <4F442592.608@3a.pl> <4F459344.5020407@rename-it.nl> <20120308115641.GB5700@omroep.nl> Message-ID: <4F58A091.7090704@rename-it.nl> On 3/8/2012 12:56 PM, Leo Baltus wrote: > Op 23/02/2012 om 02:15:48 +0100, schreef Stephan Bosch: >> The repository is at: http://hg.rename-it.nl/pigeonhole-0.3-sieve-duplicate >> >> This plugin is only a few hours old, experimental, and largely >> untested, so test it thoroughly before considering to use this. Read >> the INSTALL file for compile and installation instructions. >> >> Comments are welcome. > I did some very basic testing and it seems to work fine. > > The example in spec-bosch-sieve-duplicate.txt however says: > > if duplicate { > fileinto :create "Trash/Duplicate"; > } > > This assumes the hierarchy separator is '/', but in Maildir this defaults to '.' > > So this leads to: > failed to store into mailbox 'Trash/Duplicate': Invalid mailbox name > > I am not sure if this a bug or not, I suppose you know the rfc's better > than I do, is the sieve language supposed to be agnostic of the > internals of the storage-engine (dovecot)? For Sieve, the mailbox name is pretty much opaque. Usually, it matches what is used through IMAP. http://tools.ietf.org/html/rfc5228#section-4.1 So, in your case, just use "Trash.Duplicate" instead. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 8 14:19:28 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 08 Mar 2012 13:19:28 +0100 Subject: [Dovecot] dsync replication available for testing Message-ID: Hi -- On 08.03.2012 12:35, Timo Sirainen wrote: > On Thu, 2012-03-08 at 11:26 +0100, Michael Grimm wrote: > You can do for example: > > service config { > unix_listener config { > user = vmail > } > } I will try that later. >> It seems to me, that whenever a larger number of mails arrive on >> both >> servers simultaneously, the replicator gets into trouble [1]. I am >> unsure if one can expect that a replicator should deal with such >> stress, >> though. Or? > > Were these mails delivered via LMTP or dovecot-lda? LMTP > The locks should prevent duplicates I think, so there's something > still > going wrong. Just to be sure that I didn't misunderstand your proposed configuration: @mx1: plugin { mail_replica = remote:vmail at mx2.example.tld } @mx2: plugin { mail_replica = remote:vmail at mx1.example.tld } I do need to define one mail_replica plugin at each server pointing to the other one, correct? Regards, Michael From as at 3a.pl Thu Mar 8 14:24:13 2012 From: as at 3a.pl (Adam Szpakowski) Date: Thu, 08 Mar 2012 13:24:13 +0100 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: Message-ID: <4F58A4ED.8070704@3a.pl> On 08.03.2012 10:56, Vincent Schut wrote: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... > I read in the docs about the auto-generated-from-hg debian dovecot > packages for 2.0, 2.1 and 2.2. Which leaves me to the choice what > version to use... OK, 2.2 is development, which leaves the choice to: > 1.2.15; 2.0.x, or 2.1.x. > > I would appreciate any consideration or thoughts on what version to > choose. On several production machines we are using dovecot from debian testing repos, so 2.0.x. It's working stable for us and is quite easy to maintain. Please be careful and very selectively install packages from testing. If possible, the package dependences should be installed from stable/security. -- Adam Szpakowski From schut at sarvision.nl Thu Mar 8 15:53:54 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 08 Mar 2012 14:53:54 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F589FF9.7080608@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: On 03/08/2012 01:03 PM, Charles Marcus wrote: > On 2012-03-08 4:56 AM, Vincent Schut wrote: >> The old server was running gentoo linux (which is mainly the culprit of >> the old dovecot version: gentoo was too much trouble to keep updating); > > Please stop with the FUD... > > I've been running gentoo for 8+ years, and it is a *breeze* to keep > updated, *especially* long term (since it is a 'rolling release' type of > distro)... Right. I should've known I shouln't mention anyone's favourite distro... :-) Hey, listen, sorry I offended you... its really nothing I have against gentoo, I'm sorry it might have sounded like that. It's just that I appeared not to have the time and energy to do regular updates, and when I tried to update something some months later, I had problems which I had no time and energy to start solving. Thus I decided a rolling distro was no good combination for my server and me. Which is why I will switch to a less rolling distro. That's really all there is to say about. I do still have a rolling distro which-will-not-be-named on my desktop, which I can and do update often and easy. > > Yes, it actually does require some minimum amount of attention from the > admin, like, say, once per week or once per month updates - buy so > should *any* system... and yes, it does require a little more > willingness to learn and 'get your hands dirty' (especially for the > installation), but it is well worth it. Yes, I have learned lots from some years with gentoo. No bad feelings. Just bad combo this time. > > Oh - and Portage rocks... :) > Well, yes, so does granite. Or iron maiden. Or whatever. As long as you like it :-) But maybe you also have something useful to say on the questions I *did* ask? About dovecot versions, and/or maildir vs. dbox for example? As the subject said, I was seeking advice, not rant nor war... Best, Vincent. From campbell at cnpapers.com Thu Mar 8 16:38:36 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 08 Mar 2012 09:38:36 -0500 Subject: [Dovecot] Shared mboxes In-Reply-To: <4F57C96F.7090602@hardwarefreak.com> References: <4F551456.102@cnpapers.com> <4F55495B.10609@hardwarefreak.com> <4F561F27.5000102@cnpapers.com> <4F5670D2.9090004@hardwarefreak.com> <4F567B14.3030908@cnpapers.com> <4F57C96F.7090602@hardwarefreak.com> Message-ID: <4F58C46C.8000202@cnpapers.com> On 3/7/2012 3:47 PM, Stan Hoeppner wrote: > On 3/6/2012 3:01 PM, Steve Campbell wrote: > >> I've experienced that type of locked mailbox before on the old server. >> Users insist on accessing their email account as a pop account on their >> desktop with the "check for new mail every so many minutes" turned on >> and still keep their smartphones on while accessing it as an imap >> account so they can still download the files to their desktop when they >> return. > Using IMAP on the phone and POP on the PC doesn't make any sense. Is > there a (valid) reason why these people insist on this phone/IMAP and > PC/POP setup? This seems seriously counter intuitive/productive. The bulk of these type users are sales staff. They use their desktop when their in the office. For years, the only type of email account we used was pop just because that was the way it was. We used horde for webmail, which read these type of accounts just fine. Once they needed email in the field, it was necessary to either set up their phones to use pop and keep email on the server so that they could download the email to their desktop, or use imap on the phones. They typically don't use any folders they've created on the imap account when accessing mail on the desktop. It would be a nightmare going to each desktop, finding a time when each and every user would have the time to allow us to change things, and switching all of the accounts. It may not seem to be a good way of doing things, but it's just the way our system here has evolved. Now that we're down to skeleton-type staffing, it's not easy to find the time and manpower to accomplish change when it "ain't broke". The occasional locked mailbox was easier to resolve that the massive change to all user's accounts. This all came about because I installed a new server to replace the old, and dovecot became the pop/imap server. > >> So just to clarify, is it OK to have a maildir account setup on this >> server for these shared/imap access only accounts along with the mbox >> accounts already on there? > Yes. With Dovecot it is possible to specify mail_location on a per user > basis: > > http://wiki.dovecot.org/MailLocation > > You can even do a split mailbox type setup per user using multiple > namespaces, for example specifying that INBOX use mbox with all other > mail being stored in maildir format: > > http://wiki.dovecot.org/Namespaces > >> Thanks for the patience and help > Sure thing. Again, thanks for the help. From CMarcus at Media-Brokers.com Thu Mar 8 18:04:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 08 Mar 2012 11:04:14 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: References: <4F589FF9.7080608@Media-Brokers.com> Message-ID: <4F58D87E.3040704@Media-Brokers.com> On 2012-03-08 8:53 AM, Vincent Schut wrote: > But maybe you also have something useful to say on the questions I *did* > ask? About dovecot versions, and/or maildir vs. dbox for example? As the > subject said, I was seeking advice, not rant nor war... Yeah, sorry, and I wasn't offended, I just dislike it when someone says something like that without clarification... As for version, it is generally recommended for obvious reasons to stay within the confines of your distros package manager unless you are comfortable installing from source. I've never used Debian, so can't speak to which repos you can safely use or the implications if you do... As for what mailbox format, there is no more 'dbox', it is either sdbox (like mbox one file per folder) or mdbox (multiple files per folder) - that said, mdbox seems to be the best general purpose, but my understanding is it can complicate things if something goes wrong, but it seems to be very solid. -- Best regards, Charles From micah at riseup.net Thu Mar 8 18:27:43 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:27:43 -0500 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> Message-ID: <87ty1zys5c.fsf@algae.riseup.net> Willie Gillespie writes: > On 03/07/2012 12:43 PM, Micah Anderson wrote: >> >> When a user makes a folder called 'x.y' it actually creates a folder >> called 'x' with a folder called 'y' inside, rather than a folder called >> 'x.y'. I'm guessing this has to do with an internal folder separator >> namespace configuration, but I'm a bit confused by how this works. > > Correct. > Similar to how in Linux, I could create a folder > mkdir test1/test2 > It will create test2 inside of test1. > > The difference being that IMAP doesn't necessarily need the parent mailbox to > exist, where Linux would throw an error if test1/ didn't exist first. > > So basically, as far as I know, you can't have a folder with a "." in the name > with the namespaces you have set up. That makes sense, however I'm not sure that I need these namespaces any longer if I no longer am using the maildir format (mdbox). In either case, it seems like the internal folder separator should not be exposed to the user like this. What is happening now is the user gets something other than they expect (a folder within a folder, instead of a folder with a dot in the name) because of some unknown internal configuration. If moving to mdbox is not enough to remove these namespace configurations that cause this, then it would be good if the user was unable to create such a folder, because it was prohibited, rather than creating something other than they expect. micah From micah at riseup.net Thu Mar 8 18:29:46 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 08 Mar 2012 11:29:46 -0500 Subject: [Dovecot] seeking advice: dovecot versions; mailbox formats. References: Message-ID: <87pqcnys1x.fsf@algae.riseup.net> Vincent Schut writes: > Debian currently has dovecot 1.2.15 in its repositories; not that much > newer... No, Debian has 1.2.15 in its /stable (squeeze)/ repositories, there are newer versions available in other Debian repositories. micah From robert at schetterer.org Thu Mar 8 18:32:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Thu, 08 Mar 2012 17:32:49 +0100 Subject: [Dovecot] dot named folders In-Reply-To: <87ty1zys5c.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> Message-ID: <4F58DF31.3040203@schetterer.org> Am 08.03.2012 17:27, schrieb Micah Anderson: > Willie Gillespie writes: > >> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>> >>> When a user makes a folder called 'x.y' it actually creates a folder >>> called 'x' with a folder called 'y' inside, rather than a folder called >>> 'x.y'. I'm guessing this has to do with an internal folder separator >>> namespace configuration, but I'm a bit confused by how this works. >> >> Correct. >> Similar to how in Linux, I could create a folder >> mkdir test1/test2 >> It will create test2 inside of test1. >> >> The difference being that IMAP doesn't necessarily need the parent mailbox to >> exist, where Linux would throw an error if test1/ didn't exist first. >> >> So basically, as far as I know, you can't have a folder with a "." in the name >> with the namespaces you have set up. > > That makes sense, however I'm not sure that I need these namespaces any > longer if I no longer am using the maildir format (mdbox). > > In either case, it seems like the internal folder separator should not > be exposed to the user like this. What is happening now is the user gets > something other than they expect (a folder within a folder, instead of a > folder with a dot in the name) because of some unknown internal > configuration. > > If moving to mdbox is not enough to remove these namespace > configurations that cause this, then it would be good if the user was > unable to create such a folder, because it was prohibited, rather than > creating something other than they expect. > > micah > http://wiki.dovecot.org/Plugins/Listescape may help -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From e-frog at gmx.de Thu Mar 8 19:43:25 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 08 Mar 2012 18:43:25 +0100 Subject: [Dovecot] v2.1 latest hg: untagged reply to namespace command In-Reply-To: <4F57A640.1030202@gmx.de> References: <4F57A640.1030202@gmx.de> Message-ID: <4F58EFBD.9080205@gmx.de> On 07.03.2012 19:17, wrote e-frog: > # 2.1.1 (94de7605f50f) > 1 namespace > * NAMESPACE (("" "/")("virtual/" "/")) NIL NIL > * OK Namespace completed. > > Please note that the "OK Namespace completed." is send untagged. Ok, it's working again today with 2.1.1 (7a26c427fc78). From busseniu at in.tum.de Thu Mar 8 19:56:03 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Thu, 08 Mar 2012 18:56:03 +0100 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> Message-ID: <4F58F2B3.9070407@in.tum.de> On 03/04/2012 03:10 PM, Timo Sirainen wrote: > BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. Oh, I was not aware that this feature exists. I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Our migration process involves: 1) Converting the maildir from Courier using the Perl script 2) Converting to mdbox using dsync -R backup The POP3 ordering seems to get lost during the second step. I.e., if Dovecot is set up to server POP3 mails from a maildir having "O" flags, the POP3 ordering is as intended. After changing the configuration to mdbox format and converting the mails using dsync, the POP3 ordering is different. Is this known or am I missing something? (I tried Dovecot 2.1.1.) Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 20:46:50 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 18:46:50 +0000 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: Your message of "Sun, 04 Mar 2012 15:36:59 +0200." <5F971D9D-715A-4C06-8F3B-CF371E2EF3A8@iki.fi> Message-ID: Thank you for your help, Timo. > use Dovecot v2.0's dsync I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) on a dovecot (v1) installation; presumably with its own configuration file(s). > You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... Yes, I think we could do that; I should have thought of it myself, thanks again. I think there was one other problem with the automatic conversion which I've now remembered: I note that the first time a user connects to th eimap service dovecot creates their (virtual) home directory for them with all the right permissions. That's great and I use the existence of that directory as an indication to our MTA that the user wants delivery into the dovecot store rather than their old system mailbox. However once I tried using the convert plugin the process fails because (it seems) the conversion tries to take place before the home directory has been created. Is there any configuration change that might change this order? Can I configure the convert plugin on LDA delivery, for example, instead of as part of the "protocol imap" section? Many thanks, Steve Platt From tss at iki.fi Thu Mar 8 20:51:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 20:51:26 +0200 Subject: [Dovecot] Pop3 ordering in mdbox In-Reply-To: <4F58F2B3.9070407@in.tum.de> References: <4F4B2F62.1020204@in.tum.de> <1CEE23CD-9E64-4A57-BB60-E2226F1B3B42@iki.fi> <4F58F2B3.9070407@in.tum.de> Message-ID: <0FEB6932-0FE5-42C6-B72F-3FD914B3A7BB@iki.fi> On 8.3.2012, at 19.56, Christoph Bu?enius wrote: > On 03/04/2012 03:10 PM, Timo Sirainen wrote: >> BTW. The script should some day be updated for Dovecot v2.0.13+ which supports storing separate POP3 and IMAP message order. > > Oh, I was not aware that this feature exists. > > I was just experimenting with the "O" flag in dovecot-uidlist to see how the conversion script can be updated. I was wondering if this is only implemented for Maildir? Yeah, for now it's only for Maildir. Probably wouldn't be difficult to implement for dbox by adding it as dbox metadata (although how to add it there? dsync can't copy that). From steve.platt at mrc-bsu.cam.ac.uk Thu Mar 8 21:04:47 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Thu, 08 Mar 2012 19:04:47 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 Message-ID: I've set up a list of ciphers that excludes SSLv2 ciphers (and other weak ones) in the hope of preventing SSLv2 connections: ssl_cipher_list = TLSv1+HIGH : !SSLv2 : RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH However, this doesn't prevent the SSLv2 connection being allowed as our Nessus scans show and I'm tasked with trying to plug that "hole". I see Dovecot2 had the following change a year or so ago, in file src/login-common/ssl-proxy-openssl.c: - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); I tried making the same change to dovecot1's src tree on our test system and it seems to have the desired effect; however I am very hesitant about putting this into our production system without seeking advice here first :-) Have I missed anything that's obviously bad about doing this please? Thanks again, Steve Platt From markus at mpetri.org Thu Mar 8 21:18:12 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 8 Mar 2012 20:18:12 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 Message-ID: <20120308201812.2932e90c@legolas.home.ceotex.de> Hi, after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use shared folders with mutt anymore. 2.1 lists the shared namespace prefix once per user sharing an folder in LIST "" "%". I also noticed, that with 2.1 the user folder (Shared/) is no longer tagged as \NoSelect. Is this the intended behaviour and mutt simply cannot cope with it or is it a dovecot problem? Here an example with three users sharing a folder to the logged in user with Dovecot 2.1.1: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\HasChildren) "/" "Shared/test" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasChildren) "/" "Shared/test2" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. The same three users and config with Dovecot 2.0.18: 2 LIST "" "*" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared/test" * LIST (\Noselect \HasChildren) "/" "Shared/test2" * LIST (\Noselect \HasChildren) "/" "Shared/test3" * LIST (\HasNoChildren) "/" "Shared/test/Share" * LIST (\HasNoChildren) "/" "Shared/test2/Share2" * LIST (\HasNoChildren) "/" "Shared/test3/Share3" 2 OK List completed. 2 LIST "" "%" * LIST (\HasNoChildren) "/" "INBOX" * LIST (\Noselect \HasChildren) "/" "Shared" 2 OK List completed. Markus # 2.1.1: /opt/dovecot-2.1/etc/dovecot/dovecot.conf # OS: Linux 3.2.0-1-amd64 x86_64 Debian wheezy/sid auth_mechanisms = plain login disable_plaintext_auth = no listen = 192.168.56.11 mail_location = maildir:~/Maildir mail_plugins = acl managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { inbox = yes location = prefix = separator = / subscriptions = yes type = private } namespace { inbox = no list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/index/shared/%%u prefix = Shared/%%u/ separator = / subscriptions = no type = shared } passdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } plugin { acl = vfile acl_anyone = allow acl_shared_dict = file:/var/lib/vdovecot/shared-mailboxes.db } protocols = imap service auth { unix_listener auth-userdb { mode = 0600 user = vdovecot } } ssl = no userdb { args = /opt/dovecot-2.1/etc/dovecot/passwd driver = passwd-file } verbose_proctitle = yes protocol imap { mail_plugins = acl imap_acl } From tss at iki.fi Thu Mar 8 21:36:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 8 Mar 2012 21:36:09 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <20120308201812.2932e90c@legolas.home.ceotex.de> References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: On 8.3.2012, at 21.18, Markus Petri wrote: > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > once per user sharing an folder in LIST "" "%". > > I also noticed, that with 2.1 the user folder (Shared/) is no > longer tagged as \NoSelect. > > Is this the intended behaviour and mutt simply cannot cope with it or > is it a dovecot problem? Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even if it did. Also Dovecot probably should add \Noselect, especially if the mailbox isn't really selectable (there's some weirdness between shared/user being equal to shared/user/INBOX, but I'm not sure what to do about it). From Bennett.Tony at con-way.com Fri Mar 9 01:23:00 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Thu, 8 Mar 2012 15:23:00 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? Message-ID: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. (The output of "dovecot -n" is at the bottom of this email.) I'm trying "baby steps" to get it up, before I give it the final configuration. (My apologies: I was pointed to RFC3501 and told to get an IMAP server, build it, configure it, and bring it up) What is currently occurring when I start dovecot is: Error: service(pop3-login): listen(::, 110) failed: Address already in use Error: service(pop3-login): listen(::, 995) failed: Address already in use Error: service(imap-login): listen(::, 143) failed: Address already in use Error: service(imap-login): listen(::, 993) failed: Address already in use Fatal: Failed to start listeners Using TRUSS and recompiling with log messages I've determined that dovecot is successfully creating and binding to AF_INET sockets... but is failing when trying to do the "bind" the same port to an AF_INET6 socket. The failure is "EADDRINUSE". The logic in the dovecot sources seems driven off of the define of HAVE_IPV6 (defined in config.h by configure) So, the questions I have are: - Is this the correct behavior - If this is the correct behavior, has this been tested against AIX 6.1, and if so, does anyone have an idea of what I did wrong...??? If it has not been tested against AIX 6.1 and is NOT the correct behavior, should I just change "config.h", and undefined HAVE_IPV6 ... or is there a better way to move beyond this issue... (like a change to "configure")??? Thanks, -tony Here is the output of "dovecot -n": # 2.1.1: /attic/usr/local/etc/dovecot/dovecot.conf # OS: AIX 1 00C30F654C00 default_login_user = dovecot disable_plaintext_auth = no namespace { inbox = yes location = mailbox { special_use = \Drafts name = Drafts } mailbox { special_use = \Junk name = Junk } mailbox { special_use = \Sent name = Sent } mailbox { special_use = \Sent name = Sent Messages } mailbox { special_use = \Trash name = Trash } prefix = name = inbox } passdb { args = scheme=CRYPT username_format=%u /attic/usr/local/etc/dovecot/users driver = passwd-file } service anvil-auth-penalty { name = anvil } service auth-worker { name = auth-worker } service auth-client { name = auth } service config { name = config } service dict { name = dict } service login/proxy-notify { name = director } service dns-client { name = dns_client } service doveadm-server { name = doveadm } service imap { name = imap-login } service login/imap { name = imap } service indexer-worker { name = indexer-worker } service indexer { name = indexer } service ipc { name = ipc } service lmtp { name = lmtp } service log-errors { name = log } service pop3 { name = pop3-login } service login/pop3 { name = pop3 } service login/ssl-params { name = ssl-params } service stats-mail { name = stats } ssl_cert = References: Message-ID: <201203091030.20828.sdavies@sdc.com.au> Yes that is the google thread that I saw. I don't see the relevance of your reference to dsync. As I read the man pages for dsync it is used to sync separate servers, to make backups or to convert mailbox formats. When I upgraded from 1.2.15 to 2.1.1 I saw nothing in the doco to suggest that dsync was relevant to my scenario. In a previous thread here (Log sync errors), Timo suggested that the migration fix was to delete all .imap directories. My understanding was that this should fix any differences between 1.2.15 files and 2.1.1. If that were the case, mentioning the migration again would seem irrelevant. However, it seems that deleting the .imap files did not fix the log sync errors or the fscking warnings. Both are still happening continuously. Cheers, Stephen On Thu, 8 Mar 2012 08:26:55 PM dovecot-request at dovecot.org wrote: > Date: Wed, 07 Mar 2012 15:03:35 -0600 > From: Stan Hoeppner > To: dovecot at dovecot.org > Subject: Re: [Dovecot] Fscking warnings > Message-ID: <4F57CD27.3000207 at hardwarefreak.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On 3/6/2012 5:07 PM, Stephen Davies wrote: > > Google tells me that these "should go away" but they don't. > > > > > > > > Seems to happen continuously while a user is viewing email. > > Is this thread what "Google tells you"? > > http://dovecot.org/list/dovecot/2010-October/053909.html > > Timo is the creator of Dovecot, if you didn't know. So you can take his > words for gospel. Also note his last statement in that thread: > > "The next time you could do it with dsync to avoid these kind of > problems." > > It would seem you omitted a very important detail from your problem > report, which is that you recently performed a migration. Please don't > omit such critical details in future requests for help. Provide as much > relevant detail as possible. This speeds the process up for everyone, > and avoids guesswork on our part. > > -- > Stan > > > ------------------------------ > > Message: 10 > Date: Thu, 8 Mar 2012 00:26:55 +0100 > From: "Marc" > To: > Subject: [Dovecot] FW: Centos 6 + dovecot 2 + mail.app + imap -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583 From mmielke at sapphire.gi Fri Mar 9 02:50:47 2012 From: mmielke at sapphire.gi (Martin Mielke) Date: Fri, 9 Mar 2012 00:50:47 +0000 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. Message-ID: Hi all, I have inherited an old Dovecot installation which is causing headaches almost every day. I know that one of the rules says "Don't bother asking questions about v0.99.x versions. They're no longer supported."...but please bear with me, this will be quick as I only need some advise from experienced Dovecot gurus out there. I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... Thanks a lot in advance! Regards, Martin From ybhu at hk1.ibm.com Fri Mar 9 10:09:37 2012 From: ybhu at hk1.ibm.com (Andy YB Hu) Date: Fri, 9 Mar 2012 16:09:37 +0800 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? Message-ID: In the real world, the mails which belongs to one thread could be dispersed in different mailbox, at least Inbox and SENT, so whether the command can search different mailboxes and grap them in one talk with dovecot? If Not, any other approach to do that? BTW, what the THREAD=REFS stands for? In the RFC, http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. From janfrode at tanso.net Fri Mar 9 10:31:34 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 9 Mar 2012 09:31:34 +0100 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <20120309083134.GA8248@dibs.tanso.net> On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: > > As for what mailbox format, there is no more 'dbox', it is either > sdbox (like mbox one file per folder) or mdbox (multiple files per > folder) - Sdbox is like maildir, one message per file, while mdbox is more like mbox: http://wiki2.dovecot.org/MailboxFormat/dbox > that said, mdbox seems to be the best general purpose, but > my understanding is it can complicate things if something goes > wrong, but it seems to be very solid. It's a leap of faith to go with dovecot's own format, and no longer be able to use grep and mutt to poke in mail folders directly, but as a serverside storage format it seems like the right way to go. -jf From varia at e-healthexpert.org Fri Mar 9 11:35:17 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Fri, 9 Mar 2012 09:35:17 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: References: Message-ID: <20120309093517.30979c04@e-healthexpert.org> On Thu, 08 Mar 2012 19:04:47 +0000, Steve Platt wrote: > I've set up a list of ciphers that excludes SSLv2 ciphers (and other > weak ones) in the hope of preventing SSLv2 connections: > > ssl_cipher_list = TLSv1+HIGH : !SSLv2 : > RC4+MEDIUM : !aNULL : !eNULL : !3DES : @STRENGTH > > I tried making the same change to dovecot1's src tree on our test > system and it seems to have the desired effect; No need to change sources. Try this and see if it serves your purpose: ssl = required ssl_cipher_list = HIGH:!SSLv2:!aNULL:!MD5!DES:!3DES M. From CMarcus at Media-Brokers.com Fri Mar 9 16:45:35 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 09 Mar 2012 09:45:35 -0500 Subject: [Dovecot] OT: Distrowars - WAS: Re: seeking advice: dovecot versions; mailbox formats. In-Reply-To: <4F58D87E.3040704@Media-Brokers.com> References: <4F589FF9.7080608@Media-Brokers.com> <4F58D87E.3040704@Media-Brokers.com> Message-ID: <4F5A178F.1060404@Media-Brokers.com> On Thu, Mar 09, 2012 at 12:30AM -0500, Jan-Frode Myklebust wrote: > On Thu, Mar 08, 2012 at 11:04:14AM -0500, Charles Marcus wrote: >> As for what mailbox format, there is no more 'dbox', it is either >> sdbox (like mbox one file per folder) or mdbox (multiple files per >> folder) - > > Sdbox is like maildir, one message per file, while mdbox is more > like mbox: > > http://wiki2.dovecot.org/MailboxFormat/dbox Wow, I've no idea how that bit of incorrect data got lodged inside my head. Thanks Jan-Frode for the correction! -- Best regards, Charles From steve.platt at mrc-bsu.cam.ac.uk Fri Mar 9 17:05:26 2012 From: steve.platt at mrc-bsu.cam.ac.uk (Steve Platt) Date: Fri, 09 Mar 2012 15:05:26 +0000 Subject: [Dovecot] disabling SSLv2 in dovecot 1.2.17 In-Reply-To: Message from Mark Alan of "Fri, 09 Mar 2012 09:35:17 GMT." <20120309093517.30979c04@e-healthexpert.org> Message-ID: Hi Mark, I think I may not have been clear enough in my query, sorry! What I'm trying to do is to prevent SSLv2 connections being made to our IMAP server while allowing SSLv3 and TLSv1 connections. I think I've prevented the use of SSLv2 ciphers but this does not prevent SSLv2 protocol connections (as far as I can tell). (Once connected, the SSLv2 client finds it has no ciphers so the session fails at that point but this is not enough to satisfy our security audit. I want to disable the use of the SSLv2 protocol itself, not just the SSLv2 ciphers) steve.platt at mrc-bsu.cam.ac.uk said: > I see Dovecot2 had the following change a year or so ago, in file src/ > login-common/ssl-proxy-openssl.c: > > - SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL); > + SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > > I tried making the same change to dovecot1's src tree on our test system and > it seems to have the desired effect ... I'm testing this by using: openssl s_client -ssl2 -connect mailhost:993 This should fail immediately with "ssl handshake failure" (for a happy audit!). Thanks again, Steve From user+dovecot at localhost.localdomain.org Fri Mar 9 22:19:15 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Fri, 09 Mar 2012 21:19:15 +0100 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> Message-ID: <4F5A65C3.4080404@localhost.localdomain.org> On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From Bennett.Tony at con-way.com Fri Mar 9 22:34:56 2012 From: Bennett.Tony at con-way.com (Bennett, Tony) Date: Fri, 9 Mar 2012 12:34:56 -0800 Subject: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? In-Reply-To: <4F5A65C3.4080404@localhost.localdomain.org> References: <9E085D377965634187A85638358AE61101A289A3CC@DCXPRCL017.cnf.prod.cnf.com> <4F5A65C3.4080404@localhost.localdomain.org> Message-ID: <9E085D377965634187A85638358AE61101A291DA92@DCXPRCL017.cnf.prod.cnf.com> Bingo... It worked... Thanks, Pascal -tony -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Pascal Volk Sent: Friday, March 09, 2012 12:19 PM To: Dovecot Mailing List Subject: Re: [Dovecot] Has dovecot 2.1.1 been built and tested on AIX 6.1??? On 03/09/2012 12:23 AM Bennett, Tony wrote: > I have downloaded and built dovecot 2.1.1 using gcc on AIX 6.1. > (The output of "dovecot -n" is at the bottom of this email.) > > I'm trying "baby steps" to get it up, before I give it the final configuration. > (My apologies: I was pointed to RFC3501 and told to get an IMAP server, > build it, configure it, and bring it up) > > What is currently occurring when I start dovecot is: > Error: service(pop3-login): listen(::, 110) failed: Address already in use > Error: service(pop3-login): listen(::, 995) failed: Address already in use > Error: service(imap-login): listen(::, 143) failed: Address already in use > Error: service(imap-login): listen(::, 993) failed: Address already in use > Fatal: Failed to start listeners Edit your dovecot.conf around line 26. By default listen is set to '*, ::' If your host doesn't have IPv6 enabled use: listen = * Regards, Pascal -- The trapper recommends today: beeffeed.1206921 at localdomain.org From sca at andreasschulze.de Fri Mar 9 23:40:16 2012 From: sca at andreasschulze.de (Andreas Schulze) Date: Fri, 9 Mar 2012 22:40:16 +0100 Subject: [Dovecot] sieve and utf-7 foldernames Message-ID: <20120309214016.GA5584@doran.andreasschulze.de> Hi all, since many dovecot/pigeonhole versions I have an error: Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) setup: dovecot-2.1.1 / pigeonhole-0.3.0 postfix deliver to dovecot-lda .dovecot.sieve contains this: require ["fileinto","reject","vacation","relational","comparator-i;ascii-numeric","regex"]; if header :contains "To" "green at example.org" { fileinto "INBOX.gr&APw-n"; stop; } a mail to the mentioned address produces this logging: Mar 9 22:23:10 test dovecot: lda(foo): save: box=INBOX.gr&APw-n, uid=2, msgid=<4711 at example.org>, size=4642 also a mail_debug=yes log no more lines related to the foldername but at the end I found the mail not in the expected folder. It places in "INBOX.gr&-APw-n" notice the extra minus after & I think it's pigeonholes fault but I may be wrong ... # doveconf -n # 2.1.1: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686 i686 Debian 5.0.9 ext3 ... lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = 127.0.0.1, ::1 protocols = " imap sieve" protocol lda { mail_plugins = quota notify mail_log sieve } ... Andreas From user+dovecot at localhost.localdomain.org Sat Mar 10 02:13:43 2012 From: user+dovecot at localhost.localdomain.org (Pascal Volk) Date: Sat, 10 Mar 2012 01:13:43 +0100 Subject: [Dovecot] sieve and utf-7 foldernames In-Reply-To: <20120309214016.GA5584@doran.andreasschulze.de> References: <20120309214016.GA5584@doran.andreasschulze.de> Message-ID: <4F5A9CB7.1040804@localhost.localdomain.org> On 03/09/2012 10:40 PM Andreas Schulze wrote: > Hi all, > > since many dovecot/pigeonhole versions I have an error: > Mails are delivered into wrong folders if the foldername contain a german umlaut. ( ?, ?, ? ) > ? Behaves as documented - since Dovecot 1.2.0: Wiki > Upgrading > v1.1 to v1.2 > Sieve: * You should consider migrating from CMU Sieve to Dovecot Sieve (see the link for instructions) http://wiki.dovecot.org/LDA/Sieve/Dovecot#Migration_from_CMUSieve: * Be sure to use UTF8 for the mailbox argument of the fileinto command. Older CMUSieve installations used modified UTF7 (as IMAP does) for the mailbox parameter. If not adjusted, the new Sieve plugin will use the wrong folder name for storing the message. Regards, Pascal -- The trapper recommends today: cafefeed.1207001 at localdomain.org From tss at iki.fi Sat Mar 10 18:01:27 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:01:27 +0200 Subject: [Dovecot] Advise on upgrading from a jurassic version - please help. In-Reply-To: References: Message-ID: On 9.3.2012, at 2.50, Martin Mielke wrote: > I have read the Dovecot documentation and there are instructions to upgrade from 0.99.x to 1.x and so on... my question is: can I upgrade from 0.99.11 to 2.x directly or is it a massive leap? If so, what do I have to keep in mind? This is a production system so I should not break anything... or at least have a rollback plan... http://wiki2.dovecot.org/Upgrading/1.0 points out a few things: - rename .subscriptions -> subscriptions - rename .customflags -> dovecot-keywords - default pop3 UIDL format changed From tss at iki.fi Sat Mar 10 18:02:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:02:40 +0200 Subject: [Dovecot] Whether the THREAD command support cross-mailbox thread? In-Reply-To: References: Message-ID: <4956B88C-3B55-460C-AAD2-E8253DA8627B@iki.fi> On 9.3.2012, at 10.09, Andy YB Hu wrote: > > In the real world, the mails which belongs to one thread could be dispersed > in different mailbox, at least Inbox and SENT, so whether the command can > search different mailboxes and grap them in one talk with dovecot? > > If Not, any other approach to do that? Create a virtual mailbox containing all mails. Then you see all the mails within a thread. http://wiki2.dovecot.org/Plugins/Virtual > BTW, what the THREAD=REFS stands for? In the RFC, > http://tools.ietf.org/html/rfc5256, i didn't find this algorithm. It's from http://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05 which never made it to an actual RFC. From tss at iki.fi Sat Mar 10 18:04:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:04:44 +0200 Subject: [Dovecot] migrating/converting from system users -> virtual users In-Reply-To: References: Message-ID: <79C53550-348C-4812-AAA1-2C3D6D9F59B4@iki.fi> On 8.3.2012, at 20.46, Steve Platt wrote: >> use Dovecot v2.0's dsync > > I gather from your reply that it's OK to use Dovecot 2.0 utilities (eg dsync) > on a dovecot (v1) installation; presumably with its own configuration file(s). Yes, although in some situations it might write stuff to index files that v1.x complains about. But deleting index files afterwards fixes that. >> You could set mail_drop_priv_before_exec=yes ... chgrp vmail ... > > Yes, I think we could do that; I should have thought of it myself, thanks > again. > > I think there was one other problem with the automatic conversion which I've > now remembered: I note that the first time a user connects to th eimap service > dovecot creates their (virtual) home directory for them with all the right > permissions. That's great and I use the existence of that directory as an > indication to our MTA that the user wants delivery into the dovecot store > rather than their old system mailbox. However once I tried using the convert > plugin the process fails because (it seems) the conversion tries to take place > before the home directory has been created. > > Is there any configuration change that might change this order? No. > Can I configure the convert plugin on LDA delivery, for example, instead of as > part of the "protocol imap" section? Yes. From tss at iki.fi Sat Mar 10 18:06:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:06:20 +0200 Subject: [Dovecot] dot named folders In-Reply-To: <87aa3s2o3u.fsf@algae.riseup.net> References: <87aa3s2o3u.fsf@algae.riseup.net> Message-ID: <729DCBA4-E353-41EA-903D-0DDF897E5208@iki.fi> On 7.3.2012, at 21.43, Micah Anderson wrote: > When a user makes a folder called 'x.y' it actually creates a folder > called 'x' with a folder called 'y' inside, rather than a folder called > 'x.y'. I'm guessing this has to do with an internal folder separator > namespace configuration, but I'm a bit confused by how this works. > > I'm using 2.0.15 with mdbox and this is what I have configured for my > namespaces: > > namespace { > separator = . > prefix = > inbox = yes > } Keep this. > namespace { > separator = . > prefix = INBOX. > inbox = no > hidden = yes > list = no > } > > I migrated from courier maildirs, so perhaps I no longer need some of > these now that the conversion is finished? It depends on if you have any users whose clients are using INBOX. namespace. If there are, and you remove it, the users won't see anything except INBOX anymore. From tss at iki.fi Sat Mar 10 18:11:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:11:55 +0200 Subject: [Dovecot] Single instance storage In-Reply-To: References: Message-ID: <508532C3-B8D0-47E8-9566-6A570A3233F3@iki.fi> On 8.3.2012, at 3.30, Jean-Daniel Beaubien wrote: > I have read most of the doc on the dovecot website, and couldn't find any > info on the single instance storage feature, so I'm posting my questions > here. > > - Are these 3 parameters the only one necessary for single instance > storage? I cannot find any doc on this feature on the website; is there > anything specific I need to know about them? (the last one isn't exactly > self-explanatory). > - mail_attachment_dir = /srv/vmail/attachments > - mail_attachment_hash = %{sha256} > - mail_cache_min_mail_count = 2 > > - Is this feature ready for production? mail_cache_min_mail_count isn't related to single instance storage at all. I didn't really even remember that such a setting existed. I'm not sure if it's actually useful in any setups.. Maybe you were thinking about mail_attachment_min_size? Other than that, yeah, the mail_attachment_dir is really the only thing you need to set to enable SIS. From tss at iki.fi Sat Mar 10 18:13:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:13:39 +0200 Subject: [Dovecot] Fscking warnings In-Reply-To: <201203070937.06545.sdavies@sdc.com.au> References: <201203070937.06545.sdavies@sdc.com.au> Message-ID: On 7.3.2012, at 1.07, Stephen Davies wrote: > Google tells me that these "should go away" but they don't. > > Seems to happen continuously while a user is viewing email. > > Mar 7 09:29:52 server dovecot: imap(john): Warning: fscking index file > /home/john/Mail/INBOX/.imap/Archive/dovecot.index What Dovecot version? Anyway, something wrong in the mbox I guess. Just do rm -rf /home/john/Mail/INBOX/.imap/ From tss at iki.fi Sat Mar 10 18:20:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:20:33 +0200 Subject: [Dovecot] POP3C storage backend In-Reply-To: <4F5538DC.4060802@talpey.com> References: <4F5538DC.4060802@talpey.com> Message-ID: <7B049F23-AE14-4BCE-857C-91D70E02A7E8@iki.fi> On 6.3.2012, at 0.06, Tom Talpey wrote: > I see a new "POP3C" lib-storage client backend in dovecot 2.1, but I > don't see anything in the 2.1 doc directory or in the wiki. Can this > be used to synchronize dovecot with external pop servers? Doing away > with my current fetchmail and lmtp solution for this would be quite > interesting. > > Thanks for any pointers to configuring and using this, if so... It could possibly be used to do that with http://wiki2.dovecot.org/Plugins/Snarf Although that would probably connect to POP3 server quite often. And when IDLEing it wouldn't see new POP3 mails (that would need a small change to snarf plugin). From tss at iki.fi Sat Mar 10 18:44:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:44:21 +0200 Subject: [Dovecot] dovecot Digest, Vol 107, Issue 20 Fscking warnings In-Reply-To: <201203091030.20828.sdavies@sdc.com.au> References: <201203091030.20828.sdavies@sdc.com.au> Message-ID: <7F823A5F-36DC-4444-A13E-3182FA243EE1@iki.fi> On 9.3.2012, at 2.00, Stephen Davies wrote: > However, it seems that deleting the .imap files did not fix the log sync errors > or the fscking warnings. > > Both are still happening continuously. If you're talking about errors like these: Mar 10 18:21:38 imap(tss): Error: Log synchronization error at seq=1,offset=26896 for /home/tss/mail/.imap/INBOX/dovecot.index: Extension header update points outside header size and the following fsck error, then deleting all of the .imap directories should get rid of them (maybe you didn't delete all of them? note that each subdirectory has its own, so there's more than just ~/mail/.imap/). Anyway, this is now also fixed: http://hg.dovecot.org/dovecot-2.0/rev/dc88712581c9 http://hg.dovecot.org/dovecot-2.1/rev/1289b79241bb From tss at iki.fi Sat Mar 10 18:50:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:50:15 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F576370.8040706@Media-Brokers.com> References: <4F576370.8040706@Media-Brokers.com> Message-ID: <7F351B29-16BF-4098-8C3A-3FC84D85ADD4@iki.fi> On 7.3.2012, at 15.32, Charles Marcus wrote: > 4. They can *move* messages to other folders in that account (ie, 'file' them), and last > > (this is the tricky part) > > 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. There is unfortunately no "default ACL" feature currently. Although you could somewhat easily add an ugly hack to the code for that. And I guess it wouldn't be difficult to implement it, maybe by reading it from $mail_root/dovecot-acl-default file or something.. So without code changes you could: - create all of the necessary folders - set such ACLs that user can't create any more folders - disallow expunging in all folders From tss at iki.fi Sat Mar 10 18:51:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:51:16 +0200 Subject: [Dovecot] Lock down Shared Mail Accounts? In-Reply-To: <4F578F31.3000303@wildgooses.com> References: <4F576370.8040706@Media-Brokers.com> <4F578F31.3000303@wildgooses.com> Message-ID: <75B1D406-190B-4824-B575-E1AF1F76B207@iki.fi> On 7.3.2012, at 18.39, Ed W wrote: >> 5. No one other than a designated user or users (Master User(s)? Users in a specified Group?) can delete any messages in this account, in any of the folders. > > Have them delivered with only read permissions on the physical files? (Bet that doesn't work very well in practice or other than maildir...) The maildir file's read permission doesn't matter, the parent cur/ or new/ directory's write permission matters. And removing those prevents moving mails from new/ to cur/ and from keeping the flag states in the filename.. Not very good. From tss at iki.fi Sat Mar 10 18:53:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:53:38 +0200 Subject: [Dovecot] nfs error fcntl(read-lock) locking failed for file In-Reply-To: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> References: <.120.61.8.40.1331086943.squirrel@www.24x7server.net> Message-ID: <35158279-4A11-4806-A481-548314E27282@iki.fi> On 7.3.2012, at 4.22, Rajesh M wrote: > on server number 1 i get errors as such > Error: fcntl(read-lock) locking failed for file Input/output error > squirrelmail gives error imap connection closed and i am not able to login > > > so i set the parameters as such in the dovecot conf file and the error > stopped > > mmap_disable=yes > dotlock_use_excl = yes > lock_method = dotlock > > can somebody please advise me if the above is correct ? That should work. > or is it preferred to use fcntl with lockd That would probably be more efficient. > (note that my mailbox is maildir format) The fcntl locking is used for Dovecot index files, not for maildir files. From tss at iki.fi Sat Mar 10 18:56:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 10 Mar 2012 18:56:50 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: References: Message-ID: On 7.3.2012, at 5.19, Jean-Daniel Beaubien wrote: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." > > How will the mailbox files (m.X) files be modified when I move or delete > emails using mdbox+gzip. Will the resulting gzipped mdbox files be > rsync-able or will they need a full re-upload? > > If I plan on using rsync for backups, am I better off not using the gzip > feature (if i can spare the extra storage)??? gzipping is irrelevant, the behavior is the same with and without gzip. The purging step recreates new mail files, so the new files will need to be fully uploaded with rsync. You might want to consider using dsync instead. From btb at bitrate.net Mon Mar 12 06:00:11 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Mon, 12 Mar 2012 00:00:11 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections Message-ID: hi- i have a configuration in which i'm using different passdb/userdb settings for each of imap, lmtp and smtp [without getting too far off on a tangent, this is so ldap group membership can be used to independently control authorization for receiving [lmtp], retrieving [imap], and sending [smtp/postfix] of email. each passdb/userdb uses a different search filter.] when i use this as my config: # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-15-generic-pae i686 Ubuntu 11.10 auth_debug = yes first_valid_gid = 2000 first_valid_uid = 2000 log_timestamp = "%d.%m.%Y %H.%M.%S " login_greeting = dovecot ready mail_debug = yes protocols = " imap lmtp" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service lmtp { inet_listener lmtp { address = 127.0.0.1 ::1 port = 10026 } } ssl_cert = Hello We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib - Does anybody know of any other project to create an easy-restore for Dovecot? Regards Maria From wouter at vdschagt.com Mon Mar 12 14:09:51 2012 From: wouter at vdschagt.com (Wouter van der Schagt) Date: Mon, 12 Mar 2012 13:09:51 +0100 Subject: [Dovecot] Dovecot LDA breaking .qmail forwarding? Message-ID: <00d201cd0049$07274880$1575d980$@vdschagt.com> Good morning all, I've a problem, I'm using the Dovecot LDA in my .qmail file: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER However when I add a forward, such as: |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $EXT@$USER &wouter at vdschagt.com The mail isn't forwarded and stays in the queue on the mail server, the same is the case when I reverse the lines. The error in the log file is: @400000004f5de5f7033bc434 delivery 824026: deferral: qmail-inject:_fatal:_qq_trouble_creating_files_in_queue_(#4.3.0)/system_erro r_calling_qmail-inject/ When using another LDA, for example when specifying the Maildir, it works, but then i cannot use Sieve scripts. Any ideas? Am I doing anything wrong? Sincerely, - Wouter van der Schagt From bind at enas.net Mon Mar 12 15:02:33 2012 From: bind at enas.net (Urban Loesch) Date: Mon, 12 Mar 2012 14:02:33 +0100 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <4F5DF3E9.9030703@enas.net> Hi, perhaps the mail_log plugin is what you need. Regards Urban On 12.03.2012 12:56, Maria Arrea wrote: > Hello > > We are working in a web based restore system for our Dovecot users. In this web form a user must log-in and after successful login can estore a deleted folder from date X. We will release it under the GPL. I have a couple of questions: > > - Is there any way of Dovecot logging to write when a folder is deleted or created? We do not want to increase too much our "normal" logging level. We use Dovecot 2.0.18+mdbox+zlib > - Does anybody know of any other project to create an easy-restore for Dovecot? > > Regards > > Maria > From tss at iki.fi Mon Mar 12 15:04:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:04:10 +0200 Subject: [Dovecot] Question about folder creation/delete and logging In-Reply-To: <20120312115614.134760@gmx.com> References: <20120312115614.134760@gmx.com> Message-ID: <1331557450.2081.112.camel@innu> On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From giles at coochey.net Mon Mar 12 15:09:52 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:09:52 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) Message-ID: <4F5DF5A0.9000609@coochey.net> Hi, I'm looking for a quick tool that can connect to my IMAP account grab all the messages in a particular folder and dump them to a mbox format file? Anyone know a quick easy tool to do that? This is a spam folder that I'd like to do some Bayes spam learning on, but since I've migrated to mdbox I don't think I can do this directly on the mailbox. Any thoughts appreciated. -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From maria_arrea at gmx.com Mon Mar 12 15:18:01 2012 From: maria_arrea at gmx.com (Maria Arrea) Date: Mon, 12 Mar 2012 14:18:01 +0100 Subject: [Dovecot] Question about folder creation/delete and logging Message-ID: <20120312131801.134760@gmx.com> mail_log plugin is just what we need, thank you for your support. Yes, we are using bacula+doveadm import for this project. If you are interested, we have in production an home-brew message-tracking system for our end-users. We parse qmail / postfix / clamav / spamassassin/ dovecot / sieve logs and insert them in a mysql database, and a logged user in our webapp can see what happened with her mails (sent mails or messages waiting for arrival). Affero GPL software, of course. Some screenshots (Spanish only, sorry) here: https://gestionproyectos.us.es/attachments/download/321/Seguimiento_de_mensajes___Vista_detalla_de_mensaje_1331558163319.png https://gestionproyectos.us.es/attachments/download/145/Captura_de_pantalla_2011-06-05_a_las_14.03.42.png https://gestionproyectos.us.es/attachments/download/156/Seguimiento_de_mensajes___Administraci%C3%B3n_1308042340487.png URL of the project (Seguimiento, spanish word for "tracking"): https://gestionproyectos.us.es/projects/seguimiento Regards Maria ----- Original Message ----- From: Timo Sirainen Sent: 03/12/12 02:04 PM To: Maria Arrea Subject: Re: [Dovecot] Question about folder creation/delete and logging On Mon, 2012-03-12 at 12:56 +0100, Maria Arrea wrote: > > - Is there any way of Dovecot logging to write when a folder is > deleted or created? We do not want to increase too much our "normal" > logging level. We use Dovecot 2.0.18+mdbox+zlib You can configure mail_log plugin to only log mailbox creations and deletions. http://wiki2.dovecot.org/Plugins/MailLog > - Does anybody know of any other project to create an easy-restore > for Dovecot? I guess you're using "doveadm import"? So other than that, haven't heard of any. From tss at iki.fi Mon Mar 12 15:19:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:19:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DF5A0.9000609@coochey.net> References: <4F5DF5A0.9000609@coochey.net> Message-ID: <1331558383.2081.114.camel@innu> On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: > Hi, > > I'm looking for a quick tool that can connect to my IMAP account grab > all the messages in a particular folder and dump them to a mbox format file? > > Anyone know a quick easy tool to do that? > > This is a spam folder that I'd like to do some Bayes spam learning on, > but since I've migrated to mdbox I don't think I can do this directly on > the mailbox. > > Any thoughts appreciated. With v2.0: dsync -m spam backup mbox:~/mbox-mails/ There are also doveadm move and doveadm import commands that can do this. From giles at coochey.net Mon Mar 12 15:37:43 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:37:43 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331558383.2081.114.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> Message-ID: <4F5DFC27.4020004@coochey.net> On 12/03/2012 13:19, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:09 +0000, Giles Coochey wrote: >> Hi, >> >> I'm looking for a quick tool that can connect to my IMAP account grab >> all the messages in a particular folder and dump them to a mbox format file? >> >> Anyone know a quick easy tool to do that? >> >> This is a spam folder that I'd like to do some Bayes spam learning on, >> but since I've migrated to mdbox I don't think I can do this directly on >> the mailbox. >> >> Any thoughts appreciated. > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > There are also doveadm move and doveadm import commands that can do > this. > > OK, That would do a mailbox called spam no? I have a normal user who has an IMAP folder called 'Junk E-mail' Is there a similar syntax? -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From rob0 at gmx.co.uk Mon Mar 12 15:38:49 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 12 Mar 2012 08:38:49 -0500 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: References: Message-ID: <20120312133849.GS24983@harrier.slackbuilds.org> On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: > the problem with this is that while each of the passdb/userdb > configs for the various protocols does indeed work, if a result > is not found in one of them, the global passdb appears to then > function as a catch-all. > > how can i tell dovecot it doesn't need a global passdb? each > of the protocols' passdb/userdb configs is functioning as > desired, but having dovecot look elsewhere upon failure ends > up defeating the purpose. A simple workaround: use an empty passwd-file passdb as global. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From tss at iki.fi Mon Mar 12 15:39:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 15:39:43 +0200 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <4F5DFC27.4020004@coochey.net> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> Message-ID: <1331559583.2081.115.camel@innu> On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: > >> Any thoughts appreciated. > > With v2.0: dsync -m spam backup mbox:~/mbox-mails/ > > > > There are also doveadm move and doveadm import commands that can do > > this. > > > > > OK, That would do a mailbox called spam no? > I have a normal user who has an IMAP folder called 'Junk E-mail' > > Is there a similar syntax? Yeah, the -m parameter just specifies the mailbox name. So: dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ If different accounts have different names, then I guess you'll just have to run dsync twice with each name. From giles at coochey.net Mon Mar 12 15:42:10 2012 From: giles at coochey.net (Giles Coochey) Date: Mon, 12 Mar 2012 13:42:10 +0000 Subject: [Dovecot] Extracting mbox format from Dovecot IMAP (mdbox) In-Reply-To: <1331559583.2081.115.camel@innu> References: <4F5DF5A0.9000609@coochey.net> <1331558383.2081.114.camel@innu> <4F5DFC27.4020004@coochey.net> <1331559583.2081.115.camel@innu> Message-ID: <4F5DFD32.3010609@coochey.net> On 12/03/2012 13:39, Timo Sirainen wrote: > On Mon, 2012-03-12 at 13:37 +0000, Giles Coochey wrote: >>>> Any thoughts appreciated. >>> With v2.0: dsync -m spam backup mbox:~/mbox-mails/ >>> >>> There are also doveadm move and doveadm import commands that can do >>> this. >>> >>> >> OK, That would do a mailbox called spam no? >> I have a normal user who has an IMAP folder called 'Junk E-mail' >> >> Is there a similar syntax? > Yeah, the -m parameter just specifies the mailbox name. So: > > dsync -m 'Junk E-mail' backup mbox:~/mbox-mails/ > > If different accounts have different names, then I guess you'll just > have to run dsync twice with each name. > > Thanks - was experimenting and got it. Works a treat! -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey From micah at riseup.net Mon Mar 12 17:05:06 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:05:06 -0400 Subject: [Dovecot] dot named folders References: <87aa3s2o3u.fsf@algae.riseup.net> <4F57C7F5.4030803@es2eng.com> <87ty1zys5c.fsf@algae.riseup.net> <4F58DF31.3040203@schetterer.org> Message-ID: <87k42pyi59.fsf@algae.riseup.net> Robert Schetterer writes: > Am 08.03.2012 17:27, schrieb Micah Anderson: >> Willie Gillespie writes: >> >>> On 03/07/2012 12:43 PM, Micah Anderson wrote: >>>> >>>> When a user makes a folder called 'x.y' it actually creates a folder >>>> called 'x' with a folder called 'y' inside, rather than a folder called >>>> 'x.y'. I'm guessing this has to do with an internal folder separator >>>> namespace configuration, but I'm a bit confused by how this works. >>> >>> Correct. >>> Similar to how in Linux, I could create a folder >>> mkdir test1/test2 >>> It will create test2 inside of test1. >>> >>> The difference being that IMAP doesn't necessarily need the parent mailbox to >>> exist, where Linux would throw an error if test1/ didn't exist first. >>> >>> So basically, as far as I know, you can't have a folder with a "." in the name >>> with the namespaces you have set up. >> >> That makes sense, however I'm not sure that I need these namespaces any >> longer if I no longer am using the maildir format (mdbox). >> >> In either case, it seems like the internal folder separator should not >> be exposed to the user like this. What is happening now is the user gets >> something other than they expect (a folder within a folder, instead of a >> folder with a dot in the name) because of some unknown internal >> configuration. >> >> If moving to mdbox is not enough to remove these namespace >> configurations that cause this, then it would be good if the user was >> unable to create such a folder, because it was prohibited, rather than >> creating something other than they expect. >> >> micah >> > > http://wiki.dovecot.org/Plugins/Listescape > may help Interesting, thanks for the pointer, although I think I prefer if users are just prohibited from making a 'folder.withadot' and told that it is prohibited right away, rather than giving them a way to do it. micah -- From micah at riseup.net Mon Mar 12 17:10:46 2012 From: micah at riseup.net (Micah Anderson) Date: Mon, 12 Mar 2012 11:10:46 -0400 Subject: [Dovecot] mdbox + gzip and rsync References: Message-ID: <87fwddyhvt.fsf@algae.riseup.net> Jean-Daniel Beaubien writes: > After reading the following paragraph from the dovecot doc, I've been > wondering how it would affect rsync (when combined with gzip): > > "Expunging a message only decreases the message's refcount. The space is > later freed in "purge" step. This is typically done in a nightly cronjob > when there's less disk I/O activity. The purging first finds all files that > have refcount=0 mails. Then it goes through each file and copies the > refcount>0 mails to other mdbox files (to the same files as where newly > saved messages would also go), updates the map index and finally deletes > the original file. So there is never any overwriting or file truncation." Interesting, so it would be recommended to those using mdbox format to run a 'dovadm purge -A' every night to clean up these unused files? It seems like without this, mail storage usage will just grow infinitely. It does appear that using an rsync backup process for mdbox would not be able to detect this and backups would also grow infinitely. micah -- From tss at iki.fi Mon Mar 12 17:46:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 12 Mar 2012 17:46:51 +0200 Subject: [Dovecot] mdbox + gzip and rsync In-Reply-To: <87fwddyhvt.fsf@algae.riseup.net> References: <87fwddyhvt.fsf@algae.riseup.net> Message-ID: <1A1E352C-2A32-4C90-9357-A35C92D98875@iki.fi> On 12.3.2012, at 17.10, Micah Anderson wrote: > Jean-Daniel Beaubien writes: > >> After reading the following paragraph from the dovecot doc, I've been >> wondering how it would affect rsync (when combined with gzip): >> >> "Expunging a message only decreases the message's refcount. The space is >> later freed in "purge" step. This is typically done in a nightly cronjob >> when there's less disk I/O activity. The purging first finds all files that >> have refcount=0 mails. Then it goes through each file and copies the >> refcount>0 mails to other mdbox files (to the same files as where newly >> saved messages would also go), updates the map index and finally deletes >> the original file. So there is never any overwriting or file truncation." > > Interesting, so it would be recommended to those using mdbox format to > run a 'dovadm purge -A' every night to clean up these unused files? It > seems like without this, mail storage usage will just grow infinitely. Yes. > It does appear that using an rsync backup process for mdbox would not be > able to detect this and backups would also grow infinitely. rsync --delete would delete the old files, right? Anyway, I'd avoid using rsync for mdbox unless you're doing it on a filesystem snapshot. dsync backup should work better. From rtroy at ScienceTools.com Mon Mar 12 19:14:09 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) Subject: [Dovecot] Trouble adding sasl support via dovecot Message-ID: Hello Folks, I've been the admin of a site that uses Postfix with Dovecot on RedHat since, oh, gosh, maybe 1996? It's been a long time. I've never built it from source, though, just used the rpms (and I wonder if maybe that's my problem now). It just works, is reliable, and lets me be a very-part-time administrator. Repeatedly over the last few years I've been asked to have our mail system "join the modern age" and provide mail sending capabilities for clients that aren't on our internal network - via their smart-phones, from home, etc. OK... Well, way back when the site was set up, smtp servers didn't do any kind of "auth", but along the way to solving this problem (trying to configure pop-before-smtp, someone mentioned that Postfix now has an auth mechanism that uses Dovecot and I should use that instead! Great! ... Except that I've spent between 16 and 20 hours on this with no joy, and while I hate having to ask for help, it's time to ask what things that are obvious to the less ignorant that I must be doing wrong... Certainly, given the solid history of Postfix and Dovecot, I must be the problem! My problem statement is simply, "it should be working", but doesn't, and I don't get any announcement of "auth" when testing connections to Postfix as per directions here: http://www.postfix.org/SASL_README.html#server_test At least I haven't broken the normal functionality! I'm building a new server on the latest Fedora Core (16), but it's lacking in some hardware and won't be ready for a while, so I'm working with FC 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth scheme (which works fine and I'd hate to change it if I don't have to). The system has been up and functional on these versions for a couple of years, and quite stable, we just can't send if we're not local. When I do "postconf-a" it indicates cyrus and dovecot, so I take it that means Postfix has been built with sasl support. (I presume this means I don't have to compile it from source.) First Dovecot. Its set up to provide all protocols, but only imaps and pop3s have ports forwarded through the firewall. Plain-text auth is disabled, ssl is set to yes, ssl_listen is not specified, and the cert and key files are in the default locations - and work. No cipher list is used. Dovecot's chrooted. The protocol sections imap and pop3 take ALL the defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as I don't think it matters. "auth default {" has mechanisms set to cram-md5, digest-md5, plain, and login, with passdb passwd-file pointing to a file in /etc where the cram data goes. It's not using pam, and there's an OLD comment in the config: # Experience says we need an empty passdb - passwd group: which is followed by passdb passwd{}. Later, there's "userdb passwd {}. All of that was configured long ago and has been functional. The changes I've made to add sasl support primarily pertain to the "socket listen section of "auth default". There, the master section remains commented out while the client section has been uncommented, the path set to /var/spool/postfix/private/auth, mode set to 0660, and the user and group have been set to postfix. ...This is all as described here: http://www.postfix.org/SASL_README.html and http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL That's it for Dovecot. Now, to Postfix itself. >From the working environ, only listening on port 25, I simply added the following (as per directions already cited above): smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_security_options = noanonymous, noplaintext smtp_sasl_security_options = noanonymous, noplaintext smtpd_sasl_tls_security_options = noanonymous smtp_sasl_tls_security_options = noanonymous And, of course, permit_sasl_authenticated was added to smtpd_recipient_restrictions. I got the impression from the baove sources that Postfix will then use Dovecot's authentication mechanism via a socket it finds in its private/auth subdirectory. NOT documented in any of those places, someone suggested I must turn on TLS. OK... The documentation found here: http://www.postfix.org/TLS_README.html claims (intimates) that it's not possible to run a site on a self-signed certificate, however, there's ZERO budget for a signed certificate, so unless I can get one for ten bucks somewhere, that could be a deal-breaker here. However, we've been using self-signed certificates for a while now and wonder why an "exception" mechanism wouldn't exist. As that web page talks about "Netscape" I suspect it's very old and may no longer apply. In any event, I tried this, too (after trying without). On the good side, an available Android phone, previously reading fine, but unable to send, no longer complained when the setup was changed to the imap username and password, same server address, TLS security type, and the server port of 25. HOWEVER, no mail has passed through it successfully, it just gives no error whatsoever, so far, while the server's log reports "Relay access denied." Notably, when setting up TLS, Postfix complained when the smtpd_tls_key_file was incorrect, but did not complain when it was provided properly, suggesting it's reading and accepting my self-signed certificate and private key. Ideas, please?! And, by the way, what's port 465 all about? Some clients propose that's what should be used to send... Thanks in advance for your help, Richard From rtroy at ScienceTools.com Mon Mar 12 20:59:01 2012 From: rtroy at ScienceTools.com (Richard Troy) Date: Mon, 12 Mar 2012 11:59:01 -0700 (PDT) Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: Message-ID: Hi All, it turned out to be the order of entries in stmpd_recipient_restrictions. Regards, Richard On Mon, 12 Mar 2012, Richard Troy wrote: > Date: Mon, 12 Mar 2012 10:14:09 -0700 (PDT) > From: Richard Troy > To: postfix-users at cloud9.net, dovecot at dovecot.org > Subject: [Dovecot] Trouble adding sasl support via dovecot > > > Hello Folks, > > I've been the admin of a site that uses Postfix with Dovecot on RedHat > since, oh, gosh, maybe 1996? It's been a long time. I've never built it > from source, though, just used the rpms (and I wonder if maybe that's my > problem now). It just works, is reliable, and lets me be a very-part-time > administrator. > > Repeatedly over the last few years I've been asked to have our mail system > "join the modern age" and provide mail sending capabilities for clients > that aren't on our internal network - via their smart-phones, from home, > etc. OK... Well, way back when the site was set up, smtp servers didn't do > any kind of "auth", but along the way to solving this problem (trying to > configure pop-before-smtp, someone mentioned that Postfix now has an auth > mechanism that uses Dovecot and I should use that instead! Great! ... > Except that I've spent between 16 and 20 hours on this with no joy, and > while I hate having to ask for help, it's time to ask what things that are > obvious to the less ignorant that I must be doing wrong... Certainly, > given the solid history of Postfix and Dovecot, I must be the problem! > > My problem statement is simply, "it should be working", but doesn't, and I > don't get any announcement of "auth" when testing connections to Postfix > as per directions here: > > http://www.postfix.org/SASL_README.html#server_test > > At least I haven't broken the normal functionality! > > I'm building a new server on the latest Fedora Core (16), but it's lacking > in some hardware and won't be ready for a while, so I'm working with FC > 14, running Postfix 2.5.6, and Dovecot 1.2.8. It uses the "cram-md5" auth > scheme (which works fine and I'd hate to change it if I don't have to). > The system has been up and functional on these versions for a couple of > years, and quite stable, we just can't send if we're not local. > > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) > > First Dovecot. Its set up to provide all protocols, but only imaps and > pop3s have ports forwarded through the firewall. Plain-text auth is > disabled, ssl is set to yes, ssl_listen is not specified, and the cert and > key files are in the default locations - and work. No cipher list is used. > Dovecot's chrooted. The protocol sections imap and pop3 take ALL the > defaults, as does lda (I've ignored sendmail_path = /usr/lib/sendmail) as > I don't think it matters. "auth default {" has mechanisms set to cram-md5, > digest-md5, plain, and login, with passdb passwd-file pointing to a file > in /etc where the cram data goes. It's not using pam, and there's an OLD > comment in the config: > > # Experience says we need an empty passdb - passwd group: > > which is followed by passdb passwd{}. Later, there's "userdb passwd {}. > > All of that was configured long ago and has been functional. > > The changes I've made to add sasl support primarily pertain to the "socket > listen section of "auth default". There, the master section remains > commented out while the client section has been uncommented, the path set > to /var/spool/postfix/private/auth, mode set to 0660, and the user and > group have been set to postfix. ...This is all as described here: > > http://www.postfix.org/SASL_README.html > and > http://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL > > That's it for Dovecot. Now, to Postfix itself. > > >From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): > > smtpd_sasl_type = dovecot > smtpd_sasl_path = private/auth > smtpd_sasl_auth_enable = yes > broken_sasl_auth_clients = yes > smtpd_sasl_security_options = noanonymous, noplaintext > smtp_sasl_security_options = noanonymous, noplaintext > smtpd_sasl_tls_security_options = noanonymous > smtp_sasl_tls_security_options = noanonymous > > And, of course, permit_sasl_authenticated was added to > smtpd_recipient_restrictions. > > > I got the impression from the baove sources that Postfix will then use > Dovecot's authentication mechanism via a socket it finds in its > private/auth subdirectory. > > NOT documented in any of those places, someone suggested I must turn on > TLS. OK... > > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, however, there's ZERO budget for a signed certificate, so > unless I can get one for ten bucks somewhere, that could be a > deal-breaker here. However, we've been using self-signed certificates for > a while now and wonder why an "exception" mechanism wouldn't exist. As > that web page talks about "Netscape" I suspect it's very old and may no > longer apply. > > In any event, I tried this, too (after trying without). On the good side, > an available Android phone, previously reading fine, but unable to send, > no longer complained when the setup was changed to the imap username and > password, same server address, TLS security type, and the server port of > 25. HOWEVER, no mail has passed through it successfully, it just gives no > error whatsoever, so far, while the server's log reports "Relay access > denied." > > Notably, when setting up TLS, Postfix complained when the > smtpd_tls_key_file was incorrect, but did not complain when it was > provided properly, suggesting it's reading and accepting my self-signed > certificate and private key. > > Ideas, please?! > > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... > > > Thanks in advance for your help, > Richard > > -- Richard Troy, Chief Scientist Science Tools Corporation 510-717-6942 rtroy at ScienceTools.com, http://ScienceTools.com/ From terry at cnysupport.com Mon Mar 12 20:57:24 2012 From: terry at cnysupport.com (Terry Carmen) Date: Mon, 12 Mar 2012 14:57:24 -0400 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> Message-ID: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> On 03/04/2012 09:58 AM, Timo Sirainen wrote: > On 4.3.2012, at 16.48, Terry Carmen wrote: > >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%49.100$.example.com >>> >>> If the prefix differs, but all of the exchange server names have >>> the same length, for example 10, you can also do: >>> >>> pass_attrs = ..., \ >>> msExchHomeServerName=userdb_imapc_host=%-10$.example.com >>> There's no otherwise nice way to parse this string. >> >> >> If by prefix, you mean the >> "/O=example/OU=INT/cn=Configuration/cn=Servers/" part, then, yes, >> they're different. > > OK, so if the prefix or suffix isn't always the same length you > can't do the above. > >> I could export the data to a text file as >> username:homeexchangeserver (or whatever other format is needed). >> >> homeservers.txt: >> user1:exch1.example.com >> user2:exch1.example.com >> user3:exch1.example.com >> user4:exch2.example.com >> >> Is it possible to do a lookup in a text file to get this? > > > If you can use userdb passwd-file and export the data to that file, > it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile > > Example line: > > user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com > > Note that you can't then return any userdb fields from passdb ldap lookup. That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. What would be perfect is if I could do something like this: //////////////////////////// http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying Proxy only server . . . In this document I assume that Dovecot is installed under /opt/dovecot, by default it is installed under /usr/local when compiling from source. Examples in this document are for MySQL but configs do not differ much with PostgreSQL. SQL table structure Create SQL table like CREATE TABLE proxy ( user varchar(255) NOT NULL, host varchar(16) default NULL, destuser varchar(255) default NULL, PRIMARY KEY (user) ); ////////////////////////////////////// All I really need is a way to lookup the user's home IMAP server when given the username, as above. Does imapproxy still support this 1.x feature? Thanks! Terry From CMarcus at Media-Brokers.com Mon Mar 12 21:02:55 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Mon, 12 Mar 2012 15:02:55 -0400 Subject: [Dovecot] FIXED Re: Trouble adding sasl support via dovecot In-Reply-To: References: Message-ID: <4F5E485F.40207@Media-Brokers.com> Since you got it working, I'll just comment on a couple of things... On Mon, 12 Mar 2012, Richard Troy wrote: > When I do "postconf-a" it indicates cyrus and dovecot, so I take it that > means Postfix has been built with sasl support. (I presume this means I > don't have to compile it from source.) Correct... > From the working environ, only listening on port 25, I simply added the > following (as per directions already cited above): You really should separate AUTH to the port that is designed for it: port 587 (aka the 'submission' port/service)... just uncomment it (and its attendant lines) in master.cf > The documentation found here: > > http://www.postfix.org/TLS_README.html > > claims (intimates) that it's not possible to run a site on a self-signed > certificate, Where does it state any such thing? I've been using self-signed certs for 8+years with postfix... You do have to 'accept' the certs in the clients though, and that cn scare some users. I've had zero problems with this in Android, and none in recent versions of iOS, although earlier versions required you to install the cert manually (could be done using Safari on the iPhone)... Also, Outlook provides no simple way to Accept a Cert and store it permanently (Thunderbird does), so unless/until Outlook users import the Cert, they'll have to accept it each time they fire up Outlook and check mail. > And, by the way, what's port 465 all about? Some clients propose that's > what should be used to send... It is the *deprecated* SMTPS (smtp over SSL). All modern clients can use the submission service, but some older versions of Outlook/Outlook Express can only use 465. It doesn't hurt anything to have it enabled, but you shoiuld absolutely tell all other clients to use the normal submissions service (STARTTLS on port 587). -- Best regards, Charles From andrei.michescu at miau.ca Tue Mar 13 07:41:53 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 01:41:53 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 Message-ID: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Hello, I'm using dovecot 2.1.1 with vpopmail 5.4.30 with multiples domains and I have problems setting up synchronization in between multiple computers. All act like master (my clients can connect to any of the them and read their emails either via POP3 either via IMAP, inbound email gets on any of the machines). Each machine is on a different continent, there is no shared drive in between and the synchronization is supposed to be asynchronous via cron scripts. To simplify the case, let's consider 2 machines (mx1.a and mx2.a) with 3 virtual domains (a and b and c). On both machine domain a is the default domain (needs only username to connect to imap). Initially I synchronize mx1.a with mx2.a using rsync. I check that I can login using dovecot. I tried any of the following commands to synchronize the 2 machines: mx1.a$ doveadm -Dv sync -u user1 at a -f ssh mx2.a doveadm dsync-server -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a ssh mx2.a doveadm -u user1 at a mx1.a$ doveadm -Dv sync -u user1 at a user1 at a The only thing that happens is that the on each machine the folders get doubled with some random extension (eg. Inbox becomes Inbox_3e3ff3g3gb3bb3b22). Also, another bug, if there is a domain setup as default (auth_default_realm) dsync simply ignores the specified -u and attempts to sync the first email in the default domain. Please advise. # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From nick.z.edwards at gmail.com Tue Mar 13 09:27:28 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 17:27:28 +1000 Subject: [Dovecot] upgrade convert omissions Message-ID: It did not convert over some things: protocol imap { listen = *:143 mail_plugins = quota imap_quota imap_client_workarounds = outlook-idle ssl_listen = *:993 } protocol pop3 { pop3_uidl_format = %f mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { mail_plugins = quota cmusieve quota_full_tempfail = no log_path = /var/log/dovecot/deliver.log deliver_log_format = msgid=%m: from=%f: %$ auth_socket_path = /var/run/dovecot/auth-master } I ended up with service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } It looks like service-foobar is replacing stuff inside protocol {} but protocol still exists in examples, kind of confusing. Oh what about service-pop3, different than examples and service-imap, where's pop3s ? I guess I'll just copy the service pop3 section from an example file. Does all this mean the protocol section is not needed? or everything inside of them I had is no longer valid? No LDA specific logging converted, it did not include the quota stuff in where it is needed (mail_plugins went MIA), but did inside converted plugin section. I did note the conversion warned that workarounds = outlook-idle is no longer needed, but nothing about the other stuff. Just concerns me if it ignored some needed stuff, what else did it ignore. Nik (who is rather reluctant to use series 2 and break everything and stay with 1.2) Below is the new conf file: auth_cache_negative_ttl = 0 auth_cache_ttl = 5 mins auth_mechanisms = plain login auth_verbose = yes disable_plaintext_auth = no first_valid_uid = 95 last_valid_uid = 95 listen = *,:: log_path = /var/log/dovecot/pop3.log login_log_format_elements = user=<%u> method=%m rip=%r %c mail_location = maildir:/vmail/%d/%n/Maildir mail_nfs_index = yes mail_nfs_storage = yes maildir_very_dirty_syncs = yes mmap_disable = yes passdb { args = /etc/dovecot-sql.conf driver = sql } plugin { quota = maildir quota_rule = *:storage=1000M quota_rule2 = Trash:storage=100M quota_rule3 = Junk:ignore quota_rule4 = Spam:ignore quota_warning = storage=90%% /usr/local/bin/quotawarn-90.sh quota_warning2 = storage=75%% /usr/local/bin/quotawarn-75.sh } pop3_lock_session = yes protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } user = vmail } service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } process_limit = 1024 service_count = 1 } service imap { process_limit = 1024 } service pop3-login { process_limit = 1024 service_count = 1 } service pop3 { process_limit = 1024 } shutdown_clients = no ssl_cert = References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> Message-ID: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> On 13.3.2012, at 7.41, Michescu Andrei wrote: > Initially I synchronize mx1.a with mx2.a using rsync. I check that I can > login using dovecot. .. > The only thing that happens is that the on each machine the folders get > doubled with some random extension (eg. Inbox becomes > Inbox_3e3ff3g3gb3bb3b22). This is kind of a feature. Currently if two mailboxes have a same name, but different GUID, dsync doesn't even try to merge them but instead renames one of them. So don't do initial sync with rsync, but with dsync. Alternatively you need to first get each mailbox assigned a GUID, for example: doveadm -A mailbox status guid '*' > Also, another bug, if there is a domain setup as default > (auth_default_realm) dsync simply ignores the specified -u and > attempts to sync the first email in the default domain. That can't be possible, something else is happening. What does dsync and auth log with debugs enabled when this happens? From tss at iki.fi Tue Mar 13 09:56:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 09:56:12 +0200 Subject: [Dovecot] upgrade convert omissions In-Reply-To: References: Message-ID: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> On 13.3.2012, at 9.27, Nick Edwards wrote: > It did not convert over some things: .. What Dovecot version did you use? In my test it converted everything (v2.0.18). Copy&pasting your config to a new file, adding ssl_cert + ssl_key and it produces output that converted everything (although cmusieve should be replaced with sieve): service imap-login { inet_listener imap { address = * port = 143 } inet_listener imaps { address = * port = 993 } } ssl_cert = It looks like service-foobar is replacing stuff inside protocol {} but > protocol still exists in examples, kind of confusing. Services have replaced some settings, not protocol itself. http://wiki2.dovecot.org/Services > Oh what about service-pop3, different than examples and service-imap, You had explicitly set listen/ssl_listen only for imap, not for pop3, so that's what the conversion did. pop3 uses the defaults. > where's pop3s ? It's enabled by default. But pop3s isn't a really a "protocol", so it's no longer treated specially. > I guess I'll just copy the service pop3 section from > an example file. No need to, the defaults are fine. Same with service imap actually, you could just remove it. From tss at iki.fi Tue Mar 13 10:06:23 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 10:06:23 +0200 Subject: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: On 12.3.2012, at 20.57, Terry Carmen wrote: >> If you can use userdb passwd-file and export the data to that file, it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile >> >> Example line: >> >> user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com >> >> Note that you can't then return any userdb fields from passdb ldap lookup. > > That doesn't seem to work because I can't create the passdb file containing the user's password, since they're only known to the remote IMAP server that I want imapproxy to connect to. Well, you could allow users to log in with any password and then let it just fail later at imapc login, but that's a bit ugly. You could also use passdb imap {} + userdb passwd-file {} with some extra work. The authentication would be done against the remote imap server, while the userdb_imapc_host would be looked up from the passwd-file. > What would be perfect is if I could do something like this: > > //////////////////////////// > > http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying > Proxy only server .. > All I really need is a way to lookup the user's home IMAP server when given the username, as above. > > Does imapproxy still support this 1.x feature? This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy From nick.z.edwards at gmail.com Tue Mar 13 10:15:09 2012 From: nick.z.edwards at gmail.com (Nick Edwards) Date: Tue, 13 Mar 2012 18:15:09 +1000 Subject: [Dovecot] upgrade convert omissions In-Reply-To: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> References: <6AE33F8D-1FA0-4ECA-8D35-A3843DA3564E@iki.fi> Message-ID: On 3/13/12, Timo Sirainen wrote: > On 13.3.2012, at 9.27, Nick Edwards wrote: > >> It did not convert over some things: > .. > > What Dovecot version did you use? In my test it converted everything 1.2.17 -> 2.1.1 > > protocol pop3 { > mail_plugins = quota > pop3_client_workarounds = outlook-no-nuls oe-ns-eoh > pop3_uidl_format = %f > } > protocol lda { > auth_socket_path = /var/run/dovecot/auth-master > deliver_log_format = msgid=%m: from=%f: %$ > log_path = /var/log/dovecot/deliver.log > mail_plugins = quota cmusieve > quota_full_tempfail = no > } > >> It l any idea why it never copied over the protocol stuff? > Services have replaced some settings, not protocol itself. > http://wiki2.dovecot.org/Services thanks > >> I guess I'll just copy the service pop3 section from >> an example file. > > No need to, the defaults are fine. Same with service imap actually, you > could just remove it. OK Thanks From schut at sarvision.nl Tue Mar 13 10:46:04 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 09:46:04 +0100 Subject: [Dovecot] invalid mailbox name Message-ID: Hi, while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Migration is from dovecot 1.1.11 to 1.2.15. Thanks, Vincent. From tss at iki.fi Tue Mar 13 11:00:19 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:00:19 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 13.3.2012, at 10.46, Vincent Schut wrote: > Hi, > > while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: > > ERROR: Folder 'Organisations.RS Env & IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env & IJRS']) > > I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... > > Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as "&-". From schut at sarvision.nl Tue Mar 13 11:10:48 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:10:48 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: On 03/13/2012 10:00 AM, Timo Sirainen wrote: > On 13.3.2012, at 10.46, Vincent Schut wrote: > >> Hi, >> >> while migrating all mail from our old to our new server (using offlineimap, imap -> imap), I get the following error for one of my user's mailboxes: >> >> ERROR: Folder 'Organisations.RS Env& IJRS'[local_hoekman] could not be created. Server responded: ('NO', ['Invalid mailbox name: Organisations.RS Env& IJRS']) >> >> I suppose this is because of the ampersand in the mailbox name? Because other folder with spaces in it go just fine... >> >> Is this a fixed thing? Why was my user able to create the folder once? And why is dovecot 1.2.15 refusing to create the folder now? > > Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. > > Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". > Timo, thanks for the quick follow-up. However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: - when I want to create a folder with a "&" using a imap client, I have to type "&-"? or: - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) or: - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? or did you mean something else altogether ("don't use folders with & in their names, they're evil")? Vincent. From tss at iki.fi Tue Mar 13 11:15:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:15:45 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <66BBAF62-FBC2-452C-8A97-0A505515EC7E@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: >> Old Dovecot versions didn't enforce mailbox names to be valid, new ones do. I guess user had a broken IMAP client that created a mailbox with invalid name and now Dovecot refuses to do it again. >> >> Mailbox names are in "modified UTF-7" format. "&" character needs to be translated as"&-". >> > > Timo, > > thanks for the quick follow-up. > However, I'm afraid I don't really yet grasp your explanation about the utf-7 translation ("&" character needs to be translated as"&-"). To end my confusion, could you elaborate a bit on which of these interpretations is correct: > > - when I want to create a folder with a "&" using a imap client, I have to type "&-"? no. > or: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) no. > or: > - the foldername on disk should contain "&-" instead of just "&" to denote the ampersand? yes. > or did you mean something else altogether ("don't use folders with & in their names, they're evil")? When user types "&", the IMAP client should translate it to "&-". Also in filesystem it should be shown as "&-" (although this will be configurable in future). On input IMAP client will of course also see it as "&-" and should translate it back to "&" before making it visible to user. offlineimap works correctly in that it doesn't really need to do any kind of translation or validation, since it was originally IMAP client's fault for creating it and secondarily Dovecot's fault for allowing its creation. Although I guess offlineimap could have detected that this is an invalid mailbox name and translated it to something valid (this is how Dovecot v2.x's dsync works). From tss at iki.fi Tue Mar 13 11:20:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 11:20:47 +0200 Subject: [Dovecot] invalid mailbox name In-Reply-To: References: Message-ID: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> On 13.3.2012, at 11.10, Vincent Schut wrote: > - the imap client (offlineimap in this case) should translate the "&" into "&-" on the fly (I can do that, you can give folder translation functions in offlineimap) Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the & character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change & in there to &-, it would end up showing broken to user. Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. From schut at sarvision.nl Tue Mar 13 11:33:59 2012 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 13 Mar 2012 10:33:59 +0100 Subject: [Dovecot] invalid mailbox name In-Reply-To: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> References: <651F9D47-BF14-4177-8C75-BAE1E3C66571@iki.fi> Message-ID: On 03/13/2012 10:20 AM, Timo Sirainen wrote: > On 13.3.2012, at 11.10, Vincent Schut wrote: > >> - the imap client (offlineimap in this case) should translate the "&" into"&-" on the fly (I can do that, you can give folder translation functions in offlineimap) > > Oh, and you probably shouldn't do any automated translations, since they're more likely to just break things. This isn't just about the& character, but any non-ascii, such as a mailbox called "p??" would be translated as "p&AOQA5A-". If you go and change& in there to&-, it would end up showing broken to user. > > Unless there are a lot of these, it's easier to just rename the broken mailboxes in the source server. > > Thanks Timo. Everything's clear now. There are only a few of these, I'll just rename them (and their line in the subscriptions file). Vincent. From marcio.merlone at a1.ind.br Tue Mar 13 16:36:32 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 11:36:32 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage Message-ID: <4F5F5B70.1020304@a1.ind.br> Hi people, When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is find -type d -name .Archives\* -print0 | \ while read -d $'\0' archive; do \ mv "$archive" /dead/Emails/jhon.doe/; \ ln -s "/dead/Emails/jhon.doe/$archive" .; done It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? Thanks in advance, best regards. -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 18:49:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 18:49:54 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F5B70.1020304@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> Message-ID: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> On 13.3.2012, at 16.36, Marcio Merlone wrote: > Hi people, > > When a user archives a message from Thunderbird it moves to an IMAP folder "Archives", everyone knows that. I use dovecot 1:1.2.9-1ubuntu6.5 on Ubuntu 10.04 and want to move that folder (and respective IMAP sub-folders) to a slower storage, link it to original location and my first idea for this is > > find -type d -name .Archives\* -print0 | \ > while read -d $'\0' archive; do \ > mv "$archive" /dead/Emails/jhon.doe/; \ > ln -s "/dead/Emails/jhon.doe/$archive" .; done > > It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. I'm not sure what you mean by locks. dovecot-uidlist.lock? Anyway, the above is safe only if the IMAP client doesn't try to access the mailboxes during the move. Otherwise it can become confused. > I took a look on mailling list archives, but could not find something like that and was wondering what you guys use (if any) for such task and what are the recommendations, best practices, solutions for that? A perfectly working solution would be to (upgrade to v2.x and) switch to sdbox or mdbox format with alt storage enabled, then you could simply do: doveadm altmove -A mailbox 'Archives*' all From btb at bitrate.net Tue Mar 13 19:56:08 2012 From: btb at bitrate.net (btb at bitrate.net) Date: Tue, 13 Mar 2012 13:56:08 -0400 Subject: [Dovecot] No passdbs specified in configuration file with passdb/userdb in protocol sections In-Reply-To: <20120312133849.GS24983@harrier.slackbuilds.org> References: <20120312133849.GS24983@harrier.slackbuilds.org> Message-ID: <467D7FF4-02AB-4AB2-B1BA-26D50CC5A145@bitrate.net> On Mar 12, 2012, at 09.38, /dev/rob0 wrote: > On Mon, Mar 12, 2012 at 12:00:11AM -0400, btb at bitrate.net wrote: >> the problem with this is that while each of the passdb/userdb >> configs for the various protocols does indeed work, if a result >> is not found in one of them, the global passdb appears to then >> function as a catch-all. >> >> how can i tell dovecot it doesn't need a global passdb? each >> of the protocols' passdb/userdb configs is functioning as >> desired, but having dovecot look elsewhere upon failure ends >> up defeating the purpose. > > A simple workaround: use an empty passwd-file passdb as global. thanks, yeah. i've got what effectively accomplishes the same thing - a global ldap passdb within which the search filter always returns nothing. it sure seems like there would be a more logical method than this though. i looked briefly at the static password database, but at the moment it's intended application appears to be the opposite. if there were some argument such as deny=y or similar, it could be used. but ultimately, it would seem to make more sense to be able to simply tell dovecot that it doesn't need a global passdb, since obviously it doesn't. -b From andrei.michescu at miau.ca Tue Mar 13 20:22:16 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 13 Mar 2012 14:22:16 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: Hello, Thank you for your reply. I'm attaching you the output for the 2nd bug. All the folders that you see in there does not exists in user1 at b but they belong to first_user at a (which is NOT involved in this sync), BUT a is the default domain. Also for the first suggestion: 1) how do you sync initially the 2 machines? Because if you create the account on both machines, already the Inbox has 2 different guids 2) if you know the guid, how do you change them? Because then I can do the rsync and after I can correct the guid on the other machine Thank you, Andrei > On 13.3.2012, at 7.41, Michescu Andrei wrote: > >> Initially I synchronize mx1.a with mx2.a using rsync. I check that I can >> login using dovecot. > .. >> The only thing that happens is that the on each machine the folders get >> doubled with some random extension (eg. Inbox becomes >> Inbox_3e3ff3g3gb3bb3b22). > > This is kind of a feature. Currently if two mailboxes have a same name, > but different GUID, dsync doesn't even try to merge them but instead > renames one of them. > > So don't do initial sync with rsync, but with dsync. Alternatively you > need to first get each mailbox assigned a GUID, for example: doveadm -A > mailbox status guid '*' > >> Also, another bug, if there is a domain setup as default >> (auth_default_realm) dsync simply ignores the specified -u and >> attempts to sync the first email in the default domain. > > That can't be possible, something else is happening. What does dsync and > auth log with debugs enabled when this happens? > > > !DSPAM:4f5efb4c315461389012818! > > -------------- next part -------------- A non-text attachment was scrubbed... Name: output_doveadm.odt Type: application/vnd.oasis.opendocument.text Size: 16377 bytes Desc: not available URL: From e-frog at gmx.de Tue Mar 13 20:42:41 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 19:42:41 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors Message-ID: <4F5F9521.2060206@gmx.de> Hello Timo, I'm experimenting with 'doveadm backup' on 2.1.1 (latest hg, full dovecot -n output attached) and haven't managed to get it working. This is what I have done: 1. Create the directory /tmp/backup which is empty 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ Then I see the following errors: doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in dest (guid=9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=bd05451f2fbb574d40600000ec8d17cd) dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox 9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Trying to open a non-listed mailbox with guid=9e4b88178b905f4f456e0000381555a6 dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) dsync(testuser at ubuntu-test.localdomain): Error: msg iteration failed: Couldn't open mailbox bd05451f2fbb574d40600000ec8d17cd dsync(testuser at ubuntu-test.localdomain): Error: Mailbox INBOX changed its GUID (bd05451f2fbb574d40600000ec8d17cd -> 9e4b88178b905f4f456e0000381555a6) It somehow finds INBOX in destination however the backup directory is newly created and empty. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: doveconf-n.txt URL: From tss at iki.fi Tue Mar 13 20:51:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 20:51:13 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> Message-ID: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> On 13.3.2012, at 20.22, Michescu Andrei wrote: > I'm attaching you the output for the 2nd bug. All the folders that you see > in there does not exists in user1 at b but they belong to first_user at a (which > is NOT involved in this sync), BUT a is the default domain. The output showed debug output from the local dsync, but not from the remote. I think you'll see that if you do: sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv dsync-server -u user1 at b Also it's possible that in v2.1.1 there was some bug related to this.. You could try the latest nightly snapshot that has several fixes related to dsync: http://www.dovecot.org/nightly/ > Also for the first suggestion: > > 1) how do you sync initially the 2 machines? Because if you create the > account on both machines, already the Inbox has 2 different guids doveadm sync should be run before the destination Maildir exists at all. If vpopmail creates that, I guess it would just have to be deleted manually.. > 2) if you know the guid, how do you change them? Because then I can do the > rsync and after I can correct the guid on the other machine The GUID is generated the first time it's used, which normally means when you run dsync for the first time. Alternatively you can also use doveadm to ask for the mailbox's GUID and it gets generated: doveadm mailbox status -u user at domain guid '*' Running rsync after this is done also copies the GUID (it's stored in dovecot-uidlist). From marcio.merlone at a1.ind.br Tue Mar 13 20:58:59 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Tue, 13 Mar 2012 15:58:59 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> Message-ID: <4F5F98F3.1090601@a1.ind.br> Em 13-03-2012 13:49, Timo Sirainen escreveu: > On 13.3.2012, at 16.36, Marcio Merlone wrote: >> It works, given that the files are not locked. I could restart dovecot before so it would unlock the files, but there is no guarantee it does not get locked again before the find finishes. > I'm not sure what you mean by locks. Operating system lock of open files (lsof). >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Sounds really nice. There are no 2.0 packages for Lucid tough, so will take a look at the latest stable Ubuntu, or perhaps give the beta a try (until it comes stable). Thanks for your prompt reply. :) -- *Marcio Merlone* From tss at iki.fi Tue Mar 13 21:16:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 13 Mar 2012 21:16:54 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5F9521.2060206@gmx.de> References: <4F5F9521.2060206@gmx.de> Message-ID: On 13.3.2012, at 20.42, e-frog wrote: > This is what I have done: > 1. Create the directory /tmp/backup which is empty > 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ .. > Then I see the following errors: > > doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. Try without mailbox_list_index=yes From e-frog at gmx.de Tue Mar 13 22:19:53 2012 From: e-frog at gmx.de (e-frog) Date: Tue, 13 Mar 2012 21:19:53 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: References: <4F5F9521.2060206@gmx.de> Message-ID: <4F5FABE9.3080200@gmx.de> On 13.03.2012 20:16, wrote Timo Sirainen: > On 13.3.2012, at 20.42, e-frog wrote: > >> This is what I have done: >> 1. Create the directory /tmp/backup which is empty >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > .. >> Then I see the following errors: >> >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > Try without mailbox_list_index=yes Yes, after disabling list indexes it works. From terry at cnysupport.com Tue Mar 13 23:44:40 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 17:44:40 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> Message-ID: <4F5FBFC8.3060306@cnysupport.com> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a regular dummy proxying setup. Sure you could still do > that, but it's not imapc proxying. > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. Thanks for the help! Terry From tss at iki.fi Wed Mar 14 00:05:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 00:05:14 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F5FBFC8.3060306@cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> Message-ID: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> On 13.3.2012, at 23.44, Terry Carmen wrote: > On 03/13/2012 04:06 AM, Timo Sirainen wrote: >> This describes a regular dummy proxying setup. Sure you could still do that, but it's not imapc proxying. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > The above URL worked beautifully and Dovecot is now running as a proxy for a dozen older Exchange servers on a private network. If you find out that IMAP clients still don't work nicely with Exchange (apparently they have random problems, especially with shared mailboxes/accounts), you can still put imapc proxy in front of your currently working Dovecot proxy. :) From terry at cnysupport.com Wed Mar 14 00:29:09 2012 From: terry at cnysupport.com (Terry Carmen) Date: Tue, 13 Mar 2012 18:29:09 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> Message-ID: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> ----- Message from Timo Sirainen --------- ? ? Date: Wed, 14 Mar 2012 00:05:14 +0200 ? ? From: Timo Sirainen Reply-To: Dovecot Mailing List Subject: Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location ? ? ? To: Terry Carmen ? ? ? Cc: dovecot at dovecot.org > On 13.3.2012, at 23.44, Terry Carmen wrote: >> On 03/13/2012 04:06 AM, Timo Sirainen wrote: > This describes a >> regular dummy proxying setup. Sure you could still do that, but >> it's not imapc proxying. >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy >> The above URL worked beautifully and Dovecot is now running as a >> proxy for a dozen older Exchange servers on a private network. > If you find out that IMAP clients still don't work nicely with > Exchange (apparently they have random problems, especially with > shared mailboxes/accounts), you can still put imapc proxy in front > of your currently working Dovecot proxy. :) I'm going to hope everything is OK for a while, since my goal is to retire all the old Exchange servers and move all the users to dovecot/maildir within the next couple of months. However it's always nice to know there are options. 8-) Terry From andrei.michescu at miau.ca Wed Mar 14 07:25:09 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 01:25:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: Hello, Thank you very much... Using the nightly build and a combination of mailbox status + rsync + dsync made it happen. So the *full* procedure was: for every domain dom for every user u in dom doveadm mailbox status -u user at domain guid '*' rsync /home/vpopmail/domains/$dom/$u mx2.a:/home/vpopmail/domains/$dom/$u doveadm -Dv sync -u $u@$dom -f ssh mx2.a doveadm dsync-server -u $u@$dom loop $u loop $dom Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync keeps them in sync (cron job every 5 minutes). The problem comes when I start using the master-master model: emails starts getting duplicate with different ids. Here is the example: mx1.a receives an email for user1 (next line is the ls on the Maildir/new on mx1.a): -rw------- 1 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 mx2.a receives another email for user1: -rw------- 1 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx2,S\=273 cron job starts on mx1.a: sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 dsync-local(user1 at b): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at b): Info: INBOX: Couldn't keep all uids dsync-local(user1 at b): Warning: Mailbox changes caused a desync. You may want to run dsync again. --due to desync it runs again (via the cron script): sudo -u vpopmail doveadm -Dv sync -u user1 at b -f ssh mx2.a doveadm dsync-server -u user1 at b doveadm(vpopmail): Debug: Loading modules from directory: /usr/lib/dovecot/doveadm doveadm(vpopmail): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(vpopmail): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(user1 at b): Debug: auth input: user1 at b uid=89 gid=89 home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: Effective uid=89, gid=89, home=/home/vpopmail/domains/b/user1 doveadm(user1 at b): Debug: maildir++: root=/home/vpopmail/domains/b/user1/Maildir, index=, control=, inbox=/home/vpopmail/domains/b/user1/Maildir, alt= dsync-local(user1 at b): Debug: Namespace : Using permissions from /home/vpopmail/domains/b/user1/Maildir: mode=0700 gid=-1 The expected result would be that in user1's Mailbox on both mx1.a and mx2.a there would be 2 files... which is not happening ls on mx1.a -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331701451.24233.mx1,S\=278 -rw------- 1 vpopmail vchkpw 273 Mar 14 01:05 1331701504.32564.mx2,S\=273 -rw------- 2 vpopmail vchkpw 278 Mar 14 01:04 1331702193.M868989P24524.mx1,S\=278 ls on mx2.a -rw------- 1 vpopmail vchkpw 278 Mar 14 07:04 1331701451.24233.mx1,S\=278 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331701504.32564.mx1,S\=273 -rw------- 2 vpopmail vchkpw 273 Mar 14 07:05 1331702193.M798223P32571.mx2,S\=273 As you can see on every machine the original email gets duplicated. Please advise on how I can fix this issue. Thank you and Best regards, Andrei > On 13.3.2012, at 20.22, Michescu Andrei wrote: > >> I'm attaching you the output for the 2nd bug. All the folders that you >> see >> in there does not exists in user1 at b but they belong to first_user at a >> (which >> is NOT involved in this sync), BUT a is the default domain. > > The output showed debug output from the local dsync, but not from the > remote. I think you'll see that if you do: > > sudo -u vpopmail doveadm sync -u user1 at b -f ssh mx2.a doveadm -Dv > dsync-server -u user1 at b > > Also it's possible that in v2.1.1 there was some bug related to this.. You > could try the latest nightly snapshot that has several fixes related to > dsync: http://www.dovecot.org/nightly/ > >> Also for the first suggestion: >> >> 1) how do you sync initially the 2 machines? Because if you create the >> account on both machines, already the Inbox has 2 different guids > > doveadm sync should be run before the destination Maildir exists at all. > If vpopmail creates that, I guess it would just have to be deleted > manually.. > >> 2) if you know the guid, how do you change them? Because then I can do >> the >> rsync and after I can correct the guid on the other machine > > The GUID is generated the first time it's used, which normally means when > you run dsync for the first time. Alternatively you can also use doveadm > to ask for the mailbox's GUID and it gets generated: > > doveadm mailbox status -u user at domain guid '*' > > Running rsync after this is done also copies the GUID (it's stored in > dovecot-uidlist). > !DSPAM:4f5f972f80146209382307! > > From CMarcus at Media-Brokers.com Wed Mar 14 12:58:54 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 06:58:54 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> Message-ID: <4F6079EE.4000201@Media-Brokers.com> On 2012-03-13 6:29 PM, Terry Carmen wrote: > I'm going to hope everything is OK for a while, since my goal is to retire > all the old Exchange servers and move all the users to dovecot/maildir > within the next couple of months. > > However it's always nice to know there are options. 8-) I'm currently looking at rolling out SOGo as part of a major reworking of their current infrastructure (will also include converting their old Courier-IMAP to dovecot 2.1.x among other things)... SOGo, as far as I can tell, is the best truly free and open source 'exchange clone' available that works extremely well with Thunderbird+Lightning (which is what my Client uses currently, but they are very dissatisfied with using Google Calendar for Shared calendars), Outlook and Apple Apps, as well as Android, Blackberry and Apple mobile devices - and their upcoming v2 (in beta now) will not only provide native Outlook support (no plugin needed), it will also (optionally) provide a Samba4 Active Directory server in my main Client's office - all with absolutely no licenses required. Commercial support is available from Inverse, the company created by the developers to provide said support services. I also learned something very interesting yesterday concerning SOGo and dovecot during a sales call with a SOGo rep, but I'll wait and see if Timo cares to chime in on this one... ;) -- Best regards, Charles From marcio.merlone at a1.ind.br Wed Mar 14 13:51:30 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 08:51:30 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F5F98F3.1090601@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> Message-ID: <4F608642.5060707@a1.ind.br> Em 13-03-2012 15:58, Marcio Merlone escreveu: > Em 13-03-2012 13:49, Timo Sirainen escreveu: >> A perfectly working solution would be to (upgrade to v2.x and) switch >> to sdbox or mdbox format with alt storage enabled, then you could >> simply do: doveadm altmove -A mailbox 'Archives*' all Should this command be run every time a new folder (.Archives.2012 for example) is created or can this be automated, something like instructing deliver to do this to any Archives* is created? In other words, to cron or not to cron? Best regards. -- *Marcio Merlone* From tss at iki.fi Wed Mar 14 14:19:31 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 14:19:31 +0200 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <4F608642.5060707@a1.ind.br> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> Message-ID: <1331727571.2081.126.camel@innu> On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: > Em 13-03-2012 15:58, Marcio Merlone escreveu: > > Em 13-03-2012 13:49, Timo Sirainen escreveu: > >> A perfectly working solution would be to (upgrade to v2.x and) switch > >> to sdbox or mdbox format with alt storage enabled, then you could > >> simply do: doveadm altmove -A mailbox 'Archives*' all > Should this command be run every time a new folder (.Archives.2012 for > example) is created or can this be automated, something like instructing > deliver to do this to any Archives* is created? In other words, to cron > or not to cron? It's not deliver that creates it, it's the IMAP client. And it would have to be done when IMAP client copies mails there. But there's currently no easy way to automate that, so you'd need to do it in cron. Another possibility could be to add a new feature (plugin) that always immediately saves mails in Archives* mailboxes to alt storage. From tss at iki.fi Wed Mar 14 15:41:30 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 15:41:30 +0200 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F5FABE9.3080200@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> Message-ID: <1331732490.2081.127.camel@innu> On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: > On 13.03.2012 20:16, wrote Timo Sirainen: > > On 13.3.2012, at 20.42, e-frog wrote: > > > >> This is what I have done: > >> 1. Create the directory /tmp/backup which is empty > >> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > > .. > >> Then I see the following errors: > >> > >> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ > >> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. > > > > Try without mailbox_list_index=yes > > Yes, after disabling list indexes it works. With latest hg version it should work. From tss at iki.fi Wed Mar 14 16:09:49 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:09:49 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> Message-ID: <1331734189.2081.137.camel@innu> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync > keeps them in sync (cron job every 5 minutes). > > The problem comes when I start using the master-master model: emails > starts getting duplicate with different ids. I was testing this a bit, and I guess in your tests dsync was running during a mail delivery, which seems to make it duplicate mails sometimes. I'll probably fix this at some point (I've actually been thinking about a larger dsync redesign), but anyway: Even if dsync worked perfectly and didn't duplicate mails, it's not a great idea to do deliver mails randomly to both servers. Each time dsync notices that both sides have had new mails, it needs to reassign new IMAP UIDs for the messages, which means that IMAP clients may need to redownload the mails. Better to give one MX a higher priority so mails typically are delivered through it. From tss at iki.fi Wed Mar 14 16:29:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:29:15 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: Message-ID: <1331735355.2081.140.camel@innu> On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: > Heya, > > We are expiriencing issues with dovecot 2.1.1 on Linux with weird > filenames in home directory of username. We are using mbox IMAP > folders, with no special changes (mail_location = mbox:~/:INBOX=% > h/.mailbox). > > Mar 6 13:37:17 machine dovecot: imap(username): Panic: file > mail-storage.c: line 628 (mailbox_alloc): assertion failed: > (uni_utf8_str_is_valid(vname)) .. > AFAIK, the problem lies in processing the file list of home folder, > which can contain filenames that do not have proper UTF-8 encoding of > filenames, which causes dovecot to crash. Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This should fix it by renaming such mailboxes: http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 > On the other hand, UTF-8 filenames created on the system by hand > (using touch), are not displayed in IMAP LIST command (sample is > included in the folder structure; single letter file). This is a bit trickier problem. The mailbox names are currently stored in filesystem as IMAP's modified-UTF7. So it's not really even currently supposed to work, although it's not very nice that the mailboxes aren't visible either. Maybe I'll do something smart in future for this, like allowing both mUTF-7 and UTF-8 and remembering per-mailbox which formatting it is in. From tss at iki.fi Wed Mar 14 16:40:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 16:40:05 +0200 Subject: [Dovecot] \NoSelect on missing folders in LIST In-Reply-To: <4F55BA0C.5090606@one.com> References: <4F54B942.9070005@one.com> <4F54D434.6090300@one.com> <637D369C-0E1E-487B-A172-E4CD5BC38D1D@iki.fi> <4F54D731.6060705@one.com> <4F55BA0C.5090606@one.com> Message-ID: <1331736005.2081.144.camel@innu> On Tue, 2012-03-06 at 08:17 +0100, Peter Mogensen wrote: > On 2012-03-05 16:36, Timo Sirainen wrote: > >> Still curious about if Courier is doing something wrong which the scripts just happened to take advantage of. > > > > Neither behavior is wrong, just different. :) > > Ok... I were in doubt if I had missed something from the RFC. > However... for testing, I tried to create "INBOX.INBOX" on dovecot. > But then dovecot answers NO and complains that the folder already > exists. Though it's still not on disk and dovecot still doesn't list it > with "*". This is a bit problematic when you have prefix="INBOX." namespace. There if you access "INBOX.INBOX", its internal storage name is "INBOX". And INBOX's internal storage name is also "INBOX". So in some parts of the code they are treated as if they were both the same mailbox. Maybe I can get this fixed for v2.2. From campbell at cnpapers.com Wed Mar 14 16:46:58 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 10:46:58 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions Message-ID: <4F60AF62.80702@cnpapers.com> I've mostly finished a conversion from an old Centos 3 UW-Imap server to a new Centos 6 dovecot server. I did not copy the old ~/.mailboxlist file to ~/mail/.subscriptions file, but notice some users have the latter file now. These are all mbox folders on the old and new server. I'm getting ready to do the same to another old/new pair of servers and I'm wondering if there is an advantage of doing the copy. I'm assuming the .subscription files are created when they access their account through our webmail application, but I'm not sure if it was automatic or due to a "subscribe" action done manually. There are 49 accounts with a .mailboxlist file and only 4 with the new .subscriptions file. So either our webmail application isn't being used a lot or there's a problem with it due to the missing .subscriptions file, but the phone usually rings pretty quickly when problems arise. One last question, please. Over the years, some imap accounts had their folders directly in their home directory and the contents of the .mailboxlist file would have an entry with just the name of the folder in it (Trash, eg), and most had the folders in their ~/mail folder with an entry like "mail/Trash". Our webmail app, Horde/Imp, always seemed to take care of this. If I create the .subscription file for the users during the move to the new server, should I move the folders to the mail directory and amend their .subscriptions file to reflect that change on these odd ball accounts, and will that affect how their client is seeing these? The first server conversion was a bear due to my lack of dovecot knowledge. I've since learned a little more, and mostly found out that dovecot is a more complex application than the old imap application. There's so much more that can be done with dovecot, whereas the old imap server was mostly just load-and-go. Seems like no matter how much I read, the more I discovered I didn't know. Anyway, thanks for all the past help and any opinions anyone might decide to offer on this post. steve campbell From marcio.merlone at a1.ind.br Wed Mar 14 17:00:49 2012 From: marcio.merlone at a1.ind.br (Marcio Merlone) Date: Wed, 14 Mar 2012 12:00:49 -0300 Subject: [Dovecot] Thunderbird Archive to slower storage - sort of alternate storage In-Reply-To: <1331727571.2081.126.camel@innu> References: <4F5F5B70.1020304@a1.ind.br> <8D61E51A-2049-4744-8FF9-9D865AD466D0@iki.fi> <4F5F98F3.1090601@a1.ind.br> <4F608642.5060707@a1.ind.br> <1331727571.2081.126.camel@innu> Message-ID: <4F60B2A1.7060104@a1.ind.br> Em 14-03-2012 09:19, Timo Sirainen escreveu: > On Wed, 2012-03-14 at 08:51 -0300, Marcio Merlone wrote: >> Em 13-03-2012 15:58, Marcio Merlone escreveu: >>> Em 13-03-2012 13:49, Timo Sirainen escreveu: >>>> A perfectly working solution would be to (upgrade to v2.x and) switch >>>> to sdbox or mdbox format with alt storage enabled, then you could >>>> simply do: doveadm altmove -A mailbox 'Archives*' all >> Should this command be run every time a new folder (.Archives.2012 for >> example) is created or can this be automated, something like instructing >> deliver to do this to any Archives* is created? In other words, to cron >> or not to cron? > It's not deliver that creates it, it's the IMAP client. And it would I first though about deliver since this seems the guy who could do that, but don't matter. > have to be done when IMAP client copies mails there. But there's > currently no easy way to automate that, so you'd need to do it in cron. Just imagine that: protocol imap{ ... x_alternate_storage_always = Archives, Spam, Trash x_alternate_storage_size = 20MB x_alternate_storage_age = 1y ... } The client (Thunderbird?) sends imap commands to dovecot create such folder or to move a message to such folder. Dovecot obeys and check the config to see if the folder in question is one of those listed on x_alternate_storage_always. If the folder is not on alternate storage yet, create/move it there and them proceed what was asked to do. I imagine that an age based action would depend of a cron job in order to not overload the server each time it performs any imap command and check old messages age... > Another possibility could be to add a new feature (plugin) that always > immediately saves mails in Archives* mailboxes to alt storage. I don't know if the duck is male, I want the egg! :) I know nothing about the inner workings of dovecot, and very little about the outer working. A well designed and integrated plugin could be, but seems to be a nice core feature for dovecot, based on folder name, size or age. Many thanks for your time. Please be kind considering those ideas, I am just a poor man's server admin. :) Best regards. -- *Marcio Merlone* From campbell at cnpapers.com Wed Mar 14 17:48:43 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 11:48:43 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60BDDB.8000103@cnpapers.com> On 3/14/2012 10:46 AM, Steve Campbell wrote: > > > One last question, please. > > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". > Our webmail app, Horde/Imp, always seemed to take care of this. If I > create the .subscription file for the users during the move to the new > server, should I move the folders to the mail directory and amend > their .subscriptions file to reflect that change on these odd ball > accounts, and will that affect how their client is seeing these? > > The first server conversion was a bear due to my lack of dovecot > knowledge. I've since learned a little more, and mostly found out that > dovecot is a more complex application than the old imap application. > There's so much more that can be done with dovecot, whereas the old > imap server was mostly just load-and-go. Seems like no matter how much > I read, the more I discovered I didn't know. > > Anyway, thanks for all the past help and any opinions anyone might > decide to offer on this post. > > steve campbell I've discovered another situation. This may not be a problem, but I've got to deal with it at any rate. I find that some users have a .mailboxlist which points to folders in their home directory, and have folders in their mail directory as well. For the most part, this situation involved horde/imp "sent-mail" folders which are created when users send mail through our webmail but they more than likely have a client on either their phone or desktop that is configured as imap. The horde/imp "sent-mail" is not listed in their .mailboxlist file. So I'm guessing this will be all right to leave as is or to modify the resultant .subscription file to point to moved folders. So many things to consider for so many different situations. Thanks steve > > From terry at cnysupport.com Wed Mar 14 17:59:26 2012 From: terry at cnysupport.com (Terry Carmen) Date: Wed, 14 Mar 2012 11:59:26 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F60C05E.1090803@cnysupport.com> On 03/14/2012 06:58 AM, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. Looks interesting. I have currently have horde/imp/kronolith running with postfix/dovecot/mysql on the back end and it's been working nicely with all the clients and devices except for outlook. I'll have to take a look at sogo, because I'd really like to keep outlook for the users that want it, to cut down on support and complaints. Thanks! Terry From arnaud.abelard at univ-nantes.fr Wed Mar 14 18:06:32 2012 From: arnaud.abelard at univ-nantes.fr (=?ISO-8859-1?Q?Arnaud_Ab=E9lard?=) Date: Wed, 14 Mar 2012 17:06:32 +0100 Subject: [Dovecot] RECENT status always 0 Message-ID: <4F60C208.6010304@univ-nantes.fr> Hello, we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) then 2 minutes later: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) How can the UNSEEN count change without the RECENT count changing accordingly? Thanks in advance, Arnaud -- Arnaud Ab?lard (jabber: arnaud.abelard at univ-nantes.fr) Administrateur Syst?me - Responsable Services Web Direction des Syst?mes d'Informations Universit? de Nantes - ne pas utiliser: trapemail at univ-nantes.fr From tss at iki.fi Wed Mar 14 18:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:27:09 +0200 Subject: [Dovecot] RECENT status always 0 In-Reply-To: <4F60C208.6010304@univ-nantes.fr> References: <4F60C208.6010304@univ-nantes.fr> Message-ID: <1AE988E0-F799-45AA-A098-A3462DC13340@iki.fi> On 14.3.2012, at 18.06, Arnaud Ab?lard wrote: > Hello, > > we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 10) > > then 2 minutes later: > > . STATUS INBOX (MESSAGES UNSEEN RECENT) > * STATUS "INBOX" (MESSAGES 35106 RECENT 0 UNSEEN 11) > > How can the UNSEEN count change without the RECENT count changing accordingly? If any client has the INBOX opened, the recent count for other connections stays at 0. I guess you're expecting recent flags to work differently than how IMAP RFC specifies them. (It's entirely possible that Courier implemented them in the wrong way.) From jom at grosjo.net Wed Mar 14 18:36:55 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 22:21:55 +0545 Subject: [Dovecot] FTS crash Message-ID: Hi Timo, I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. When I remove the plugin from the dovecot.conf, all works fine. No debug available here, but let me know how can I help you. Joan From andrei.michescu at miau.ca Wed Mar 14 18:45:07 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 12:45:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Hello, Nope dsync was not running during the email delivery on that account. I've simulated in a controlled environment. Servers are having different priorities, but this was a basic scenario to test the master-master synchronization. Think that for incoming SMTP I can even restrict which server is the master (forcing all other to redeliver to this one). BUT, for a distributed IMAP cluster there is no way to restrict users to perform changes on only one server. This would defeat the model and the purpose of a distributed cluster... One idea might be to have the IDs dependent on server where they appear first time so that they keep the ID once they get replicated. Here there are many options: - the DB model = each server has a set of ids that can give (either ranges, either increment with step different then 1) - the vpopmail/qmail model = append the server name (as you saw in the previous email in the listings the email files contained the hostname mx1.a and mx2.a) Thank you, Andrei > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> Now, as long as I touch the mailbox of user1 only on mx1.a doveadm sync >> keeps them in sync (cron job every 5 minutes). >> >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Each time dsync > notices that both sides have had new mails, it needs to reassign new > IMAP UIDs for the messages, which means that IMAP clients may need to > redownload the mails. Better to give one MX a higher priority so mails > typically are delivered through it. > > > > !DSPAM:4f60a6b137151972926802! > > From tss at iki.fi Wed Mar 14 18:56:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:56:58 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: On 14.3.2012, at 18.45, Michescu Andrei wrote: > Nope dsync was not running during the email delivery on that account. I've > simulated in a controlled environment. How? You mean simply deliver mail to server A and to server B and run dsync and it duplicates it? I can't reproduce it that way, only if I run dsync during a flood of new mails. > Think that for incoming SMTP I can even restrict which server is the > master (forcing all other to redeliver to this one). BUT, for a > distributed IMAP cluster there is no way to restrict users to perform > changes on only one server. This would defeat the model and the purpose of > a distributed cluster... For IMAP it's not much of a problem, because user typically still uses only one client actively, so clients aren't uploading mails to multiple servers at the same time. > One idea might be to have the IDs dependent on server where they appear > first time so that they keep the ID once they get replicated. Here there > are many options: The messages have GUIDs that stay the same always, but IMAP UIDs are required to be ascending from client's point of view, and several clients rely on that, so when UID conflict happens the only safe thing to do is to assign new UIDs for all of the conflicting mails. From tss at iki.fi Wed Mar 14 18:58:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 18:58:36 +0200 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: On 14.3.2012, at 18.36, Joan Moreau wrote: > I updated my 2.1 from HG, and now the FTS (Squat) plugin > makes a segmentation fault. To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > No debug available here, but let me know > how can I help you. gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From CMarcus at Media-Brokers.com Wed Mar 14 19:00:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 14 Mar 2012 13:00:25 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <4F60CEA9.3080008@Media-Brokers.com> On 2012-03-14 10:46 AM, Steve Campbell wrote: > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes... dovecot doesn't like it when stuff other than mail is in the home folder: http://wiki2.dovecot.org/VirtualUsers/Home -- Best regards, Charles From andrei.michescu at miau.ca Wed Mar 14 19:26:42 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 13:26:42 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <91240ca12b62eb144b87ab4e56cbe9b8.squirrel@web.miau.ca> Message-ID: <2c95e6565a8b3783b05b2eafc2d4833e.squirrel@web.miau.ca> > On 14.3.2012, at 18.45, Michescu Andrei wrote: > >> Nope dsync was not running during the email delivery on that account. >> I've >> simulated in a controlled environment. > > How? You mean simply deliver mail to server A and to server B and run > dsync and it duplicates it? I can't reproduce it that way, only if I run > dsync during a flood of new mails. > YES. simply deliver mail to server A and then to server B (to the same user_1). After run dsync and you get exactly what you saw in my previous email. That's why I included the ls for the both servers, so that you can see the email files too. because each server duplicates only its own email (so brings the email from the other server and duplicates its own email). >> Think that for incoming SMTP I can even restrict which server is the >> master (forcing all other to redeliver to this one). BUT, for a >> distributed IMAP cluster there is no way to restrict users to perform >> changes on only one server. This would defeat the model and the purpose >> of >> a distributed cluster... > > For IMAP it's not much of a problem, because user typically still uses > only one client actively, so clients aren't uploading mails to multiple > servers at the same time. > hehe... one would think so, but when you have road-warriors that roam you can not insure that the server where they connect for IMAP (closest based on geo-ip) is the same as the server that you have picked for inbound SMTP. So you already have 2 servers that mess-up with user's mailbox. The second case where you can not control this is for mobile devices that flip-in/out of wi-fi (my iPhone is in Canada when it is on 3G and in Europe when it is WiFi due to vpn tunneling, and this can change every couple of minutes... :( ) >> One idea might be to have the IDs dependent on server where they appear >> first time so that they keep the ID once they get replicated. Here there >> are many options: > > The messages have GUIDs that stay the same always, but IMAP UIDs are > required to be ascending from client's point of view, and several clients > rely on that, so when UID conflict happens the only safe thing to do is to > assign new UIDs for all of the conflicting mails. well I don't know much about IMAP standard (you guys are the experts :)! here). If the GUID stays the same then this can be used to prevent the duplication error. Also, as you can detect if the email is new or not (a client has already seen it or not): in the case that no one has seen it then it is safe to assign any UIDs that fits. In case that on only one server it has been seen then you can give it that UIDs on all servers, and reassign all the unseen ones. So the only messed-up case is if on both servers the message has been seen with different UIDs :( Thank you very much for your time and patience. I know that our setup is pretty atypical. And think that this model with only 2 servers I'm showing you is only for simplicity as the real deployment has multiple servers geographically sparse connected by slow intercontinental internet links... :)) Otherwise we'd use a distribute file system and have only a unified storage :P Best regards. Andrei From jom at grosjo.net Wed Mar 14 19:31:47 2012 From: jom at grosjo.net (Joan Moreau) Date: Wed, 14 Mar 2012 23:16:47 +0545 Subject: [Dovecot] FTS crash In-Reply-To: References: Message-ID: I have been unlucky in deed. Problem solved with recent changes Le 14/03/2012 22:43, Timo Sirainen a ?crit : > On 14.3.2012, at 18.36, Joan Moreau wrote: > >> I updated my 2.1 from HG, and now the FTS (Squat) plugin makes a segmentation fault. > > To which version exactly? Because I broke FTS two days ago and fixed it yesterday, maybe you were unlucky enough to get a broken version. > >> No debug available here, but let me know how can I help you. > > gdb backtrace of the crash is always helpful: http://dovecot.org/bugreport.html From e-frog at gmx.de Wed Mar 14 21:33:49 2012 From: e-frog at gmx.de (e-frog) Date: Wed, 14 Mar 2012 20:33:49 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <1331732490.2081.127.camel@innu> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> Message-ID: <4F60F29D.2010409@gmx.de> On 14.03.2012 14:41, wrote Timo Sirainen: > On Tue, 2012-03-13 at 21:19 +0100, e-frog wrote: >> On 13.03.2012 20:16, wrote Timo Sirainen: >>> On 13.3.2012, at 20.42, e-frog wrote: >>> >>>> This is what I have done: >>>> 1. Create the directory /tmp/backup which is empty >>>> 2. Run doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>> .. >>>> Then I see the following errors: >>>> >>>> doveadm -v backup -u testuser at ubuntu-test.localdomain mdbox:/tmp/backup/ >>>> dsync(testuser at ubuntu-test.localdomain): Error: Can't delete mailbox INBOX: INBOX can't be deleted. >>> >>> Try without mailbox_list_index=yes >> >> Yes, after disabling list indexes it works. > > With latest hg version it should work. > Hi Timo, The "can't delete mailbox INBOX" error is gone now with changeset c077ca9bc306 and it's working successfully on the account from yesterday where it also worked with mailbox_list_index=no. However using a different account (more mail and mailboxes) I'm seeing dbox corruption errors. I have tested with mailbox_list_index=yes and no and it's the same for both. So this might be unrelated to this setting. Attached are logs from doveadm backup runs. First to an empty directory and 2 consecutive runs. Thanks, e-frog -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: log.txt URL: From campbell at cnpapers.com Wed Mar 14 21:53:00 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 14 Mar 2012 15:53:00 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60CEA9.3080008@Media-Brokers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> Message-ID: <4F60F71C.8090306@cnpapers.com> On 3/14/2012 1:00 PM, Charles Marcus wrote: > On 2012-03-14 10:46 AM, Steve Campbell wrote: >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > > Yes... dovecot doesn't like it when stuff other than mail is in the > home folder: > > http://wiki2.dovecot.org/VirtualUsers/Home I'm not sure these are virtual users, so that link may have confused me. All accounts on these servers have real unix accounts. Their inbox is /var/spool/mail/unix-user-name. Their imap folders, the ones that they create using an imap client or webmail, are either in ~ or ~/mail. Their original .mailboxlist is always in ~. Based on that, I should probably copy any imap folders not in ~/mail to that folder, duplicate ~/.mailboxlist to the file ~/mail/.subscriptions, and amend any .subscriptions file contents to just have the name of the folders (without any "mail/folder" reference in it). My example would then be as follows /home/steve = folder /home/steve/Drafts = original folder /home/steve/AnyFolder = original folder /home/steve/.mailboxlist = original file /home/steve/mail = folder (either original or created) /home/steve/mail/.subscriptions = copied contents of .mailboxlist file /home/steve/mail/Drafts = copied folder of original /home/steve/mail/AnyFolder = copied folder of original Contents of original .mailboxlist and new .subscriptions: Drafts AnyFolder If the imap folders were in ~/mail, then the original .mailboxlist would have been mail/Drafts mail/AnyFolder but after the corrections to the .subscriptions file, they would be as above (without reference to the mail folder). Is this correct? thanks for the help steve From trashcan at odo.in-berlin.de Wed Mar 14 22:36:30 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 21:36:30 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <1331734189.2081.137.camel@innu> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> Message-ID: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Hi -- On 14.03.2012, at 15:09, Timo Sirainen wrote: > On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: >> The problem comes when I start using the master-master model: emails >> starts getting duplicate with different ids. > I was testing this a bit, and I guess in your tests dsync was running > during a mail delivery, which seems to make it duplicate mails > sometimes. I'll probably fix this at some point (I've actually been > thinking about a larger dsync redesign), Good to hear ;-) > but anyway: > > Even if dsync worked perfectly and didn't duplicate mails, it's not a > great idea to do deliver mails randomly to both servers. Sometimes croncobs are running on both servers at the same time producing locally delivered mails simultaneously, though. Ok, one can modify run times accordingly ... > Better to give one MX a higher priority so mails typically are delivered > through it. That's what I did. Now dsync/replicator is performing great, if the mail volume is rather low. I'm very satisfied, because this is the best performance ever. (Before I was running unison and dsync 2.0.) But, whenever the high priority server will show delays during stress situations like huge mail loads, the low priority server will receive loads of mails as well. A dsync/replicator setup will then most probably produce duplicates (and multiples). That is a rather unrealistic scenario for my little severs, but others might have more difficulties. And spammers don't care about mx priorities at all :-( Regards, Michael From tss at iki.fi Wed Mar 14 23:14:10 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 14 Mar 2012 23:14:10 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: On 14.3.2012, at 22.36, Michael Grimm wrote: > And spammers don't care about mx priorities at all :-( But spams go to spam mailbox where duplicates don't really matter. :) From trashcan at odo.in-berlin.de Wed Mar 14 23:26:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:26:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Hi - On 14.03.2012, at 22:14, Timo Sirainen wrote: > On 14.3.2012, at 22.36, Michael Grimm wrote: >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) True ;-) But spam mails might interfere with syncing of legitimate mail arriving at the same time. Regards, Michael From trashcan at odo.in-berlin.de Wed Mar 14 23:30:41 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:30:41 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <247C60FA-6319-445E-9327-AE630522CE1C@odo.in-berlin.de> Message-ID: <5EE60F3E-A25F-4E6D-98E9-2246B75ACA10@odo.in-berlin.de> Hi -- On 14.03.2012, at 22:26, Michael Grimm wrote: > But spam mails might interfere with syncing of legitimate > mail arriving at the same time. Forget about that part, I was wrong because duplicates are produced in corresponding mailboxes, only. Sorry for the noise, Michael From andrei.michescu at miau.ca Wed Mar 14 23:32:40 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Wed, 14 Mar 2012 17:32:40 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: > On 14.3.2012, at 22.36, Michael Grimm wrote: > >> And spammers don't care about mx priorities at all :-( > > But spams go to spam mailbox where duplicates don't really matter. :) In an ideal world yes... or no. In our deployment spam is simply header tagged and left in INBOX. Each user can decide after what they want to do with it (client side rules). And, in the end, it is the same discussion, because the spam mailbox get replicated too and if the spam gets duplicated we are in the worst scenarios... knowing that spam represents 95% of all email traffic (in a real-world public-facing system). ;) hehe... in the meanwhile I looked a little on the ietf and there are different RFCs out there on disconnected clients and UIDPLUS and other nice features ;) let me know if you are interested to get some help in implementing it :D From andrei at lctax.ro Wed Mar 14 22:58:09 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Wed, 14 Mar 2012 16:58:09 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> Message-ID: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> > Hi -- > > On 14.03.2012, at 15:09, Timo Sirainen wrote: >> On Wed, 2012-03-14 at 01:25 -0400, Michescu Andrei wrote: > >>> The problem comes when I start using the master-master model: emails >>> starts getting duplicate with different ids. > >> I was testing this a bit, and I guess in your tests dsync was running >> during a mail delivery, which seems to make it duplicate mails >> sometimes. I'll probably fix this at some point (I've actually been >> thinking about a larger dsync redesign), > > Good to hear ;-) > >> but anyway: >> >> Even if dsync worked perfectly and didn't duplicate mails, it's not a >> great idea to do deliver mails randomly to both servers. > > Sometimes croncobs are running on both servers at the same time > producing locally delivered mails simultaneously, though. Ok, one > can modify run times accordingly ... Why do you run the crontab on all the servers? You can run a start-based system where only one ("main"-master) syncs all the other masters. And like this you avoid the time synch'ing of crontabs (especially if you don't always know how longer it will take for a dsync to finish). > >> Better to give one MX a higher priority so mails typically are delivered >> through it. > > That's what I did. Now dsync/replicator is performing great, if the > mail volume is rather low. I'm very satisfied, because this is the > best performance ever. (Before I was running unison and dsync 2.0.) > > But, whenever the high priority server will show delays during stress > situations like huge mail loads, the low priority server will receive > loads of mails as well. A dsync/replicator setup will then most probably > produce duplicates (and multiples). That is a rather unrealistic > scenario for my little severs, but others might have more difficulties. This is another nice case to "motivate" Timo to look for solutions ;) I tried to push 2 ideas in the same direction earlier :P Especially that he confirmed that every single email has a unique GID (which should help prevent duplication/multiplication)... > > And spammers don't care about mx priorities at all :-( Actually, statistically speaking, spammers select the low priority ones. > > Regards, > Michael > Nice to hear that we are not the only ones out there to try to run something like this over dovecot :P Thnx, Andrei From trashcan at odo.in-berlin.de Wed Mar 14 23:51:22 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Wed, 14 Mar 2012 22:51:22 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> Message-ID: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Hi -- On 14.03.2012, at 21:58, Michescu Andrei wrote: >> Sometimes croncobs are running on both servers at the same time >> producing locally delivered mails simultaneously, though. Ok, one >> can modify run times accordingly ... > > Why do you run the crontab on all the servers? You can run a start-based > system where only one ("main"-master) syncs all the other masters. You misunderstood. I was referring to system cronjob's mail reports from cron.daily jobs like security reports et al. Those reports normally run at identical times. >> And spammers don't care about mx priorities at all :-( > > Actually, statistically speaking, spammers select the low priority ones. Actually: you are right ;-) > Nice to hear that we are not the only ones out there to try to run > something like this over dovecot :P Yes. I never loved the idea of a clusterfs for my small mail servers, I always considered such clusterfs an overkill. Well, my servers do reside in the same housing building, thus it could be done without performance loss. But a scenario of worldwide distributed mail servers desires a dsync/replicator scheme, IMHO ;-) Regards, Michael From jtam.home at gmail.com Thu Mar 15 01:33:23 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 14 Mar 2012 16:33:23 -0700 (PDT) Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: Steve Campbell writes: > Their imap folders, the ones that they create using an imap client or > webmail, are either in ~ or ~/mail. Their original .mailboxlist is > always in ~. Based on that, I should probably copy any imap folders not > in ~/mail to that folder, duplicate ~/.mailboxlist to the file > ~/mail/.subscriptions, and amend any .subscriptions file contents to > just have the name of the folders (without any "mail/folder" reference > in it). > > My example would then be as follows > > /home/steve = folder > /home/steve/Drafts = original folder > /home/steve/AnyFolder = original folder > /home/steve/.mailboxlist = original file > /home/steve/mail = folder (either original or created) > /home/steve/mail/.subscriptions = copied contents of .mailboxlist > file > /home/steve/mail/Drafts = copied folder of original > /home/steve/mail/AnyFolder = copied folder of original > > Contents of original .mailboxlist and new .subscriptions: > > Drafts > AnyFolder > > If the imap folders were in ~/mail, then the original .mailboxlist would > have been > > mail/Drafts > mail/AnyFolder > > but after the corrections to the .subscriptions file, they would be as > above (without reference to the mail folder). > > Is this correct? That depends -- are you aliasing namespaces so that prefix={"", "mail/", etc.} all map to a user's ~/mail folder? You may be creating a confusing situation where a client with a null IMAP prefix has 2 copies of a mailbox. Joseph Tam From kgc at corp.sonic.net Thu Mar 15 01:51:38 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Wed, 14 Mar 2012 16:51:38 -0700 Subject: [Dovecot] Just in time AV scanning Message-ID: <20120314235138.GE39671@corp.sonic.net> I'm curious if anyone has any plugins for AV integration directly into dovecot. Our old pop servers have been scanning messges as they're moved from new->cur in the inbox and, at least where user's aren't poping every few seconds, there is occasionally enough time between scanning through the MXs to message retreval to snag a few more virues with updated definitions before they reach customers. Anyone doing anything similar? -- Kelsey Cummings - kgc at corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407 From jtl+dovecot at uvm.edu Thu Mar 15 03:24:34 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 14 Mar 2012 21:24:34 -0400 Subject: [Dovecot] director lmtp -> smtp problem Message-ID: <4F6144D2.2080900@uvm.edu> Hi Timo & Dovecot users, We have a 2-node director setup which front-ends for 4 nodes which share a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. Approximately 40k users, but typically only a few thousand active at any time. The director nodes run sendmail, which deliver mail "locally" using LMTP to the director, which then feeds to SMTP on the real servers (also sendmail.) Why sendmail? Because procmail is used for mail filtering and as the delivery agent. Here's the problem, on the director: Mar 14 20:40:08 imapdir2 dovecot: lmtp(10692): Connect from local Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy->data_input ->eof) Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3d99a) [0x7f79156c499a] -> /usr/lib/doveco t/libdovecot.so.0(+0x3d9e6) [0x7f79156c49e6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f791569df8f] -> dovecot/lmtp() [0x406e77] -> /usr/l ib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xd4) [0x7f79156d0044] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x5b) [0x7f79156d 0c3b] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f79156cfca8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7915 6bdfc3] -> dovecot/lmtp(main+0x154) [0x403f84] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7914ef8cdd] -> dovecot/lmtp() [0x403d69] Mar 14 20:40:38 imapdir2 sendmail[6905]: q2D8KodI018432: SYSERR(root): timeout writing message to localhost: Broken pipe Most mail goes through OK, but some messages do not and end up queued until they run into the queue time limit. So far as I have been able to tell, all of the messages have this failure when the following conversation takes place between sendmail (on director), the Dovecot LMTP proxy, and sendmail on the backend node (SMTP): (names mangled to protect the guilty) (first, sendmail -> director LMTP) > [root at imapdir2 ~]# sendmail -qIq2EFZt1p004708 -v > > Running /var/spool/mqueue/qd2/q2EFZt1p004708 (sequence 1 of 1) > ... Connecting to > /var/lib/dovecot/lmtp-socket via cyrusv2... > 220 imapdir2.uvm.edu Dovecot LMTP ready > >>> LHLO imapdir2.uvm.edu > 250-imapdir2.uvm.edu > 250-8BITMIME > 250-ENHANCEDSTATUSCODES > 250 PIPELINING > >>> MAIL From: > 250 2.1.0 OK > >>> RCPT To: > >>> DATA > 250 2.1.5 OK > 354 OK > timeout writing message to localhost: Broken pipe > ... Deferred > Closing connection to localhost The conversation between the director (LMTP) and the backend (sendmail SMTP) goes like this: > 250-penguinc.uvm.edu Hello imapdir2.uvm.edu [132.198.100.150], pleased > to meet you > 250-ENHANCEDSTATUSCODES > 250-PIPELINING > 250-8BITMIME > 250-SIZE 10485760 > 250-ETRN > 250-AUTH DIGEST-MD5 CRAM-MD5 > 250-DELIVERBY > 250 HELP > MAIL FROM: > 250 2.1.0 ... > Sender ok > RCPT TO: > 552 5.2.2 User ntssdfwe mailbox is full At this point Dovecot should return the failed RCPT TO: status back to sendmail over LMTP, but instead it sits there (waiting for a timeout to expire?) and eventually dies. doveconf -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.2.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.2 (Santiago) base_dir = /var/run/dovecot/ default_client_limit = 6000 default_process_limit = 10240 director_mail_servers = penguina.uvm.edu penguinb.uvm.edu penguinc.uvm.edu penguind.uvm.edu director_servers = imapdir1.uvm.edu imapdir2.uvm.edu lmtp_proxy = yes login_trusted_networks = [REDACTED] passdb { args = proxy=y nopassword=y protocol=smtp driver = static } service anvil { client_limit = 40000 } service auth { client_limit = 45960 unix_listener auth-userdb { group = mail mode = 0660 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9090 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director service_count = 0 } service imap { process_limit = 10240 vsz_limit = 1 G } service lmtp { client_limit = 1 inet_listener lmtp { port = 24 } unix_listener /var/lib/dovecot/lmtp-socket { group = root mode = 0600 user = root } } service pop3-login { executable = pop3-login director service_count = 0 } service pop3 { process_limit = 5000 } shutdown_clients = no ssl_cert = <[REDACTED].pem ssl_key = <[REDACTED].key userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol lmtp { auth_socket_path = director-userdb } protocol imap { mail_max_userip_connections = 100 } Hope you can help, Jim Lawson From alexis.lelion at gmail.com Thu Mar 15 11:48:52 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 10:48:52 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules Message-ID: Hello, In my current setup, I have a spam filter upstream that adds a specific header - X-Spam-Level on every incoming mail. Based on this level, the mail will be moved to the user spam folder using sieve by doing "fileinto :create 'spam';" Unfortunately, some legitimate email may end up in this spam folder, so I have kind of a whitelist that performs an explicit keep over specific trusted domains. So, my complete spam filtering rule is : if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ]{ ??? keep; elseif header :contains "X-Spam-Level" ["0","1","2"] { ??? fileinto :create "__spam__"; } This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf This works as expected except that it doesn't take into account users filtering for domains that were matched for the explicit keep. For example, I have the following rule : if address :domain "From" "trusted.tld" { ??? fileinto "trusted" } But mail coming from that domain are still delivered in my mailbox. Is there something I'm missing here? I guess yes, otherwise it would work as I want ^_^ Any help/comment is appreciated Thanks! Alexis From tss at iki.fi Thu Mar 15 12:02:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:02:16 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F6144D2.2080900@uvm.edu> References: <4F6144D2.2080900@uvm.edu> Message-ID: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Hi, On 15.3.2012, at 3.24, Jim Lawson wrote: > We have a 2-node director setup which front-ends for 4 nodes which share > a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. .. > Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > (proxy->data_input > ->eof) I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. From paul at actionlans.com Thu Mar 15 02:29:19 2012 From: paul at actionlans.com (paul) Date: Thu, 15 Mar 2012 10:59:19 +1030 Subject: [Dovecot] firefox dovecot-sieve Message-ID: <1331771362.1955.1.camel@paul15.localdomain> Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org On 14.3.2012, at 4.45, paul wrote: > HI. I have just started to play with sieve and everything seems ok when > logging on using telnet localhost 4190 and an encoded username/password. > If I try to connect with firefox at localhost:4190 I get > "IMPLEMENTATION" "Dovecot Pigeonhole" > "SIEVE" "fileinto reject envelope encoded-character vacation subaddress > comparator-i;ascii-numeric relational regex imap4flags copy include > variables body enotify environment mailbox date ihave" > "NOTIFY" "mailto" > "SASL" "PLAIN LOGIN" > "STARTTLS" > "VERSION" "1.0" > OK "Dovecot ready." > NO "Error in MANAGESIEVE command received by server." > NO "Error in MANAGESIEVE command received by server." > NO "Invalid characters in atom" > BYE "Too many invalid MANAGESIEVE commands." > my dovecot -n shows > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.42.9-1.fc15.i686.PAE i686 Fedora release 15 (Lovelock) > auth_mechanisms = plain login > disable_plaintext_auth = no > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mail_privileged_group = mail > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date ihave > mbox_write_locks = fcntl > passdb { > driver = pam > } > plugin { > sieve = ~/.dovecot.sieve > sieve_dir = ~/sieve > } > protocols = imap pop3 lmtp sieve > service managesieve-login { > inet_listener sieve { > port = 4190 > } > } > ssl_cert = ssl_key = userdb { > driver = passwd > } > Have I missed something obvious or does Firefox clash with Managesieve? > Thanks. Paul > From aydin.demirel at endersys.com Thu Mar 15 12:09:11 2012 From: aydin.demirel at endersys.com (=?UTF-8?B?QXlkxLFuIERlbWlyZWw=?=) Date: Thu, 15 Mar 2012 12:09:11 +0200 Subject: [Dovecot] Login Failed Message-ID: <4F61BFC7.1000607@endersys.com> Hi; We are using scripts for login successes. Is there a feature for login failed status or can it be developed? Regards -- *Ayd?n Demirel Endersys Ltd. Sistem Destek M?hendisi/ System Support Engineer* * *<> Phone : +90 216 470 9423 | GSM : +90 530 401 8203 Fax : +90 216 470 9508 | Web : http://www.endersys.com Blog : http://blog.endersys.com Twitter : http://www.twitter.com/endersys LPI : The #1 Linux Certification for IT Professionals LPI (Linux Professional Institute) Turkey http://www.lpi-turkey.com From CMarcus at Media-Brokers.com Thu Mar 15 12:20:38 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:20:38 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60F71C.8090306@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> <4F60CEA9.3080008@Media-Brokers.com> <4F60F71C.8090306@cnpapers.com> Message-ID: <4F61C276.90309@Media-Brokers.com> On 2012-03-14 3:53 PM, Steve Campbell wrote: > I'm not sure these are virtual users, so that link may have confused me. > All accounts on these servers have real unix accounts. Their inbox is > /var/spool/mail/unix-user-name. Doesn't matter, the same thing applies... don't put mail directly in their 'home' folder, put it in a subfolder (ie, /home/user/mail)... You *will* have problems if you leave those as they are... -- Best regards, Charles From tss at iki.fi Thu Mar 15 12:29:55 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:29:55 +0200 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <4F60AF62.80702@cnpapers.com> References: <4F60AF62.80702@cnpapers.com> Message-ID: <1331807395.10319.3.camel@innu> On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: > I've mostly finished a conversion from an old Centos 3 UW-Imap server to > a new Centos 6 dovecot server. This is messy stuff to do. There are ways you could make Dovecot behave identically to UW-IMAP (mail_full_filesystem_access=yes), but for future and for security it's better if you don't do that. > I did not copy the old ~/.mailboxlist > file to ~/mail/.subscriptions file, but notice some users have the > latter file now. These are all mbox folders on the old and new server. Copying it for users who haven't already readded their subscriptions would be a good idea. > Over the years, some imap accounts had their folders directly in their > home directory and the contents of the .mailboxlist file would have an > entry with just the name of the folder in it (Trash, eg), and most had > the folders in their ~/mail folder with an entry like "mail/Trash". Our > webmail app, Horde/Imp, always seemed to take care of this. If I create > the .subscription file for the users during the move to the new server, > should I move the folders to the mail directory and amend their > .subscriptions file to reflect that change on these odd ball accounts, > and will that affect how their client is seeing these? Yes, move all of the mboxes to mail/ directory. With the compatibility namespaces it should work so that clients don't notice changes: http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility There are also a few old mailing list threads detailing all kinds of issues and solutions related to UW-IMAP -> Dovecot migration.. From tss at iki.fi Thu Mar 15 12:31:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:31:40 +0200 Subject: [Dovecot] Login Failed In-Reply-To: <4F61BFC7.1000607@endersys.com> References: <4F61BFC7.1000607@endersys.com> Message-ID: <1331807500.10319.4.camel@innu> On Thu, 2012-03-15 at 12:09 +0200, Ayd?n Demirel wrote: > We are using scripts for login successes. Is there a feature for login > failed status or can it be developed? Login failures are only visible in auth and login processes. Probably better to implement it in auth process. And there it depends on what passdb you use. You could for example switch to passdb checkpassword, which allows you to easily run scripts for both success and failure. From tss at iki.fi Thu Mar 15 12:33:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:33:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <20120314235138.GE39671@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> Message-ID: <1331807624.10319.6.camel@innu> On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: > I'm curious if anyone has any plugins for AV integration directly into > dovecot. > > Our old pop servers have been scanning messges as they're moved from > new->cur in the inbox and, at least where user's aren't poping every > few seconds, there is occasionally enough time between scanning through > the MXs to message retreval to snag a few more virues with updated > definitions before they reach customers. > > Anyone doing anything similar? http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a script that modifies a mail while it's being read. You could make it run a virus check, and if that happens you could change the virus MIME part to be full of spaces (better not to change message size, line count or MIME structure). From stephan at rename-it.nl Thu Mar 15 12:43:05 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 11:43:05 +0100 Subject: [Dovecot] firefox dovecot-sieve In-Reply-To: <1331771362.1955.1.camel@paul15.localdomain> References: <1331771362.1955.1.camel@paul15.localdomain> Message-ID: <4F61C7B9.9050605@rename-it.nl> Op 3/15/2012 1:29 AM, paul schreef: > Send this to dovecot at dovecot.org, not dovecot-owner at dovecot.org > > On 14.3.2012, at 4.45, paul wrote: > >> HI. I have just started to play with sieve and everything seems ok when logging on using telnet localhost 4190 and an encoded username/password. If I try to connect with firefox at localhost:4190 I get >> "IMPLEMENTATION" "Dovecot Pigeonhole" >> "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave" >> "NOTIFY" "mailto" >> "SASL" "PLAIN LOGIN" >> "STARTTLS" >> "VERSION" "1.0" >> OK "Dovecot ready." >> NO "Error in MANAGESIEVE command received by server." >> NO "Error in MANAGESIEVE command received by server." >> NO "Invalid characters in atom" >> BYE "Too many invalid MANAGESIEVE commands." >> Have I missed something obvious or does Firefox clash with Managesieve? Yes you have :). Firefox speaks HTTP (and quite a few other protocols), but not ManageSieve. You'll need to run a Sieve editor on your webserver if you want to edit Sieve scripts using your browser. Regards, Stephan. From Ralf.Hildebrandt at charite.de Thu Mar 15 12:44:21 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 11:44:21 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? Message-ID: <20120315104421.GW21113@charite.de> Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:47:26 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:47:26 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 Mar 15 09:51:01 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock Mar 15 09:51:01 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 during that time, I wasn't able to access the mailbox using imap. I then issued doveadm kick username and all over sudden the mailbox was accessible (via IMAP) # doveadm kick username kicked connections from the following users: username # /usr/local/scripts/find_abnormal_imap Mar 15 11:38:48 postamt dovecot: imap: Warning: Killed with signal 15 (by pid=24545 uid=0 code=kill) -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 12:47:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 12:47:07 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315104421.GW21113@charite.de> References: <20120315104421.GW21113@charite.de> Message-ID: <1331808427.10319.7.camel@innu> On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > during that time, I wasn't able to access the mailbox using imap. I > then issued Maildir? > doveadm kick username > > and all over sudden the mailbox was accessible (via IMAP) So one of them had the INBOX locked. Do you have pop3_lock_session=yes? From CMarcus at Media-Brokers.com Thu Mar 15 12:51:11 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 06:51:11 -0400 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F502485.9070503@hardwarefreak.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> Message-ID: <4F61C99F.2040409@Media-Brokers.com> On 2012-03-01 8:38 PM, Stan Hoeppner wrote: > Get yourself a qualified network architect. Pay for a full network > traffic analysis. He'll attach sniffers at multiple points in your > network to gather traffic/error/etc data. Then you'll discuss the new > office, which employees/types with move there, and you'll be able to > know almost precisely the average and peak bandwidth needs over the MAN > link. He'll very likely tell you the same thing I have, that a single > gigabit MAN link is plenty. If you hire him to do the work, he'll > program the proper QOS setup to match the traffic patterns gleaned from > the sniffers. Finally had time to properly review your answers here Stan. The time you took for the in-depth reply is very much appreciated - and I'm sure you got a kick out of the level of my ignorance... ;) As for hiring a network architect, I will absolutely be doing as you recommend (was already planning on it), but with the information I'm now armed with, at least I'll have a better chance of knowing if they know what they are doing/talking about... I'm still planning for the two physical servers (one at each location), but you have convinced me that trying to run two live mail systems is an unnecessary and even unwanted level of complexity. The DC VM will still be hot (it is always best to have two DCs in a windows domain environment anyway) so I'll get automatic real time off site backup of all of the users data (since it will all be on DFS), but for the mail services, I'll just designate one as live, and one as the hot/standby that is kept in sync using dsync. This way I'll automatically get off site back up for each site for the users data stored in the DFS, and have a second mail system ready to go if something happens to the primary. Again, thanks Stan... I am constantly amazed at the level of expertise and quality of advice available *for free* in the open source world, as is available on these lists. -- Best regards, Charles From Ralf.Hildebrandt at charite.de Thu Mar 15 13:01:18 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 12:01:18 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331808427.10319.7.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> Message-ID: <20120315110118.GX21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 11:44 +0100, Ralf Hildebrandt wrote: > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > > during that time, I wasn't able to access the mailbox using imap. I > > then issued > > Maildir? Yep. > > doveadm kick username > > > > and all over sudden the mailbox was accessible (via IMAP) > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > Yes. # makes Dovecot lock the mailbox for the whole session pop3_lock_session = yes Shouldn't be doing this I guess? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From tss at iki.fi Thu Mar 15 13:04:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 13:04:43 +0200 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <20120315110118.GX21113@charite.de> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> Message-ID: <1331809483.10319.9.camel@innu> On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > Yes. > # makes Dovecot lock the mailbox for the whole session > pop3_lock_session = yes > > Shouldn't be doing this I guess? If you do it then a single POP3 session can keep the mailbox locked pretty much forever. If you don't do it, you're violating POP3 RFC, but I don't think anyone really cares about that.. I guess this setting should really use a separate POP3-only lock when it's enabled. From stephan at rename-it.nl Thu Mar 15 13:11:16 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 12:11:16 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: Message-ID: <4F61CE54.4010607@rename-it.nl> Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > Hello, > > In my current setup, I have a spam filter upstream that adds a > specific header - X-Spam-Level on every incoming mail. Based on this > level, the mail will be moved to the user spam folder using sieve by > doing "fileinto :create 'spam';" > Unfortunately, some legitimate email may end up in this spam folder, > so I have kind of a whitelist that performs an explicit keep over > specific trusted domains. So, my complete spam filtering rule is : require ["fileinto", "mailbox"]; if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { keep; } elsif header :contains "X-Spam-Level" ["0","1","2"] { fileinto :create "__spam__"; } Fixed a few syntax issues there before I could test this. > This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is > my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf What version are you using? The above statement hints that it is recent, probably Dovecot v2.1 with matching Pigeonhole. > This works as expected except that it doesn't take into account users > filtering for domains that were matched for the explicit keep. For > example, I have the following rule : > if address :domain "From" "trusted.tld" { > fileinto "trusted" > } > But mail coming from that domain are still delivered in my mailbox. At my end, this is correctly delivered in the "trusted" folder, provided that this folder exists. Are you sure that the user's personal script even executes correctly? For example, the above script omits a ';'. The script also fails when there is n no "trusted" folder. Check the log files for errors. The default action in the event of an error is to store the message in INBOX, which may well be what you're seeing here. Regards, Stephan. From CMarcus at Media-Brokers.com Thu Mar 15 13:21:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 07:21:07 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> Message-ID: <4F61D0A3.7070503@Media-Brokers.com> On 2012-03-14 5:51 PM, Michael Grimm wrote: > You misunderstood. I was referring to system cronjob's mail reports > from cron.daily jobs like security reports et al. Those reports > normally run at identical times. But are these really 'duplicate' mails? It sounds to me like they are individual to each system. I'm also confused - are you actually delivering the exact *same* mail to two (or multiple) *different* servers simultaneously? If only one copy of the mail gets delivered, regardless of which server it gets delivered to, when dsync runs, there would be no duplicates, right? I'm asking for clarification because I was considering a similar setup. -- Best regards, Charles From campbell at cnpapers.com Thu Mar 15 13:21:17 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 07:21:17 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: References: Message-ID: <4F61D0AD.1050705@cnpapers.com> On 3/14/2012 7:33 PM, Joseph Tam wrote: > Steve Campbell writes: > >> Their imap folders, the ones that they create using an imap client or >> webmail, are either in ~ or ~/mail. Their original .mailboxlist is >> always in ~. Based on that, I should probably copy any imap folders not >> in ~/mail to that folder, duplicate ~/.mailboxlist to the file >> ~/mail/.subscriptions, and amend any .subscriptions file contents to >> just have the name of the folders (without any "mail/folder" reference >> in it). >> >> My example would then be as follows >> >> /home/steve = folder >> /home/steve/Drafts = original folder >> /home/steve/AnyFolder = original folder >> /home/steve/.mailboxlist = original file >> /home/steve/mail = folder (either original or created) >> /home/steve/mail/.subscriptions = copied contents of .mailboxlist >> file >> /home/steve/mail/Drafts = copied folder of original >> /home/steve/mail/AnyFolder = copied folder of original >> >> Contents of original .mailboxlist and new .subscriptions: >> >> Drafts >> AnyFolder >> >> If the imap folders were in ~/mail, then the original .mailboxlist would >> have been >> >> mail/Drafts >> mail/AnyFolder >> >> but after the corrections to the .subscriptions file, they would be as >> above (without reference to the mail folder). >> >> Is this correct? > > That depends -- are you aliasing namespaces so that prefix={"", > "mail/", etc.} all map to a user's ~/mail folder? You may be creating a > confusing situation where a client with a null IMAP prefix has 2 copies > of a mailbox. > > Joseph Tam I have the following set: mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u namespace { type = private separator = / prefix = "#mbox/" location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } namespace { type = private separator = / prefix = mail/ hidden = yes list = no # for v1.1+ } namespace { type = private separator = / prefix = ~/mail/ hidden = yes list = yes # for v1.1+ location = mbox:~/mail:INBOX=/var/mail/%u } namespace { type = private separator = / prefix = ~%u/mail/ hidden = yes list = no # for v1.1+ } These are mostly what's defined as the "Backward Compatability" namespaces in the wiki. Are you saying that I should probably have something like the following then: namespace { type = private separator = / prefix = location = mbox:~/mail:INBOX=/var/mail/%u inbox = yes hidden = yes list = no } And is the multiple "inbox = yes" in the differing namespaces a no-no? Based on the comments in the 10-mail.conf file, it seems to say it is a problem, but if a user has any prefix defined, even the blank prefix, wouldn't that mean they use only that set of parameters defined in the namespace being used? So far, I've only changed one prefix in the building to the #mbox prefix and that was because of the weird layout of files they had. I'm hoping one day to understand all of this. Dovecot, as I stated before, is much more complex that the imap server used previously. It allows one to use all of the facilities of the imap protocol, and much more, but unfortunately, for admins like me that are just moving to these new imap servers, most of those extras were either unknown to me or unused. Again, thanks all for the patience and help. steve From alexis.lelion at gmail.com Thu Mar 15 13:42:14 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 12:42:14 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61CE54.4010607@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> Message-ID: Hello Stephan, Thanks for your answer, and sorry for forgetting to specify which dovecot version I was using :-/ I'm using Dovecot 2.0.15, with PigeonHole. The syntax issues are some typos I made while writing this email, I double checked, and indeed, my production script was slightly different from what I wrote in the first place. I can confirm that the scripts compile properly with sievec, and also that the folder does exist, but just to be sure this is not an issue, I added the ":create" option to the user's fileinto. I have no errors in my logs, the only thing displayed is tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' Is there any way to increase verbosity for sieve only? Thanks On Thu, Mar 15, 2012 at 12:11 PM, Stephan Bosch wrote: > Op 3/15/2012 10:48 AM, Alexis Lelion schreef: > >> Hello, >> >> In my current setup, I have a spam filter upstream that adds a >> specific header - X-Spam-Level on every incoming mail. Based on this >> level, the mail will be moved to the user spam folder using sieve by >> doing "fileinto :create 'spam';" >> Unfortunately, some legitimate email may end up in this spam folder, >> so I have kind of a whitelist that performs an explicit keep over >> specific trusted domains. So, my complete spam filtering rule is : > > require ["fileinto", "mailbox"]; > > if address :domain :contains "From" ["mycompany.tld", "trusted.tld" ] { > ? ?keep; > } elsif header :contains "X-Spam-Level" ["0","1","2"] { > ? ?fileinto :create "__spam__"; > } > > Fixed a few syntax issues there before I could test this. > > >> This rule is stored in "/var/lib/dovecot/sieve/before.sieve", which is >> my "sieve_before" file as defined in /etc/dovecot/conf.d/90-sieve.conf > > What version are you using? The above statement hints that it is recent, > probably Dovecot v2.1 with matching Pigeonhole. > > >> This works as expected except that it doesn't take into account users >> filtering for domains that were matched for the explicit keep. For >> example, I have the following rule : >> if address :domain "From" "trusted.tld" { >> ? ? fileinto "trusted" >> } >> But mail coming from that domain are still delivered in my mailbox. > > > At my end, this is correctly delivered in the "trusted" folder, provided > that this folder exists. Are you sure that the user's personal script even > executes correctly? For example, the above script omits a ';'. The script > also fails when there is n no "trusted" folder. Check the log files for > errors. The default action in the event of an error is to store the message > in INBOX, which may well be what you're seeing here. > > Regards, > > Stephan. From jtl+dovecot at uvm.edu Thu Mar 15 13:50:22 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 07:50:22 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> Message-ID: <4F61D77E.8020805@uvm.edu> On 3/15/12 6:02 AM, Timo Sirainen wrote: > Hi, > > On 15.3.2012, at 3.24, Jim Lawson wrote: >> We have a 2-node director setup which front-ends for 4 nodes which share >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > .. >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >> (proxy->data_input >> ->eof) > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > I'll give it a shot. For the purposes of doing a rolling upgrade, is it reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for the duration, or should I split-brain them during the upgrade? Jim From Ralf.Hildebrandt at charite.de Thu Mar 15 14:00:40 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 15 Mar 2012 13:00:40 +0100 Subject: [Dovecot] 2.1: timeout waiting for lock? In-Reply-To: <1331809483.10319.9.camel@innu> References: <20120315104421.GW21113@charite.de> <1331808427.10319.7.camel@innu> <20120315110118.GX21113@charite.de> <1331809483.10319.9.camel@innu> Message-ID: <20120315120040.GA21113@charite.de> * Timo Sirainen : > On Thu, 2012-03-15 at 12:01 +0100, Ralf Hildebrandt wrote: > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Error: Couldn't open INBOX: Timeout while waiting for lock > > > > Mar 15 09:46:11 postamt dovecot: pop3(username): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > > So one of them had the INBOX locked. Do you have pop3_lock_session=yes? > > > > Yes. > > # makes Dovecot lock the mailbox for the whole session > > pop3_lock_session = yes > > > > Shouldn't be doing this I guess? > > If you do it then a single POP3 session can keep the mailbox locked > pretty much forever. If you don't do it, you're violating POP3 RFC, but > I don't think anyone really cares about that.. Indeed. All I care about is that the user gets his/her mail. Which he didn't. I disabled it. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From campbell at cnpapers.com Thu Mar 15 14:15:39 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 08:15:39 -0400 Subject: [Dovecot] .mailboxlist -> .subscriptions In-Reply-To: <1331807395.10319.3.camel@innu> References: <4F60AF62.80702@cnpapers.com> <1331807395.10319.3.camel@innu> Message-ID: <4F61DD6B.2020606@cnpapers.com> On 3/15/2012 6:29 AM, Timo Sirainen wrote: > On Wed, 2012-03-14 at 10:46 -0400, Steve Campbell wrote: >> I've mostly finished a conversion from an old Centos 3 UW-Imap server to >> a new Centos 6 dovecot server. > This is messy stuff to do. There are ways you could make Dovecot behave > identically to UW-IMAP (mail_full_filesystem_access=yes), but for future > and for security it's better if you don't do that. > >> I did not copy the old ~/.mailboxlist >> file to ~/mail/.subscriptions file, but notice some users have the >> latter file now. These are all mbox folders on the old and new server. > Copying it for users who haven't already readded their subscriptions > would be a good idea. > >> Over the years, some imap accounts had their folders directly in their >> home directory and the contents of the .mailboxlist file would have an >> entry with just the name of the folder in it (Trash, eg), and most had >> the folders in their ~/mail folder with an entry like "mail/Trash". Our >> webmail app, Horde/Imp, always seemed to take care of this. If I create >> the .subscription file for the users during the move to the new server, >> should I move the folders to the mail directory and amend their >> .subscriptions file to reflect that change on these odd ball accounts, >> and will that affect how their client is seeing these? > Yes, move all of the mboxes to mail/ directory. With the compatibility > namespaces it should work so that clients don't notice changes: > > http://wiki2.dovecot.org/Namespaces -> Backwards Compatibility > > There are also a few old mailing list threads detailing all kinds of > issues and solutions related to UW-IMAP -> Dovecot migration.. I'd replied to an earlier thread, and in it, I'd asked a question about a "blank" prefix namespace and the backward compatability namespaces. I'm not sure whether my "mail_location" takes precedence over namespaces (with or without a "location" parm), especially since I don't define a "blank" prefix defined. It's been working, or at least I'm not getting calls, so maybe I'm OK. In any event, I believe if I move all of these folders to ~/mail, ensure the .subscriptions file is matching, that at least people using Thunderbird will re-read the file and set their folders properly. Not sure about other clients. Thanks for the help. steve > > From tss at iki.fi Thu Mar 15 14:22:11 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:22:11 +0200 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: References: <20120308201812.2932e90c@legolas.home.ceotex.de> Message-ID: <1331814131.10319.15.camel@innu> On Thu, 2012-03-08 at 21:36 +0200, Timo Sirainen wrote: > On 8.3.2012, at 21.18, Markus Petri wrote: > > > after upgrading from 2.0.18 to 2.1.1 I noticed that I could not use > > shared folders with mutt anymore. 2.1 lists the shared namespace prefix > > once per user sharing an folder in LIST "" "%". > > > > I also noticed, that with 2.1 the user folder (Shared/) is no > > longer tagged as \NoSelect. > > > > Is this the intended behaviour and mutt simply cannot cope with it or > > is it a dovecot problem? > > Both. Dovecot shouldn't send duplicates, but mutt shouldn't break even > if it did. This is a bit difficult to fix. I'll probably leave it until v2.2. > Also Dovecot probably should add \Noselect, especially if the mailbox > isn't really selectable (there's some weirdness between shared/user > being equal to shared/user/INBOX, but I'm not sure what to do about > it). These should fix this: http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 From tss at iki.fi Thu Mar 15 14:25:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:25:12 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F61D77E.8020805@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> Message-ID: <1331814312.10319.18.camel@innu> On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: > On 3/15/12 6:02 AM, Timo Sirainen wrote: > > Hi, > > > > On 15.3.2012, at 3.24, Jim Lawson wrote: > >> We have a 2-node director setup which front-ends for 4 nodes which share > >> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. > > .. > >> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file > >> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: > >> (proxy->data_input > >> ->eof) > > I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. > > > I'll give it a shot. For the purposes of doing a rolling upgrade, is it > reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for > the duration, or should I split-brain them during the upgrade? I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The current hg version has some extra features, but it doesn't use them until all of the directors have upgraded to the new version. From tss at iki.fi Thu Mar 15 14:58:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 14:58:06 +0200 Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build In-Reply-To: References: Message-ID: <1331816286.10319.23.camel@innu> On Mon, 2012-03-05 at 00:32 +0000, Andreas M. Kirchwitz wrote: > Thanks for this patch. I've applied it to the dovecot-20120303 > nightly snapshot. The good news is, compilation works fine. > The bad news is, the libraries and binaries don't work because > they don't find the custom SSL libraries. .. > $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch > $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install You would have needed to run autogen.sh again. It works with me now that I tried in a test server with OpenSSL in non-standard dir. From mhlavink at redhat.com Thu Mar 15 15:34:32 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Thu, 15 Mar 2012 14:34:32 +0100 Subject: [Dovecot] dovecot and systemd Message-ID: <4F61EFE8.1000901@redhat.com> Hi all, dovecot supports systemd socket activation. Together with standard unit activation (like old sysv init script), there are two ways how to configure dovecot(only interface:port, not whole configuration). This can result in situation where those configurations does not say the same. Question is what should happen then? For example, lets have dovecot configured to listen for imap(s) and lets have systemd dovecot socket configured to listen for all protocols - pop3(s) and imap(s). When dovecot is configured to start on boot, systemd will start it and dovecot will listen on imap(s) ports. But when dovecot.socket is enabled, it'll listen on pop3(s) too and when new pop3 connection comes, it'll pass it to dovecot and dovecot will serve it. The question is: Should this happen? What exactly should happen when dovecot.conf does not match dovecot.socket configuration? Michal From markus at mpetri.org Thu Mar 15 15:46:27 2012 From: markus at mpetri.org (Markus Petri) Date: Thu, 15 Mar 2012 14:46:27 +0100 Subject: [Dovecot] Shared folder prefix listed multiple times with dovecot 2.1.1 In-Reply-To: <1331814131.10319.15.camel@innu> References: <20120308201812.2932e90c@legolas.home.ceotex.de> <1331814131.10319.15.camel@innu> Message-ID: <20120315144627.6173dc44@legolas.home.ceotex.de> On Thu, 15 Mar 2012 14:22:11 +0200 Timo Sirainen wrote: > > Also Dovecot probably should add \Noselect, especially if the > > mailbox isn't really selectable (there's some weirdness between > > shared/user being equal to shared/user/INBOX, but I'm not sure what > > to do about it). > > These should fix this: > > http://hg.dovecot.org/dovecot-2.1/rev/65a75939ac2c > http://hg.dovecot.org/dovecot-2.1/rev/55586f4a86f1 > Yes, those fix the problem. Thanks. From trashcan at odo.in-berlin.de Thu Mar 15 15:46:56 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 14:46:56 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61D0A3.7070503@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> Message-ID: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Hi -- On 15.03.2012 12:21, Charles Marcus wrote: > On 2012-03-14 5:51 PM, Michael Grimm > wrote: >> You misunderstood. I was referring to system cronjob's mail reports >> from cron.daily jobs like security reports et al. Those reports >> normally run at identical times. > > But are these really 'duplicate' mails? It sounds to me like they are > individual to each system. > > I'm also confused - are you actually delivering the exact *same* mail > to two (or multiple) *different* servers simultaneously? If only one > copy of the mail gets delivered, regardless of which server it gets > delivered to, when dsync runs, there would be no duplicates, right? Well, let me explain it in more detail: Given there are two servers called mx1 and mx2. They both have cron.daily jobs running, and let's say those cronjobs are meant to create at 3:00 a postfix-logwatch report on every server. Thus, the cronjob at mx1 sends his final report to the admin of mx1, and the one at mx2 to the admin of mx2. I happen to be the one who will finally receive those reports, and therefore I did tell sieve to drop them into some folder of mine, let's say REPORTS. Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the mailfolder REPORTS. Important: both mails are different but they arrive in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. And, let's call the report from mx1 cronjob "mx1-report" and that from mx2 "mx2-report". I had dsync running every minute. Thus at 3:00 the final sync has been initiated, and at 3:01 dsync will find two mails to sync in REPORTS. In 99.9% of all synchronizations the final result at both server's REPORTS mailbox is as expected and as follows: mx1-report 3:01 mx2-report 3:01 But occasionally, and what I refer to as duplicates, I did find either ... mx1-report 3:01 mx1-report 3:01 mx2-report 3:01 ... or ... mx1-report 3:01 mx2-report 3:01 mx2-report 3:01 Actually, that was when I started to investigate how dsync will behave when many mails arrive at two servers simultaneously with identical final mailboxes. The day I switched to the new replicator/dsync technique, those duplicates are history, but I'm still able to produce duplicates (and multiples) if I simultaneously produce *many* mails at every server with identical mailbox destinations in a minute (see my other report a couple of days ago). Timo is suspecting the combination of arriving mails while running dsync to be a possible cause of such duplicates, if I didn't get him wrong. Again, if your servers aren't receiving loads of mails for the very same mailboxes within very short time, the current dsync/replicator works great. HTH and regards, Michael From tss at iki.fi Thu Mar 15 15:59:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 15:59:38 +0200 Subject: [Dovecot] doveadm -A stops processing at first uid References: Message-ID: <1331819978.10319.29.camel@innu> On Mon, 2012-03-05 at 18:01 -0800, Joseph Tam wrote: > On Sun, 4 Mar 2012, Timo Sirainen writes: > > > > I would like to run various doveadm commands that involves all (mail) users like > > > > > > doveadm expunge -A mailbox Trash savedbefore 30d > > > > > > but any doveadm command that uses "-A" to iterate through all users will > > > stop processing at the first account with UID > > > What userdb are you using? userdb passwd should already skip users that > > aren't in the valid range. And what Dovecot version are you using? > > passwd-file under dovecot 2.0.16. Ah. The skipping only works in v2.1. Also you mean you're using passwd-file for /etc/passwd? You shouldn't really be doing that. From CMarcus at Media-Brokers.com Thu Mar 15 16:04:01 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 15 Mar 2012 10:04:01 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4F61F6D1.9010703@Media-Brokers.com> On 2012-03-15 9:46 AM, Michael Grimm wrote: > Thus, at 3:01 one report from mx1 will be delivered at mx1 into mailfolder > REPORTS and at 3:01 one report from mx2 will be delivered at mx2 into the > mailfolder REPORTS. Important: both mails are different but they arrive > in the mailfolder REPORTS at the same time, one at mx1 the other at mx2. > And, let's call the report from mx1 cronjob "mx1-report" and that from > mx2 "mx2-report". so these are LOCAL mails delivered to local user accounts? The easiest thing to do for this is simply alias the local address(es) so that they all go to one single server/account (I would use only virtual, but you can do it with system accounts too). I see lots of potential problems doing it the way you are doing it. -- Best regards, Charles From tss at iki.fi Thu Mar 15 16:05:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:05:29 +0200 Subject: [Dovecot] dovecot and systemd In-Reply-To: <4F61EFE8.1000901@redhat.com> References: <4F61EFE8.1000901@redhat.com> Message-ID: <1331820329.10319.32.camel@innu> On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: > What exactly should happen when > dovecot.conf does not match dovecot.socket configuration? Dovecot's systemd code was written by one of you Redhat guys. I had some similar thoughts when I applied the patch, but didn't really know what to do about it, so I didn't do anything. So: I don't know. Maybe some other project has solved this somehow already? Dovecot anyway needs its own internal UNIX listeners. Should all internal inet listeners be disabled? Could Dovecot somehow talk to systemd and ask what listeners it's using for Dovecot and log warnings if they don't match? From stephan at rename-it.nl Thu Mar 15 16:17:55 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 15:17:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: References: <4F61CE54.4010607@rename-it.nl> Message-ID: <4F61FA13.5060204@rename-it.nl> On 3/15/2012 12:42 PM, Alexis Lelion wrote: > Hello Stephan, > > Thanks for your answer, and sorry for forgetting to specify which > dovecot version I was using :-/ > I'm using Dovecot 2.0.15, with PigeonHole. > > The syntax issues are some typos I made while writing this email, I > double checked, and indeed, my production script was slightly > different from what I wrote in the first place. I can confirm that the > scripts compile properly with sievec, and also that the folder does > exist, but just to be sure this is not an issue, I added the ":create" > option to the user's fileinto. > > I have no errors in my logs, the only thing displayed is > tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into > mailbox 'INBOX' > > Is there any way to increase verbosity for sieve only? You can test Sieve outside normal delivery using the sieve-test tool; include the global sieve_before script using a -s argument. Alternatively, you can use the vnd.dovecot.debug extension as follows: require ["fileinto", "mailbox", "vnd.dovecot.debug"]; if address :domain "From" "trusted.tld" { fileinto :create "trusted"; debug_log "Tried to save in \"trusted\""; } You need to add the vnd.dovecot.debug extension to sieve_extensions in your 90-sieve.conf, e.g.: sieve_extensions = +vnd.dovecot.debug This will produce the following output in the user's personal sieve log (typically ~/.dovecot.sieve.log): sieve: info: started log at Mar 15 15:13:29. main_script: line 5: info: DEBUG: Tried to save in "trusted". info: msgid=unspecified: stored mail into mailbox 'trusted'. If the DEBUG line is missing at your end, the fileinto is not executed at all. If it is, and things are still delivered in INBOX, something else is going on. Regards, Stephan. From trashcan at odo.in-berlin.de Thu Mar 15 16:24:01 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 15:24:01 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4F61F6D1.9010703@Media-Brokers.com> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4F61F6D1.9010703@Media-Brokers.com> Message-ID: Hi -- On 15.03.2012 15:04, Charles Marcus wrote: > On 2012-03-15 9:46 AM, Michael Grimm > wrote: >> Thus, at 3:01 one report from mx1 will be delivered at mx1 into >> mailfolder >> REPORTS and at 3:01 one report from mx2 will be delivered at mx2 >> into the >> mailfolder REPORTS. Important: both mails are different but they >> arrive >> in the mailfolder REPORTS at the same time, one at mx1 the other at >> mx2. >> And, let's call the report from mx1 cronjob "mx1-report" and that >> from >> mx2 "mx2-report". > > so these are LOCAL mails delivered to local user accounts? All locally produced mails are aliased to the very same virtual user, namely myself. > The easiest thing to do for this is simply alias the local > address(es) > so that they all go to one single server/account (I would use only > virtual, but you can do it with system accounts too). That is exactly what I'm doing, I'm running virtual, only. No local user accounts here. Every locally produced system mail end in virtual mailboxes of myself. In the given example "mx1-report" is delivered to REPORTS at mx1 and "mx2-report" to REPORTS at mx2. Now, I want to access them via IMAP for instance at my mx1 mail account. Without dsync I would only be able to access "mx1-report", thus I do need to sync REPORTS to see both at mx1. > I see lots of potential problems doing it the way you are doing it. Hmm, now, I don't understand you. Regards, Michael From alexis.lelion at gmail.com Thu Mar 15 16:38:55 2012 From: alexis.lelion at gmail.com (Alexis Lelion) Date: Thu, 15 Mar 2012 15:38:55 +0100 Subject: [Dovecot] sieve.before script is taking preceedence over user defined rules In-Reply-To: <4F61FA13.5060204@rename-it.nl> References: <4F61CE54.4010607@rename-it.nl> <4F61FA13.5060204@rename-it.nl> Message-ID: Thanks for this useful information, I will give it a try On Thu, Mar 15, 2012 at 3:17 PM, Stephan Bosch wrote: > On 3/15/2012 12:42 PM, Alexis Lelion wrote: >> >> Hello Stephan, >> >> Thanks for your answer, and sorry for forgetting to specify which >> dovecot version I was using :-/ >> I'm using Dovecot 2.0.15, with PigeonHole. >> >> The syntax issues are some typos I made while writing this email, I >> double checked, and indeed, my production script was slightly >> different from what I wrote in the first place. I can confirm that the >> scripts compile properly with sievec, and also that the folder does >> exist, but just to be sure this is not an issue, I added the ":create" >> option ?to the user's fileinto. >> >> I have no errors in my logs, the only thing displayed is >> tThbJ1myYU+ZPwAA6RJXdw: sieve: msgid=unspecified: stored mail into >> mailbox 'INBOX' >> >> Is there any way to increase verbosity for sieve only? > > > You can test Sieve outside normal delivery using the sieve-test tool; > include the global sieve_before script using a -s argument. > > Alternatively, you can use the vnd.dovecot.debug extension as follows: > > require ["fileinto", "mailbox", "vnd.dovecot.debug"]; > > > if address :domain "From" "trusted.tld" { > ?fileinto :create "trusted"; > ?debug_log "Tried to save in \"trusted\""; > } > > You need to add the vnd.dovecot.debug extension to sieve_extensions in your > 90-sieve.conf, e.g.: > > sieve_extensions = +vnd.dovecot.debug > > This will produce the following output in the user's personal sieve log > (typically ~/.dovecot.sieve.log): > > sieve: info: started log at Mar 15 15:13:29. > main_script: line 5: info: DEBUG: Tried to save in "trusted". > info: msgid=unspecified: stored mail into mailbox 'trusted'. > > If the DEBUG line is missing at your end, the fileinto is not executed at > all. If it is, and things are still delivered in INBOX, something else is > going on. > > Regards, > > Stephan. > > From tss at iki.fi Thu Mar 15 16:53:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 16:53:53 +0200 Subject: [Dovecot] v2.1.2 released Message-ID: <1331823233.10319.40.camel@innu> http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig There are a ton of proxying related improvements in this release. You should now be able to do pretty much anything you want with Dovecot proxy/director. This release also includes the initial version of dsync-based replication. I'm already successfully using it for @dovecot.fi mails, but it still has some problems. See http://dovecot.org/list/dovecot/2012-March/064243.html for some details how to configure it. + Initial implementation of dsync-based replication. For now this should be used only on non-critical systems. + Proxying: POP3 now supports sending remote IP+port from proxy to backend server via Dovecot-specific XCLIENT extension. + Proxying: proxy_maybe=yes with host= (instead of IP) works now properly. + Proxying: Added auth_proxy_self setting + Proxying: Added proxy_always extra field (see wiki docs) + Added director_username_hash setting to specify what part of the username is hashed. This can be used to implement per-domain backends (which allows safely accessing shared mailboxes within domain). + Added a "session ID" string for imap/pop3 connections, available in %{session} variable. The session ID passes through Dovecot IMAP/POP3 proxying to backend server. The same session ID is can be reused after a long time (currently a bit under 9 years). + passdb checkpassword: Support "credentials lookups" (for non-plaintext auth and for lmtp_proxy lookups) + fts: Added fts_index_timeout setting to abort search if indexing hasn't finished by then (default is to wait forever). - doveadm sync: If mailbox was expunged empty, messages may have become back instead of also being expunged in the other side. - director: If user logged into two directors while near user expiration, the directors might have redirected the user to two different backends. - imap_id_* settings were ignored before login. - Several fixes to mailbox_list_index=yes - Previous v2.1.x didn't log all messages at shutdown. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tss at iki.fi Thu Mar 15 17:23:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 17:23:50 +0200 Subject: [Dovecot] v2.0.19 released Message-ID: <1331825030.10319.42.camel@innu> http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.19.tar.gz.sig Hopefully one of the last v2.0.x releases. - IMAP: ENABLE CONDSTORE/QRESYNC + STATUS for a mailbox might not have seen latest external changes to it, like new mails. - imap_id_* settings were ignored before login. - doveadm altmove did too much work sometimes, retrying moves it had already done. - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. From tom at talpey.com Thu Mar 15 18:04:44 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 12:04:44 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 Message-ID: <4F62131C.2090008@talpey.com> I'm seeing a few warnings emitted when building for x86. They're pretty obvious, but if you want the configure options etc, I can provide those. In Dovecot 2.1.2 (I also see some of these in 2.1.1): 1) src/lib-index/mail-cache-fields.c (comparison between two last_used fields) mail-cache-fields.c: In function 'mail_cache_header_fields_read': mail-cache-fields.c:406: warning: comparison between signed and unsigned 2) src/director/user-directory.c (comparison with ioloop_time) user-directory.c: In function 'user_directory_user_is_recently_updated': user-directory.c:147: warning: comparison between signed and unsigned 3) src/replication/replicator/replicator-brain.c (comparison with ioloop_time) replicator-brain.c: In function 'doveadm_replicate': replicator-brain.c:113: warning: comparison between signed and unsigned 4) src/replication/replicator/replicator-queue.c (comparison with ioloop_time) replicator-queue.c: In function 'replicator_queue_pop': replicator-queue.c:201: warning: comparison between signed and unsigned In Pigeonhole 0.3.0: 5) src/managesieve-login/client-authenticate.c (passing size_t * not uoff_t *) client-authenticate.c: In function 'managesieve_client_auth_read_response': client-authenticate.c:214: warning: passing argument 3 of 'i_stream_get_size' from incompatible pointer type From tss at iki.fi Thu Mar 15 18:25:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 18:25:21 +0200 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F62131C.2090008@talpey.com> References: <4F62131C.2090008@talpey.com> Message-ID: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> On 15.3.2012, at 18.04, Tom Talpey wrote: > I'm seeing a few warnings emitted when building for x86. They're pretty > obvious, but if you want the configure options etc, I can provide those. > > In Dovecot 2.1.2 (I also see some of these in 2.1.1): Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. From dluke at geeklair.net Thu Mar 15 18:33:20 2012 From: dluke at geeklair.net (Daniel J. Luke) Date: Thu, 15 Mar 2012 12:33:20 -0400 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: On Mac OS X 10.5.8 / darwin 9.8.0, I'm getting this error on startup again: dovecot[74267]: master: Fatal: kevent(EV_ADD, READ, 19) failed: Invalid argument dovecot.conf contains: service stats { fifo_listener stats-mail { mode = 0 } } which fixed the issue with 2.1.1 adding the following seems to have fixed things: service aggregator { fifo_listener replication-notify-fifo { mode = 0 } } from looking at config/all-settings.c it looks like I should maybe also add the following (but I have not tried it). service director { fifo_listener login/proxy-notify { mode = 0 } } It would be really nice if this failed more gracefully so the config tweaks weren't necessary. (I can work on a patch if it's something that would be accepted and if someone can point me in the right direction). -- Daniel J. Luke +========================================================+ | *---------------- dluke at geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ From mcazzador at gmail.com Thu Mar 15 18:42:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 17:42:03 +0100 Subject: [Dovecot] replication howto Message-ID: Hello, excuse me but there is some documentation about replication now? I dont' understand where i must put the lines below (dovecot.conf? , 20-imap?) Excuse but it's not so clear for me cause i'm a new dovecot user. Another question, i use virtual users on mysql backend , so for replication i need to give ssh at every virtual users? Or i can use a only use a system ssh user? Thank's service aggregator { # give enough permissions for mail processes fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup process_min_avail = 1 } plugin { # host1 replicates to host2 mail_replica = remote:vmail at host2.example.com # host2 replicates to host1 #mail_replica = remote:vmail at host1.example.com } #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service doveadm { # if you're using a single virtual user, set this to # start ssh as vmail (not root) user = vmail } -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From trashcan at odo.in-berlin.de Thu Mar 15 19:09:21 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:09:21 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi -- On 15.03.2012, at 17:42, Matteo Cazzador wrote: > Hello, excuse me but there is some documentation about replication now? Not that I'm aware of. > I dont' understand where i must put the lines below (dovecot.conf? , > 20-imap?) You can put them wherever you wish, as long as you include that part of your configuration. Myself, I'm still using a single dovecot.conf, only. > Another question, i use virtual users on mysql backend , so for > replication i need to give ssh at every virtual users? > Or i can use a only use a system ssh user? If I'm not mistaken, you can use a single ssh user, and you could use the vmail user for instance. That's what I do, and I'm using sqlite for userdb. Here's my configuration: ----------------------------------------------------------------------- If you choose to run ssh on a different port from the default one, you need: ## ssh command line used in dsync replication (ssh port added) # dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} If not, you can start here: ## --- DSYNC REPLICATION ---------------------------------------- # # aggregator, replicator, doveadm, and config needed, and # dsync_remote_cmd if running ssh via non-default port # service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } The following part is for server 1, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server2.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } The following part is for server 2, only: ## --- PLUGINS ---------------------------------------- # # dsync replication plugin # plugin { # this host replicates to remote host # mail_replica = remote:vmail at server1.domain # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hours } HTH, Michael From mcazzador at gmail.com Thu Mar 15 19:16:17 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:16:17 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: Hi, thank's a lot! for your detailed answer. About ssh (excuse for my english) i think you correctly understand what is "my problem" with virtual user (i have no system user ) and there are not ssh account. So i must use a dedicate account for replication (ssh) that must act sync for all virtual mail account. Thank' s i try you suggest now! Il 15 marzo 2012 18:09, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 17:42, Matteo Cazzador wrote: > >> Hello, excuse me but there is some documentation about replication now? > > Not that I'm aware of. > >> I dont' understand where i must put the lines below (dovecot.conf? , >> 20-imap?) > > You can put them wherever you wish, as long as you include that part > of your configuration. Myself, I'm still using a single dovecot.conf, > only. > >> Another question, i use virtual users on mysql backend , so for >> replication i need to give ssh at every virtual users? >> Or i can use a only use a system ssh user? > > If I'm not mistaken, you can use a single ssh user, and you could use > the vmail user for instance. That's what I do, and I'm using sqlite for > userdb. > > Here's my configuration: > ----------------------------------------------------------------------- > > If you choose to run ssh on a different port from the default one, you need: > > ? ## ssh command line used in dsync replication (ssh port added) > ? # > ? dsync_remote_cmd = ssh -p 1234 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > > > If not, you can start here: > > ? ## --- DSYNC REPLICATION ---------------------------------------- > ? # > ? # aggregator, replicator, doveadm, and config needed, and > ? # dsync_remote_cmd if running ssh via non-default port > ? # > ? service aggregator { > ? ? ? ?# give enough permissions for mail processes > ? ? ? ?# > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? } > ? service replicator { > ? ? ? ?# start replication at startup > ? ? ? ?# > ? ? ? ?process_min_avail = 1 > ? } > ? service doveadm { > ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail > ? ? ? ?# (not root) > ? ? ? ?# > ? ? ? ?user = vmail > ? } > ? service config { > ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm > ? ? ? ?# > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > ? } > > > > The following part is for server 1, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server2.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > > > The following part is for server 2, only: > > ? ## --- PLUGINS ---------------------------------------- > ? # > ? # dsync replication plugin > ? # > ? plugin { > ? ? ? ?# this host replicates to remote host > ? ? ? ?# > ? ? ? ?mail_replica = remote:vmail at server1.domain > > ? ? ? ?# run full synchronization mode every other hour > ? ? ? ?# (default is every 24 hours) > ? ? ? ?# > ? ? ? ?replication_full_sync_interval = 1 hours > ? } > > HTH, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jtl+dovecot at uvm.edu Thu Mar 15 19:23:01 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:23:01 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <1331814312.10319.18.camel@innu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> Message-ID: <4F622575.7050405@uvm.edu> On 3/15/12 8:25 AM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 07:50 -0400, Jim Lawson wrote: >> On 3/15/12 6:02 AM, Timo Sirainen wrote: >>> Hi, >>> >>> On 15.3.2012, at 3.24, Jim Lawson wrote: >>>> We have a 2-node director setup which front-ends for 4 nodes which share >>>> a clustered filesystem (GFS). All nodes run Dovecot 2.0.18. >>> .. >>>> Mar 14 20:40:38 imapdir2 dovecot: lmtp(10692): Panic: file >>>> lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: >>>> (proxy->data_input >>>> ->eof) >>> I pretty much rewrote the LMTP proxying code in v2.1, so there's a very good chance that it's already been fixed. >>> >> I'll give it a shot. For the purposes of doing a rolling upgrade, is it >> reasonable to expect a 2.0.18 director to peer with a 2.1.1 director for >> the duration, or should I split-brain them during the upgrade? > I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The > current hg version has some extra features, but it doesn't use them > until all of the directors have upgraded to the new version. > Trying with v2.1.2 (peer is v2.0.18): Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left == NULL && dir->right == NULL)) Mar 15 13:15:53 imapdir2 dovecot: director: Fatal: master: service(director): child 513 killed with signal 6 (core not dumped) Mar 15 13:15:53 imapdir2 dovecot: director: Error: Director 132.198.100.149:9090/right disconnected Which is OK, I can run them split-brained (rules in iptables to prevent directors from talking) while I move users around. It'll mean poor performance for GFS for the duration, but that's better than an outage. The good news is, the lmtp problem I wrote about above appears to be fixed. Thanks !!! Jim From trashcan at odo.in-berlin.de Thu Mar 15 19:28:37 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 18:28:37 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: Message-ID: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:16, Matteo Cazzador wrote: > with virtual user (i have no system user ) and there are not ssh > account. So i must use a dedicate account for replication (ssh) > that must act sync for all virtual mail account. Yes, that's what I use. I did create a dedicated account for vmail with all the necessary ssh stuff in ~vmail/.ssh One remark I forgot to mention in my last mail: >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> # >> user = vmail >> } This part is only needed, if you choose to run device doveadm as user vmail like I do. >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> # >> unix_listener config { >> user = vmail >> } >> } Regards, Michael From andrei at lctax.ro Thu Mar 15 19:49:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 13:49:58 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: > The day I switched to the new replicator/dsync technique, those > duplicates > are history, but I'm still able to produce duplicates (and multiples) > if Hello, Can you get a little bit more in details about this replicator/dsync techique? As my main problem is that EVERYTHING (that gets created on different servers in the same time) gets duplicated. I only do replication using the doveadm sync command. My servers are geographically distributed as you might remember from previous posts so I run doveadm every 5 minutes, and only 1 instance of doveadm runs at any given times (so let's say that due to a HUGE volume the doveamd take 30 minutes to complete, then all in-between 5minutes are skipped). Thnx, Andrei From tss at iki.fi Thu Mar 15 19:52:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:52:58 +0200 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <4F622575.7050405@uvm.edu> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> Message-ID: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> On 15.3.2012, at 19.23, Jim Lawson wrote: >> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >> current hg version has some extra features, but it doesn't use them >> until all of the directors have upgraded to the new version. >> > Trying with v2.1.2 (peer is v2.0.18): > > Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line > 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left > == NULL && dir->right == NULL)) This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? From tss at iki.fi Thu Mar 15 19:53:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 19:53:57 +0200 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: On 15.3.2012, at 19.49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. > > I only do replication using the doveadm sync command. Try at least v2.1.2 first, since it has some fixes. Also post your doveconf -n output. From jtl+dovecot at uvm.edu Thu Mar 15 19:55:57 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 15 Mar 2012 13:55:57 -0400 Subject: [Dovecot] director lmtp -> smtp problem In-Reply-To: <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> References: <4F6144D2.2080900@uvm.edu> <08C251F0-40E4-46CD-8D25-C5B04129DE1A@iki.fi> <4F61D77E.8020805@uvm.edu> <1331814312.10319.18.camel@innu> <4F622575.7050405@uvm.edu> <0DC0FED9-8900-402C-AE70-7E00A35042BA@iki.fi> Message-ID: <4F622D2D.80802@uvm.edu> On 3/15/12 1:52 PM, Timo Sirainen wrote: > On 15.3.2012, at 19.23, Jim Lawson wrote: > >>> I'm almost certain that v2.1.1 talks compatible protocol with v2.0. The >>> current hg version has some extra features, but it doesn't use them >>> until all of the directors have upgraded to the new version. >>> >> Trying with v2.1.2 (peer is v2.0.18): >> >> Mar 15 13:15:53 imapdir2 dovecot: director: Panic: file director.c: line >> 295 (director_sync): assertion failed: (!dir->ring_synced || (dir->left >> == NULL && dir->right == NULL)) > This points to a more generic problem. How did this happen? You have two directors, stopped & upgraded one, started it up and it crashed? > That's correct. Configs are the same between directors (same as I sent in the original msg) Jim From mcazzador at gmail.com Thu Mar 15 19:57:22 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Thu, 15 Mar 2012 18:57:22 +0100 Subject: [Dovecot] replication howto In-Reply-To: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi, yes it'a good idea but i'm using now root i hope this not invalid all I obtain this error but maybe i need some pause Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF Thank's a lot! Il 15 marzo 2012 18:28, Michael Grimm ha scritto: > Hi -- > > On 15.03.2012, at 18:16, Matteo Cazzador wrote: > >> with virtual user (i have no system user ) and there are not ssh >> account. So i must use a dedicate account for replication (ssh) >> that must act sync for all virtual mail account. > > Yes, that's what I use. I did create a dedicated account for vmail > with all the necessary ssh stuff in ~vmail/.ssh > > One remark I forgot to mention in my last mail: > >>> ? service doveadm { >>> ? ? ? ?# if you're using a single virtual user, set this to start ssh as vmail >>> ? ? ? ?# (not root) >>> ? ? ? ?# >>> ? ? ? ?user = vmail >>> ? } > > This part is only needed, if you choose to run device doveadm as user > vmail like I do. > >>> ? service config { >>> ? ? ? ?# needed to grant access to /var/run/dovecot/config for service doveadm >>> ? ? ? ?# >>> ? ? ? ?unix_listener config { >>> ? ? ? ? ? ? ? ?user = vmail >>> ? ? ? ?} >>> ? } > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From campbell at cnpapers.com Thu Mar 15 21:06:31 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 15:06:31 -0400 Subject: [Dovecot] Lack of external documentation? Message-ID: <4F623DB7.9060707@cnpapers.com> Firstly, this isn't meant to be critical, and I realize the subject line probably suggest criticism, so... I was sort of forced into using dovecot as my imap/pop server due to upgrading 3 versions of OS on my mail servers. So far, that's not bad. What surprises me is that one of the first things I usually do whenever I start using different software is to purchase a book that seems to suit me. Searching all of the common places like amazon, ebay, etc for manuals turned up little to nothing on dovecot. I'm wondering why and is this so new that people just haven't written books about it yet? The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Thanks for all the help I've received so far and I think I'm really going to like dovecot. Once I get the hang of it, I'll probably reduce the amount of noise on the list by half. steve campbell From terry at cnysupport.com Thu Mar 15 21:27:37 2012 From: terry at cnysupport.com (Terry Carmen) Date: Thu, 15 Mar 2012 15:27:37 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <4F6242A9.6090209@cnysupport.com> On 03/15/2012 03:06 PM, Steve Campbell wrote: > Firstly, this isn't meant to be critical, and I realize the subject > line probably suggest criticism, so... > > I was sort of forced into using dovecot as my imap/pop server due to > upgrading 3 versions of OS on my mail servers. So far, that's not bad. > What surprises me is that one of the first things I usually do > whenever I start using different software is to purchase a book that > seems to suit me. Searching all of the common places like amazon, > ebay, etc for manuals turned up little to nothing on dovecot. > > I'm wondering why and is this so new that people just haven't written > books about it yet? > > The one thing I'm a little critical of, though, is that trying to make > heads or tails of dovecot by following the online documentation is a > little problematic. I'm constantly jumping to another page and then > back to the original page, and for the most part, I just don't know > enough about it all yet to know what I'm looking for. The best docs are on the wiki and this mailing list. If you find the information in the wiki to be lacking, the best thing you can do is find the solution yourself and/or on this mailing list, and then make a wiki entry so the next person will know how to solve the same problem you had. Dovecot is a complex piece of software, and understanding some functionality requires reading the wiki, asking on the mailing list and/or examining the source code. You can also obtain paid support from these companies: http://dovecot.org/support.html I'll be the first to admit that complex and specialized configurations are sometimes difficult to figure out, however this list has always been a tremendous amount of help. Terry From stan at hardwarefreak.com Thu Mar 15 21:44:57 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 15 Mar 2012 14:44:57 -0500 Subject: [Dovecot] Multiple locations, 2 servers - planning questions... In-Reply-To: <4F61C99F.2040409@Media-Brokers.com> References: <4F4BB559.6050405@Media-Brokers.com> <4F4EDBBF.40004@hardwarefreak.com> <4F4F60F3.5050508@Media-Brokers.com> <4F502485.9070503@hardwarefreak.com> <4F61C99F.2040409@Media-Brokers.com> Message-ID: <4F6246B9.5080309@hardwarefreak.com> On 3/15/2012 5:51 AM, Charles Marcus wrote: > On 2012-03-01 8:38 PM, Stan Hoeppner wrote: >> Get yourself a qualified network architect. Pay for a full network >> traffic analysis. He'll attach sniffers at multiple points in your >> network to gather traffic/error/etc data. Then you'll discuss the new >> office, which employees/types with move there, and you'll be able to >> know almost precisely the average and peak bandwidth needs over the MAN >> link. He'll very likely tell you the same thing I have, that a single >> gigabit MAN link is plenty. If you hire him to do the work, he'll >> program the proper QOS setup to match the traffic patterns gleaned from >> the sniffers. > > Finally had time to properly review your answers here Stan. > > The time you took for the in-depth reply is very much appreciated - and Multi-site setups can be tricky as they often temp folks to do unnecessary things they otherwise would not. Just trying to help keep your sails pointed in the right direction. :) #1 rule when building a multi-site network: only duplicate hardware and services at the remote site(s) when absolutely necessary. > I'm sure you got a kick out of the level of my ignorance... ;) Not at all. I'm sure there is some subject or another where you would demonstrate my ignorance. From another perspective, if there was no ignorance left on the planet then there would be nothing left for anyone to learn. That would make for a boring world. > As for hiring a network architect, I will absolutely be doing as you > recommend (was already planning on it), but with the information I'm now > armed with, at least I'll have a better chance of knowing if they know > what they are doing/talking about... Now that you are aware of network analysis using sniffers, allow me to throw you a curve ball. For a network of your size, less than 70 users IIRC, with a typical application mix but with SMB/NFS traffic/file sizes a little above 'average', a qualified engineer probably won't need to plug sniffers into your network to determine the size MAN pipe and what traffic shaping you'll need. He'll have already done a near identical setup dozens of times. The good news is this saves you a few grand. Analysis with sniffers ain't cheap, even for small networks. And sniffers are normally only deployed to identify the cause of network problems, not very often for architectural or capacity planning. But, asking him about doing a full analysis using sniffers, and hearing his response, may lead to a valuable discussion nonetheless. Have your MAN and internet providers' (if not the same company) pricing sheet(s) in hand when you meet with the engineer. Depending on fast ethernet MAN, GbE MAN, and internet pipe pricing, he may have some compelling options/recommendations for you, possibly quite different, less costly, and more redundant than what you have been considering up to this point. > I'm still planning for the two physical servers (one at each location), Again, if you don't _need_ hardware and services at the 2nd site to achieve the current service level at the primary site, do not add these things to the 2nd site. I really want to put a bunch of exclamation points here but I hate exclamation points in technical emails--actually I just hate them, period. ;) > but you have convinced me that trying to run two live mail systems is an > unnecessary and even unwanted level of complexity. Running an active/active Dovecot cluster doesn't guarantee an unnecessary nor unwanted additional complexity. The need for clustering should go through a justification process just like anything else: what's the benefit, total 'cost', what's the ROI, etc. Lots of people here do active/active clustering every day with great success. Connecting the cluster nodes over a MAN link, however, does introduce unnecessary complexity. Locating one node in another building many blocks away is unnecessary. Putting the nodes in the same rack/room is smart, and easily accomplished in your environment, gives you the redundancy above, but without the potentially problematic MAN link as the cluster interconnect. Granted you'll need to build two new (preferably identical) systems from scratch and setup shared storage (DRBD or a SAN array) and GFS2 or OCFS, etc. Given your environment, there are only two valid reasons for locating equipment and duplicating data and services at a remote site: 1. Unrecoverable network failure (due to single MAN link) 2. Unrecoverable primary site failure (natural or man made disaster) #1 is taken care of by redundant MAN links #2 you've never planned for to this date (probability is *low*) and you need _everything_ duplicated at the remote site Duplicating servers for high(er) user throughput/lower latency to/from servers isn't a valid reason for remote site duplication in your case because you are able to afford plenty of bandwidth and link redundancy between the sites. The relative low cost and high bandwidth of the MAN link outweighs any benefit of service replication due to the latter's complexity level. Here are some other 'rules': 1. Don't duplicate servers at remote sites to mitigate network link failure when sites are close and redundant bandwidth is afforadable 2. Do duplicate network links to mitigate link failure when sites are close and bandwidth is affordable 3. Implement and test a true disaster avoidance and recovery plan > The DC VM will still > be hot (it is always best to have two DCs in a windows domain > environment anyway) so I'll get automatic real time off site backup of > all of the users data (since it will all be on DFS), but for the mail > services, I'll just designate one as live, and one as the hot/standby > that is kept in sync using dsync. This way I'll automatically get off > site back up for each site for the users data stored in the DFS, and > have a second mail system ready to go if something happens to the primary. Again, you're not looking at this network design from the proper perspective. See rules 1-3 above. Off site backups/replication are used exclusively to mitigate data loss due to catastrophic facility failure, not server failure, enabling rapid system recovery when new equipment has arrived. Many business insurers have catastrophic IT equipment replacement plans and relationships with the big 5 hardware vendors, enabling you to get new new equipment racked and begin your restore from offsite tape, within as little as 24 hours of notification. Think of how FEMA stages emergency supplies all around the country. Now think 10 times better, faster. Such services increase your premiums, but if you're serious about disaster avoidance and recovery, this is the only way to go. IBM, HP, maybe Dell, Sun (used to anyway), have dedicated account reps for disaster recovery. They work with you to keep an inventory of all of your systems and storage. Your records are constantly updated when your products are EOL'd or superseded or you replace or add hardware, and a list is maintained of current hardware best matched to replace all of your now burned, flooded, tornado shredded, hurricane blasted equipment, right down to bare metal restore capability, if possible/applicable. You plan to replicate filesystem user data and mailbox data to a 2nd site to mitigate single server failures. Why does that need to be done to an offsite location/system? It doesn't. There is no benefit whatsoever. You can accomplish this in the same rack/room and get by with a smaller MAN pipe saving time, money, and administrative burden. The restore procedure will be faster if all machines are in the same rack/room and you're using tape, and you won't slow users down with restore traffic going over the MAN link. If you really want off-site backup, for what it's meant to accomplish, get a network attached tape library/silo, or a speedy high cap LTO-4/5 tape drive in each server, put a real backup rotation and restore plan in place, and store backup tapes in a secure facility. A remote "hot site" is great when it's in a different city, better yet region, or in a hardened facility in any locale. Your hot site is only a few blocks away. If your primary site it taken out by anything other than fire, such as a tornado, earthquake, hurricane being more likely in your case, chances are your hot site may go down soon after the primary. If you want/need a real off site backup solution, rotate tapes to an everything-proof facility. Here are 3 companies in the Atlanta area that offer media rotation storage services. Watch the Offsite Tape Vaulting video at IronMountain: http://www.ironmountain.com/Knowledge-Center/Reference-Library/View-by-Document-Type/Demonstrations-Videos/Tours/Offsite-Tape-Vaulting.aspx http://www.askads.net/media-rotation/ http://www.adamsdatamanagement.com/tape-rotation-atlanta-ga.htm > Again, thanks Stan... I am constantly amazed at the level of expertise > and quality of advice available *for free* in the open source world, as > is available on these lists. Always glad to assist my brethren in this digital kingdom. Whichever architecture/topology you choose, remote replicated systems or not, I hope my input has given you some good information on which to base your decisions. -- Stan From list at airstreamcomm.net Thu Mar 15 21:48:57 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 15 Mar 2012 14:48:57 -0500 Subject: [Dovecot] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> On Thu, 15 Mar 2012 16:53:53 +0200, Timo Sirainen wrote: > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. Are there any performance metrics around dsync replication, such as how many users this has been tested on, or how long the replication take to occur? Also I have not been able to determine from reading the mailinglist whether or not dsync replication works with different types of mailboxes (maildir, dbox, mbox), what is supported? From tss at iki.fi Thu Mar 15 21:55:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 21:55:57 +0200 Subject: [Dovecot] v2.1.2 released In-Reply-To: <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> References: <1331823233.10319.40.camel@innu> <0e153c2894a556889762f16315149caf@mail.airstreamcomm.net> Message-ID: <6489C385-E8C3-425E-8D2D-B3A242A6E0AF@iki.fi> On 15.3.2012, at 21.48, wrote: > Are there any performance metrics around dsync replication, such as how > many users this has been tested on, or how long the replication take to > occur? The performance isn't optimal yet. You can probably replicate some hundreds of users ok, maybe thousands, but depends. > Also I have not been able to determine from reading the mailinglist > whether or not dsync replication works with different types of mailboxes > (maildir, dbox, mbox), what is supported? Maildir and dbox is supported, mbox probably works okayish but since it doesn't have proper message GUIDs you could run into trouble. From tom at talpey.com Thu Mar 15 22:23:19 2012 From: tom at talpey.com (Tom Talpey) Date: Thu, 15 Mar 2012 16:23:19 -0400 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> Message-ID: <4F624FB7.9000408@talpey.com> On 3/15/2012 12:25 PM, Timo Sirainen wrote: > On 15.3.2012, at 18.04, Tom Talpey wrote: > >> I'm seeing a few warnings emitted when building for x86. They're pretty >> obvious, but if you want the configure options etc, I can provide those. >> >> In Dovecot 2.1.2 (I also see some of these in 2.1.1): > > Thanks, fixed in hg. I guess I should add x86 vm building these nightly as well.. Confirmed, Dovecot builds cleanly for me now. Thanks Timo! The pigeonhole warning appears to be harmless and I'll wait for Stefan to confirm/address. From giles at coochey.net Thu Mar 15 22:30:19 2012 From: giles at coochey.net (Giles Coochey) Date: Thu, 15 Mar 2012 20:30:19 +0000 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F62515B.1050207@coochey.net> On 15/03/2012 19:27, Terry Carmen wrote: > On 03/15/2012 03:06 PM, Steve Campbell wrote: >> Firstly, this isn't meant to be critical, and I realize the subject >> line probably suggest criticism, so... >> >> I was sort of forced into using dovecot as my imap/pop server due to >> upgrading 3 versions of OS on my mail servers. So far, that's not >> bad. What surprises me is that one of the first things I usually do >> whenever I start using different software is to purchase a book that >> seems to suit me. Searching all of the common places like amazon, >> ebay, etc for manuals turned up little to nothing on dovecot. >> >> I'm wondering why and is this so new that people just haven't written >> books about it yet? >> >> The one thing I'm a little critical of, though, is that trying to >> make heads or tails of dovecot by following the online documentation >> is a little problematic. I'm constantly jumping to another page and >> then back to the original page, and for the most part, I just don't >> know enough about it all yet to know what I'm looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always > been a tremendous amount of help. > > Terry > > > > > > What he said +1. I don't want to be-little IMAP software or the work that Timo has done to get dovecot to the IMAP server world, but IMAP in general is a small enough subject to only really warrant two maybe three books - the most recent of which was written 5-7 years ago. The original release of dovecot was around 2002, but I don't think it became as widely adopted as Courier / Cyrus until around 2010. I wouldn't be surprised that if there is a next edition of "The Book of IMAP" or the O'reilly "Managing IMAP" that there would probably be an equal share section on dovecot than any other server out there. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From jerry at seibercom.net Thu Mar 15 22:46:18 2012 From: jerry at seibercom.net (Jerry) Date: Thu, 15 Mar 2012 16:46:18 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F6242A9.6090209@cnysupport.com> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <20120315164618.705ca356@scorpio> On Thu, 15 Mar 2012 15:27:37 -0400 Terry Carmen articulated: > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > Firstly, this isn't meant to be critical, and I realize the subject > > line probably suggest criticism, so... > > > > I was sort of forced into using dovecot as my imap/pop server due > > to upgrading 3 versions of OS on my mail servers. So far, that's > > not bad. What surprises me is that one of the first things I > > usually do whenever I start using different software is to purchase > > a book that seems to suit me. Searching all of the common places > > like amazon, ebay, etc for manuals turned up little to nothing on > > dovecot. > > > > I'm wondering why and is this so new that people just haven't > > written books about it yet? > > > > The one thing I'm a little critical of, though, is that trying to > > make heads or tails of dovecot by following the online > > documentation is a little problematic. I'm constantly jumping to > > another page and then back to the original page, and for the most > > part, I just don't know enough about it all yet to know what I'm > > looking for. > > The best docs are on the wiki and this mailing list. If you find the > information in the wiki to be lacking, the best thing you can do is > find the solution yourself and/or on this mailing list, and then make > a wiki entry so the next person will know how to solve the same > problem you had. > > Dovecot is a complex piece of software, and understanding some > functionality requires reading the wiki, asking on the mailing list > and/or examining the source code. You can also obtain paid support > from these companies: http://dovecot.org/support.html > > I'll be the first to admit that complex and specialized > configurations are sometimes difficult to figure out, however this > list has always been a tremendous amount of help. The lack of truly informative documentation has been the Achilles' heel of open-source software since its inception. I feel your pain. I have always loved a hard copy, i.e. book documenting the subject I am studying. Jumping from screen to screen sucks, plus how do I highlight a passage on the monitor for future reference? There have been a few books written to document Postfix, but to the best of my knowledge, none exist for Dovecot. -- Jerry ? Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __________________________________________________________________ From e-frog at gmx.de Thu Mar 15 22:46:22 2012 From: e-frog at gmx.de (e-frog) Date: Thu, 15 Mar 2012 21:46:22 +0100 Subject: [Dovecot] 2.1.1: doveadm backup errors In-Reply-To: <4F60F29D.2010409@gmx.de> References: <4F5F9521.2060206@gmx.de> <4F5FABE9.3080200@gmx.de> <1331732490.2081.127.camel@innu> <4F60F29D.2010409@gmx.de> Message-ID: <4F62551E.1000102@gmx.de> On 14.03.2012 20:33, wrote e-frog: > On 14.03.2012 14:41, wrote Timo Sirainen: >> >> With latest hg version it should work. >> > > Hi Timo, > > The "can't delete mailbox INBOX" error is gone now with changeset > c077ca9bc306 and it's working successfully on the account from yesterday > where it also worked with mailbox_list_index=no. > > However using a different account (more mail and mailboxes) I'm seeing > dbox corruption errors. I have tested with mailbox_list_index=yes and no > and it's the same for both. So this might be unrelated to this setting. > Attached are logs from doveadm backup runs. First to an empty directory > and 2 consecutive runs. > Further testing (now with 2.1.2) shows it only seems to work for a single mailbox. e.g. doveadm -v backup -u testuser at ubuntu-test.localdomain -m 'INBOX' mdbox:/tmp/backup dsync(testuser at ubuntu-test.localdomain): Info: INBOX: only in source (guid=c63f581c030b774b572a0000ec8d17cd) -> no errors This works for every single mailbox in this account. The errors only occur without -m 'mailbox'. Using maildir as destination format however seems to work fine on the whole account. doveadm -v backup -u testuser at ubuntu-test.localdomain maildir:/tmp/backup -> no errors Thanks, e-frog From trashcan at odo.in-berlin.de Thu Mar 15 22:48:31 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:48:31 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: Hi -- On 15.03.2012, at 18:57, Matteo Cazzador wrote: > Hi, yes it'a good idea but i'm using now root i hope this not > invalid all Actually it's a bad idea to use root for ssh from a security point of view. A hacked root account isn't fun. Thus, normally one needs to explicitly change the config of the sshd daemon to allow root logins (at least with FreeBSD what I'm using). Thus, I do recommend to use an unprivileged user like vmail. > I obtain this error but maybe i need some pause ;-) > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found root doesn't not find doveadm at the remote server. As mentioned above you better create an account for vmail and allow that user to find doveadm in its path. > Mar 15 18:55:28 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF That's an error due to not finding doveadm at the remote site. Regards, Michael From trashcan at odo.in-berlin.de Thu Mar 15 22:55:17 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Thu, 15 Mar 2012 21:55:17 +0100 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Hi -- On 15.03.2012, at 18:49, Michescu Andrei wrote: > Can you get a little bit more in details about this replicator/dsync > techique? http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html and http://www.dovecot.org/img/dsync-director-replication-ssh.png helped me a lot understand the idea behind it. > As my main problem is that EVERYTHING (that gets created on > different servers in the same time) gets duplicated. As Timo recommended already, you better upgrade to 2.1.2 first. I can confirm that he fixed a lot compared to older dsync versions. Regards, Michael From tss at iki.fi Thu Mar 15 23:01:39 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:01:39 +0200 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <4F623DB7.9060707@cnpapers.com> References: <4F623DB7.9060707@cnpapers.com> Message-ID: <67E4C4F8-A9CE-4912-9B3F-05770041C383@iki.fi> On 15.3.2012, at 21.06, Steve Campbell wrote: > The one thing I'm a little critical of, though, is that trying to make heads or tails of dovecot by following the online documentation is a little problematic. I'm constantly jumping to another page and then back to the original page, and for the most part, I just don't know enough about it all yet to know what I'm looking for. Perhaps it would be helpful to have some more talkative howtos for some of the typical configurations, that don't only list the options that are given but actually talks about why things are done the way they are? I've tried to avoid duplication of text in wiki, because if something changes it's difficult to update it everywhere, but in howtos I guess it wouldn't be too bad. Or maybe the wiki could be restructured in some way to make it easier to follow. I think I'm the worst possible person to figure out anything like that, because I don't know what the difficult parts are. I'd think the Dovecot wiki is good if you know what you want to do and just want to know some specifics, but I guess it can be difficult to figure things out otherwise. > Does anyone know of any manuals/books that have been written that might introduce me to most of the stuff in dovecot? A few people have talked about writing a Dovecot book and I've promised to help them, but no one's actually written one as far as I know. > So far, the list has been great, but once the "dsync" threads started popping up, I find there's even more I don't know about. Features that aren't yet even fully implemented don't really have documentation for them. From tss at iki.fi Thu Mar 15 23:05:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:05:54 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> Message-ID: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> On 15.3.2012, at 22.48, Michael Grimm wrote: > On 15.03.2012, at 18:57, Matteo Cazzador wrote: > >> Hi, yes it'a good idea but i'm using now root i hope this not >> invalid all > > Actually it's a bad idea to use root for ssh from a security point > of view. A hacked root account isn't fun. Thus, normally one needs > to explicitly change the config of the sshd daemon to allow root > logins (at least with FreeBSD what I'm using). Thus, I do recommend > to use an unprivileged user like vmail. Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. From hoogendyk at bio.umass.edu Thu Mar 15 23:13:34 2012 From: hoogendyk at bio.umass.edu (Chris Hoogendyk) Date: Thu, 15 Mar 2012 17:13:34 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <4F625B7E.5060902@bio.umass.edu> On 3/15/12 4:46 PM, Jerry wrote: > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > >> On 03/15/2012 03:06 PM, Steve Campbell wrote: >>> Firstly, this isn't meant to be critical, and I realize the subject >>> line probably suggest criticism, so... >>> >>> I was sort of forced into using dovecot as my imap/pop server due >>> to upgrading 3 versions of OS on my mail servers. So far, that's >>> not bad. What surprises me is that one of the first things I >>> usually do whenever I start using different software is to purchase >>> a book that seems to suit me. Searching all of the common places >>> like amazon, ebay, etc for manuals turned up little to nothing on >>> dovecot. >>> >>> I'm wondering why and is this so new that people just haven't >>> written books about it yet? >>> >>> The one thing I'm a little critical of, though, is that trying to >>> make heads or tails of dovecot by following the online >>> documentation is a little problematic. I'm constantly jumping to >>> another page and then back to the original page, and for the most >>> part, I just don't know enough about it all yet to know what I'm >>> looking for. >> The best docs are on the wiki and this mailing list. If you find the >> information in the wiki to be lacking, the best thing you can do is >> find the solution yourself and/or on this mailing list, and then make >> a wiki entry so the next person will know how to solve the same >> problem you had. >> >> Dovecot is a complex piece of software, and understanding some >> functionality requires reading the wiki, asking on the mailing list >> and/or examining the source code. You can also obtain paid support >> from these companies: http://dovecot.org/support.html >> >> I'll be the first to admit that complex and specialized >> configurations are sometimes difficult to figure out, however this >> list has always been a tremendous amount of help. > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. I like books, but, especially in the case of actively developed software such as Dovecot, they become outdated very quickly. I have two editions of the Unix System Administration Handbook (can't remember the last time I looked at them), and two editions of Backup & Recovery. I use Amanda for backup. It has been developed actively over the last several years, and the Backup & Recovery chapter on Amanda is sorely out of date. The wiki, the users mailing list, and the man pages are the only way to really be up-to-date. With the book, you won't know anything about any changes or additions since the book was written, which would have been at least many months before it was published. I'm into online documentation every day. I'm a Solaris admin, but I've been jumping from Solaris 10 to Ubuntu without any books, and I've been jumping from ZFS to LVM without any books. That's a significant transition. But it seems I can find almost everything online. Sometimes another admin gives me an explanation and a link. It's just the way things are. The digital world is moving too fast to be frozen in print. -- --------------- Chris Hoogendyk - O__ ---- Systems Administrator c/ /'_ --- Biology& Geology Departments (*) \(*) -- 140 Morrill Science Center ~~~~~~~~~~ - University of Massachusetts, Amherst --------------- Erd?s 4 From amk at spamfence.net Thu Mar 15 23:43:02 2012 From: amk at spamfence.net (Andreas M. Kirchwitz) Date: Thu, 15 Mar 2012 21:43:02 +0000 (UTC) Subject: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build References: <1331816286.10319.23.camel@innu.invalid> Message-ID: Timo Sirainen wrote: >> $ patch -p1 -s < ../dovecot-20120303-e540404debb7.patch >> $ env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/Dovecot-20120303 --with-ssl=openssl --with-ssldir=/usr/local/Dovecot-20120303/etc/dovecot/certs && make && make install > > You would have needed to run autogen.sh again. It works with me now that > I tried in a test server with OpenSSL in non-standard dir. Sorry, I didn't know that with "autogen.sh". Just grabbed Dovecot 2.1.2 (which is all properly set up - so I couldn't do anything wrong ;-) and compiled it. Compilation works. Great! The binaries find all their libraries. But two libraries are not quite okay. They don't find their SSL libs: libdovecot-lda.so libdovecot-storage.so Since libdovecot-lda.so doesn't contain the words libssl or libcrypto, I guess that ldd just complains because it uses libdovecot-storage.so. Thus, libdovecot-storage.so is the (only) one left with an incomplete library search path. Luckily, all binaries use some additional libraries which come with a proper library path. So the whole things works, but it's more like some kind of magic. It would be great if libdovecot-storage.so could be fixed as well to make things finally perfect. Thanks for all your effort. (I know this isn't top priority as most people use precompiled stuff and never run into such kind of things.) Greetings, Andreas From p at state-of-mind.de Thu Mar 15 23:46:59 2012 From: p at state-of-mind.de (Patrick Ben Koetter) Date: Thu, 15 Mar 2012 22:46:59 +0100 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <20120315214658.GC3750@state-of-mind.de> * Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. Dovecot is a moving target and it is hard to produce any print that represents what Dovecot can do when the print finally will be released. I know, because I am one of the two authors who wrote "The Book of Postfix" and we found it hard if almost impossible to keep up with Wietse's pace when he wrote major parts of Postfix. For now, I believe, the wiki and the mailing list is as good as it gets. Later when Dovecot settles a book might be something to write and something to spend money on because it lasts for a while. man pages would be a good thing, but given Dovecots configuration syntax and flexibility this might be an even harder task. Its probably easier to describe certain aspects of configuration or use cases than list all options and their possible occurences. p at rick -- state of mind () http://www.state-of-mind.de Franziskanerstra?e 15 Telefon +49 89 3090 4664 81669 M?nchen Telefax +49 89 3090 4666 Amtsgericht M?nchen Partnerschaftsregister PR 563 From david at blue-labs.org Thu Mar 15 23:49:54 2012 From: david at blue-labs.org (David Ford) Date: Thu, 15 Mar 2012 17:49:54 -0400 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <4F626402.4030606@blue-labs.org> in ~privilgeduser/.ssh/authorized keys: from= cmd=dsync.sh pubkey... On 03/15/2012 05:05 PM, Timo Sirainen wrote: > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. And if you allow ssh > login only with public key authentication I don't think there are much > security issues. And finally, it would be possible to write a small > wrapper that allows the root's public key auth to only execute > dsync-user.sh script that can't do anything except sync a specified > user's mails. From andrei at lctax.ro Thu Mar 15 23:52:39 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 17:52:39 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> Message-ID: <64c9b1a4813862ad254f591c6a5ffc02.squirrel@web.miau.ca> Hello Timo, I have update the repository with hg pull -u, recompiled and redeployed and somehow the dovecot -n still shows 2.1.1... :( I ran exactly the same test: starting for 1 clean user1, I create 2 emails, one on mx1.a and one on mx2.a and I sync them with doveadm. The output is exactly as previously sent :( Here is my conf: # 2.1.1: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } Thank you, Andrei > On 15.3.2012, at 19.49, Michescu Andrei wrote: > >> Can you get a little bit more in details about this replicator/dsync >> techique? As my main problem is that EVERYTHING (that gets created on >> different servers in the same time) gets duplicated. >> >> I only do replication using the doveadm sync command. > > Try at least v2.1.2 first, since it has some fixes. Also post your > doveconf -n output. > > > !DSPAM:4f622cb881591647615726! > > From tss at iki.fi Thu Mar 15 23:55:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 15 Mar 2012 23:55:26 +0200 Subject: [Dovecot] replication howto In-Reply-To: <4F626402.4030606@blue-labs.org> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Plus the scripts that 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) and 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. On 15.3.2012, at 23.49, David Ford wrote: > in ~privilgeduser/.ssh/authorized keys: > > from= cmd=dsync.sh pubkey... > > On 03/15/2012 05:05 PM, Timo Sirainen wrote: >> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. > From stephan at rename-it.nl Fri Mar 16 00:17:40 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Thu, 15 Mar 2012 23:17:40 +0100 Subject: [Dovecot] Compiler warnings in dovecot-2.1.2 and pigeonhole 0.3.0 In-Reply-To: <4F624FB7.9000408@talpey.com> References: <4F62131C.2090008@talpey.com> <7146CEBF-4794-4EEB-8F6D-21272151976F@iki.fi> <4F624FB7.9000408@talpey.com> Message-ID: <4F626A84.1070705@rename-it.nl> On 3/15/2012 9:23 PM, Tom Talpey wrote: > On 3/15/2012 12:25 PM, Timo Sirainen wrote: >> On 15.3.2012, at 18.04, Tom Talpey wrote: >> >>> I'm seeing a few warnings emitted when building for x86. They're pretty >>> obvious, but if you want the configure options etc, I can provide >>> those. >>> >>> In Dovecot 2.1.2 (I also see some of these in 2.1.1): >> >> Thanks, fixed in hg. I guess I should add x86 vm building these >> nightly as well.. > > Confirmed, Dovecot builds cleanly for me now. Thanks Timo! > > The pigeonhole warning appears to be harmless and I'll wait for Stefan > to confirm/address. Thanks, fixed: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/75c1a2fd9b26 Regards, Stephan. From andrei at lctax.ro Fri Mar 16 01:20:00 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Thu, 15 Mar 2012 19:20:00 -0400 Subject: [Dovecot] problems with SSH-based clustering dovecot 2.1.1 In-Reply-To: <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> References: <2771442c9d1b0e130e10ea6db7a16fec.squirrel@web.miau.ca> <7838F62F-BF33-4D66-B44F-0E0A36DB2E37@iki.fi> <9ABA688F-E73B-45C9-B369-2703999D7EFD@iki.fi> <1331734189.2081.137.camel@innu> <7D79ED97-E114-466E-9966-27D542D92CE8@odo.in-berlin.de> <6ddf6434eeb40113178b80e2d97ec7a3.squirrel@web.miau.ca> <6947BFFF-C255-4E89-9DD4-06CD30FD02BA@odo.in-berlin.de> <4F61D0A3.7070503@Media-Brokers.com> <34f1ed7bd80091a33ab9fca46a0e831f@mx1.enfer-du-nord.net> <4EEED55B-C0BE-4126-8467-EBA2C5D1D987@odo.in-berlin.de> Message-ID: hello, So I upgraded to 2.1.2 (not from repository because that one still says 2.1.1, but from the release). I ran exactly the same test with exactly the same behaviour. (new account, synced successfully on 2 servers, deliver 1 email to each server, run doveadm sync)... Please find below the dovecot-uidlists: on mx1.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852540.19862.mx2,S=272 2 G1331852540.19862.mx2,S=272 :1331852573.M89342P19877.mx2,S=272 3 :1331852488.30409.mx1,S=268 on mx2.a: 3 V1331851700 N1 Gc9e2a526b471624f70760000498f706b 1 :1331852488.30409.mx1,S=268 2 :1331852540.19862.mx2,S=272 3 G1331852488.30409.mx1,S=268 :1331852572.M622052P30410.mx1,S=268 As you can see both servers duplicated the email that was delivered first to them (1 in both cases, because the user1 is a clean account). There is the same effect in the folders: initial there is only one file on each server and after sync there are 3 files instead of only 2... Also, after the sync, there should be 2 new emails (N2 if I interpret correctly that N1 means only one new). Thank you. Andrei PS: also I need to run dsync twice, because first time I receive: dsync-local(user1 at a): Info: INBOX: Ignored 1 modseq changes dsync-local(user1 at a): Info: INBOX: Couldn't keep all uids dsync-local(user1 at a): Warning: Mailbox changes caused a desync. You may want to run dsync again. The config is below: # 2.1.2: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 2.6.38-b i686 Slackware 13.0.0.0.0 auth_debug = yes auth_debug_passwords = yes auth_default_realm = a first_valid_gid = 89 first_valid_uid = 89 last_valid_gid = 89 last_valid_uid = 89 listen = * log_path = /dev/stderr login_greeting = WebMail MX1.A login_trusted_networks = 192.168.20.64/26 mail_debug = yes mail_gid = vchkpw mail_location = maildir:~/Maildir mail_privileged_group = vchkpw mail_uid = vpopmail passdb { driver = vpopmail } protocols = imap pop3 service auth-worker { unix_listener auth-worker { user = vpopmail } user = vpopmail } service auth { user = vpopmail } service imap-login { user = vpopmail } service pop3-login { user = vpopmail } ssl = no userdb { driver = vpopmail } From gedalya at gedalya.net Fri Mar 16 01:55:09 2012 From: gedalya at gedalya.net (Gedalya) Date: Thu, 15 Mar 2012 19:55:09 -0400 Subject: [Dovecot] IMAP to Maildir Migration preserving UIDs? In-Reply-To: <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> Message-ID: <4F62815D.7020002@gedalya.net> On 01/28/2012 12:45 PM, Timo Sirainen wrote: > On 27.1.2012, at 2.00, Gedalya wrote: > >> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> >> Program received signal SIGSEGV, Segmentation fault. >> mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 >> 213 mailbox-log.c: No such file or directory. >> in mailbox-log.c > This crash is now fixed, so there's no need to give /tmp/imapc path anymore: > http://hg.dovecot.org/dovecot-2.1/rev/7b94d1c8a6e7 > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current versions are putting the body of the last message in "Sent Items" in place of every single email in INBOX. In other words, for every email that sits in INBOX in the source, I get a copy of the last email in "Sent Items" instead. This happens for every account I try to migrate. Very strange. I noticed this only now, and the last package I have left in the local apt cache which still works is 2.1.rc7-0~auto+0. From ml at smtp.fakessh.eu Fri Mar 16 02:31:59 2012 From: ml at smtp.fakessh.eu (ml) Date: Fri, 16 Mar 2012 01:31:59 +0100 Subject: [Dovecot] [Dovecot-news] v2.1.2 released In-Reply-To: <1331823233.10319.40.camel@innu> References: <1331823233.10319.40.camel@innu> Message-ID: <1331857919.16694.5.camel@localhost> Le jeudi 15 mars 2012 ? 16:53 +0200, Timo Sirainen a ?crit : > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz > http://dovecot.org/releases/2.1/dovecot-2.1.2.tar.gz.sig > > There are a ton of proxying related improvements in this release. You > should now be able to do pretty much anything you want with Dovecot > proxy/director. > > This release also includes the initial version of dsync-based > replication. I'm already successfully using it for @dovecot.fi mails, > but it still has some problems. See > http://dovecot.org/list/dovecot/2012-March/064243.html for some details > how to configure it. > > + Initial implementation of dsync-based replication. For now this > should be used only on non-critical systems. > + Proxying: POP3 now supports sending remote IP+port from proxy to > backend server via Dovecot-specific XCLIENT extension. > + Proxying: proxy_maybe=yes with host= (instead of IP) > works now properly. > + Proxying: Added auth_proxy_self setting > + Proxying: Added proxy_always extra field (see wiki docs) > + Added director_username_hash setting to specify what part of the > username is hashed. This can be used to implement per-domain > backends (which allows safely accessing shared mailboxes within > domain). > + Added a "session ID" string for imap/pop3 connections, available > in %{session} variable. The session ID passes through Dovecot > IMAP/POP3 proxying to backend server. The same session ID is can be > reused after a long time (currently a bit under 9 years). > + passdb checkpassword: Support "credentials lookups" (for > non-plaintext auth and for lmtp_proxy lookups) > + fts: Added fts_index_timeout setting to abort search if indexing > hasn't finished by then (default is to wait forever). > - doveadm sync: If mailbox was expunged empty, messages may have > become back instead of also being expunged in the other side. > - director: If user logged into two directors while near user > expiration, the directors might have redirected the user to two > different backends. > - imap_id_* settings were ignored before login. > - Several fixes to mailbox_list_index=yes > - Previous v2.1.x didn't log all messages at shutdown. > - mbox: Fixed accessing Dovecot v1.x mbox index files without errors. > > i build with succes the last release for centos 5 work fine and best ns.fakessh.eu/rpms/dovecot-2.1.2-1.centme.el5.src.rpm thanks Timo -- http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC2626742 gpg --keyserver pgp.mit.edu --recv-key C2626742 http://urlshort.eu fakessh @ http://gplus.to/sshfake http://gplus.to/sshswilting http://gplus.to/john.swilting https://lists.fakessh.eu/mailman/ This list is moderated by me, but all applications will be accepted provided they receive a note of presentation -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From campbell at cnpapers.com Fri Mar 16 03:08:15 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Thu, 15 Mar 2012 21:08:15 -0400 Subject: [Dovecot] Lack of external documentation? In-Reply-To: <20120315164618.705ca356@scorpio> References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> <20120315164618.705ca356@scorpio> Message-ID: <1331860095.4f62927f9acd0@perdition.cnpapers.net> Quoting Jerry : > On Thu, 15 Mar 2012 15:27:37 -0400 > Terry Carmen articulated: > > > On 03/15/2012 03:06 PM, Steve Campbell wrote: > > > Firstly, this isn't meant to be critical, and I realize the subject > > > line probably suggest criticism, so... > > > > > > I was sort of forced into using dovecot as my imap/pop server due > > > to upgrading 3 versions of OS on my mail servers. So far, that's > > > not bad. What surprises me is that one of the first things I > > > usually do whenever I start using different software is to purchase > > > a book that seems to suit me. Searching all of the common places > > > like amazon, ebay, etc for manuals turned up little to nothing on > > > dovecot. > > > > > > I'm wondering why and is this so new that people just haven't > > > written books about it yet? > > > > > > The one thing I'm a little critical of, though, is that trying to > > > make heads or tails of dovecot by following the online > > > documentation is a little problematic. I'm constantly jumping to > > > another page and then back to the original page, and for the most > > > part, I just don't know enough about it all yet to know what I'm > > > looking for. > > > > The best docs are on the wiki and this mailing list. If you find the > > information in the wiki to be lacking, the best thing you can do is > > find the solution yourself and/or on this mailing list, and then make > > a wiki entry so the next person will know how to solve the same > > problem you had. > > > > Dovecot is a complex piece of software, and understanding some > > functionality requires reading the wiki, asking on the mailing list > > and/or examining the source code. You can also obtain paid support > > from these companies: http://dovecot.org/support.html > > > > I'll be the first to admit that complex and specialized > > configurations are sometimes difficult to figure out, however this > > list has always been a tremendous amount of help. > > The lack of truly informative documentation has been the Achilles' heel > of open-source software since its inception. I feel your pain. I have > always loved a hard copy, i.e. book documenting the subject I am > studying. Jumping from screen to screen sucks, plus how do I highlight > a passage on the monitor for future reference? There have been a few > books written to document Postfix, but to the best of my knowledge, none > exist for Dovecot. > > -- > Jerry ??? > > Disclaimer: off-list followups get on-list replies or get ignored. > Please do not ignore the Reply-To header. > __________________________________________________________________ > So many great replies, but I'll pick this one to use as my reply-to since it mirrors mostly how I feel about my experiences so far when it comes to learning Dovecot. I installed a new server, going from Centos 3 to Centos 6. I found that Postfix was the preferred SMTP server and Dovecot was the preferred imap/pop server. I gave Postfix my best shot, but didn't really have it tested well enough to stick with it, so I dropped back to Sendmail, something I'm somewhat familiar with. I've read multiple versions of O'Reilly's Sendmail books along with the Sendmail Cookbook. I have to admit that it was these books that made me realize the power of Sendmail. Post l website to further learn, but I had to get the basics first to do what needed to be done to get the job into a working server. Dovecot is an application that probably would work out of the box for me if I didn't have to use data from the previous server. So I had to use more than the standard options to make this work. Finding those options was the main gripe I had with the wiki - there are just so many options to make Dovecot the complete server. That's a good thing. Just remember, us noobies-to-Dovecot have to discover all of those options. I mentioned that I was happy with the wiki and the list when it comes to answering my questions. But I'm sure the list will get tired of me asking what must appear to be redundant, simple, obnoxious questions. The index-like wiki page is most helpful. I knew dovecot has been around for a while, but didn't know how mature it was. The fact that Centos/Red Hat uses it as a default says quite a bit about it's reliability, so I'll stick with it. One of the the things I was planning on doing was combining two servers, which services one domain on one server and services two other domains on the other, into one server, and have the other as a server-in-waiting. So along comes this dsync thread, and now it appears that Dovecot might make that all easier. I see all the potential Dovecot has, but learning it is a little difficult for us new users. Once I get the hang of it, I'm sure I want need to search for the things I need to find, but for now, a good book would have been nice and a lot easier. I give all the praise to Timo that he deserves. (I'm guessing he's either the developer, the lead guru on the list or something of that stature). I like what I'm seeing, I'm just not always seeing what I need. Again, this is not critical in nature. I'm just stating what this particular rookie is conveying to the list about my experience (and lack of experience) in getting where I need to be with Dovecot. Thanks for such a great application, all the great replies and help so far. steve ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From lists at wiesinger.com Fri Mar 16 08:02:10 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Fri, 16 Mar 2012 07:02:10 +0100 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage Message-ID: <4F62D762.7080607@wiesinger.com> Hello, After fixing configuration and other issues I'm still having one problem with imap executable and pine: less .pinerc # Changed config: #rsh-command=/usr/sbin/dovecot --exec-mail imap rsh-command=/usr/local/bin/imap Calling imap still fails as non root: imap /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied collect2: ld returned 1 exit statusn Any ideas to fix it? BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Thnx. Ciao, Gerhard From tss at iki.fi Fri Mar 16 10:05:20 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 10:05:20 +0200 Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <4F62D762.7080607@wiesinger.com> References: <4F62D762.7080607@wiesinger.com> Message-ID: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: > After fixing configuration and other issues I'm still having one problem with imap executable and pine: > less .pinerc > # Changed config: > #rsh-command=/usr/sbin/dovecot --exec-mail imap > rsh-command=/usr/local/bin/imap That's correct. > Calling imap still fails as non root: > imap > /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied > collect2: ld returned 1 exit statusn Huh? That looks like imap is running ld to link something. It shouldn't be doing that. > BTW: What is the recommended dovecot storage for dovecot 2.x (upgradeable from mbox)? Maildir for reliability, sdbox/mdbox for performance. From mcazzador at gmail.com Fri Mar 16 10:37:47 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 09:37:47 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, thank's everybody, today afternoon i apply the suggest and i test solution. I post the actual configuration that i will test: vmail users is present too, i create ssh-keygen for users vmail and relative home directory and permit ssh with no password with user vmail on two servers. Then i use the configuration below i leave comment the line below or i need to active it excuse but i don't understand clear cause my terrible english? #dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} and apply this on two servers service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } } Thank's everyboy Il 15 marzo 2012 22:55, Timo Sirainen ha scritto: > Plus the scripts that > > 1) when calling ssh dsync first writes the username to stdout (before dsync starts communicating) > > and > > 2) dsync.sh on remote first reads the username from stdin, before execing dsync itself > > Because it's not possible to give -u $username parameter in the authorized_keys cmd itself. That's the only changing parameter that is needed. > > On 15.3.2012, at 23.49, David Ford wrote: > >> in ~privilgeduser/.ssh/authorized keys: >> >> from= cmd=dsync.sh pubkey... >> >> On 03/15/2012 05:05 PM, Timo Sirainen wrote: >>> Then again it's safer to use system user accounts than a single vmail account that has access to everyone's emails. And if you allow ssh login only with public key authentication I don't think there are much security issues. And finally, it would be possible to write a small wrapper that allows the root's public key auth to only execute dsync-user.sh script that can't do anything except sync a specified user's mails. >> > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From jernej.porenta at arnes.si Fri Mar 16 11:09:07 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Fri, 16 Mar 2012 10:09:07 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1331735355.2081.140.camel@innu> References: <1331735355.2081.140.camel@innu> Message-ID: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> On Mar 14, 2012, at 3:29 PM, Timo Sirainen wrote: > On Tue, 2012-03-06 at 14:28 +0100, Jernej Porenta wrote: >> Heya, >> >> We are expiriencing issues with dovecot 2.1.1 on Linux with weird >> filenames in home directory of username. We are using mbox IMAP >> folders, with no special changes (mail_location = mbox:~/:INBOX=% >> h/.mailbox). >> >> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >> (uni_utf8_str_is_valid(vname)) > .. >> AFAIK, the problem lies in processing the file list of home folder, >> which can contain filenames that do not have proper UTF-8 encoding of >> filenames, which causes dovecot to crash. > > Yes, Dovecot shouldn't crash even if there are non-UTF8 mailboxes. This > should fix it by renaming such mailboxes: > http://hg.dovecot.org/dovecot-2.1/rev/c077ca9bc306 We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. Whenever . LIST "" "*" is issued, dovecot crashes: Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b5466f3119c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Any clues? > >> On the other hand, UTF-8 filenames created on the system by hand >> (using touch), are not displayed in IMAP LIST command (sample is >> included in the folder structure; single letter file). > > This is a bit trickier problem. The mailbox names are currently stored > in filesystem as IMAP's modified-UTF7. So it's not really even currently > supposed to work, although it's not very nice that the mailboxes aren't > visible either. Maybe I'll do something smart in future for this, like > allowing both mUTF-7 and UTF-8 and remembering per-mailbox which > formatting it is in. I think we can leave this issue out, since I don't believe the users will be creating folders directly from interactive SSH sessions and rather use IMAP to create folders. So, this use-case is not very likely to occur. Thank you for your help... Cheers, Jernej From nmilas at noa.gr Fri Mar 16 11:26:45 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 11:26:45 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x Message-ID: <4F630755.7070909@noa.gr> Hi, A quick question: Are there any incompatibilities in config settings among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require changes of any config settings (as upgrading from v1.x to 2.0.x did)? Thanks, Nick From mstevens at imt-systems.com Fri Mar 16 11:39:53 2012 From: mstevens at imt-systems.com (Morten Stevens) Date: Fri, 16 Mar 2012 10:39:53 +0100 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <4F630755.7070909@noa.gr> References: <4F630755.7070909@noa.gr> Message-ID: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> On 16.03.2012 10:26, Nikolaos Milas wrote: > Hi, > > A quick question: Are there any incompatibilities in config settings > among versions 2.0.x and 2.1.x (and subsequently v2.2.x)? > > That is, upgrading Dovecot 2.0.x to 2.1.x software, will also require > changes of any config settings (as upgrading from v1.x to 2.0.x did)? Hi, See: http://wiki2.dovecot.org/Upgrading/2.1 Best regards, Morten From nicku at nicku.org Fri Mar 16 12:08:36 2012 From: nicku at nicku.org (Nick Urbanik) Date: Fri, 16 Mar 2012 21:08:36 +1100 Subject: [Dovecot] imaptest: performance testing Message-ID: <20120316100836.GA12049@nicku.org> Dear Folks, Using head of imaptest with dovecot 2.1, I am attempting to put a heavy load on the server, with insufficient success. I made 15000 user accounts, put them in to a file, one per line. Then I ran imaptest with ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 pass=SECRETPASSWORD But CPU load is only reaching 20. I put an mbox containing 67 messages into ~/mail/dovecot-crlf So what is the best way to put a really heavy load on an imap server? The documentation describes testing correctness of the server without really emphasising performance. I would like to do something like dnsperf, which adds a linearly increasing load until the server is unable to cope. This machine has 24G RAM and 8 cores. I'll be grateful for any constructive suggestions. -- Nick Urbanik http://nicku.org nicku at nicku.org GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24 From odhiambo at gmail.com Fri Mar 16 12:51:46 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 13:51:46 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 Message-ID: I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) to Apple Mail. Having looked at all options, I have resorted to the, perhaps, most difficult way: Create folders on the IMAP server, copy e-mails into them from Outlook, connect Apple Mail and do the reverse. However, I have hit a wall. I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. In my first attempt, I have 1792 messages in the "Sent Items" folder for Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items folder, but the process dies at some point. When that happens, Outlook pops a screen saying "IMAP session state is inconsistent, please relogin". Dovecot says: Mar 16 13:30:26 jaribu dovecot: master: Warning: Killed with signal 15 (by pid=72242 uid=0 code=kill) Mar 16 13:30:29 master: Info: Dovecot v2.1.2 starting up Mar 16 13:31:15 auth-worker(72594): Info: mysql(localhost): Connected to database exim4u Mar 16 13:31:15 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72595 Mar 16 13:31:15 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:15 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:15 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72646 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid= -1 Mar 16 13:31:30 imap: Debug: Loading modules from directory: /opt/dovecot2.1/lib/dovecot Mar 16 13:31:30 imap: Debug: Module loaded: /opt/dovecot2.1/lib/dovecot/lib10_quota_plugin.so Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Effective uid=26, gid=26, home=/var/spool/virtual/kictanet.or.ke/wash Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: quota: No quota setting - plugin disabled Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: fs: root=/var/spool/virtual/kictanet.or.ke/wash/mdbox, index=, control=, inbox=, alt= Mar 16 13:31:30 imap(wash at kictanet.or.ke): Debug: Namespace : Using permissions from /var/spool/virtual/kictanet.or.ke/wash/mdbox: mode=0700 gid=-1 Mar 16 13:31:30 imap-login: Info: Login: user=, method=PLAIN, rip=192.168.40.194, lip=192.168.40.252, mpid=72647 Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around offset=894): msg header has bad magic value Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage: rebuilding indexes Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: /var/spool/virtual/kictanet.or.ke/wash/mdbox/mailboxes/SentItems/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 16 13:33:23 imap(wash at kictanet.or.ke): Info: Disconnected: IMAP session state is inconsistent, please relogin. bytes=13816863/907529 My doveconf output is here -> http://pastebin.com/6yNP5ygt -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From mlopez at gtdinternet.com Fri Mar 16 13:07:15 2012 From: mlopez at gtdinternet.com (=?ISO-8859-1?Q?Mauricio_L=F3pez_Riffo?=) Date: Fri, 16 Mar 2012 08:07:15 -0300 Subject: [Dovecot] POP3 Performance Message-ID: <4F631EE3.40806@gtdinternet.com> Hi, We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of bandwith) Attach of dovecot -n output: # 2.0.18: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) auth_debug_passwords = yes auth_default_realm = portalplata.cl auth_realms = portalplata.cl auth_verbose = yes auth_verbose_passwords = plain auth_worker_max_count = 100 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot.log default_process_limit = 200 default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_gid = 12 first_valid_uid = 8 lock_method = dotlock login_greeting = Dovecot mta10 mail_cache_min_mail_count = 5 mail_debug = yes mail_fsync = always mail_full_filesystem_access = yes mail_gid = 12 mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u mail_nfs_storage = yes mail_plugins = " quota" mail_uid = 8 maildir_copy_with_hardlinks = no passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = box from subject quota = maildir } postmaster_address = mail at mail.com protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth { unix_listener auth-userdb { mode = 0600 user = exim } } service imap-login { service_count = 0 } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = root } service imap { executable = imap imap-postlogin } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } service_count = 0 } service pop3 { process_limit = 1024 } ssl_cert = References: <4F623DB7.9060707@cnpapers.com> <4F6242A9.6090209@cnysupport.com> Message-ID: <4F631F27.2050506@Media-Brokers.com> On 2012-03-15 3:27 PM, Terry Carmen wrote: > On 2012-03-15 3:06 PM, Steve Campbell wrote: >> Does anyone know of any manuals/books that have been written that >> might introduce me to most of the stuff in dovecot? > I'll be the first to admit that complex and specialized configurations > are sometimes difficult to figure out, however this list has always been > a tremendous amount of help. I agree completely. The ability to come to places like this and get answers directly from the software developer(s) is one of the main reasons I love open source software. And I will also say that Timo (yes, Steve, he is *the* dovecot developer, although he has had some excellent help for a while now) and this list is one of the most civil & respectful of any list I've been on, and the quality of support/answers is second to none. The postfix list is imo just as good as far as the quality of support, but they are very strict on 'form' - ie, no top-posting, you're expected (and often reminded) to read the instructions in the welcome message as to 'How to report a problem' and to actually follow those instructions - and quite often their replies seem harsh and unfriendly. I'd actually like to see dovecot have a similarly detailed welcome message (complete with a link to a detailed wiki page on 'How to Report a Problem' along with some helpful troubleshooting tips), but as much as I dislike top-posters (especially those who blindly quote the entire message they are replying to), I'm glad that this list is a bit less strict on form, and just seems more friendly. I for one would *love* to see some kind of 'The Book of Dovecot' (like 'The Book of Postfix'), but one reason I can see that would keep someone from wanting to write one is that dovecot (like most popular open source software) is still a very fast moving target as compared to the useful life of a book. Maybe his commercial support company can provide the resources for writing one once the target slows down a bit - or maybe even start off writing [a][some] smaller 'Basic Configuration' guide[s] for the things that aren't such fast moving targets that could eventually become chapters in a more comprehensive book. That would I think be a (admittedly probably fairly small) revenue generator, but hopefully at least enough to pay for itself and maybe provide a small profit. Another option I can think of would be for Timo to provide a method for people to pay a small fee for his support company to write up a custom 'How-To' for someone based on a list of requirements. I would imagine this as a web page that is put together with the appropriate questions, the answers for which are necessary to accomplish the goal. Of course, the other option is for other people to step up and 'fix the wiki' or 'write the Book' (or How-Tos), instead of just complaining about the lack (no offense, your 'complaint' wasn't all that bad). Yeah, I know this is the standard answer on free/open source software support lists, but it is the standard answer for a reason. On 2012-03-15 9:08 PM, Steve Campbell wrote: > I found that Postfix was the preferred SMTP server and Dovecot was > the preferred imap/pop server. I gave Postfix my best shot, but > didn't really have it tested well enough to stick with it, so I > dropped back to Sendmail, something I'm somewhat familiar with. I understand the argument for sticking with something you're familiar with, but I don't think you gave postfix a fair shot either - and it *does* have a number of excellent books written for it, so you don't have that excuse for postfix... ;). It is *much* easier to configure and run than sendmail, is much more performant and supposedly much more secure (just going by what I've read), and can do most anything that sendmail does (even supports milters). > Dovecot is an application that probably would work out of the box for > me if I didn't have to use data from the previous server. So I had to > use more than the standard options to make this work. Finding those > options was the main gripe I had with the wiki - there are just so > many options to make Dovecot the complete server. That's a good > thing. Just remember, us noobies-to-Dovecot have to discover all of > those options. You always have the option to get commercial support for fast resolutions to complex problems like this... ;) -- Best regards, Charles From robert at schetterer.org Fri Mar 16 14:10:49 2012 From: robert at schetterer.org (Robert Schetterer) Date: Fri, 16 Mar 2012 13:10:49 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F632DC9.4070108@schetterer.org> Am 16.03.2012 12:07, schrieb Mauricio L?pez Riffo: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see that > the performance is very poor and receive complaint about delays of > autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection (betwen > mtas and nfs storage) All account information work in LDAP plataform > (two servers in replicated mode, no high average or delays detected in > this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all of > service suffer a high load average (about 8 - 20 load average in each > dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, > autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, like > a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english > looks like you need to play with some config stuff and do more debug on your possible bottlenecks, what did you allready played with dovecot to high performance guess Timo will help about config settings after all for short to read http://wiki.dovecot.org/Authentication/Caching http://wiki.dovecot.org/LoginProcess http://wiki2.dovecot.org/Services any reasons for that mail_full_filesystem_access = yes ? general nfs may not optimal, but that a long story also virtual machines have their pros and contras maildir is not so good in performance -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From janfrode at tanso.net Fri Mar 16 14:11:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 16 Mar 2012 13:11:07 +0100 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <20120316121107.GA23566@dibs.tanso.net> One quick fix to try, if it's the login-time that's killing you, is to enable auth caching: http://wiki2.dovecot.org/Authentication/Caching that should offload your backend LDAP-servers from doing bind() on ever login, had a huge login performance impact for us. We use "auth_cache_size = 100 M", which gives us 99% cache hits: dovecot: auth: Authentication cache hits 3654591/3669119 (99%) dovecot: auth: Authentication cache inserts: positive: 588030 80931909B, negative: 912 49888B -jf From mcazzador at gmail.com Fri Mar 16 14:12:40 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 13:12:40 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, i obtain the same error Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: command not found Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: dsync-local(matteo at netlite.locale): Error: read() from worker server failed: EOF i've create vmail users (i've virtual domain netlite.locale (postfix), mysql backend i receive ana send mail correctly i use imap protocol), get ssh connection with publick key, i verify that with su - vmail , vmail find doveadm, i post my dovecot.conf file because i don't know what is wrong vmail exist on every server with publick key (server one => 10.0.0.118 server two => 10.0.0.122) dovecot.conf -> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail # (not root) user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm unix_listener config { user = vmail } } plugin { # this host replicates to remote host # mail_replica = remote:vmail at 10.0.0.122 # run full synchronization mode every other hour # (default is every 24 hours) # replication_full_sync_interval = 1 hour } idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 thank's -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Fri Mar 16 14:49:55 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 14:49:55 +0200 Subject: [Dovecot] Upgrading from 2.0.x to 2.1.x In-Reply-To: <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> References: <4F630755.7070909@noa.gr> <72b20840e790071a9b56e12ebf087eb9@imt-systems.com> Message-ID: <4F6336F3.6040601@noa.gr> On 16/3/2012 11:39 ??, Morten Stevens wrote: > See: http://wiki2.dovecot.org/Upgrading/2.1 Thank you Morten. This was exactly what I was looking for. By the way, searching in the wiki2 for "Upgrade" does not locate the upgrade pages, except "Upgrading/1.0". The pages are found when searching for "upgrading". I am wondering whether it would be possible to add keyword(s) to the respective articles, in order to provide better search results. Thanks again, Nick From campbell at cnpapers.com Fri Mar 16 14:54:21 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Fri, 16 Mar 2012 08:54:21 -0400 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <4F6337FD.4070404@cnpapers.com> On 3/16/2012 7:07 AM, Mauricio L?pez Riffo wrote: > Hi, > > We actually have a mail hosting solutions with aprox. 100 thousand > of email account, where about 90% of a customers use POP3 like email > configuration. About a few mounths (we perfomed a lot of migration > throught mbox email software to Maildir with dovecot) but i can see > that the performance is very poor and receive complaint about delays > of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, > VMware as a virtualization (the mtas are a virtual machines lives in > netapp too) about 4T of data mails and a 10G network connection > (betwen mtas and nfs storage) All account information work in LDAP > plataform (two servers in replicated mode, no high average or delays > detected in this servers) > > When the traffic have a peak of 1800 concurrent connections POP3, all > of service suffer a high load average (about 8 - 20 load average in > each dovecot) and authenticacion takes about 2 -10 seconds (in low > traffic, autenticacion takes about 60 miliseconds) > > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with > 6G RAM (virtual machine) and share's hardware with a exim instance, > like a MTA relay system (autenticated relay) > > Usage of network have peaks of 80Mbits (all dmz network have 1Gbits of > bandwith) > > Attach of dovecot -n output: > > # 2.0.18: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.18-308.1.1.el5 x86_64 CentOS release 5.8 (Final) > auth_debug_passwords = yes > auth_default_realm = portalplata.cl > auth_realms = portalplata.cl > auth_verbose = yes > auth_verbose_passwords = plain > auth_worker_max_count = 100 > base_dir = /var/run/dovecot/ > debug_log_path = /var/log/dovecot.log > default_process_limit = 200 > default_vsz_limit = 512 M > disable_plaintext_auth = no > first_valid_gid = 12 > first_valid_uid = 8 > lock_method = dotlock > login_greeting = Dovecot mta10 > mail_cache_min_mail_count = 5 > mail_debug = yes > mail_fsync = always > mail_full_filesystem_access = yes > mail_gid = 12 > mail_location = > maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes > mail_plugins = " quota" > mail_uid = 8 > maildir_copy_with_hardlinks = no > passdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > plugin { > mail_log_events = delete undelete expunge copy mailbox_delete > mailbox_rename > mail_log_fields = box from subject > quota = maildir > } > postmaster_address = mail at mail.com > protocols = imap pop3 > sendmail_path = /usr/lib/sendmail > service auth { > unix_listener auth-userdb { > mode = 0600 > user = exim > } > } > service imap-login { > service_count = 0 > } > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } > service imap { > executable = imap imap-postlogin > } > service pop3-login { > inet_listener pop3s { > port = 995 > ssl = yes > } > service_count = 0 > } > service pop3 { > process_limit = 1024 > } > ssl_cert = ssl_key = userdb { > args = /etc/dovecot/dovecot-ldap.conf.ext > driver = ldap > } > verbose_proctitle = yes > protocol lmtp { > mail_plugins = " quota" > } > protocol lda { > mail_plugins = " quota" > } > protocol imap { > imap_capability = > mail_max_userip_connections = 10 > mail_plugins = " quota autocreate notify quota imap_quota mail_log" > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } > } > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } > > > Output of account information in a LDAP: > > # nettester, email.net, MAIL, USERS, cl > dn: uid=nettester,dc=email.net,o=MAIL,o=USERS,c=cl > dc: email.net > mailMessageStore: /export/mdir/3/12/nettester at email.net/Maildir > uid: nettester > cn: nettester at email.net > sn: nettester at email.net > gidNumber: 12 > homeDirectory: /export/mdir/3/12/nettester at email.net > mail: nettester at email.net > uidNumber: 8 > objectClass: mailUser > objectClass: posixAccount > objectClass: mailSetting > loginShell: /bin/false > description: enable > service: pop3 > service: imap > service2: webmail > mailRate: 200 > mailQuota: 1024M > deliveryMode: none > mailReplyText: . > > Any suggestions? All ideas will be have a good receptions ;) > > > Pd: Sorry my english It doesn't seem to matter what type of hardware you might have, NFS can cause real bottlenecks, even to the point that your machine may report disk errors. Unfortunately, it's an evil necessity in some shops, but any way to eliminate NFS when large throughput is occurring will definitely help. Make sure you're running the latest version of NFS on all machines since V3 and V4 don't always like each other. I don't have a solution for it's replacement other than expensive hardware solutions. steve From mcazzador at gmail.com Fri Mar 16 15:02:03 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:02:03 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, with this changes first step is passed: I decomment this #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} i active and add absolute path of doveadm dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm dsync-server -u%u -l%{lock_timeout} -n%{namespace} But now from server 1 obtain Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no home directory Note: if i send a mail from server2 mail goes correcly in local (server 2) home virtual directory I note that when i launch manually from server1 sync, mysql on server 2 make correct sql to find home user dir On server 2 SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE username = 'matteo at netlite.locale' result are: +------------------------+------+------+------------------------------------------+ | maildir | uid | gid | mail | +------------------------+------+------+------------------------------------------+ | netlite.locale/matteo/ | 1000 | 1000 | /home/domini-posta/netlite.locale/matteo | +------------------------+------+------+------------------------------------------+ this configuration is teh same for server 1 and 2 thank's Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: > Hi, i obtain the same error > > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: > command not found > Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: > dsync-local(matteo at netlite.locale): Error: read() from worker server > failed: EOF > > > i've create vmail users (i've virtual domain netlite.locale (postfix), > mysql backend i receive ana send mail correctly i use imap protocol), > get ssh connection with publick key, > i verify that with su - vmail , vmail find doveadm, i post my > dovecot.conf file because i don't know what is wrong > > vmail exist on every server with publick key > > (server one => 10.0.0.118 > > server two => 10.0.0.122) > > dovecot.conf -> > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > service aggregator { > # give enough permissions for mail processes > # > ? ? ? ?fifo_listener replication-notify-fifo { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > ? ? ? ?unix_listener replication-notify { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 > ? ? ? ?} > } > service replicator { > # start replication at startup > # > ? ? ? ?process_min_avail = 1 > } > > > service doveadm { > # if you're using a single virtual user, set this to start ssh as vmail > # (not root) > > ? ? ? ?user = vmail > } > > service config { > # needed to grant access to /var/run/dovecot/config for service doveadm > > ? ? ? ?unix_listener config { > ? ? ? ? ? ? ? ?user = vmail > ? ? ? ?} > } > > plugin { > ? ? ? # this host replicates to remote host > # > ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 > > # run full synchronization mode every other hour > # (default is every 24 hours) > # > ? ? ? ?replication_full_sync_interval = 1 hour > } > > > idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 > > thank's > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:07:24 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:07:24 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F631EE3.40806@gtdinternet.com> References: <4F631EE3.40806@gtdinternet.com> Message-ID: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > We actually have a mail hosting solutions with aprox. 100 thousand of email account, where about 90% of a customers use POP3 like email configuration. About a few mounths (we perfomed a lot of migration throught mbox email software to Maildir with dovecot) but i can see that the performance is very poor and receive complaint about delays of autentications of accounts. > > The solution lives in Metrocluster Netapp storage, filesystem NFS, VMware as a virtualization (the mtas are a virtual machines lives in netapp too) about 4T of data mails and a 10G network connection (betwen mtas and nfs storage) All account information work in LDAP plataform (two servers in replicated mode, no high average or delays detected in this servers) Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > When the traffic have a peak of 1800 concurrent connections POP3, all of service suffer a high load average (about 8 - 20 load average in each dovecot) and authenticacion takes about 2 -10 seconds (in low traffic, autenticacion takes about 60 miliseconds) What does the CPU usage and NFS IOPS usage look like during those times? Meaning is the problem related to disk usage or something else? Note that for POP3 connections you don't get the "OK Logged in" reply until all of the message sizes have been read into memory. > Also, each dovecot instance lives un MTA server CentOS 5.8 x86_64 with 6G RAM (virtual machine) and share's hardware with a exim instance, like a MTA relay system (autenticated relay) Are you randomly redirecting users to different Dovecot servers? Dovecot director would work better: http://wiki2.dovecot.org/Director > auth_worker_max_count = 100 Auth workers are irrelevant with LDAP. > lock_method = dotlock fcntl would be faster, if your NFS setup can handle it. > mail_cache_min_mail_count = 5 I'm not really sure if it's a good idea to ever set this anything else than 0. Of course if you have detected that this actually decreases disk IO I'd be interested to see numbers. > mail_full_filesystem_access = yes If your users are sharing the same UID, this means all the users can access each others' mails now! Even if they have different UIDs this is unlikely to be helpful. > mail_location = maildir:%h/Maildir:INDEX=/data/cache/indexes/%2d/%1u/%2u/%u > mail_nfs_storage = yes Is /data also on NFS? Or does each server have its own local indexes? > maildir_copy_with_hardlinks = no This makes IMAP COPY command slower. Is there a reason why you've disabled it? > service imap-postlogin { > executable = script-login /usr/local/bin/postlogin.sh > user = root > } post-login script makes logins slower. What do you do in it? Why only for IMAP, not POP3? > plugin { > autocreate = Sent Items > autocreate2 = Spam > autocreate3 = Drafts > autocreate4 = Trash > autosubscribe = Sent Items > autosubscribe2 = Spam > autosubscribe3 = Drafts > autosubscribe4 = Trash > } These also slow logins down a little bit. v2.1 fixes that. > protocol pop3 { > mail_plugins = " quota quota notify mail_log" > pop3_reuse_xuidl = yes > pop3_save_uidl = yes > pop3_uidl_format = %u > } pop3_uidl_format=%u is a little bad, since it doesn't include %v. And you can improve pop3 performance with: pop3_no_flag_changes=yes And if the maildir filenames don't contain S=1234 sizes, this also makes a huge difference: pop3_fast_size_lookups=yes From mcazzador at gmail.com Fri Mar 16 15:10:04 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Fri, 16 Mar 2012 14:10:04 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> <4F626402.4030606@blue-labs.org> Message-ID: Hi, Solved! i add at my sql SELECT maildir, 1000 AS uid, 1000 AS gid, '/home/domini-posta/netlite.locale/matteo' as mail, '/home/domini-posta/netlite.locale/matteo' as home FROM mailbox WHERE username = 'matteo at netlite.locale' Now i've see first replication going!!! thank's everybody I hope my test help someone. Now i proceedd at use the replication system. Il 16 marzo 2012 14:02, Matteo Cazzador ha scritto: > Hi, with this changes first step is passed: > > I decomment this > > #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server > -u%u -l%{lock_timeout} -n%{namespace} > > i active and add absolute path of doveadm > > dsync_remote_cmd = ssh -p 22 -l%{login} %{host} /usr/local/bin/doveadm > dsync-server -u%u -l%{lock_timeout} -n%{namespace} > > But now ?from server 1 obtain > > Error: remote: dsync-remote(matteo at netlite.locale): Error: User has no > home directory > > Note: if i send a mail from server2 mail goes correcly in local > (server 2) home virtual directory > > I note that when i launch manually from server1 sync, mysql on server > 2 make correct sql to find home user dir > > On server 2 > > SELECT maildir, 1000 AS uid, 1000 AS gid, > '/home/domini-posta/netlite.locale/matteo' as mail FROM mailbox WHERE > username = 'matteo at netlite.locale' > > result are: > > +------------------------+------+------+------------------------------------------+ > | maildir ? ? ? ? ? ? ? ?| uid ?| gid ?| mail > ? ? ? ? ? | > +------------------------+------+------+------------------------------------------+ > | netlite.locale/matteo/ | 1000 | 1000 | > /home/domini-posta/netlite.locale/matteo | > +------------------------+------+------+------------------------------------------+ > > this configuration is teh same for server 1 and 2 > > thank's > > Il 16 marzo 2012 13:12, Matteo Cazzador ha scritto: >> Hi, i obtain the same error >> >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: remote: bash: doveadm: >> command not found >> Mar 16 13:02:01 Gentoo_cyrus_imap dovecot: >> dsync-local(matteo at netlite.locale): Error: read() from worker server >> failed: EOF >> >> >> i've create vmail users (i've virtual domain netlite.locale (postfix), >> mysql backend i receive ana send mail correctly i use imap protocol), >> get ssh connection with publick key, >> i verify that with su - vmail , vmail find doveadm, i post my >> dovecot.conf file because i don't know what is wrong >> >> vmail exist on every server with publick key >> >> (server one => 10.0.0.118 >> >> server two => 10.0.0.122) >> >> dovecot.conf -> >> >> #dsync_remote_cmd = ssh -p 22 -l%{login} %{host} doveadm dsync-server >> -u%u -l%{lock_timeout} -n%{namespace} >> >> service aggregator { >> # give enough permissions for mail processes >> # >> ? ? ? ?fifo_listener replication-notify-fifo { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> ? ? ? ?unix_listener replication-notify { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ? ? ? ? ? ? ? ? ?mode = 0600 >> ? ? ? ?} >> } >> service replicator { >> # start replication at startup >> # >> ? ? ? ?process_min_avail = 1 >> } >> >> >> service doveadm { >> # if you're using a single virtual user, set this to start ssh as vmail >> # (not root) >> >> ? ? ? ?user = vmail >> } >> >> service config { >> # needed to grant access to /var/run/dovecot/config for service doveadm >> >> ? ? ? ?unix_listener config { >> ? ? ? ? ? ? ? ?user = vmail >> ? ? ? ?} >> } >> >> plugin { >> ? ? ? # this host replicates to remote host >> # >> ? ? ? ?mail_replica = remote:vmail at 10.0.0.122 >> >> # run full synchronization mode every other hour >> # (default is every 24 hours) >> # >> ? ? ? ?replication_full_sync_interval = 1 hour >> } >> >> >> idem on the server 2 changing mail_replica = remote:vmail at 10.0.0.118 >> >> thank's >> >> >> -- >> Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. >> ****************************************** >> Ing. Matteo Cazzador >> Email: mcazzador at gmail.com >> ****************************************** > > > > -- > Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. > ****************************************** > Ing. Matteo Cazzador > Email: mcazzador at gmail.com > ****************************************** -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From tss at iki.fi Fri Mar 16 15:14:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:14:12 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> Message-ID: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> On 16.3.2012, at 11.09, Jernej Porenta wrote: >>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>> (uni_utf8_str_is_valid(vname)) >> .. > We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. > > Whenever . LIST "" "*" is issued, dovecot crashes: > Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot I don't think this is the same Panic as the original one? What is the Panic message now? From tss at iki.fi Fri Mar 16 15:17:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:17:50 +0200 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <20120316100836.GA12049@nicku.org> References: <20120316100836.GA12049@nicku.org> Message-ID: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> On 16.3.2012, at 12.08, Nick Urbanik wrote: > Using head of imaptest with dovecot 2.1, I am attempting to put a > heavy load on the server, with insufficient success. > > I made 15000 user accounts, put them in to a file, one per line. Then > I ran imaptest with > > ./imaptest userfile=../../imap-test-userlist-15001.txt clients=15001 > pass=SECRETPASSWORD A single imaptest process can't handle that many simultaneous clients. You'd need to run multiple imaptests in parallel. > So what is the best way to put a really heavy load on an imap server? > The documentation describes testing correctness of the server without > really emphasising performance. Yes, imaptest is mainly meant to test server correctness (i.e. for me to test that Dovecot is bugfree). It spends a lot of time checking and tracking things that is irrelevant when you simply want to load the server. You could add no_tracking parameter to get rid of some of it. From tss at iki.fi Fri Mar 16 15:24:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 15:24:53 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: Message-ID: <1331904293.26095.2.camel@innu> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > I have a situation where I need to migrate e-mails from Outlook 2011 (Mac) > to Apple Mail. Having looked at all options, I have resorted to the, > perhaps, most difficult way: Create folders on the IMAP server, copy > e-mails into them from Outlook, connect Apple Mail and do the reverse. > However, I have hit a wall. > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent Items > folder, but the process dies at some point. When that happens, Outlook pops > a screen saying "IMAP session state is inconsistent, please relogin". .. > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: Corrupted > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4 (around > offset=894): msg header has bad magic value Well, this isn't good. The mdbox was empty when you first started copying the mails? This is a normal local FreeBSD filesystem (not NFS or something else weird)? What happens if you now run: doveadm force-resync -u wash at kictanet.or.ke INBOX Does it show any errors? If not, and if you try to copy the mails again, does it still fail? It's interesting if you can reproduce this. I wonder if it's because of FreeBSD or if it's related to single instance storage. From nmilas at noa.gr Fri Mar 16 15:39:38 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 16 Mar 2012 15:39:38 +0200 Subject: [Dovecot] ldapi support Message-ID: <4F63429A.6040304@noa.gr> Hi, Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) connections for LDAP lookups (user, password, auth etc.) or only ldap/ldaps (over TCP)? If yes, how do we specify ldapi://localhost in Dovecot configuration files? Thanks, Nick From odhiambo at gmail.com Fri Mar 16 15:51:18 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 16:51:18 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > Yes, it was pretty much empty..actually, I just configured the Dovecot instance yesterday evening and only tested for "correct operation" by sending a test mail to myself and login to POP3/IMAP. Pretty much pristine. And yes, if is FreeBSD ufs. [wash at jaribu ~]$ mount /dev/label/rootfs0 on / (ufs, local, noatime, journaled soft-updates) devfs on /dev (devfs, local, multilabel) procfs on /proc (procfs, local) linprocfs on /compat/linux/proc (linprocfs, local) /dev/ada1s1a on /disk2 (ufs, local, noatime, soft-updates) > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > Unfortunately, the user has taken the MacBook now so I am unable to test copying again. > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > I should be able to test this again at some point next week, but just to add, I changed the storage to Maildir and I was able to copy all the mails to the IMAP folder without any issue so I think it's something with SIS. PS: I wish I could test this with Outlook running on Windows, but I guess that introduces a completely different environment than what I had on the MacBook, right? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From odhiambo at gmail.com Fri Mar 16 16:00:41 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:00:41 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331904293.26095.2.camel@innu> References: <1331904293.26095.2.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: > > I have a situation where I need to migrate e-mails from Outlook 2011 > (Mac) > > to Apple Mail. Having looked at all options, I have resorted to the, > > perhaps, most difficult way: Create folders on the IMAP server, copy > > e-mails into them from Outlook, connect Apple Mail and do the reverse. > > However, I have hit a wall. > > > > I am running Dovecot 2.1.2 on FreeBSD, with mdbox storage. > > > > In my first attempt, I have 1792 messages in the "Sent Items" folder for > > Outlook. I have tried (3 times) to copy the messages to the IMAP/Sent > Items > > folder, but the process dies at some point. When that happens, Outlook > pops > > a screen saying "IMAP session state is inconsistent, please relogin". > .. > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > Corrupted > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > offset=894): msg header has bad magic value > > Well, this isn't good. The mdbox was empty when you first started > copying the mails? This is a normal local FreeBSD filesystem (not NFS or > something else weird)? > > What happens if you now run: > > doveadm force-resync -u wash at kictanet.or.ke INBOX > > Does it show any errors? If not, and if you try to copy the mails again, > does it still fail? > > It's interesting if you can reproduce this. I wonder if it's because of > FreeBSD or if it's related to single instance storage. > > Ok. I have been able to reproduce it anyway. Environment: Windows 8 Consumer Preview, Outlook 2010. I had 415 e-mails in the Inbox. While copying, Outlook popped up an error: IMAP session state is inconsistent, please relogin. Protocol: IMAP Server: 192.168.40.252 Port: 143 Error Code: 0x800CCCDD ...and dovecot.log details at that time can be found here - http://196.200.26.114/~wash/dovecot.log.txt And the output of the force-resync command is: [root at jaribu] /usr/home/wash# cp /var/log/dovecot.log ~wash/public_html/dovecot.log.txt [root at jaribu] /usr/home/wash# /opt/dovecot2.1/bin/doveadm force-resync -u wash at kictanet.or.ke INBOX doveadm(wash at kictanet.or.ke): Warning: mdbox /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage: rebuilding indexes doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.5 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.5 doveadm(wash at kictanet.or.ke): Error: Corrupted dbox file /var/spool/virtual/ kictanet.or.ke/wash/mdbox/storage/m.10 (around offset=30): Invalid dbox version doveadm(wash at kictanet.or.ke): Error: mdbox rebuild: Failed to fix file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.10 -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From CMarcus at Media-Brokers.com Fri Mar 16 16:03:12 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 10:03:12 -0400 Subject: [Dovecot] imaptest: performance testing In-Reply-To: <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> References: <20120316100836.GA12049@nicku.org> <9E1ECCB9-352C-474D-AC2D-95D285030577@iki.fi> Message-ID: <4F634820.5040605@Media-Brokers.com> On 2012-03-16 9:17 AM, Timo Sirainen wrote: > imaptest is mainly meant to test server correctness (i.e. for me to > test that Dovecot is bugfree). It spends a lot of time checking and > tracking things that is irrelevant when you simply want to load the > server. You could add no_tracking parameter to get rid of some of it. Maybe imaptest could be duped+modified somehow to produce a new imap_load_test utility...? -- Best regards, Charles From giles at coochey.net Fri Mar 16 16:11:40 2012 From: giles at coochey.net (Giles Coochey) Date: Fri, 16 Mar 2012 14:11:40 +0000 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <4F634A1C.8060501@coochey.net> On 16/03/2012 14:00, Odhiambo Washington wrote: > On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: > >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> I have a situation where I need to migrate e-mails from Outlook 2011 >> (Mac) >> Personally I would just use readpst to export the standard Outlook personal storage folders to mbox format... -- Best Regards, Giles Coochey NetSecSpec Ltd UK Mobile: +44 7983 877 438 Business Email: giles.coochey at netsecspec.co.uk Email/MSN/Live Messenger: giles at coochey.net Skype: gilescoochey -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4928 bytes Desc: S/MIME Cryptographic Signature URL: From mhlavink at redhat.com Fri Mar 16 16:48:00 2012 From: mhlavink at redhat.com (Michal Hlavinka) Date: Fri, 16 Mar 2012 15:48:00 +0100 Subject: [Dovecot] dovecot and systemd In-Reply-To: <1331820329.10319.32.camel@innu> References: <4F61EFE8.1000901@redhat.com> <1331820329.10319.32.camel@innu> Message-ID: <4F6352A0.5020200@redhat.com> On 03/15/2012 03:05 PM, Timo Sirainen wrote: > On Thu, 2012-03-15 at 14:34 +0100, Michal Hlavinka wrote: >> What exactly should happen when >> dovecot.conf does not match dovecot.socket configuration? > > Dovecot's systemd code was written by one of you Redhat guys. I had some > similar thoughts when I applied the patch, but didn't really know what > to do about it, so I didn't do anything. So: I don't know. Maybe some > other project has solved this somehow already? > > Dovecot anyway needs its own internal UNIX listeners. Should all > internal inet listeners be disabled? Could Dovecot somehow talk to > systemd and ask what listeners it's using for Dovecot and log warnings > if they don't match? I don't know that match about systemd. I'll forward this to systemd mailing list and I will let you know once I know more. From odhiambo at gmail.com Fri Mar 16 16:50:50 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 17:50:50 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F634A1C.8060501@coochey.net> References: <1331904293.26095.2.camel@innu> <4F634A1C.8060501@coochey.net> Message-ID: On Fri, Mar 16, 2012 at 17:11, Giles Coochey wrote: > On 16/03/2012 14:00, Odhiambo Washington wrote: > >> On Fri, Mar 16, 2012 at 16:24, Timo Sirainen wrote: >> >> On Fri, 2012-03-16 at 13:51 +0300, Odhiambo Washington wrote: >>> >>>> I have a situation where I need to migrate e-mails from Outlook 2011 >>>> >>> (Mac) >>> >>> Personally I would just use readpst to export the standard Outlook > personal storage folders to mbox format... > > Outlook 2011 (Mac OS X - Lion) can export everything into (!pst) .olm I haven't no clue whether .olm and .pst are one and the same, but I highly doubt. With Outlook 2011, the guys at Redmond intended to lock the user to Outlook! I have seen Outlook->Apple Mail migration procedures fraught with e-mail loses. Well, not quite a loss because you still have the e-mail inside Outlook, but that idea of running Outlook side-by-side with Apple Mail is not what I want to subject the user to. I also don't want to make them start searching for the e-mails that might not have been migrated and forward them to themselves. Migration should be complete & safe - no losses. There are commercial software out there that's said to be good at this process, but I wanted the "free"method. If there is *readpst *that can migrate from .olm to mbox, then I am willing to give it a shot, but I also want to see if Dovecot gets a fix for whatever problem I am facing. -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 17:05:42 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:05:42 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> Message-ID: <1331910342.26095.34.camel@innu> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > Corrupted > > > dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around > > > offset=894): msg header has bad magic value > > > Ok. I have been able to reproduce it anyway. It looks like mdbox is completely broken in your setup. Don't try to use it until this is solved, or you'll probably end up losing mails. Could you try if you can easily reproduce this using imaptest? http://imapwiki.org/ImapTest Simply run it for an empty test account as: imaptest host=localhost user=testuser pass=testpass Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else can report that they can reproduce this problem that would be helpful.. From tss at iki.fi Fri Mar 16 17:09:14 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:09:14 +0200 Subject: [Dovecot] ldapi support In-Reply-To: <4F63429A.6040304@noa.gr> References: <4F63429A.6040304@noa.gr> Message-ID: <1331910554.26095.35.camel@innu> On Fri, 2012-03-16 at 15:39 +0200, Nikolaos Milas wrote: > Hi, > > Quick question: Does Dovecot support ldapi (i.e. via Unix Sockets) > connections for LDAP lookups (user, password, auth etc.) or only > ldap/ldaps (over TCP)? > > If yes, how do we specify ldapi://localhost in Dovecot configuration files? OpenLDAP library handles the connections internally. It probably works the same way in Dovecot as in other software that uses OpenLDAP. So I don't know, try ldapi://. From lists at wildgooses.com Fri Mar 16 17:22:07 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:22:07 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F6079EE.4000201@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> Message-ID: <4F635A9F.2020406@wildgooses.com> On 14/03/2012 10:58, Charles Marcus wrote: > On 2012-03-13 6:29 PM, Terry Carmen wrote: >> I'm going to hope everything is OK for a while, since my goal is to >> retire >> all the old Exchange servers and move all the users to dovecot/maildir >> within the next couple of months. >> >> However it's always nice to know there are options. 8-) > > I'm currently looking at rolling out SOGo as part of a major reworking > of their current infrastructure (will also include converting their > old Courier-IMAP to dovecot 2.1.x among other things)... > > SOGo, as far as I can tell, is the best truly free and open source > 'exchange clone' available that works extremely well with > Thunderbird+Lightning (which is what my Client uses currently, but > they are very dissatisfied with using Google Calendar for Shared > calendars), Outlook and Apple Apps, as well as Android, Blackberry and > Apple mobile devices - and their upcoming v2 (in beta now) will not > only provide native Outlook support (no plugin needed), it will also > (optionally) provide a Samba4 Active Directory server in my main > Client's office - all with absolutely no licenses required. Commercial > support is available from Inverse, the company created by the > developers to provide said support services. > > I also learned something very interesting yesterday concerning SOGo > and dovecot during a sales call with a SOGo rep, but I'll wait and see > if Timo cares to chime in on this one... ;) > If the answer is that he will write a Z-Push/Activesync module for SOGo then I'm all ears! I have been watching SOGo for some time and the main thing I would miss is that every phone I have ever owned has largely limited/broken Funambol based sync and annoyingly working Activesync capability (I own a stream of Nokias...). It seems that although I don't like it, I need activesync support if I want my contacts/calendar on my phone... (I think I can do caldav on some of them, but not cardav on my N9) Apart from that it's a very neat system! Ed W From tss at iki.fi Fri Mar 16 17:26:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 17:26:33 +0200 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <1331911593.26095.47.camel@innu> On Fri, 2012-03-16 at 15:22 +0000, Ed W wrote: > > I also learned something very interesting yesterday concerning SOGo > > and dovecot during a sales call with a SOGo rep, but I'll wait and see > > if Timo cares to chime in on this one... ;) > > > > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) We're also very much wishing for SOGo Activesync, but I'm not planning on writing it myself (but maybe we'll hire someone who will). Annoyingly Microsoft has patented Activesync, so I guess it can't be legally used at least in USA without paying MS. From lists at wildgooses.com Fri Mar 16 17:30:42 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 15:30:42 +0000 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <1331807624.10319.6.camel@innu> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> Message-ID: <4F635CA2.8080401@wildgooses.com> On 15/03/2012 10:33, Timo Sirainen wrote: > On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote: >> I'm curious if anyone has any plugins for AV integration directly into >> dovecot. >> >> Our old pop servers have been scanning messges as they're moved from >> new->cur in the inbox and, at least where user's aren't poping every >> few seconds, there is occasionally enough time between scanning through >> the MXs to message retreval to snag a few more virues with updated >> definitions before they reach customers. >> >> Anyone doing anything similar? > http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a > script that modifies a mail while it's being read. You could make it run > a virus check, and if that happens you could change the virus MIME part > to be full of spaces (better not to change message size, line count or > MIME structure). > > Couple of other ideas: 1) Could use one of the (buggy and variously unsupported) on access virus scanners. I think Dazuko is now abandoned, but this is a new one mentioned via the Clamav site: http://www.fsl.cs.sunysb.edu/docs/avfs-security04/index.html 2) Extremely racey, but if you were on maildir you could use some kind of pre-login scripting to kick off a scan on login. Touch some lock file so that you can tell when last scanned and only scan if the definitions have been updated since you last scanned? 3) There are some POP proxies which offer inline virus scanning. Could place one in front of your mail server. Presumably this will expose you to all the bugs in that proxy... Good luck Ed W From CMarcus at Media-Brokers.com Fri Mar 16 17:45:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Fri, 16 Mar 2012 11:45:58 -0400 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F635A9F.2020406@wildgooses.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> Message-ID: <4F636036.4020604@Media-Brokers.com> On 2012-03-16 11:22 AM, Ed W wrote: > If the answer is that he will write a Z-Push/Activesync module for SOGo > then I'm all ears! I have been watching SOGo for some time and the main > thing I would miss is that every phone I have ever owned has largely > limited/broken Funambol based sync and annoyingly working Activesync > capability (I own a stream of Nokias...). It seems that although I > don't like it, I need activesync support if I want my contacts/calendar > on my phone... (I think I can do caldav on some of them, but not cardav > on my N9) While I agree it would be nice, why not just switch to a supported phone and be done with it? ;) When we roll out SOGo, we'll only be supporting the officially supported mobile clients (android, iphone/ipad, blackberry and windows mobile)... -- Best regards, Charles From odhiambo at gmail.com Fri Mar 16 17:57:29 2012 From: odhiambo at gmail.com (Odhiambo Washington) Date: Fri, 16 Mar 2012 18:57:29 +0300 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: On Fri, Mar 16, 2012 at 18:05, Timo Sirainen wrote: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: > > > > Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: > > > Corrupted > > > > dbox file /var/spool/virtual/ > kictanet.or.ke/wash/mdbox/storage/m.4(around > > > > offset=894): msg header has bad magic value > > > > > Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > > > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I had to do that, but things don't appear good at all. [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest host=localhost user=wash at kictanet.or.ke pass=XXX Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox You are welcome to access my box if you like, but I must warn you, things will be slow. Internationa links are slow as a result of fiber cuts in MSA, KE! -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254733744121/+254722743223 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I can't hear you -- I'm using the scrambler. Please consider the environment before printing this email. -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 652 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 18:09:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:09:58 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <9AD845E2-7922-4E4A-A7ED-F1C2EE55F31A@iki.fi> On 16.3.2012, at 17.57, Odhiambo Washington wrote: >> imaptest host=localhost user=testuser pass=testpass >> >> Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else >> can report that they can reproduce this problem that would be helpful.. >> > Must I edit src/settings.h to reflect my #define MBOX_PATH value? I think I > had to do that, but things don't appear good at all. No, you can also specify it as mbox=path parameter. > [root at jaribu] ~wash/Tools/Dovecot/2.1/imaptest-20120129# ./src/imaptest > host=localhost user=wash at kictanet.or.ke pass=XXX > Fatal: Empty mbox file: /var/spool/virtual/kictanet.or.ke/wash/mdbox But don't point the mbox there! You'll need to download http://www.dovecot.org/tmp/dovecot-crlf file and point the mbox to that. It's used to upload mails. From e-frog at gmx.de Fri Mar 16 18:16:49 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 17:16:49 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <1331910342.26095.34.camel@innu> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> Message-ID: <4F636771.1000308@gmx.de> On 16.03.2012 16:05, wrote Timo Sirainen: > On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>> Corrupted >>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>> offset=894): msg header has bad magic value >>> This is kind of the same error message I'm getting with doveadm backup. I can reproduce this at will. Not sure however if this is related. http://www.dovecot.org/list/dovecot/2012-March/064462.html >> Ok. I have been able to reproduce it anyway. > > It looks like mdbox is completely broken in your setup. Don't try to use > it until this is solved, or you'll probably end up losing mails. > > Could you try if you can easily reproduce this using imaptest? > http://imapwiki.org/ImapTest > > Simply run it for an empty test account as: > > imaptest host=localhost user=testuser pass=testpass > > Maybe I should set up a FreeBSD VM to try this myself. Or if anyone else > can report that they can reproduce this problem that would be helpful.. > From lists at wildgooses.com Fri Mar 16 18:17:19 2012 From: lists at wildgooses.com (Ed W) Date: Fri, 16 Mar 2012 16:17:19 +0000 Subject: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location In-Reply-To: <4F636036.4020604@Media-Brokers.com> References: <20120301173527.Horde.yhMsCVeGiNBPT-mvY-IFEYA@www.cnysupport.com> <89C2A7FD-47B2-4C87-985F-68DC8642F0FF@iki.fi> <20120304094817.Horde.7UgsdFeGiNBPU4Cxi1uiPmA@www.cnysupport.com> <2DB76CF0-A905-4836-AF9F-29A2F0A2B3C9@iki.fi> <20120312145724.Horde.a8Ybb1eGiNBPXkcUUESE4QA@www.cnysupport.com> <4F5FBFC8.3060306@cnysupport.com> <9432AB09-4EE8-475E-941E-1EFA731901C7@iki.fi> <20120313182909.Horde.lT1HNFeGiNBPX8o11Mb2A7A@www.cnysupport.com> <4F6079EE.4000201@Media-Brokers.com> <4F635A9F.2020406@wildgooses.com> <4F636036.4020604@Media-Brokers.com> Message-ID: <4F63678F.9080104@wildgooses.com> On 16/03/2012 15:45, Charles Marcus wrote: > On 2012-03-16 11:22 AM, Ed W wrote: >> If the answer is that he will write a Z-Push/Activesync module for SOGo >> then I'm all ears! I have been watching SOGo for some time and the main >> thing I would miss is that every phone I have ever owned has largely >> limited/broken Funambol based sync and annoyingly working Activesync >> capability (I own a stream of Nokias...). It seems that although I >> don't like it, I need activesync support if I want my contacts/calendar >> on my phone... (I think I can do caldav on some of them, but not cardav >> on my N9) > > While I agree it would be nice, why not just switch to a supported > phone and be done with it? ;) > > When we roll out SOGo, we'll only be supporting the officially > supported mobile clients (android, iphone/ipad, blackberry and windows > mobile)... > That implies you will be using cardav/caldav on those phones? I thought Android support was quite weak for those? I definitely don't like the idea of supporting activesync, but it seems like the only widely supported solution to pushing calendar and contacts updates to clients? Caldav gets you part of the way there, but cardav seems badly supported and there is no push support with either... Out of curiousity, what kind of performance are you getting out of the web interface and any tricks you used to improve "perceived" performance? My quick testing gave something circa 150-200ms response times from SOGo (forget exactly now) and as a result it was perceivable and just very slightly laggy (versus a desktop mail program!!). I get slightly better perceived performance from Roundcube (which also seems more amenable to building extension plugins) Seems a bit of a surprise that a compiled language delivers results slightly less quickly than PHP... Did you find any magic knobs to twist to get performance up there with gmail? Cheers Ed W From tss at iki.fi Fri Mar 16 18:57:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 18:57:57 +0200 Subject: [Dovecot] v2.1.3 released Message-ID: <1235E4DC-130A-4CE7-9C22-C6180062D914@iki.fi> http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.3.tar.gz.sig Do not use v2.1.2 with multi-dbox format, it's broken. I didn't notice that a small optimization I did a few days ago broke mdbox in some situations (when mdbox first created a new m.X file, and later in same session saved another message to it). It's quite a high priority for me to run some automated testing before releasing new versions, for example a small imaptest run with mdbox would have caught this. Perhaps the next release will already have the automated testing. From tss at iki.fi Fri Mar 16 19:00:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:00:57 +0200 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: <4F636771.1000308@gmx.de> References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: On 16.3.2012, at 18.16, e-frog wrote: > On 16.03.2012 16:05, wrote Timo Sirainen: >> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>> Corrupted >>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>> offset=894): msg header has bad magic value >>>> > > This is kind of the same error message I'm getting with doveadm backup. > I can reproduce this at will. Not sure however if this is related. > > http://www.dovecot.org/list/dovecot/2012-March/064462.html Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. From e-frog at gmx.de Fri Mar 16 19:24:38 2012 From: e-frog at gmx.de (e-frog) Date: Fri, 16 Mar 2012 18:24:38 +0100 Subject: [Dovecot] Problem copying e-mails to IMAP - Dovecot 2.1.2 In-Reply-To: References: <1331904293.26095.2.camel@innu> <1331910342.26095.34.camel@innu> <4F636771.1000308@gmx.de> Message-ID: <4F637756.3020707@gmx.de> On 16.03.2012 18:00, wrote Timo Sirainen: > On 16.3.2012, at 18.16, e-frog wrote: > >> On 16.03.2012 16:05, wrote Timo Sirainen: >>> On Fri, 2012-03-16 at 17:00 +0300, Odhiambo Washington wrote: >>>>>> Mar 16 13:33:23 jaribu dovecot: imap(wash at kictanet.or.ke): Error: >>>>> Corrupted >>>>>> dbox file /var/spool/virtual/kictanet.or.ke/wash/mdbox/storage/m.4(around >>>>>> offset=894): msg header has bad magic value >>>>> >> >> This is kind of the same error message I'm getting with doveadm backup. >> I can reproduce this at will. Not sure however if this is related. >> >> http://www.dovecot.org/list/dovecot/2012-March/064462.html > > Yeah, it's the same. Looks like I just hadn't stress tested mdbox myself last few days. Ok, just tested with 2.1.3 and it works again. Thanks Timo! From kgc at corp.sonic.net Fri Mar 16 19:49:46 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:49:46 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <4F637D3A.1000301@corp.sonic.net> On 03/16/12 06:07, Timo Sirainen wrote: > Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. Timo, can you explain why Maildir isn't a good for POP3 in this context? Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. -K From kgc at corp.sonic.net Fri Mar 16 19:52:58 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 10:52:58 -0700 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F635CA2.8080401@wildgooses.com> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> Message-ID: <4F637DFA.9070800@corp.sonic.net> On 03/16/12 08:30, Ed W wrote: > 2) Extremely racey, but if you were on maildir you could use some kind > of pre-login scripting to kick off a scan on login. Touch some lock file > so that you can tell when last scanned and only scan if the definitions > have been updated since you last scanned? I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Still need to take a look at Timo's patch set. -K From tss at iki.fi Fri Mar 16 19:54:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:54:06 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <4F637D3A.1000301@corp.sonic.net> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> Message-ID: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> On 16.3.2012, at 19.49, Kelsey Cummings wrote: > On 03/16/12 06:07, Timo Sirainen wrote: >> Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. > > Timo, can you explain why Maildir isn't a good for POP3 in this context? Compared to mbox/mdbox: It needs to read and delete multiple separate files, which is typically much slower than reading and deleting a single file. > Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. You mean deleting the messages after +OK, instead of before? Does it really make a difference?.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. From tss at iki.fi Fri Mar 16 19:55:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 19:55:44 +0200 Subject: [Dovecot] Just in time AV scanning In-Reply-To: <4F637DFA.9070800@corp.sonic.net> References: <20120314235138.GE39671@corp.sonic.net> <1331807624.10319.6.camel@innu> <4F635CA2.8080401@wildgooses.com> <4F637DFA.9070800@corp.sonic.net> Message-ID: <044DBE43-7DEA-45D5-A34A-41F7E0FB8345@iki.fi> On 16.3.2012, at 19.52, Kelsey Cummings wrote: > On 03/16/12 08:30, Ed W wrote: >> 2) Extremely racey, but if you were on maildir you could use some kind >> of pre-login scripting to kick off a scan on login. Touch some lock file >> so that you can tell when last scanned and only scan if the definitions >> have been updated since you last scanned? > > I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Whatever you do: Don't modify existing message files (without renaming them so they appear as new mails). IMAP (and Dovecot) require that messages never change. From kgc at corp.sonic.net Fri Mar 16 21:04:19 2012 From: kgc at corp.sonic.net (Kelsey Cummings) Date: Fri, 16 Mar 2012 12:04:19 -0700 Subject: [Dovecot] POP3 Performance In-Reply-To: <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <4F637D3A.1000301@corp.sonic.net> <1DC1CA8D-EC97-4158-86F7-0699C2D68FA9@iki.fi> Message-ID: <4F638EB3.6040802@corp.sonic.net> On 03/16/12 10:54, Timo Sirainen wrote: >> Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. > > You mean deleting the messages after +OK, instead of before? Does it really make a difference? In the context of a clients "send and receive" phase taking a (small) fraction of a second less time, perhaps, but it is a small difference in any case. It was one of many small changes we made to try to improve interactive performance. >.. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. True, we decided that loosing that ability didn't really matter. (Like not counting newlines as two bytes in the message size.) -K From alex.handle at gmail.com Fri Mar 16 22:00:05 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:00:05 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection Message-ID: Hi Dovecot-list! My setup consists of a dovecot server with lmtp delivery and 3 postfix mta servers in front. Previously the mtas authenticated (SASL) through the courier-authdaemond software to our mysql database. To get support for more password formats i migrated to dovecot for SASL authentification. Our postfix mtas connect to dovecot through a tcp-socket smtpd_sasl_auth_enable = yes smtpd_sasl_type = dovecot smtpd_sasl_path = inet:10.11.100.230:12345 smtpd_sasl_security_options = noanonymous smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = broken_sasl_auth_clients = yes 10.11.100.230 is our dovecot server. Please look at the attached doveconf.log to see my auth service configuration. I did the configuration according to the postfix SASL README. http://www.postfix.org/SASL_README.html#server_dovecot I tested the setup and everything worked fine but after 2 days i noticed these error messages in my mail.log: dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of existing connection and also these messages from postfix: SASL LOGIN authentication failed: Connection lost to authentication server I get the dovecot error message about 3000 times a day and postfix message about 270 times. Please see my attached mail.log for a detailed trace. Thank you for your help :) Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: doveconf.log Type: application/octet-stream Size: 14009 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mail.log Type: application/octet-stream Size: 3120 bytes Desc: not available URL: From tss at iki.fi Fri Mar 16 22:14:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 16 Mar 2012 22:14:28 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: Message-ID: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> On 16.3.2012, at 22.00, Alex Ha wrote: > dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of > existing connection Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. -------------- next part -------------- A non-text attachment was scrubbed... Name: tcp-auth.diff Type: application/octet-stream Size: 718 bytes Desc: not available URL: From alex.handle at gmail.com Fri Mar 16 22:39:42 2012 From: alex.handle at gmail.com (Alex Ha) Date: Fri, 16 Mar 2012 21:39:42 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 22.00, Alex Ha wrote: > >> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >> existing connection > > Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. > Thanks Timo! I will try the patch and report to you. Alex From lists at wiesinger.com Sat Mar 17 08:31:12 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Sat, 17 Mar 2012 07:31:12 +0100 (CET) Subject: [Dovecot] Update problem from 1.2 => 2.0.19 and recommended imap storage In-Reply-To: <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> References: <4F62D762.7080607@wiesinger.com> <6246DF8F-30A6-4EDE-8E0F-B31AC2312343@iki.fi> Message-ID: On Fri, 16 Mar 2012, Timo Sirainen wrote: > On 16.3.2012, at 8.02, Gerhard Wiesinger wrote: >> Calling imap still fails as non root: >> imap >> /usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: Permission denied >> collect2: ld returned 1 exit statusn > > Huh? That looks like imap is running ld to link something. It shouldn't be doing that. After starting it once as root the following files are created and it works also as non root: ls -l /usr/local/bin/.libs/ total 1160 -rwxr-xr-x 1 root root 235848 Aug 25 2010 lt-doveconf -rwxr-xr-x 1 root root 938454 Mar 16 07:03 lt-imap Before only one of these files was generated (I think lt-doveconf). Strange. Any ideas? Ciao, Gerhard -- http://www.wiesinger.com/ From hsn at filez.com Sat Mar 17 08:36:22 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 07:36:22 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs Message-ID: <4F6430E6.6040100@filez.com> Is there way to import old plain mboxes via dsync? It complains about lack of index files: ponto:(admin)~>dsync mirror mbox:~/mail dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files From tss at iki.fi Sat Mar 17 12:32:06 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 12:32:06 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F6430E6.6040100@filez.com> References: <4F6430E6.6040100@filez.com> Message-ID: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> On 17.3.2012, at 8.36, Radim Kolar wrote: > Is there way to import old plain mboxes via dsync? It complains about lack of index files: > > ponto:(admin)~>dsync mirror mbox:~/mail > dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are not permanent without index files > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? From varia at e-healthexpert.org Sat Mar 17 16:14:24 2012 From: varia at e-healthexpert.org (Mark Alan) Date: Sat, 17 Mar 2012 14:14:24 +0000 Subject: [Dovecot] POP3 Performance In-Reply-To: <6996C9EF-C624-415C-A904-D24513685979@iki.fi> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> Message-ID: <20120317141424.1c629e46@e-healthexpert.org> On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: > On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > pop3_no_flag_changes=yes Is it the same as pop3_no_flag_updates=yes ? M. From tss at iki.fi Sat Mar 17 16:40:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 17 Mar 2012 16:40:59 +0200 Subject: [Dovecot] POP3 Performance In-Reply-To: <20120317141424.1c629e46@e-healthexpert.org> References: <4F631EE3.40806@gtdinternet.com> <6996C9EF-C624-415C-A904-D24513685979@iki.fi> <20120317141424.1c629e46@e-healthexpert.org> Message-ID: <1FE9581E-498E-4AE7-800D-4038DA32BB73@iki.fi> On 17.3.2012, at 16.14, Mark Alan wrote: > On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen wrote: >> On 16.3.2012, at 13.07, Mauricio L?pez Riffo wrote: > >> pop3_no_flag_changes=yes > > Is it the same as pop3_no_flag_updates=yes ? Yeah. I wrote it from my memory. From hsn at filez.com Sat Mar 17 19:29:07 2012 From: hsn at filez.com (Radim Kolar) Date: Sat, 17 Mar 2012 18:29:07 +0100 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> Message-ID: <4F64C9E3.7080102@filez.com> > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? indexes never existed because these mboxes were never used by dovecot, its not conversion from one format to another, its import. Maybe open bug to add feature "dsync import" which will not depend on existing indexes? From kayasaman at gmail.com Sat Mar 17 20:31:33 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 18:31:33 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice Message-ID: <4F64D885.8000501@gmail.com> Hi, I am currently in the process of setting up an IMAP repository for round 100 users.... Currently the user authentication method is being handled via a Windows Domain Controller. The host OS for Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? Can anyone give me a hand with this? Regards, Kaya From sven at svenhartge.de Sat Mar 17 21:36:15 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 20:36:15 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> Message-ID: <88l5d3on0ev8@mids.svenhartge.de> Kaya Saman wrote: > I am currently in the process of setting up an IMAP repository for > round 100 users.... > Currently the user authentication method is being handled via a > Windows Domain Controller. > The host OS for Dovecot will either be FreeBSD or CentOS. > Would Dovecot be able to authenticate to either the DC directly or > would we need to go through LDAP?? Why not join the server to the domain and simply use PAM? Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you. > Additionally what would be the best method to store the **mail** > information? - as in MySQL database or Maildir format; coinciding with > this what is the best backup method in order to be able to do 'dump' > backups or restore single emails?? Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress. To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots. Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore. Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself. Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory. Some, like mbox or mdbox are a little bit more complex, but with the correct doveadm command you are nevertheless able to restore single mails. Gr??e, Sven. -- Sigmentation fault. Core dumped. From pstm.spain at gmail.com Sat Mar 17 21:42:27 2012 From: pstm.spain at gmail.com (PSTM) Date: Sat, 17 Mar 2012 20:42:27 +0100 Subject: [Dovecot] Problem managing mbox Message-ID: <4F64E923.5060401@gmail.com> Hello, I have a problem with dovecot. seems that do not erase mail that mail client request to be erased. And I have this errors: > Error: Next message unexpectedly corrupted in mbox file Info: > dovecot-2.1.1-2.0.cf.fc16.i686 > root 5979 0.0 0.1 3208 1260 ? Ss 20:18 0:00 > /usr/sbin/dovecot -F > dovenull 5985 0.0 0.2 7060 2280 ? S 20:18 0:00 > dovecot/imap-login > vmail 5988 0.0 0.1 7888 1848 ? S 20:18 0:00 > dovecot/imap permissions on mail dir: > total 4 > drwxr-xr-x 9 vmail mail 4096 ene 21 21:43 vmail Any suggestion? Regards, -- -- http://www.0pc.eu/ From kayasaman at gmail.com Sat Mar 17 21:55:35 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 19:55:35 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <88l5d3on0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> Message-ID: <4F64EC37.5010309@gmail.com> On 03/17/2012 07:36 PM, Sven Hartge wrote: > Kaya Saman wrote: > >> I am currently in the process of setting up an IMAP repository for >> round 100 users.... >> Currently the user authentication method is being handled via a >> Windows Domain Controller. >> The host OS for Dovecot will either be FreeBSD or CentOS. >> Would Dovecot be able to authenticate to either the DC directly or >> would we need to go through LDAP?? > Why not join the server to the domain and simply use PAM? > > Using ActiveDirectory through LDAP is a bit of a pain so I would avoid > this if I where you. Danke Sven :-) I don't actually have much AD/LDAP integration experience so I will try your method! >> Additionally what would be the best method to store the **mail** >> information? - as in MySQL database or Maildir format; coinciding with >> this what is the best backup method in order to be able to do 'dump' >> backups or restore single emails?? > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. > > To have a consistent backup, your mail storage should be able to > snapshot the volume the mail is stored on, so use LVM or an external > storage unit capable of snapshots. Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. > > Then backup the content of the snapshot using any program you like. > I use Bacula for long-term offsite storage and a local rsnapshot to keep > 7 days worth of mail for a quick restore. To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server. > > Whether you are able to restore single mails or the complete storage is > no property or feature of the mailbox format itself. > > Some formats are simpler to handle, like Maildir++, where you just drop > the file containing a mail into a directory. You mention Maildir++... is this Maildir format or something new which I haven't heard about yet? > > Some, like mbox or mdbox are a little bit more complex, but with the > correct doveadm command you are nevertheless able to restore single > mails. > > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sat Mar 17 23:03:21 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 22:03:21 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> Message-ID: <98l5i9bn0ev8@mids.svenhartge.de> Kaya Saman wrote: > On 03/17/2012 07:36 PM, Sven Hartge wrote: >> Kaya Saman wrote: >>> I am currently in the process of setting up an IMAP repository for >>> round 100 users.... Currently the user authentication method is >>> being handled via a Windows Domain Controller. The host OS for >>> Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to >>> authenticate to either the DC directly or would we need to go >>> through LDAP?? >> Why not join the server to the domain and simply use PAM? >> Using ActiveDirectory through LDAP is a bit of a pain so I would >> avoid this if I where you. > I don't actually have much AD/LDAP integration experience so I will > try your method! Question: do you need public or shared folders? Using samba and winbindd to join a domain creates real users on your server and as far as I know configuring shared folders with real users is a bit of a pain, especially of you need shared flags (like Seen, Replied, etc.) (Someone [Timo?] please correct me.) >>> Additionally what would be the best method to store the **mail** >>> information? - as in MySQL database or Maildir format; coinciding >>> with this what is the best backup method in order to be able to do >>> 'dump' backups or restore single emails?? >> Storing mails inside SQL? Not supported by dovecot and not very wise, >> IMHO. DBmail does this, but to be honest, I never heard any good >> feedback from admins using that product. From what I have been told, you >> need quite the beefy server to get a decent performance out of DBmail, >> compared to the needs of a "traditional" setup like with dovecot or >> courier-mail, but I digress. >> >> To have a consistent backup, your mail storage should be able to >> snapshot the volume the mail is stored on, so use LVM or an external >> storage unit capable of snapshots. > Hmm..... so FreeBSD coupled together with a ZFS repo for mail should > take care of 'Snapshot' issues. Yes. Or using LVM on Linux. >> Then backup the content of the snapshot using any program you like. >> I use Bacula for long-term offsite storage and a local rsnapshot to >> keep 7 days worth of mail for a quick restore. > To be honest I was considering rsync'ing the dir containing users > mailboxes to either another storage pool or server. No need to rsync, if you use ZFS. Just create a new snapshot and you are done. Bet thing about ZFS: you get deduplication for free, so the needed space to store the backups will not grow as fast. But you still may want to store the mails offsite/offserver for desaster recovery. Either use doveadm backup for that purpose or use rsnapshot, again gaining you deduplication on the target server. >> Whether you are able to restore single mails or the complete storage is >> no property or feature of the mailbox format itself. >> >> Some formats are simpler to handle, like Maildir++, where you just drop >> the file containing a mail into a directory. > You mention Maildir++... is this Maildir format or something new which I > haven't heard about yet? Maildir++ extends the original Maildir with things like Quota and ACLs and was first implemented in Courier. http://www.courier-mta.org/imap/README.maildirquota.html All current MTAs and POP3/IMAP servers implement this variant. Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are). In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands. I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever". Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sat Mar 17 23:24:25 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:24:25 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <98l5i9bn0ev8@mids.svenhartge.de> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> Message-ID: <4F650109.7090702@gmail.com> Thanks so much Sven for your indepth and complete responses! > Question: do you need public or shared folders? I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage. Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange. To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc. > > Using samba and winbindd to join a domain creates real users on your > server and as far as I know configuring shared folders with real users > is a bit of a pain, especially of you need shared flags (like Seen, > Replied, etc.) (Someone [Timo?] please correct me.) Actually we might have an LDAP server already taking care of the AD<->UNIX integration..... I don't know yet it's only my first week :-) > >>>> Additionally what would be the best method to store the **mail** >>>> information? - as in MySQL database or Maildir format; coinciding >>>> with this what is the best backup method in order to be able to do >>>> 'dump' backups or restore single emails?? >>> Storing mails inside SQL? Not supported by dovecot and not very wise, >>> IMHO. DBmail does this, but to be honest, I never heard any good >>> feedback from admins using that product. From what I have been told, you >>> need quite the beefy server to get a decent performance out of DBmail, >>> compared to the needs of a "traditional" setup like with dovecot or >>> courier-mail, but I digress. >>> >>> To have a consistent backup, your mail storage should be able to >>> snapshot the volume the mail is stored on, so use LVM or an external >>> storage unit capable of snapshots. >> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >> take care of 'Snapshot' issues. > Yes. Or using LVM on Linux. Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too...... > >>> Then backup the content of the snapshot using any program you like. >>> I use Bacula for long-term offsite storage and a local rsnapshot to >>> keep 7 days worth of mail for a quick restore. >> To be honest I was considering rsync'ing the dir containing users >> mailboxes to either another storage pool or server. > No need to rsync, if you use ZFS. Just create a new snapshot and you are > done. Bet thing about ZFS: you get deduplication for free, so the needed > space to store the backups will not grow as fast. Ok so that solves that! :-) > But you still may want to store the mails offsite/offserver for desaster > recovery. They are currently being stored on the parent company mail server so this will be the/off-site/ disaster recovery system in a way :-P > > Either use doveadm backup for that purpose or use rsnapshot, again > gaining you deduplication on the target server. I will research this - thank you for that info :-) > >>> Whether you are able to restore single mails or the complete storage is >>> no property or feature of the mailbox format itself. >>> >>> Some formats are simpler to handle, like Maildir++, where you just drop >>> the file containing a mail into a directory. >> You mention Maildir++... is this Maildir format or something new which I >> haven't heard about yet? > Maildir++ extends the original Maildir with things like Quota and ACLs > and was first implemented in Courier. > http://www.courier-mta.org/imap/README.maildirquota.html > > All current MTAs and POP3/IMAP servers implement this variant. > > Depending on the amount of mail a user collects inside a folder, Maildir > is not the best storage format. You may want to check into mdbox, if > your users are kind of "mail hoarders" (like some of my users are). > > In my opinion, Maildir has outlived its usefullnes. It was fine when > users had 1,000 mails in some 10 folders, but today, users collect over > 100,000 mails a year and Maildir is causing serious I/O trouble and the > need to heavily fine tune your storage and filesystems to cope with > those demands. > > I cannot thank Timo enough for inventing mdbox, as this format breaks > this viciuos cycle and, as someone else said "it ends the battle at the > I/O front forever". So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!} Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability. However, if like you say mdbox is the way to go then I will put a strong case together! > > Gr??e, > Sven. > Regards, Kaya From stsiol at yahoo.co.uk Sat Mar 17 23:51:28 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sat, 17 Mar 2012 21:51:28 +0000 (GMT) Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F64D885.8000501@gmail.com> References: <4F64D885.8000501@gmail.com> Message-ID: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> >Hi, > >I am currently in the process of setting up an IMAP repository for round 100 users.... > >Currently the user authentication method is being handled via a Windows Domain Controller. > >The host OS for Dovecot will either be FreeBSD or CentOS. > > >Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? > > >Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? > > >Can anyone give me a hand with this? > > >Regards, > > >Kaya Hi Kaya, I can't force you to follow a specific path. All I can do, is tell you my experience on this. Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for Webmail, OpenLDAP for LDAP (no windows software there) and CentOS v5.5 32-bit onwards. User base is about 30 users. System uptime without a glitch reached at some point (had to reboot the server for maintenance reasons) about 200 days. I am sure it would go beyond 365 days. Hope this helps, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From kayasaman at gmail.com Sat Mar 17 23:53:44 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 21:53:44 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> References: <4F64D885.8000501@gmail.com> <1332021088.21200.YahooMailNeo@web132206.mail.ird.yahoo.com> Message-ID: <4F6507E8.4060204@gmail.com> On 03/17/2012 09:51 PM, Spyros Tsiolis wrote: >> Hi, >> >> I am currently in the process of setting up an IMAP repository for round 100 users.... >> >> Currently the user authentication method is being handled via a Windows Domain Controller. >> >> The host OS for Dovecot will either be FreeBSD or CentOS. >> >> >> Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? >> >> >> Additionally > what would be the best method to store the **mail** information? - as > in MySQL database or Maildir format; coinciding with this what is the > best backup method in order to be able to do 'dump' backups or restore > single emails?? >> >> Can anyone give me a hand with this? >> >> >> Regards, >> >> >> Kaya > > > > Hi Kaya, > > I can't force you to follow a specific path. > All I can do, is tell you my experience on this. > > Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for > Webmail, OpenLDAP for LDAP (no windows software > there) and CentOS v5.5 32-bit onwards. > User base is about 30 users. > System uptime without a glitch reached at some point > (had to reboot the server for maintenance reasons) about > 200 days. I am sure it would go beyond 365 days. > > Hope this helps, > > spyros > > > > ---- > "I merely function as a channel that filters > music through the chaos of noise" > - Vangelis Thanks for that Spyros! Regards, Kaya From sven at svenhartge.de Sun Mar 18 00:28:04 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sat, 17 Mar 2012 23:28:04 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: Kaya Saman wrote: > >> Question: do you need public or shared folders? > I don't need anything apart from an IMAP storage solution. I don't > intend to tie in Dovecot with an MTA either as I will simply be using > this for storage. > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. So, how do you plan to get the mails into this storage? offlineimap? imapsync? mbsync? fetchmail? >>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>> take care of 'Snapshot' issues. >> Yes. Or using LVM on Linux. > Yeah.... true but I specified ZFS as I'm a fan and also am quite > comfortable with Solaris/*BSD too...... If you know ZFS and are familiar with it, then, by all means, go for it. >> Depending on the amount of mail a user collects inside a folder, >> Maildir is not the best storage format. You may want to check into >> mdbox, if your users are kind of "mail hoarders" (like some of my >> users are). >> >> In my opinion, Maildir has outlived its usefullnes. It was fine when >> users had 1,000 mails in some 10 folders, but today, users collect >> over 100,000 mails a year and Maildir is causing serious I/O trouble >> and the need to heavily fine tune your storage and filesystems to >> cope with those demands. >> >> I cannot thank Timo enough for inventing mdbox, as this format breaks >> this viciuos cycle and, as someone else said "it ends the battle at >> the I/O front forever". > So mdbox is a 'new' mailbox standard? ie. one can replace Maildir > format with this and use mdbox instead. {Note to self: time to > browse!} mdbox is a format invented by Timo for dovecot. But dovecot can use nearly all common mailbox formats (except MH, but no one uses that one today). > Since where I'm implementing this is mainly an MS based environment > they are concerned about /flat/ files.... which MS seems to typically > do (although never used MS before so I wouldn't know). So there is > some concern over performance, efficiency and manageability. Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no doubt about this. But even with this crappy format, dovecot is able to deliver astounding performance by use of separete index files which allow it to access the storage in an efficient manner. mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten. mdbox solves this problem by a) storing all meta-data in the index and b) by only ever appending to a mdbox storage file, c) never truncating an existing mdbox storage file and d) using more than one mdbox storage file. Max size and TTL are configurable. But this also means deleted mails are still inside a mdbox storage file and need to be finally removed by copying all remaining files into a new file. This process has to be manually run during low traffic hours, for example using a cronjob. You can say, mdbox is like mbox on steroids. ;) Flat files are not evil or bad or slow per se, but you have to use them the right way. > However, if like you say mdbox is the way to go then I will put a > strong case together! You may want to start with something familiar and convert later, which is no problem with dovecot. Gr??e, Sven -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 00:35:37 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sat, 17 Mar 2012 22:35:37 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F6511B9.1020801@gmail.com> On 03/17/2012 10:28 PM, Sven Hartge wrote: > Kaya Saman wrote: >> >>> Question: do you need public or shared folders? >> I don't need anything apart from an IMAP storage solution. I don't >> intend to tie in Dovecot with an MTA either as I will simply be using >> this for storage. >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > So, how do you plan to get the mails into this storage? offlineimap? > imapsync? mbsync? fetchmail? Since everything is blocked at the Exchange end, users will have to manually transfer for now through MS Outlook. Currently that's what they're doing to their PST's.... > >>>> Hmm..... so FreeBSD coupled together with a ZFS repo for mail should >>>> take care of 'Snapshot' issues. >>> Yes. Or using LVM on Linux. >> Yeah.... true but I specified ZFS as I'm a fan and also am quite >> comfortable with Solaris/*BSD too...... > If you know ZFS and are familiar with it, then, by all means, go for it. :-) > >>> Depending on the amount of mail a user collects inside a folder, >>> Maildir is not the best storage format. You may want to check into >>> mdbox, if your users are kind of "mail hoarders" (like some of my >>> users are). >>> >>> In my opinion, Maildir has outlived its usefullnes. It was fine when >>> users had 1,000 mails in some 10 folders, but today, users collect >>> over 100,000 mails a year and Maildir is causing serious I/O trouble >>> and the need to heavily fine tune your storage and filesystems to >>> cope with those demands. >>> >>> I cannot thank Timo enough for inventing mdbox, as this format breaks >>> this viciuos cycle and, as someone else said "it ends the battle at >>> the I/O front forever". >> So mdbox is a 'new' mailbox standard? ie. one can replace Maildir >> format with this and use mdbox instead. {Note to self: time to >> browse!} > mdbox is a format invented by Timo for dovecot. But dovecot can use > nearly all common mailbox formats (except MH, but no one uses that one > today). Ok so if you claim that mdbox is the 'best' mailbox storage solution then I'll look at implementing this. > >> Since where I'm implementing this is mainly an MS based environment >> they are concerned about /flat/ files.... which MS seems to typically >> do (although never used MS before so I wouldn't know). So there is >> some concern over performance, efficiency and manageability. > Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no > doubt about this. > > But even with this crappy format, dovecot is able to deliver astounding > performance by use of separete index files which allow it to access the > storage in an efficient manner. > > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. > > mdbox solves this problem by a) storing all meta-data in the index and > b) by only ever appending to a mdbox storage file, c) never > truncating an existing mdbox storage file and d) using more than one > mdbox storage file. Max size and TTL are configurable. > > But this also means deleted mails are still inside a mdbox storage file > and need to be finally removed by copying all remaining files into a new > file. This process has to be manually run during low traffic hours, for > example using a cronjob. > > You can say, mdbox is like mbox on steroids. ;) > > Flat files are not evil or bad or slow per se, but you have to use them > the right way. Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-( Need to make a strong case! > >> However, if like you say mdbox is the way to go then I will put a >> strong case together! > You may want to start with something familiar and convert later, which > is no problem with dovecot. Maildir is what I'm familiar with currently and mbox format - though only use mbox as an unfortunate side product of /system mail/ accounts. Works well with Alpine client though! > > Gr??e, > Sven > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:04:22 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:04:22 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: Kaya Saman wrote: >> Flat files are not evil or bad or slow per se, but you have to use >> them the right way. > Thanks a lot for that info. I will research more into this but I maybe > overridden at some point :-( > Need to make a strong case! Hmm. Just because Microsofts way of usage of flat file database sucks does not mean any usage of flat files is bad or evil or slow, if done right. Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox But as I wrote before, it is quite easy to convert from one format to the other: http://wiki2.dovecot.org/Migration/MailFormat Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:18:30 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:18:30 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> Message-ID: <4F6529D6.60609@gmail.com> On 03/18/2012 12:04 AM, Sven Hartge wrote: > Kaya Saman wrote: > >>> Flat files are not evil or bad or slow per se, but you have to use >>> them the right way. >> Thanks a lot for that info. I will research more into this but I maybe >> overridden at some point :-( >> Need to make a strong case! > Hmm. > > Just because Microsofts way of usage of flat file database sucks does > not mean any usage of flat files is bad or evil or slow, if done right. Coming from a UNIX background I deal quite a lot with this kind of stuff so there's not problem for me. However, where I'm trying to deploy this system is a primarily MS based enterprise meaning that as the only UNIX engineer onsite and the newest addition to the team I have to convince people of working with UNIX technologies or somehow increase UNIX awareness. As a bi-product I know nothing about MS tech. only what it told to me by my colleagues :-) > > Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox I checked that out after your last email... I started Google'ing a little. :-) Looks like it would be a good solution! > > But as I wrote before, it is quite easy to convert from one format to > the other: http://wiki2.dovecot.org/Migration/MailFormat Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system??? To be honest, I run Zimbra @home for my OpenSource work and really enjoy it; in conjunction with Dovecot on FreeBSD which I run imapsync to backup **all** emails to. It works really well...... :-) I have messed around with Postfix, Dovecot and Horde3 in the past which also was really nice. > > Gr??e, > Sven. > Regards, Kaya From sven at svenhartge.de Sun Mar 18 02:32:41 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 01:32:41 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: Kaya Saman wrote: > Once we get setup this may come in quite handy! Not sure what's going > on currently as everyone above me is still quite set in using an SQL > DB as a mail storage system??? RDBMS where not designed for such a task. Using a relational database as a storage method for big chunks of data is very unwise, in my opinion. It degrades them to just being some sort of filing cabinet. Now, wouldn't it be nice, if we had something like that, a filing cabinet where we can store large chunks of data and randomly read and write them in a fast manner? Oh yes, I remember, it is called a "filesystem". Let's use some of those to store the mail data. It will be soooo awesome! ;-) Ok, back being serious: there is nothing wrong with using a RDBMS in the way it was intented, to store user credentials, quota values, account settings, forwarding addresses, address book data, bookmarks, etc. Gr??e, Sven. -- Sigmentation fault. Core dumped. From kayasaman at gmail.com Sun Mar 18 02:42:29 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 00:42:29 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F652F75.7060901@gmail.com> On 03/18/2012 12:32 AM, Sven Hartge wrote: > Kaya Saman wrote: > >> Once we get setup this may come in quite handy! Not sure what's going >> on currently as everyone above me is still quite set in using an SQL >> DB as a mail storage system??? > RDBMS where not designed for such a task. Using a relational database > as a storage method for big chunks of data is very unwise, in my > opinion. It degrades them to just being some sort of filing cabinet. > > Now, wouldn't it be nice, if we had something like that, a filing > cabinet where we can store large chunks of data and randomly read and > write them in a fast manner? > > Oh yes, I remember, it is called a "filesystem". Let's use some of those > to store the mail data. It will be soooo awesome! ;-) I think for the serious engineer there's Linux if even more serious there's UNIX and for the rest there's MS..... Actually as a medical term MS is something not that great to have; why does that also equate to IT/Computing too ;-P > > > Ok, back being serious: there is nothing wrong with using a RDBMS in the > way it was intented, to store user credentials, quota values, account > settings, forwarding addresses, address book data, bookmarks, etc. I agree! My humble opinion for a personal preference setup in this instance: FreeBSD 8.2 x64 as base OS UFS2 running on root drive Create ZFS pools for storage Have users mailboxes on the ZFS pools Enable ZFS caching and snapshots Dovecot to manage IMAPv4 --- Get rid of MS altogether! ....Then start working a really cool implementation of UNIX/Linux only infrastructure :-) > > > Gr??e, > Sven. > Regards, Kaya From stan at hardwarefreak.com Sun Mar 18 11:16:55 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sun, 18 Mar 2012 04:16:55 -0500 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F650109.7090702@gmail.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <4F65A807.9020206@hardwarefreak.com> On 3/17/2012 4:24 PM, Kaya Saman wrote: > Long story but we don't have any control over our mail server which is > handled by the parent company abroad and is on MS Exchange. > > To use an IMAP storage solution is the only way to get rid of pesky MS > .pst files which have been causing everyone grief and havoc. It's been many years since I used, or supported, MS Outlook. That said, for the 10+ years I did support it, ~1996-2006, the corporate version of Outlook, not to be confused with Outlook Express, did not store any mail in local .PST files unless specifically configured to do so. By default it keeps all mail in the user account in the Exchange server store. Thus I would assume these Outlook clients have been manually configured to use .PST files to keep copies of mail locally, for faster access and to keep inefficient MS Exchange (MAPI) traffic off the WAN link? Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off? If the problem isn't with the .PST format for storing the emails, why not simply setup a local Samba server and configure the Outlook clients to store users' PSTs on Samba shares? Better yet, if you already have a file server for home directories, simply use a folder redirection policy to put the PST files in folders in users' home directories. This is an extremely common practice in the MS world because all Microsoft Windows apps store everything in the user profile directory by default, which again, causes big problems with roaming profiles, which many/most enterprises use. -- Stan From kayasaman at gmail.com Sun Mar 18 11:46:13 2012 From: kayasaman at gmail.com (Kaya Saman) Date: Sun, 18 Mar 2012 09:46:13 +0000 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F65AEE5.30208@gmail.com> On 03/18/2012 09:16 AM, Stan Hoeppner wrote: > On 3/17/2012 4:24 PM, Kaya Saman wrote: > >> Long story but we don't have any control over our mail server which is >> handled by the parent company abroad and is on MS Exchange. >> >> To use an IMAP storage solution is the only way to get rid of pesky MS >> .pst files which have been causing everyone grief and havoc. > It's been many years since I used, or supported, MS Outlook. That said, > for the 10+ years I did support it, ~1996-2006, the corporate version of > Outlook, not to be confused with Outlook Express, did not store any mail > in local .PST files unless specifically configured to do so. By default > it keeps all mail in the user account in the Exchange server store. > > Thus I would assume these Outlook clients have been manually configured > to use .PST files to keep copies of mail locally, for faster access and > to keep inefficient MS Exchange (MAPI) traffic off the WAN link? > > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? > > If the problem isn't with the .PST format for storing the emails, why > not simply setup a local Samba server and configure the Outlook clients > to store users' PSTs on Samba shares? > > Better yet, if you already have a file server for home directories, > simply use a folder redirection policy to put the PST files in folders > in users' home directories. This is an extremely common practice in the > MS world because all Microsoft Windows apps store everything in the user > profile directory by default, which again, causes big problems with > roaming profiles, which many/most enterprises use. > Thanks Stan for this, currently our users have about 270MB space located on the Exchange server which we have no control over. Therefor users are currently manually backing up their information to .pst. Since all contact/calendar/other information is already stored on the server the IMAP solution is a better one. It also means that a more UNIX/Linux centric approach is being ask for/tolerated which is where I come in being the only full-bread UNIX engineer on site. I couldn't comment on the MS side of things as I have never really used MS stuff before but my plan using ZFS and FreeBSD should be the best. Again it's going to be **scalable** storage which is perfect! -Also easy to maintain; otherwise I don't think there would be anyone left who will be able to admin the SMB/local directory method (I can't as I don't understand MS) - additionally there isn't much space available and bandwidth either meaning we would purchase a dedicated server or build a dedicated server for this (well I would :-) ). Regards, Kaya From tss at iki.fi Sun Mar 18 17:15:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 17:15:46 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> Message-ID: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> On 18.3.2012, at 0.28, Sven Hartge wrote: > mbox has big problems with concurrent writes, the bigger the mbox is, > the more problems you get. This is mainly caused by the meta-data of a > message (meaning flags, status, etc.) which is stored inside the mbox > file itself. Flagging a message as read or replied causes the whole mbox > file to be rewritten. Dovecot moves only minimal amount of data within mbox. A flag change writes only a few bytes to mbox file, it doesn't rewrite it the whole file. Only time when the entire mbox file is rewritten is when you expunge the first message. From sven at svenhartge.de Sun Mar 18 17:19:38 2012 From: sven at svenhartge.de (Sven Hartge) Date: Sun, 18 Mar 2012 16:19:38 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: Timo Sirainen wrote: > On 18.3.2012, at 0.28, Sven Hartge wrote: >> mbox has big problems with concurrent writes, the bigger the mbox is, >> the more problems you get. This is mainly caused by the meta-data of >> a message (meaning flags, status, etc.) which is stored inside the >> mbox file itself. Flagging a message as read or replied causes the >> whole mbox file to be rewritten. > Dovecot moves only minimal amount of data within mbox. A flag change > writes only a few bytes to mbox file, it doesn't rewrite it the whole > file. Only time when the entire mbox file is rewritten is when you > expunge the first message. OK, then my memory was clouded by my exposure to the brokenness of uw-imapd and uw-popd, who rewrite an mbox file very often, resulting in abysmal performance. Gr??e, Sven. -- Sigmentation fault. Core dumped. From CMarcus at Media-Brokers.com Sun Mar 18 17:32:21 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:32:21 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F65A807.9020206@hardwarefreak.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F65A807.9020206@hardwarefreak.com> Message-ID: <4F660005.8070902@Media-Brokers.com> On 2012-03-18 5:16 AM, Stan Hoeppner wrote: > Is your problem with the PST files themselves, or merely the fact > they're stored on the local PC, probably in the users' roaming profiles, > thus creating the problem of large data movement during logon/off? If so, using redirected folders (if you're not using them, you should be) would alleviate this problem nicely, even in a large environment. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 18 17:36:25 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 18 Mar 2012 11:36:25 -0400 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> Message-ID: <4F6600F9.1010604@Media-Brokers.com> On 2012-03-18 11:15 AM, Timo Sirainen wrote: > Only time when the entire mbox file is rewritten is when you > expunge the first message. Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Although I have no desire to use mbox (planning on using mdbox), so it isn't important to me... ;) -- Best regards, Charles From janfrode at tanso.net Sun Mar 18 19:52:49 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 18 Mar 2012 18:52:49 +0100 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <20120318175249.GA15524@dibs.tanso.net> On Sun, Mar 18, 2012 at 11:36:25AM -0400, Charles Marcus wrote: > > Hmmm... wonder if there would be a way to add some kind of 'dummy' > first message that dovecot would simply ignore (not show to the > user), that would prevent that bevaior? That's what uw-imap does. It creates a message with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA", which is very annoying if your users has direct access to the mbox's... http://www.washington.edu/imap/IMAP-FAQs/index.html#6.14 -jf From tss at iki.fi Sun Mar 18 20:07:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 20:07:34 +0200 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: <4F6600F9.1010604@Media-Brokers.com> References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <20AF813D-7BE6-4147-B19B-E7B5F99E8078@iki.fi> <4F6600F9.1010604@Media-Brokers.com> Message-ID: <50F4C47F-EE04-4384-9465-89AFCF5C8B67@iki.fi> On 18.3.2012, at 17.36, Charles Marcus wrote: > On 2012-03-18 11:15 AM, Timo Sirainen wrote: >> Only time when the entire mbox file is rewritten is when you >> expunge the first message. > > Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior? Dovecot has such a dummy first message exactly like UW-IMAP. But what I meant is that if you expunge the first message and you want to free the space used by it, there's no other choice than to rewrite all of the messages after it. (And of course the first message isn't special in any way, there's just about as much of rewriting if you delete the 2nd or 3rd or other messages from the beginning of the mbox.) From arekm at maven.pl Sun Mar 18 23:00:35 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:00:35 +0100 Subject: [Dovecot] mdbox and filesystem quota Message-ID: <201203182200.36011.arekm@maven.pl> http://wiki2.dovecot.org/MailboxFormat/dbox "Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. " What happens if there is filesystem hard quota that is exceeded? Will dovecot allow to delete mails to free space without a need to wait for cronjob to do the purge? -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:45:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:45:54 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182200.36011.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> Message-ID: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > http://wiki2.dovecot.org/MailboxFormat/dbox > > "Expunging a message only decreases the message's refcount. The space is later > freed in "purge" step. This is typically done in a nightly cronjob when > there's less disk I/O activity. " > > What happens if there is filesystem hard quota that is exceeded? Will dovecot > allow to delete mails to free space without a need to wait for cronjob to do > the purge? No. Also the purging itself won't work, because it needs to write new data first before it can delete old data. Don't run out of disk space! From arekm at maven.pl Sun Mar 18 23:52:38 2012 From: arekm at maven.pl (Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?=) Date: Sun, 18 Mar 2012 22:52:38 +0100 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> Message-ID: <201203182252.38995.arekm@maven.pl> On Sunday 18 of March 2012, Timo Sirainen wrote: > On 18.3.2012, at 23.00, Arkadiusz Mi?kiewicz wrote: > > http://wiki2.dovecot.org/MailboxFormat/dbox > > > > "Expunging a message only decreases the message's refcount. The space is > > later freed in "purge" step. This is typically done in a nightly cronjob > > when there's less disk I/O activity. " > > > > What happens if there is filesystem hard quota that is exceeded? Will > > dovecot allow to delete mails to free space without a need to wait for > > cronjob to do the purge? > > No. Also the purging itself won't work, because it needs to write new data > first before it can delete old data. Don't run out of disk space! Can dovecot treat soft quota like hard quota for user then? Or better enforce quota based on filesystem quot information. With xfs I can set quota but turn enforcement off. All fs quota counters work but no enforcement is being made by xfs itself. -- Arkadiusz Mi?kiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ From tss at iki.fi Sun Mar 18 23:56:48 2012 From: tss at iki.fi (Timo Sirainen) Date: Sun, 18 Mar 2012 23:56:48 +0200 Subject: [Dovecot] mdbox and filesystem quota In-Reply-To: <201203182252.38995.arekm@maven.pl> References: <201203182200.36011.arekm@maven.pl> <7677B18D-88B8-401D-BE29-48A4BB856F7F@iki.fi> <201203182252.38995.arekm@maven.pl> Message-ID: <4BC1C8AE-4AD5-4A51-8954-FFAA4B84F35D@iki.fi> On 18.3.2012, at 23.52, Arkadiusz Mi?kiewicz wrote: >>> "Expunging a message only decreases the message's refcount. The space is >>> later freed in "purge" step. This is typically done in a nightly cronjob >>> when there's less disk I/O activity. " >>> >>> What happens if there is filesystem hard quota that is exceeded? Will >>> dovecot allow to delete mails to free space without a need to wait for >>> cronjob to do the purge? >> >> No. Also the purging itself won't work, because it needs to write new data >> first before it can delete old data. Don't run out of disk space! > > Can dovecot treat soft quota like hard quota for user then? > > Or better enforce quota based on filesystem quot information. With xfs I can > set quota but turn enforcement off. All fs quota counters work but no > enforcement is being made by xfs itself. Yes, Dovecot does the quota enforcement itself. I'm not entirely sure if it uses soft or hard quota though. From trashcan at odo.in-berlin.de Mon Mar 19 10:35:34 2012 From: trashcan at odo.in-berlin.de (Michael Grimm) Date: Mon, 19 Mar 2012 09:35:34 +0100 Subject: [Dovecot] replication howto In-Reply-To: <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi -- On 15.03.2012 22:05, Timo Sirainen wrote: > On 15.3.2012, at 22.48, Michael Grimm wrote: >> Actually it's a bad idea to use root for ssh from a security point >> of view. A hacked root account isn't fun. Thus, normally one needs >> to explicitly change the config of the sshd daemon to allow root >> logins (at least with FreeBSD what I'm using). Thus, I do recommend >> to use an unprivileged user like vmail. > > Then again it's safer to use system user accounts than a single vmail > account that has access to everyone's emails. Root has access to everyone's mail as well. > And if you allow ssh login only with public key authentication I > don't think there are much security issues. And finally, it would > be possible to write a small wrapper that allows the root's public > key auth to only execute dsync-user.sh script that can't do anything > except sync a specified user's mails. All those safety measures can be applied for the vmail user as well. Actually, that's what I did in my case, plus allowing ssh only between both mail servers (firewall rule). Regards, Michael From lcaron at unix-scripts.info Mon Mar 19 11:06:20 2012 From: lcaron at unix-scripts.info (Laurent CARON) Date: Mon, 19 Mar 2012 10:06:20 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot Message-ID: <20120319095939.maneexuo@trusted.unix-scripts.info> Hi, I'm currently having a fairly simple setup: - users (real, not virtual) - Maildir storage (over NFS) - 1 namespace I'm currently trying to render the storage snapshots available through dovecot (to allow my users to browse their mail history). dovecot.conf: namespace { inbox = yes location = prefix = INBOX. type = private } I did the following modifications: dovecot.conf: namespace snaps-h0 { prefix = INBOX.EmailBackup.h0. hidden = no list = yes inbox = no location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u type = private } Problem: I don't see the content of the inbox folder contained in the snapshots (subfolders are perfectly viewed). Do any of you have a clue on how to render it visible ? Thanks Laurent From jernej.porenta at arnes.si Mon Mar 19 11:58:56 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 10:58:56 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> Message-ID: <3974AB53-476A-4945-A828-11425C667165@arnes.si> On Mar 16, 2012, at 2:14 PM, Timo Sirainen wrote: > On 16.3.2012, at 11.09, Jernej Porenta wrote: > >>>> Mar 6 13:37:17 machine dovecot: imap(username): Panic: file >>>> mail-storage.c: line 628 (mailbox_alloc): assertion failed: >>>> (uni_utf8_str_is_valid(vname)) >>> .. >> We tried version 2.1.2, which unfortunately does not fix the issue with weird characters. >> >> Whenever . LIST "" "*" is issued, dovecot crashes: >> Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671eb8c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b54671ead83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f2a0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f376cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b5466f37846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b5466f38241] -> /opt/dovecot > > I don't think this is the same Panic as the original one? What is the Panic message now? Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) It is the same. We will try 2.1.3 today and report the results... Regards, Jernej From mcazzador at gmail.com Mon Mar 19 12:50:39 2012 From: mcazzador at gmail.com (Matteo Cazzador) Date: Mon, 19 Mar 2012 11:50:39 +0100 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: Hi, i've a simple question, what do you mean for dovecot director setup? 'i've a doubt. The solution that i'm testing is using 3 mail server in different geoghrapic locations. An user can travel in varius location, and i want his imap mail reside on mail server in every locations. S? i use you solution about replication. First server (by dns record) that receive mail sync it on the other servers, and when user consult is mail by imap protocol everything is sync on all servers. Do you suggest to use a horizontal structure for it like i explain or is better to have a single node external mail server and customer locations server like slave? Thank's Il 19 marzo 2012 09:35, Michael Grimm ha scritto: > Hi -- > > > On 15.03.2012 22:05, Timo Sirainen wrote: >> >> On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>> Actually it's a bad idea to use root for ssh from a security point >>> of view. A hacked root account isn't fun. Thus, normally one needs >>> to explicitly change the config of the sshd daemon to allow root >>> logins (at least with FreeBSD what I'm using). Thus, I do recommend >>> to use an unprivileged user like vmail. >> >> >> Then again it's safer to use system user accounts than a single vmail >> account that has access to everyone's emails. > > > Root has access to everyone's mail as well. > > >> And if you allow ssh login only with public key authentication I >> don't think there are much security issues. And finally, it would >> be possible to write a small wrapper that allows the root's public >> key auth to only execute dsync-user.sh script that can't do anything >> except sync a specified user's mails. > > > All those safety measures can be applied for the vmail user as well. > Actually, that's what I did in my case, plus allowing ssh only between > both mail servers (firewall rule). > > Regards, > Michael > -- Rispetta l'ambiente: se non ti ? necessario, non stampare questa mail. ****************************************** Ing. Matteo Cazzador Email: mcazzador at gmail.com ****************************************** From nmilas at noa.gr Mon Mar 19 13:20:22 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 13:20:22 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages Message-ID: <4F671676.3060809@noa.gr> Hi, We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've been using LTB OpenLDAP packages (http://ltb-project.org/wiki/download#openldap), which get installed in non-standard file system locations. So, I would like to re-build Dovecot packages based on these OpenLDAP libraries, esp. because I see that dovecot RPM packages are built using OpenLDAP v2.3 libraries. I am not much experienced in building RPMs and preparing spec files. In http://dl.atrpms.net/all/dovecot.spec I see: ------------------------------------------------ BuildRequires: openldap-devel, cyrus-sasl-devel ... Obsoletes: %{name}-pgsql < %{epoch}:%{version}-%{release}, %{name}-mysql < %{epoch}:%{version}-%{release}, %{name}-sqlite < %{epoch}:%{version}-%{release}, %{name}-ldap < %{epoch}:%{version}-%{release}, $ Conflicts: %{name}-pgsql > %{epoch}:%{version}-%{release}, %{name}-mysql > %{epoch}:%{version}-%{release}, %{name}-sqlite > %{epoch}:%{version}-%{release}, %{name}-ldap > %{epoch}:%{version}-%{release}, $ ------------------------------------------------ So, I can change the former reference (openldap-devel) to: openldap-ltb-debuginfo, cyrus-sasl-devel Question 1: What other changes should we make in order to specify that we will be using LDAP libraries from: /usr/local/openldap/lib64 and include files from: /usr/local/openldap/include (rather than from /usr/lib64 and /usr/include, respectively, which are the standard file paths used in openldap-devel) Question 2: How the Obsoletes and Conflicts lines should be changed? Question 3: It seems to me (by reading the spec file) that the final Dovecot RPM (and the included executables) does not need any LDAP dynamic library in order to run with LDAP support (because I don't see any dependencies on openldap package). Can somebody please confirm? Any other associated info would be appreciated. Thanks, Nick From rob0 at gmx.co.uk Mon Mar 19 14:20:50 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:20:50 -0500 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: <20120319122050.GM4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 09:35:34AM +0100, Michael Grimm wrote: > On 15.03.2012 22:05, Timo Sirainen wrote: > >On 15.3.2012, at 22.48, Michael Grimm wrote: > > >>Actually it's a bad idea to use root for ssh from a security > >>point of view. A hacked root account isn't fun. Thus, normally > >>one needs to explicitly change the config of the sshd daemon to > >>to allow root logins (at least with FreeBSD what I'm using). > >>Thus, I do recommend to use an unprivileged user like vmail. > > > >Then again it's safer to use system user accounts than a single > >vmail account that has access to everyone's emails. > > Root has access to everyone's mail as well. I think you are missing the point, that being: if all your mail are belong to vmail, somebody set up us the bomb if the vmail account is compromised. (Obviously that's true with a root compromise as well, but that is unavoidable. Effects of a root compromise can be limited with technologies like Apparmor and SELinux, but that is difficult to configure properly and only provides limited benefit: compromised root can do everything real root was allowed to do.) The point is: vmail has added a SECOND vulnerable point from which disaster can ensue. If mailbox ownership is distributed among multiple UID/GID, compromise of any one of those only endangers the mails to which it had access. > >And if you allow ssh login only with public key authentication I > >don't think there are much security issues. And finally, it would > >be possible to write a small wrapper that allows the root's public > >key auth to only execute dsync-user.sh script that can't do > >anything except sync a specified user's mails. > > All those safety measures can be applied for the vmail user as > well. Actually, that's what I did in my case, plus allowing ssh > only between both mail servers (firewall rule). Sure, but there too, all your email eggs are in the vmail basket. No, disaster is not imminent nor even likely to ensue, but the fact stands that you and millions of other virtual-only sites do have this additional potential vulnerability. It is well supported in Dovecot to be able to use a unique UID and GID for every virtual mailbox, but management of such a system presents more challenges than the single-vmail-user approach. Consequently the popular virtual frontends don't support it. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From cjeanneret at internux.ch Mon Mar 19 12:28:34 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Mon, 19 Mar 2012 12:28:34 +0200 Subject: [Dovecot] Problem with sieve Message-ID: Hello List! I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA doesn't run sieve, and thus doesn't filter my emails. Here's the sieve configuration: plugin { # Used by both the Sieve plugin and the ManageSieve protocol sieve=/var/local/vmail/%n/dovecot.sieve sieve_dir=/var/local/vmail/%n/sieve sieve_extensions = +notify +imapflags } The managesiege: protocol managesieve { # Specify an alternative address:port the daemon must listen on # (default: *:2000) listen = localhost:2000 managesieve_logout_format = bytes ( in=%i : out=%o ) } (this one is working fine, I can edit the filters through roundcube webmail, and the correct file (/var/local/vmail/%n/dovecot.sieve) is edited) the lda part: protocol lda { postmaster_address = foo at bar.com mail_plugins = sieve } I think all is in place to allow dovecot to use sieve... ? One more thing: dovecot --version 1.2.15 Any help will be welcomed :). Thanks in advance ! Cheers, C. From ngu.antoine at gmail.com Mon Mar 19 14:27:08 2012 From: ngu.antoine at gmail.com (Antoine Nguyen) Date: Mon, 19 Mar 2012 13:27:08 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: 2012/3/19 C?dric Jeanneret > Hello List! > > I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > doesn't run sieve, and thus doesn't filter my emails. > > Here's the sieve configuration: > > plugin { > # Used by both the Sieve plugin and the ManageSieve protocol > sieve=/var/local/vmail/%n/**dovecot.sieve > sieve_dir=/var/local/vmail/%n/**sieve > sieve_extensions = +notify +imapflags > } > > The managesiege: > > protocol managesieve { > # Specify an alternative address:port the daemon must listen on > # (default: *:2000) > listen = localhost:2000 > managesieve_logout_format = bytes ( in=%i : out=%o ) > } > (this one is working fine, I can edit the filters through roundcube > webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > edited) > > the lda part: > > protocol lda { > postmaster_address = foo at bar.com > mail_plugins = sieve > } > > I think all is in place to allow dovecot to use sieve... ? > > One more thing: > > dovecot --version > 1.2.15 > > > Any help will be welcomed :). > > Thanks in advance ! > > Cheers, > > C. > Have you checked the MTA configuration. Does it use dovecot's LDA ? Antoine From rob0 at gmx.co.uk Mon Mar 19 14:32:14 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 07:32:14 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F671676.3060809@noa.gr> References: <4F671676.3060809@noa.gr> Message-ID: <20120319123213.GN4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 01:20:22PM +0200, Nikolaos Milas wrote: > We are (still) mainly using CentOS 5 (5.8 x86_64). As CentOS / > RHEL 5 standard OpenLDAP packages are rather old (2.3.x), we've > been using LTB OpenLDAP packages > (http://ltb-project.org/wiki/download#openldap), which get > installed in non-standard file system locations. ISTM that herein lies the whole problem. Why did you not rpmbuild your OpenLDAP? That would have avoided all further fuss. Another observation I can offer, unwelcome as it may be: your OS choice was not a good one when you want the features of recent software. Perhaps you should rethink that choice. You have invested much effort in this task. > So, I would like to re-build Dovecot packages based on these > OpenLDAP libraries, esp. because I see that dovecot RPM packages > are built using OpenLDAP v2.3 libraries. > > I am not much experienced in building RPMs and preparing spec > files. And that is really more a question for a CentOS forum than here. > In http://dl.atrpms.net/all/dovecot.spec I see: > > ------------------------------------------------ > BuildRequires: openldap-devel, cyrus-sasl-devel The latter requirement seems curious to me. In what way does Dovecot use Cyrus SASL? -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From jernej.porenta at arnes.si Mon Mar 19 15:27:06 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Mon, 19 Mar 2012 14:27:06 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <3974AB53-476A-4945-A828-11425C667165@arnes.si> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: On Mar 19, 2012, at 10:58 AM, Jernej Porenta wrote: > Mar 19 10:56:35 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=14732, secured > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > Mar 19 10:56:40 server dovecot: imap(user): Error: Raw backtrace: /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a91610870] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a916108c6] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot.so.0 [0x2b7a9160fd83] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9134f0e5] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c6cc] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0 [0x2b7a9135c846] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2b7a9135d241] -> /opt/dovecot-2.1.2/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2b7a9135619c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt > Mar 19 10:56:40 server dovecot: imap(user): Fatal: master: service(imap): child 14732 killed with signal 6 (core dumps disabled) > > It is the same. We will try 2.1.3 today and report the results... Same thing with 2.1.3 (. LIST "" "*"): Mar 19 14:08:59 server dovecot: imap-login: Login: user=, method=PLAIN, rip=193.2.1.110, lip=193.2.1.83, mpid=28438, secured Mar 19 14:09:04 server dovecot: imap(username): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) Mar 19 14:09:04 server dovecot: imap(username): Error: Raw backtrace: /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071811870] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae0718118c6] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot.so.0 [0x2ae071810d83] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae0715500c5] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d6ac] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0 [0x2ae07155d826] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(fs_list_iter_init+0x4b1) [0x2ae07155e221] -> /opt/dovecot-2.1.3/lib/dovecot/libdovecot-storage.so.0(mailbox_list_iter_init_multiple+0xec) [0x2ae07155717c] -> dovecot/imap [0x40bbb6] -> dovecot/imap(cmd_list_full+0x520) [0x40c1f0] -> dovecot/imap(cmd_list+0xb) [0x40c3eb] -> dovecot/imap(command_exec+0x37) [0x410497] -> dovecot/imap [0x40f4ed] -> dovecot/imap [0x40f5a2] -> dovecot/imap(client_handle_input+0x3f) [0x40f6ef] -> dovecot/imap(client_input+0x62) [0x410072] -> /opt Mar 19 14:09:04 server dovecot: imap(username): Fatal: master: service(imap): child 28438 killed with signal 6 (core dumps disabled) The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz # dovecot -n # 2.1.3: /opt/dovecot-2.1.3/etc/dovecot/dovecot.conf # OS: Linux 2.6.18-274.17.1.el5 x86_64 CentOS release 5.7 (Final) default_login_user = nobody disable_plaintext_auth = no login_greeting = Server ready. login_trusted_networks = x.y.z.p/32 mail_location = mbox:~/:INBOX=%h/.mailbox:INDEX=/opt/dovecot2-indexes/%1u/%u mail_plugins = quota mail_privileged_group = dovecot mbox_write_locks = fcntl namespace inbox { inbox = yes location = prefix = separator = / type = private } passdb { args = session=yes dovecot driver = pam } plugin { quota = fs } protocols = imap pop3 service imap-login { inet_listener imap { port = 143 } inet_listener imaps { ssl = no } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { ssl = no } } ssl = no userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_max_userip_connections = 15 mail_plugins = quota imap_quota } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } Regards, Jernej From nmilas at noa.gr Mon Mar 19 15:47:24 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 15:47:24 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319123213.GN4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> Message-ID: <4F6738EC.5050200@noa.gr> On 19/3/2012 2:32 ??, /dev/rob0 wrote: > ISTM that herein lies the whole problem. Why did you not rpmbuild > your OpenLDAP? That would have avoided all further fuss. Thanks for the reply. First, how would I rpmbuild my openldap v2.4.x as a standard CentOS 5 package (i.e. replacing native openldap-2.3.43-25)? If I were more experienced, I could have tried to engineer openldap-2.3.43-25.el5.src.rpm to upgrade the system to use 2.4.x... But still, I haven't seen any OpenLDAP packages attempting to do so, probably because of the tight integration of CentOS with some openldap v2.3 libraries. I think it's good that third-party packages (even of the same software) give the ability to not mess with standard system. The same is true for reputable Symas OpenLDAP packages. So, I simply use LTB OpenLDAP, even though it's installed at non-standard locations. (This has an added benefit of easy migration. You can setup any/all of those on the same system and decide which one to enable at any time.) > Another observation I can offer, unwelcome as it may be: your OS > choice was not a good one when you want the features of recent > software. Perhaps you should rethink that choice. You have invested > much effort in this task. I like CentOS from many aspects as an enterprise server OS. I wouldn't change it. Yet, it's important to me to be able to build/combine non-standard packages. Even with CentOS 6, I would still continue to use LTB OpenLDAP for a number of reasons. It's true that I've invested much effort in this task, but mostly because my knowledge on this subject is very basic. Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. We just try to make things better (and make our life a bit more difficult) :-) > > And that is really more a question for a CentOS forum than here. > True, but I am hoping that there might be some Dovecot RHEL/CentOS packagers in this list, and that would help resolve issues more effectively, as it is a Dovecot-specific (even if for a package thereof) question. So, any help will be appreciated! > The latter requirement seems curious to me. In what way does Dovecot > use Cyrus SASL? Hmm, I can't tell. I hope atrpm packager(s), if present on this list, can provide some feedback. Thanks again, Nick From tss at iki.fi Mon Mar 19 15:53:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:53:40 +0200 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> Message-ID: <1332165220.26095.71.camel@innu> On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: > > Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) > > > > It is the same. We will try 2.1.3 today and report the results... > The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d From tss at iki.fi Mon Mar 19 15:57:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 15:57:58 +0200 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <20120319095939.maneexuo@trusted.unix-scripts.info> References: <20120319095939.maneexuo@trusted.unix-scripts.info> Message-ID: <1332165478.26095.73.camel@innu> On Mon, 2012-03-19 at 10:06 +0100, Laurent CARON wrote: > I did the following modifications: > dovecot.conf: > namespace snaps-h0 { > prefix = INBOX.EmailBackup.h0. > hidden = no > list = yes > inbox = no > location = maildir:/home/.snapshot/hourly.0/%u/Maildir:INDEX=/var/tmp/dovecot/indexes/hourly.0/%u:CONTROL=/var/dovecot/control/hourly.0/%u > type = private > } > > Problem: > I don't see the content of the inbox folder contained in the snapshots > (subfolders are perfectly viewed). > > Do any of you have a clue on how to render it visible ? So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| new} directories? The INBOX should be accessible via the "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a newer Dovecot version. From tss at iki.fi Mon Mar 19 16:00:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:00:00 +0200 Subject: [Dovecot] Problem managing mbox In-Reply-To: <4F64E923.5060401@gmail.com> References: <4F64E923.5060401@gmail.com> Message-ID: <1332165600.26095.75.camel@innu> On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: > Hello, > > I have a problem with dovecot. seems that do not erase mail that mail > client request to be erased. Are you sure the clients have actually issued the EXPUNGE command, rather than simply marked the mail with \Deleted flag? > And I have this errors: > > Error: Next message unexpectedly corrupted in mbox file > Info: mbox code isn't perfect, but if this doesn't happen often it shouldn't matter much. doveconf -n output might have been helpful in giving more suggestions. From tss at iki.fi Mon Mar 19 16:02:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 19 Mar 2012 16:02:43 +0200 Subject: [Dovecot] importing plain mboxes to dovecot maildirs In-Reply-To: <4F64C9E3.7080102@filez.com> References: <4F6430E6.6040100@filez.com> <18416F94-424E-44F5-8C75-E835683E970A@iki.fi> <4F64C9E3.7080102@filez.com> Message-ID: <1332165763.26095.77.camel@innu> On Sat, 2012-03-17 at 18:29 +0100, Radim Kolar wrote: > > dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs are not permanent without index files > > Well, you can work around if by letting it create indexes. Hm. Why exactly can't it create indexes? Do you have some setting disabling them? > indexes never existed because these mboxes were never used by dovecot, > its not conversion from one format to another, its import. > > Maybe open bug to add feature "dsync import" which will not depend on > existing indexes? dsync doesn't need existing indexes, it wants to create indexes. If it can't because of e.g. permission issues, you should be able to work around it with: dsync mirror mbox:~/mail:INDEX=/tmp/indexes I might change dsync at some point to work even without permanent mailbox GUIDs, but there are many other more important things to do. From lcaron at lncsa.com Mon Mar 19 16:05:38 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 15:05:38 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <1332165478.26095.73.camel@innu> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> Message-ID: <4F673D32.2060300@lncsa.com> On 19/03/2012 14:57, Timo Sirainen wrote: > So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| > new} directories? The INBOX should be accessible via the > "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a > newer Dovecot version. > It's not actually. Sorry for the obvoius info I didn't give. I'm currently using dovecot 2.0.7 Regards, Laurent From rob0 at gmx.co.uk Mon Mar 19 17:28:20 2012 From: rob0 at gmx.co.uk (/dev/rob0) Date: Mon, 19 Mar 2012 10:28:20 -0500 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <4F6738EC.5050200@noa.gr> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> Message-ID: <20120319152820.GP4603@harrier.slackbuilds.org> On Mon, Mar 19, 2012 at 03:47:24PM +0200, Nikolaos Milas wrote: > On 19/3/2012 2:32 ??, /dev/rob0 wrote: > > >ISTM that herein lies the whole problem. Why did you not rpmbuild > >your OpenLDAP? That would have avoided all further fuss. > > Thanks for the reply. > > First, how would I rpmbuild my openldap v2.4.x as a standard CentOS > 5 package (i.e. replacing native openldap-2.3.43-25)? If I were > more experienced, I could have tried to engineer > openldap-2.3.43-25.el5.src.rpm to upgrade the system to use That's what I would have tried. > 2.4.x... But still, I haven't seen any OpenLDAP packages attempting > to do so, probably because of the tight integration of CentOS with > some openldap v2.3 libraries. I don't have anything to tell you there, and I note that we are now fully off-topic. :) > I think it's good that third-party packages (even of the same > software) give the ability to not mess with standard system. The same > is true for reputable Symas OpenLDAP packages. > > So, I simply use LTB OpenLDAP, even though it's installed at > non-standard locations. Failing the SRPM translation, why not just install into the CentOS standard locations? ... oops, I typed too fast ... > (This has an added benefit of easy migration. You can setup any/all > of those on the same system and decide which one to enable at any > time.) So you are in fact using both the CentOS OpenLDAP and your own version? This does not sound good at all. :( > >Another observation I can offer, unwelcome as it may be: your > >OS choice was not a good one when you want the features of > >recent software. Perhaps you should rethink that choice. You > >have invested much effort in this task. > > I like CentOS from many aspects as an enterprise server OS. I > wouldn't change it. I don't doubt that CentOS/RHEL offers many benefits, but my point here is that in this endeavor you are seeing the drawbacks. > Yet, it's important to me to be able to build/combine non-standard > packages. Even with CentOS 6, I would still continue to use LTB > OpenLDAP for a number of reasons. > > It's true that I've invested much effort in this task, but mostly > because my knowledge on this subject is very basic. And there too, the better forum, with more of the skills you need, would be the CentOS one. :) > Note that Dovecot RPM works fine as is (compiled with OpenLDAP 2.3), > i.e. there is no real need in re-building it using OpenLDAP 2.4 libs. > We just try to make things better (and make our life a bit more > difficult) :-) > > > > >And that is really more a question for a CentOS forum than here. > > > > True, but I am hoping that there might be some Dovecot RHEL/CentOS > packagers in this list, and that would help resolve issues more > effectively, as it is a Dovecot-specific (even if for a package > thereof) question. > > So, any help will be appreciated! > > >The latter requirement seems curious to me. In what way does > >Dovecot use Cyrus SASL? > > Hmm, I can't tell. I hope atrpm packager(s), if present on this > list, can provide some feedback. I was thinking maybe Timo would know. As far as I can tell it doesn't. I do see in configure.in's check for LDAP, a search for sasl.h or sasl/sasl.h, so it appears that Cyrus SASL might be required to build Dovecot's LDAP support. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: From nmilas at noa.gr Mon Mar 19 18:01:01 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Mon, 19 Mar 2012 18:01:01 +0200 Subject: [Dovecot] Building Dovecot RHEL RPMs with custom LDAP packages In-Reply-To: <20120319152820.GP4603@harrier.slackbuilds.org> References: <4F671676.3060809@noa.gr> <20120319123213.GN4603@harrier.slackbuilds.org> <4F6738EC.5050200@noa.gr> <20120319152820.GP4603@harrier.slackbuilds.org> Message-ID: <4F67583D.2000309@noa.gr> On 19/3/2012 5:28 ??, /dev/rob0 wrote: > So you are in fact using both the CentOS OpenLDAP and your own > version? This does not sound good at all. :( I talked about migration, didn't I? (Helps in test environments too!) > And there too, the better forum, with more of the skills you need, > would be the CentOS one. :) > > ... > > And that is really more a question for a CentOS forum than here. > I guess I might have to subscribe to CentOS forum/mailing list... (I haven't needed to yet.) Thanks anyway, Nick From hsn at filez.com Mon Mar 19 18:01:18 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:01:18 +0100 Subject: [Dovecot] INBOX cant be created Message-ID: <4F67584E.7030309@filez.com> Inbox does not exists on disk, but following command sequence will not create it. 2 select INBOX 2 NO Mailbox doesn't exist: INBOX 3 create INBOX 3 NO [ALREADYEXISTS] Mailbox already exists: INBOX i think its bug From lcaron at lncsa.com Mon Mar 19 18:11:04 2012 From: lcaron at lncsa.com (Laurent CARON) Date: Mon, 19 Mar 2012 17:11:04 +0100 Subject: [Dovecot] Accessing maildir snapshots through dovecot In-Reply-To: <4F673D32.2060300@lncsa.com> References: <20120319095939.maneexuo@trusted.unix-scripts.info> <1332165478.26095.73.camel@innu> <4F673D32.2060300@lncsa.com> Message-ID: <4F675A98.10302@lncsa.com> On 19/03/2012 15:05, Laurent CARON wrote: > On 19/03/2012 14:57, Timo Sirainen wrote: >> So the INBOX mails would be in /home/.snapshot/hourly.0/%u/Maildir/{cur| >> new} directories? The INBOX should be accessible via the >> "INBOX.EmailBackup.h0" folder itself. If it's not, you may need to use a >> newer Dovecot version. >> > > It's not actually. > > Sorry for the obvoius info I didn't give. > > I'm currently using dovecot 2.0.7 > > Regards, > > Laurent Upgrading did the trick. Thanks From hsn at filez.com Mon Mar 19 18:22:44 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 17:22:44 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67584E.7030309@filez.com> References: <4F67584E.7030309@filez.com> Message-ID: <4F675D54.4020203@filez.com> doveadm does not works too: sudo doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied sudo doveadm mailbox create -u admin INBOX.2 (works) From patrickdk at patrickdk.com Mon Mar 19 18:37:28 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Mon, 19 Mar 2012 12:37:28 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> Message-ID: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> I'm having this problem also, with a very very few users. But in my case the email isn't double gzip, just single like normal. Error: read(.../.Deleted Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output error (uid=250) All I have to do is rename the file to add back the lost S= part and all is fine. This has happened in the inbox, deleted, and trash folders so far. and always after a change, the S= exists for new emails. It's like it's loosing it on adding the read flag, and mailbox moves But out of millions of emails, only a very few are like this, that I know of, around 6 emails. I manually fixed them, will be looking to see if this issue comes back. Quoting Timo Sirainen : > On 2.3.2012, at 12.43, Ralf Hildebrandt wrote: > >>> Alternatively you can just tell Dovecot not to care about it: >>> maildir_broken_filename_sizes=yes. Although you probably can't do >>> that if you have compressed mails. >> >> In the case above that mail was gzipped twice :( > > Yes, looks like Dovecot can't correctly fix the wrong S size for > gzipped mails. I don't know if I should bother fixing it, especially > since in your case the doubly-gzipped mails will look corrupted to > user.. From alexwbaule at gmail.com Mon Mar 19 19:04:12 2012 From: alexwbaule at gmail.com (Alex Baule) Date: Mon, 19 Mar 2012 14:04:12 -0300 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F675D54.4020203@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: doveadm(admin): Error: Can't create mailbox INBOX: Permission denied The INBOX exists but has a wrong owner. Em 19 de mar?o de 2012 13:22, Radim Kolar escreveu: > doveadm does not works too: > > sudo doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > sudo doveadm mailbox create -u admin INBOX.2 > (works) > From hsn at filez.com Mon Mar 19 20:23:04 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 19 Mar 2012 19:23:04 +0100 Subject: [Dovecot] INBOX cant be created In-Reply-To: References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> Message-ID: <4F677988.9080403@filez.com> > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > > The INBOX exists but has a wrong owner. nope ponto# cd /var/mail ponto# mv admin/ admin.X ponto# doveadm mailbox create -u admin INBOX doveadm(admin): Error: Can't create mailbox INBOX: Permission denied but it might be that ordinary user admin cant create directories in /var/mail message from IMAP reply is wrong for sure because mailbox does not exists: ponto# cd /var/mail ponto# mv admin admin.x ponto# telnet localhost imap 3 select inbox 3 NO Mailbox doesn't exist: INBOX 4 create INBOX 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX From Ralf.Hildebrandt at charite.de Mon Mar 19 20:35:47 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Mon, 19 Mar 2012 19:35:47 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> Message-ID: <20120319183547.GA28363@charite.de> * Patrick Domack : > I'm having this problem also, with a very very few users. > > But in my case the email isn't double gzip, just single like normal. > > Error: read(.../.Deleted > Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output > error (uid=250) > > All I have to do is rename the file to add back the lost S= part and > all is fine. > This has happened in the inbox, deleted, and trash folders so far. > and always after a change, the S= exists for new emails. It's like > it's loosing it on adding the read flag, and mailbox moves Yes, I'm also seeing it now with mailboxes where no mail is doubly gzipped. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From alex.handle at gmail.com Mon Mar 19 21:16:58 2012 From: alex.handle at gmail.com (Alex Ha) Date: Mon, 19 Mar 2012 20:16:58 +0100 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: On Fri, Mar 16, 2012 at 9:39 PM, Alex Ha wrote: > On Fri, Mar 16, 2012 at 9:14 PM, Timo Sirainen wrote: >> On 16.3.2012, at 22.00, Alex Ha wrote: >> >>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>> existing connection >> >> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >> > > Thanks Timo! I will try the patch and report to you. > Hi Timo! I tried the patch with 2.0.19 and the dovecot error messages disappeared. I still get a lot of this postfix warnings: SASL LOGIN authentication failed: Connection lost to authentication server but only for ips which tried a sasl brute force attack. "Connection lost to authentication server" could this be because of the dovecot auth penalties? so far i did not get any complaints from users. Thanks for your help! Alex From dovecot at r.paypc.com Mon Mar 19 23:11:25 2012 From: dovecot at r.paypc.com (Robin) Date: Mon, 19 Mar 2012 14:11:25 -0700 Subject: [Dovecot] Creating an IMAP repo for ~100 users need some advice In-Reply-To: References: <4F64D885.8000501@gmail.com> <88l5d3on0ev8@mids.svenhartge.de> <4F64EC37.5010309@gmail.com> <98l5i9bn0ev8@mids.svenhartge.de> <4F650109.7090702@gmail.com> <4F6511B9.1020801@gmail.com> <4F6529D6.60609@gmail.com> Message-ID: <4F67A0FD.8050900@r.paypc.com> On 3/17/2012 12:36 PM, Sven Hartge wrote: > Storing mails inside SQL? Not supported by dovecot and not very wise, > IMHO. DBmail does this, but to be honest, I never heard any good > feedback from admins using that product. From what I have been told, you > need quite the beefy server to get a decent performance out of DBmail, > compared to the needs of a "traditional" setup like with dovecot or > courier-mail, but I digress. Ugh, I've tried the product. It works pretty well, until you move more than a small handful of users and email hives to it, and you hit some hard walls pretty fast with how many inbound emails/second it can handle for even burly server configurations. Those hard walls occur at too low a threshold for me. The product's mailing list is supportive and there are many dedicated DBMail users who step in an answer questions, but be prepared for "BUY MORE RAM" as the answer to concerns about performance. When 128GB of RAM is needed for a small organisation's email setup to perform well, I am strongly inclined to move on to the next product. Best practices for it seem to revolve around being able to have your ENTIRE email + index content resident in RAM. Well, gosh. Why didn't I think of that before instead of wasting all of this time worrying about design and efficiency? And if you're hoping that it will make text searches "automagically" fast, think again. Timo's FTS_SQUAT blows it out of the water by orders of magnitude, even with mailbox sizes of around 300K emails (20GB), let alone something like Lucene or Solr. I understand why it seems like a great idea to store email this way, but realise that the bulk of email is NOT structured or inherently relational. =R= From jsimmons at goblin.punk.net Tue Mar 20 01:04:29 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:04:29 -0700 Subject: [Dovecot] Using plaintext auth and SSL Message-ID: <201203191604.29407.jsimmons@goblin.punk.net> I'm working with a company that presently has a Linux mailserver which all users have (no shell) accounts on. Mail is accessed via pop3 with plaintext authentication. They want to move to a system using imap with SSL. I'm building them a new server. I'd like to offer both for a while so we can work the bugs out and migrate users over to SSL imap over time. It appears that in order to limit the imap connections to SSL I will need to run two separate instances of Dovecot. Is this correct? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From jsimmons at goblin.punk.net Tue Mar 20 01:37:05 2012 From: jsimmons at goblin.punk.net (Jeff Simmons) Date: Mon, 19 Mar 2012 16:37:05 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67BE5E.4000501@knutejohnson.com> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> Message-ID: <201203191637.05129.jsimmons@goblin.punk.net> On Monday, March 19, 2012 04:16:46 pm you wrote: > On 3/19/2012 4:04 PM, Jeff Simmons wrote: > > I'm working with a company that presently has a Linux mailserver which > > all users have (no shell) accounts on. Mail is accessed via pop3 with > > plaintext authentication. They want to move to a system using imap with > > SSL. I'm building them a new server. I'd like to offer both for a while > > so we can work the bugs out and migrate users over to SSL imap over > > time. It appears that in order to limit the imap connections to SSL I > > will need to run two separate instances of Dovecot. Is this correct? > > I only have SSL or TLS connections enabled and I only have one copy of > Dovecot running. Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls only authentication imap. The 'allow plaintext authentication' configuration directive appears to be global, meaning I will need to run two instances of dovecot for a while. Is that correct, or can this be done on a single instance of dovecot? -- Jeff Simmons jsimmons at goblin.punk.net Simmons Consulting - Network Engineering, Administration, Security "You guys, I don't hear any noise. Are you sure you're doing it right?" -- My Life With The Thrill Kill Kult From dovecot at knutejohnson.com Tue Mar 20 01:56:01 2012 From: dovecot at knutejohnson.com (Knute Johnson) Date: Mon, 19 Mar 2012 16:56:01 -0700 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67C791.2000609@knutejohnson.com> On 3/19/2012 4:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > I'm pretty sure if you set disable_plain_text_auth = no that you can log in to the appropriate ports with SSL or without. Sorry I sent the first reply to you, wasn't paying attention. -- Knute Johnson From eliezer at ngtech.co.il Tue Mar 20 02:16:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:16:21 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67CC55.3070706@ngtech.co.il> On 20/03/2012 01:37, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > there is no connection between the plaintext auth to the ssl\tls layer. you can just change the in the service section of the 10-master.conf file of the imap to no imap at all and use only imaps listener with port for your choose such as 143 or 993 and you will have a only imap over ssl. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:18:39 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:18:39 +0200 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F677988.9080403@filez.com> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> Message-ID: <4F67CCDF.2010309@ngtech.co.il> On 19/03/2012 20:23, Radim Kolar wrote: > >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >> >> The INBOX exists but has a wrong owner. > nope > ponto# cd /var/mail > ponto# mv admin/ admin.X > ponto# doveadm mailbox create -u admin INBOX > doveadm(admin): Error: Can't create mailbox INBOX: Permission denied get into the maildir folder and use: ls -la to see all the directories and permissions. it might be with a starting "." what will make it "invisible" to regular ls. Regards, Eliezer > > but it might be that ordinary user admin cant create directories in > /var/mail > message from IMAP reply is wrong for sure because mailbox does not exists: > > ponto# cd /var/mail > ponto# mv admin admin.x > ponto# telnet localhost imap > 3 select inbox > 3 NO Mailbox doesn't exist: INBOX > 4 create INBOX > 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 02:45:16 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 02:45:16 +0200 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67D31C.2030302@ngtech.co.il> On 20/03/2012 02:16, Eliezer Croitoru wrote: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. one mistake, change the imap service to port 0 and port 143 will be disabled with regular imap service > > Regards, > Eliezer > -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From florob at babelmonkeys.de Tue Mar 20 04:26:12 2012 From: florob at babelmonkeys.de (Florian Zeitz) Date: Tue, 20 Mar 2012 03:26:12 +0100 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <4F67CC55.3070706@ngtech.co.il> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> <4F67CC55.3070706@ngtech.co.il> Message-ID: <4F67EAC4.1050208@babelmonkeys.de> Am 20.03.2012 01:16, schrieb Eliezer Croitoru: > On 20/03/2012 01:37, Jeff Simmons wrote: >> On Monday, March 19, 2012 04:16:46 pm you wrote: >>> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>>> I'm working with a company that presently has a Linux mailserver which >>>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>>> plaintext authentication. They want to move to a system using imap with >>>> SSL. I'm building them a new server. I'd like to offer both for a while >>>> so we can work the bugs out and migrate users over to SSL imap over >>>> time. It appears that in order to limit the imap connections to SSL I >>>> will need to run two separate instances of Dovecot. Is this correct? >>> >>> I only have SSL or TLS connections enabled and I only have one copy of >>> Dovecot running. >> >> Let me rephrase that. I want to run plaintext authentication pop3 and >> ssl/tls >> only authentication imap. The 'allow plaintext authentication' >> configuration >> directive appears to be global, meaning I will need to run two >> instances of >> dovecot for a while. Is that correct, or can this be done on a single >> instance >> of dovecot? >> > there is no connection between the plaintext auth to the ssl\tls layer. > you can just change the in the service section of the 10-master.conf > file of the imap to no imap at all and use only imaps listener with port > for your choose such as 143 or 993 and you will have a only imap over ssl. > Because it is going to drive me insane if I don't ask: Is there really no way to archive this with a modern (aka. STARTTLS based) IMAP setup? From gedalya at gedalya.net Tue Mar 20 05:19:42 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:19:42 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67F74E.10403@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > This is all you have to do: protocol imap { ssl=required } See: http://wiki2.dovecot.org/SSL Globally, you can leave disable_plaintext_auth = no, and leave protocol pop3 {} alone. Your clients will be able to log in to pop3 with any authentication mechanism you have enabled, and imap will be accessible only with SSL/TLS, either over port 143 with STARTTLS or over port 993 with implicit SSL. I actually took the trouble to verify this on my local server before posting, and it turns out the wiki didn't lie. From gedalya at gedalya.net Tue Mar 20 05:42:38 2012 From: gedalya at gedalya.net (Gedalya) Date: Mon, 19 Mar 2012 23:42:38 -0400 Subject: [Dovecot] Using plaintext auth and SSL In-Reply-To: <201203191637.05129.jsimmons@goblin.punk.net> References: <201203191604.29407.jsimmons@goblin.punk.net> <4F67BE5E.4000501@knutejohnson.com> <201203191637.05129.jsimmons@goblin.punk.net> Message-ID: <4F67FCAE.9060205@gedalya.net> On 03/19/2012 07:37 PM, Jeff Simmons wrote: > On Monday, March 19, 2012 04:16:46 pm you wrote: >> On 3/19/2012 4:04 PM, Jeff Simmons wrote: >>> I'm working with a company that presently has a Linux mailserver which >>> all users have (no shell) accounts on. Mail is accessed via pop3 with >>> plaintext authentication. They want to move to a system using imap with >>> SSL. I'm building them a new server. I'd like to offer both for a while >>> so we can work the bugs out and migrate users over to SSL imap over >>> time. It appears that in order to limit the imap connections to SSL I >>> will need to run two separate instances of Dovecot. Is this correct? >> I only have SSL or TLS connections enabled and I only have one copy of >> Dovecot running. > Let me rephrase that. I want to run plaintext authentication pop3 and ssl/tls > only authentication imap. The 'allow plaintext authentication' configuration > directive appears to be global, meaning I will need to run two instances of > dovecot for a while. Is that correct, or can this be done on a single instance > of dovecot? > Turns out you can also use the disable_plaintext_auth = yes directive under protocol imap {}, but as noted by others previously, this is related specifically to plaintext authentication methods, and is not the same as requiring SSL/TLS for the entire session. If my understanding is correct, disable_plaintext_auth means your clients can authenticate with non-plaintext e.g. with CRAM-MD5 and proceed with an unsecured session. From jeetuindian at gmail.com Tue Mar 20 06:33:47 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 10:03:47 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hi guys, I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix in 2 system both system is having same configuration and os. I want to use dsync in mirror mode via ssh but I am not able to do it. When I followed the dsync wiki then I an unable to find the username : *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com dsync -u username* In the above command what will b user name and id_dsa.dovecot I am unable to understand. -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andrei.michescu at miau.ca Tue Mar 20 07:19:47 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Tue, 20 Mar 2012 01:19:47 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: <679071689c2e93f66654f318d277e8fc.squirrel@web.miau.ca> Hello Jitendra, I'm trying to do the same thing but I hit a wall as the sync seems to be doubling some emails in my case. My issues is still open with Timo (I HOPE!) As you seem to be stuck in an earlier stage here are my advices: 1) add the id_rsa to the .ssh folder to the user that runs the sync to test that this step is ok you should be able to $ ssh mailuser at example.com without being asked for a certificate / password (if you wonder how to achieve this here is a sample tutorial http://jaybyjayfresh.com/2009/02/04/logging-in-without-a-password-certificates-ssh/) 2) username is the user that has the email (if you work in a virtual environment generally is user at domain). 3) the syntax that I found to be working for me is the following: doveadm -Dv sync -u user1 at dom1 -f ssh mx1.a doveadm dsync-server -u user1 at dom1 To explain a little bit clearer the setup: - you start with 2 server mx1.a and mx2.a. On both servers you have vpopmail as the virtual user management for the virtual domain dom1 - from mx1.a you can ssh vpopmail at mx2.a directly without being prompted for a certificate or password) - user1 at dom1 is a virtual user defined both on mx1.a and on mx2.a (which means that you can deliver emails to this user both at mx1.a and mx2.a and you can also read them through imap on both servers). Hope this makes it a little bit more clear. Have fun, Andrei > Hi guys, > > I am using dovecot-2.1.0 in centos 5.7, I configured dovecot with postfix > in 2 system both system is having same configuration and os. I want to use > dsync in mirror mode via ssh but I am not able to do it. When I followed > the dsync wiki then I an unable to find the username : > > *dsync -u username mirror ssh -i id_dsa.dovecot mailuser at example.com > dsync -u username* > > In the above command what will b user name and id_dsa.dovecot I am unable > to understand. > > > > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > From achekalin at lazurit.com Tue Mar 20 07:28:30 2012 From: achekalin at lazurit.com (Alexander Chekalin) Date: Tue, 20 Mar 2012 08:28:30 +0300 Subject: [Dovecot] Per-user IMAP enable - is it possible? Message-ID: <4F68157E.5090806@lazurit.com> Just wonder if it is possible to enable/disable IMAP4 on Dovecot (2.0.x as far) on per-user basis? The deal is simple: our policy is not to store a lot of mailing on mailserver (the user should store it locally), thus the 'use POP3' approach, but for a vary few users it is permitted to use IMAP4. But users sometimes simple miss the point that some mail clients (e.g. TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of mailings and no local store of it on user's workstation. Sound too complicated, but setting up two Dovecots is not something I'd love to do as well. Thank you for any ideas, Alexander From gedalya at gedalya.net Tue Mar 20 07:43:10 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 01:43:10 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F68157E.5090806@lazurit.com> References: <4F68157E.5090806@lazurit.com> Message-ID: <4F6818EE.6090801@gedalya.net> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: > Just wonder if it is possible to enable/disable IMAP4 on Dovecot > (2.0.x as far) on per-user basis? > > The deal is simple: our policy is not to store a lot of mailing on > mailserver (the user should store it locally), thus the 'use POP3' > approach, but for a vary few users it is permitted to use IMAP4. But > users sometimes simple miss the point that some mail clients (e.g. TB) > 'prefer' to use IMAP4 first, and afterward I see mailbox full of > mailings and no local store of it on user's workstation. > > Sound too complicated, but setting up two Dovecots is not something > I'd love to do as well. > > Thank you for any ideas, > Alexander There would be various ways to do this, the specifics would depend on what kind of passdb you use. If you happen to be using a SQL database, you could do something like this: Add an allow_imap column, and change the password_query in dovecot-sql.conf.ext to something like this: password_query = SELECT password FROM user WHERE username = '%n' AND domain = '%d' \ AND ('%s' != 'imap' or allow_imap=1) This would make the user appear to not exist when trying to log in via IMAP. http://wiki2.dovecot.org/Variables From gedalya at gedalya.net Tue Mar 20 08:18:12 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 02:18:12 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6818EE.6090801@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> Message-ID: <4F682124.4010406@gedalya.net> On 3/20/2012 1:43 AM, Gedalya wrote: > On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >> (2.0.x as far) on per-user basis? >> >> The deal is simple: our policy is not to store a lot of mailing on >> mailserver (the user should store it locally), thus the 'use POP3' >> approach, but for a vary few users it is permitted to use IMAP4. But >> users sometimes simple miss the point that some mail clients (e.g. >> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >> mailings and no local store of it on user's workstation. >> >> Sound too complicated, but setting up two Dovecots is not something >> I'd love to do as well. >> >> Thank you for any ideas, >> Alexander > There would be various ways to do this, the specifics would depend on > what kind of passdb you use. > > If you happen to be using a SQL database, you could do something like > this: Add an allow_imap column, and change the password_query in > dovecot-sql.conf.ext to something like this: > > password_query = SELECT password FROM user WHERE username = '%n' AND > domain = '%d' \ > AND ('%s' != 'imap' or allow_imap=1) > > This would make the user appear to not exist when trying to log in via > IMAP. > > http://wiki2.dovecot.org/Variables > Or like this, might be more appropriate. password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, NULL, 'y') as nologin \ FROM user WHERE username = '%n' AND domain = '%d' http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin From cjeanneret at internux.ch Tue Mar 20 08:49:10 2012 From: cjeanneret at internux.ch (=?UTF-8?Q?C=C3=A9dric_Jeanneret?=) Date: Tue, 20 Mar 2012 08:49:10 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: On 19.03.2012 14:27, Antoine Nguyen wrote: > 2012/3/19 C?dric Jeanneret > >> Hello List! >> >> I have a tiny-teeny problem with dovecot + sieve: it seems that the >> LDA >> doesn't run sieve, and thus doesn't filter my emails. >> >> Here's the sieve configuration: >> >> plugin { >> # Used by both the Sieve plugin and the ManageSieve protocol >> sieve=/var/local/vmail/%n/**dovecot.sieve >> sieve_dir=/var/local/vmail/%n/**sieve >> sieve_extensions = +notify +imapflags >> } >> >> The managesiege: >> >> protocol managesieve { >> # Specify an alternative address:port the daemon must listen on >> # (default: *:2000) >> listen = localhost:2000 >> managesieve_logout_format = bytes ( in=%i : out=%o ) >> } >> (this one is working fine, I can edit the filters through roundcube >> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) >> is >> edited) >> >> the lda part: >> >> protocol lda { >> postmaster_address = foo at bar.com >> mail_plugins = sieve >> } >> >> I think all is in place to allow dovecot to use sieve... ? >> >> One more thing: >> >> dovecot --version >> 1.2.15 >> >> >> Any help will be welcomed :). >> >> Thanks in advance ! >> >> Cheers, >> >> C. >> > > Have you checked the MTA configuration. Does it use dovecot's LDA ? > > Antoine Hello Antoine (and List), Well, it should use dovecot, as the mails are delivered to the user inbox - and I don't think postfix knows about them.. How may I be sure otherwise that postfix really uses dovecot? Cheers, C. From evocage at gmail.com Tue Mar 20 09:06:42 2012 From: evocage at gmail.com (evolution age) Date: Tue, 20 Mar 2012 12:36:42 +0530 Subject: [Dovecot] Dovecot with postfix setup Message-ID: Hey frnds, Could you suggest me any ebook or documentation for the setup of dovecot with postfix on centos 5.7 . I need it. -- Warm Regards Jitendra Kumar Bhaskar cell :- +91-8886742555 From gedalya at gedalya.net Tue Mar 20 09:12:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:12:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: References: Message-ID: <4F682DF3.2030409@gedalya.net> On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > On 19.03.2012 14:27, Antoine Nguyen wrote: >> 2012/3/19 C?dric Jeanneret >> >>> Hello List! >>> >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA >>> doesn't run sieve, and thus doesn't filter my emails. >>> >>> Here's the sieve configuration: >>> >>> plugin { >>> # Used by both the Sieve plugin and the ManageSieve protocol >>> sieve=/var/local/vmail/%n/**dovecot.sieve >>> sieve_dir=/var/local/vmail/%n/**sieve >>> sieve_extensions = +notify +imapflags >>> } >>> >>> The managesiege: >>> >>> protocol managesieve { >>> # Specify an alternative address:port the daemon must listen on >>> # (default: *:2000) >>> listen = localhost:2000 >>> managesieve_logout_format = bytes ( in=%i : out=%o ) >>> } >>> (this one is working fine, I can edit the filters through roundcube >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is >>> edited) >>> >>> the lda part: >>> >>> protocol lda { >>> postmaster_address = foo at bar.com >>> mail_plugins = sieve >>> } >>> >>> I think all is in place to allow dovecot to use sieve... ? >>> >>> One more thing: >>> >>> dovecot --version >>> 1.2.15 >>> >>> >>> Any help will be welcomed :). >>> >>> Thanks in advance ! >>> >>> Cheers, >>> >>> C. >>> >> >> Have you checked the MTA configuration. Does it use dovecot's LDA ? >> >> Antoine > > Hello Antoine (and List), > > Well, it should use dovecot, as the mails are delivered to the user > inbox - and I don't think postfix knows about them.. > How may I be sure otherwise that postfix really uses dovecot? > > Cheers, > > C. If you are using dovecot 1.2, I think lda should show lines like the following in your log. Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: msgid=: stored mail into mailbox 'INBOX' 'deliver' refers to lda. From cjeanneret at internux.ch Tue Mar 20 09:20:41 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:20:41 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F682DF3.2030409@gedalya.net> References: <4F682DF3.2030409@gedalya.net> Message-ID: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:12:51 -0400 Gedalya wrote: > On 3/20/2012 2:49 AM, C?dric Jeanneret wrote: > > On 19.03.2012 14:27, Antoine Nguyen wrote: > >> 2012/3/19 C?dric Jeanneret > >> > >>> Hello List! > >>> > >>> I have a tiny-teeny problem with dovecot + sieve: it seems that the LDA > >>> doesn't run sieve, and thus doesn't filter my emails. > >>> > >>> Here's the sieve configuration: > >>> > >>> plugin { > >>> # Used by both the Sieve plugin and the ManageSieve protocol > >>> sieve=/var/local/vmail/%n/**dovecot.sieve > >>> sieve_dir=/var/local/vmail/%n/**sieve > >>> sieve_extensions = +notify +imapflags > >>> } > >>> > >>> The managesiege: > >>> > >>> protocol managesieve { > >>> # Specify an alternative address:port the daemon must listen on > >>> # (default: *:2000) > >>> listen = localhost:2000 > >>> managesieve_logout_format = bytes ( in=%i : out=%o ) > >>> } > >>> (this one is working fine, I can edit the filters through roundcube > >>> webmail, and the correct file (/var/local/vmail/%n/dovecot.**sieve) is > >>> edited) > >>> > >>> the lda part: > >>> > >>> protocol lda { > >>> postmaster_address = foo at bar.com > >>> mail_plugins = sieve > >>> } > >>> > >>> I think all is in place to allow dovecot to use sieve... ? > >>> > >>> One more thing: > >>> > >>> dovecot --version > >>> 1.2.15 > >>> > >>> > >>> Any help will be welcomed :). > >>> > >>> Thanks in advance ! > >>> > >>> Cheers, > >>> > >>> C. > >>> > >> > >> Have you checked the MTA configuration. Does it use dovecot's LDA ? > >> > >> Antoine > > > > Hello Antoine (and List), > > > > Well, it should use dovecot, as the mails are delivered to the user > > inbox - and I don't think postfix knows about them.. > > How may I be sure otherwise that postfix really uses dovecot? > > > > Cheers, > > > > C. > > If you are using dovecot 1.2, I think lda should show lines like the > following in your log. > > Mar 11 14:14:06 mailstor1 dovecot: deliver(user at domain.tld): sieve: > msgid=: stored mail into mailbox 'INBOX' > > 'deliver' refers to lda. > Hmm... strange, doesn't show up like that in logs: Mar 20 08:14:54 sqdf3 postfix/smtpd[27509]: connect from host.foo.bar[...] Mar 20 08:15:16 sqdf3 postfix/smtpd[27509]: 892335659F4: client=host.foo.bar[...] Mar 20 08:15:35 sqdf3 postfix/cleanup[27516]: 892335659F4: message-id=<> Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 08:15:35 sqdf3 postfix/virtual[27518]: 892335659F4: to=, relay=virtual, delay=27, delays=27/0.03/0/0.14, dsn=2.0.0, status=sent (delivered to maildir) Mar 20 08:15:35 sqdf3 postfix/qmgr[11614]: 892335659F4: removed Maybe the problem is there (postfix main.cf): virtual_transport = virtual I have virtual users and a mailman running on the same domain :/. If postfix doesn't use dovecot, how comes that mails are put in the right place ? From 24x7server at 24x7server.net Tue Mar 20 09:35:56 2012 From: 24x7server at 24x7server.net (Rajesh M) Date: Tue, 20 Mar 2012 13:05:56 +0530 (Asi) Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 Message-ID: <.120.61.90.33.1332228956.squirrel@24x7server.net> hi my system is a centos 5 with qmailtoaster i migrated my email server with around 5000 users from dovecot version 1.2 to version 2 i have two separate 2 tb hdd's storing webmail data of these users. the load on the server goes very high over 100 during peak load times and the imap connections get dropped frequently, webmail becomes very slow. however pop3 download works ok in the dovecot log file i get errors as such Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 expunge attempts) i had configured dovecot 1.2 using source but i had installed version 2 using qmailtoaster rpm i am a bit confused as to what settings are to be done for a very busy server note that the cpu system usage and ram usage are less but IO wait goes to around 80-90 percent can you please guide me or post some dovecot version 2 config file settings that are relevant to a busy server. rajesh From gedalya at gedalya.net Tue Mar 20 09:36:51 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 03:36:51 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683393.60403@gedalya.net> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > status=sent (delivered to maildir) Your log clearly says postfix is delivering directly to the maildir. Postfix must be configured to find the user's maildir using things like virtual_mailbox_base and / or virtual_mailbox_maps ? From cjeanneret at internux.ch Tue Mar 20 09:44:58 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 08:44:58 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683393.60403@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> Message-ID: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 03:36:51 -0400 Gedalya wrote: > On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > > status=sent (delivered to maildir) > Your log clearly says postfix is delivering directly to the maildir. > Postfix must be configured to find the user's maildir using things like > virtual_mailbox_base and / or virtual_mailbox_maps ? > Geez.. right - taking it from ldap... should only comment out the following lines: virtual_mailbox_base = / virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf ? From gedalya at gedalya.net Tue Mar 20 10:01:08 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:01:08 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F683944.2030408@gedalya.net> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 03:36:51 -0400 > Gedalya wrote: > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>> status=sent (delivered to maildir) >> Your log clearly says postfix is delivering directly to the maildir. >> Postfix must be configured to find the user's maildir using things like >> virtual_mailbox_base and / or virtual_mailbox_maps ? >> > Geez.. right - taking it from ldap... should only comment out the following lines: > virtual_mailbox_base = / > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > ? If you want postfix to use the dovecot LDA then you have to set it up, you can get a general idea here http://wiki.dovecot.org/LDA/Postfix and adapt it to your circumstances. Basically you have to add the appropriate lines to master.cf and put virtual_transport = dovecot in main.cf. From cjeanneret at internux.ch Tue Mar 20 10:23:30 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 09:23:30 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F683944.2030408@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> Message-ID: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 04:01:08 -0400 Gedalya wrote: > On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: > > On Tue, 20 Mar 2012 03:36:51 -0400 > > Gedalya wrote: > > > >> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: > >>> status=sent (delivered to maildir) > >> Your log clearly says postfix is delivering directly to the maildir. > >> Postfix must be configured to find the user's maildir using things like > >> virtual_mailbox_base and / or virtual_mailbox_maps ? > >> > > Geez.. right - taking it from ldap... should only comment out the following lines: > > virtual_mailbox_base = / > > virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf > > > > ? > If you want postfix to use the dovecot LDA then you have to set it up, > you can get a general idea here > http://wiki.dovecot.org/LDA/Postfix > and adapt it to your circumstances. > > Basically you have to add the appropriate lines to master.cf and put > virtual_transport = dovecot in main.cf. > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. It _should_ take transport_maps = hash:/etc/postfix/transport which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( From gedalya at gedalya.net Tue Mar 20 10:50:33 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 04:50:33 -0400 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6844D9.7050504@gedalya.net> On 3/20/2012 4:23 AM, Cedric Jeanneret wrote: > On Tue, 20 Mar 2012 04:01:08 -0400 > Gedalya wrote: > >> On 3/20/2012 3:44 AM, Cedric Jeanneret wrote: >>> On Tue, 20 Mar 2012 03:36:51 -0400 >>> Gedalya wrote: >>> >>>> On 3/20/2012 3:20 AM, Cedric Jeanneret wrote: >>>>> status=sent (delivered to maildir) >>>> Your log clearly says postfix is delivering directly to the maildir. >>>> Postfix must be configured to find the user's maildir using things like >>>> virtual_mailbox_base and / or virtual_mailbox_maps ? >>>> >>> Geez.. right - taking it from ldap... should only comment out the following lines: >>> virtual_mailbox_base = / >>> virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf >>> >>> ? >> If you want postfix to use the dovecot LDA then you have to set it up, >> you can get a general idea here >> http://wiki.dovecot.org/LDA/Postfix >> and adapt it to your circumstances. >> >> Basically you have to add the appropriate lines to master.cf and put >> virtual_transport = dovecot in main.cf. >> > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > It _should_ take > transport_maps = hash:/etc/postfix/transport > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( Once you have dovecot set up in your master.cf you should be able to put as the last line simply domain.com dovecot What do your logs look like when this is set? From cjeanneret at internux.ch Tue Mar 20 11:08:01 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 10:08:01 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6844D9.7050504@gedalya.net> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> Message-ID: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> > > won't work as I also have a mailman on the same domain - and dovecot doesn't know about mailman lists... That's why I used virtual. > > It _should_ take > > transport_maps = hash:/etc/postfix/transport > > which specify "use mailman for foo at domain.com" and "use dovecot for @domain.com" (at the end)... but doesn't seem to work as expected :( > Once you have dovecot set up in your master.cf you should be able to put > as the last line simply > domain.com dovecot > What do your logs look like when this is set? > I guess it may be easier if I paste my whole config in here: postfix main.cf: smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases mydestination = public_fqdn, localhost mynetworks = 127.0.0.0/8 inet_interfaces = all recipient_delimiter = + smtpd_sasl_security_options = noanonymous myorigin = domain.ltd smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_sasl_path = private/auth smtpd_sasl_type = dovecot owner_request_special = no smtpd_tls_CAfile = /etc/postfix/ssl/ca.crt smtpd_tls_cert_file = /etc/postfix/ssl/cert.crt smtpd_tls_key_file = /etc/postfix/ssl/keyForApache2.key smtpd_use_tls = yes smtpd_sasl_tls_security_options = $smtpd_sasl_security_options smtpd_sasl_auth_enable = yes default_process_limit = 5 smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/access, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unlisted_recipient smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache broken_sasl_auth_clients = yes transport_maps = hash:/etc/postfix/transport virtual_transport = dovecot transport_maps = hash:/etc/postfix/transport virtual_alias_maps = ldap:/etc/postfix/ldap-aliases.cf, hash:/var/lib/mailman/data/virtual-mailman virtual_gid_maps = static:104 virtual_minimum_uid = 8 virtual_uid_maps = static:8 virtual_mailbox_base = / mailman_destination_recipient_limit = 1 virtual_mailbox_domains = avocats-ch.ch virtual_mailbox_maps = ldap:/etc/postfix/ldap-accounts.cf With this configuration, here's what I get: sending to a virtual user: Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/smtpd[1525]: DC9285659F4: client=remote.host.ltd[...] Mar 20 10:02:48 sqdf3 postfix/cleanup[1528]: DC9285659F4: message-id=<> Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: from=, size=279, nrcpt=1 (queue active) Mar 20 10:02:49 sqdf3 dovecot: deliver(camptocamp): msgid=: saved mail to INBOX Mar 20 10:02:49 sqdf3 postfix/pipe[1529]: DC9285659F4: to=, relay=dovecot, delay=0.45, delays=0.18/0.01/0/0.26, dsn=2.0.0, status=sent (delivered via dovecot service) Mar 20 10:02:49 sqdf3 postfix/qmgr[1462]: DC9285659F4: removed so it uses dovecot. but mail is not filtered as it should :(( And now, sending a mail to a mailman list: Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: connect from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: improper command pipelining after EHLO from remote.host.ltd[...] Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: NOQUEUE: reject: RCPT from remote.host.ltd[...]: 550 5.1.1 : Recipient address rejected: User unknown in virtual mailbox table; from= to= proto=ESMTP helo= Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: warning: non-SMTP command from remote.host.ltd[...]: Subject: testing mailman Mar 20 10:06:25 sqdf3 postfix/smtpd[1525]: disconnect from remote.host.ltd[...] may I cry ? :( From luca.palazzo at unict.it Tue Mar 20 11:09:12 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 20 Mar 2012 10:09:12 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash Message-ID: <4F684938.9000208@unict.it> Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. Log lines are like: Mar 20 10:05:45 mailgw-lb dovecot: pop3-login: Fatal: master: service(pop3-login): child 27764 killed with signal 11 (core dumps disabled) Mar 20 10:06:17 mailgw-lb dovecot: imap-login: Fatal: master: service(imap-login): child 28468 killed with signal 11 (core dumps disabled) Stack trace of a crashed process is like: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1220163904 (LWP 27764)] 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 #4 0xb77ca9b5 in server_input (proxy=0x0) at login-proxy.c:93 #5 0xb7793762 in io_loop_call_io (io=0x8094180) at ioloop.c:380 #6 0xb7794cc9 in io_loop_handler_run (ioloop=0x8055480) at ioloop-epoll.c:213 #7 0xb77936f9 in io_loop_run (ioloop=0x8055480) at ioloop.c:399 #8 0xb777e4c8 in master_service_run (service=0x80553b0, callback=0xb77cc110 ) at master-service.c:544 #9 0xb77cbcee in login_binary_run (binary=0x804ad80, argc=2, argv=0x80551c0) at main.c:406 #10 0x08049812 in main (argc=0, argv=0x0) at client.c:303 The strange part of the story is that not all process crash. I'm trying to figure out if only TLS/SSL process crash. Any idea? Thanks Luca From eliezer at ngtech.co.il Tue Mar 20 11:42:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:42:21 +0200 Subject: [Dovecot] Problem with sieve In-Reply-To: <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F6850FD.9010602@ngtech.co.il> On 20/03/2012 11:08, Cedric Jeanneret wrote: > I guess it may be easier if I paste my whole config in here: you didnt sent the virtual_transport file content. i will quote from the man pages of the transport: [quote] user at domain transport:nexthop Deliver mail for user at domain through transport to nexthop. [\quote] means you can specify specific transport such as maliman to specific user. but because you are using the virtual maps table\lookup you also must have a vaild ldap user with the same name for the list. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From eliezer at ngtech.co.il Tue Mar 20 11:45:21 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Tue, 20 Mar 2012 11:45:21 +0200 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F682124.4010406@gedalya.net> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> Message-ID: <4F6851B1.4030509@ngtech.co.il> On 20/03/2012 08:18, Gedalya wrote: > On 3/20/2012 1:43 AM, Gedalya wrote: >> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>> (2.0.x as far) on per-user basis? >>> >>> The deal is simple: our policy is not to store a lot of mailing on >>> mailserver (the user should store it locally), thus the 'use POP3' >>> approach, but for a vary few users it is permitted to use IMAP4. But >>> users sometimes simple miss the point that some mail clients (e.g. >>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>> mailings and no local store of it on user's workstation. >>> >>> Sound too complicated, but setting up two Dovecots is not something >>> I'd love to do as well. >>> >>> Thank you for any ideas, >>> Alexander >> There would be various ways to do this, the specifics would depend on >> what kind of passdb you use. >> >> If you happen to be using a SQL database, you could do something like >> this: Add an allow_imap column, and change the password_query in >> dovecot-sql.conf.ext to something like this: >> >> password_query = SELECT password FROM user WHERE username = '%n' AND >> domain = '%d' \ >> AND ('%s' != 'imap' or allow_imap=1) >> >> This would make the user appear to not exist when trying to log in via >> IMAP. >> >> http://wiki2.dovecot.org/Variables >> > Or like this, might be more appropriate. > > password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, > NULL, 'y') as nologin \ > FROM user WHERE username = '%n' AND domain = '%d' > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin > but this will disallow also pop3... Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations elilezer ngtech.co.il From gedalya at gedalya.net Tue Mar 20 11:49:45 2012 From: gedalya at gedalya.net (Gedalya) Date: Tue, 20 Mar 2012 05:49:45 -0400 Subject: [Dovecot] Per-user IMAP enable - is it possible? In-Reply-To: <4F6851B1.4030509@ngtech.co.il> References: <4F68157E.5090806@lazurit.com> <4F6818EE.6090801@gedalya.net> <4F682124.4010406@gedalya.net> <4F6851B1.4030509@ngtech.co.il> Message-ID: <4F6852B9.1050809@gedalya.net> On 3/20/2012 5:45 AM, Eliezer Croitoru wrote: > On 20/03/2012 08:18, Gedalya wrote: >> On 3/20/2012 1:43 AM, Gedalya wrote: >>> On 3/20/2012 1:28 AM, Alexander Chekalin wrote: >>>> Just wonder if it is possible to enable/disable IMAP4 on Dovecot >>>> (2.0.x as far) on per-user basis? >>>> >>>> The deal is simple: our policy is not to store a lot of mailing on >>>> mailserver (the user should store it locally), thus the 'use POP3' >>>> approach, but for a vary few users it is permitted to use IMAP4. But >>>> users sometimes simple miss the point that some mail clients (e.g. >>>> TB) 'prefer' to use IMAP4 first, and afterward I see mailbox full of >>>> mailings and no local store of it on user's workstation. >>>> >>>> Sound too complicated, but setting up two Dovecots is not something >>>> I'd love to do as well. >>>> >>>> Thank you for any ideas, >>>> Alexander >>> There would be various ways to do this, the specifics would depend on >>> what kind of passdb you use. >>> >>> If you happen to be using a SQL database, you could do something like >>> this: Add an allow_imap column, and change the password_query in >>> dovecot-sql.conf.ext to something like this: >>> >>> password_query = SELECT password FROM user WHERE username = '%n' AND >>> domain = '%d' \ >>> AND ('%s' != 'imap' or allow_imap=1) >>> >>> This would make the user appear to not exist when trying to log in via >>> IMAP. >>> >>> http://wiki2.dovecot.org/Variables >>> >> Or like this, might be more appropriate. >> >> password_query = SELECT password, if('%s' != 'imap' or allow_imap=1, >> NULL, 'y') as nologin \ >> FROM user WHERE username = '%n' AND domain = '%d' >> >> http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/NoLogin >> > but this will disallow also pop3... > > > Eliezer > No. It will return NULL unless the service is 'imap' and allow_imap != 1. nologin=NULL has no effect, so everything is allowed. From cjeanneret at internux.ch Tue Mar 20 12:00:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 11:00:21 +0100 Subject: [Dovecot] Problem with sieve In-Reply-To: <4F6850FD.9010602@ngtech.co.il> References: <4F682DF3.2030409@gedalya.net> <20120320082041.74bc26f9@cholatse.wrk.lsn.camptocamp.com> <4F683393.60403@gedalya.net> <20120320084458.4b3b8dd1@cholatse.wrk.lsn.camptocamp.com> <4F683944.2030408@gedalya.net> <20120320092330.6c7fa79b@cholatse.wrk.lsn.camptocamp.com> <4F6844D9.7050504@gedalya.net> <20120320100801.10815c64@cholatse.wrk.lsn.camptocamp.com> <4F6850FD.9010602@ngtech.co.il> Message-ID: <20120320110021.529fba41@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 11:42:21 +0200 Eliezer Croitoru wrote: > On 20/03/2012 11:08, Cedric Jeanneret wrote: > > I guess it may be easier if I paste my whole config in here: > you didnt sent the virtual_transport file content. > i will quote from the man pages of the transport: > [quote] > user at domain transport:nexthop > Deliver mail for user at domain through transport to nexthop. > [\quote] > means you can specify specific transport such as maliman to specific user. > but because you are using the virtual maps table\lookup you also must > have a vaild ldap user with the same name for the list. > > Regards, > Eliezer > Hello, while trying to remove/add options to my postfix, this part is now working now - it uses dovecot for virtual users, and mailman for lists :). Now that's good, I'll go back to sieve and ensure there's no missing configuration in dovecot. I had to remove "virtual_transport" option, and remove an "@" in my transport map. I'll come back on this thread if I still have problem with sieve. Cheers, C. From cjeanneret at internux.ch Tue Mar 20 13:05:28 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 12:05:28 +0100 Subject: [Dovecot] dovecot, sieve and vacation Message-ID: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Hi there ! Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to after searching a bit on the net, I stumbled on another (old) thread: http://www.mail-archive.com/dovecot at dovecot.org/msg25955.html I'm not really sure it's the same problem, as I'm sending the mail from another host (via telnet, for testing purpose). Other sieve rules (such as flagging, moving and so on) work fine. Only vacation is crapy. Here's the roundcube generated rule: require ["vacation"]; if true { vacation :days 2 "on holidays!"; } Any help welcome :) Cheers, C. Informations: # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-6-pve i686 Debian 6.0.4 simfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): localhost:2000 ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login first_valid_uid: 8 mail_privileged_group: mail mail_uid: mail mail_gid: mail mail_location: maildir:/var/local/vmail/%n mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve managesieve_logout_format(default): bytes=%i/%o managesieve_logout_format(imap): bytes=%i/%o managesieve_logout_format(pop3): bytes=%i/%o managesieve_logout_format(managesieve): bytes ( in=%i : out=%o ) lda: postmaster_address: foo at bar.com mail_plugins: sieve auth default: mechanisms: plain login user: mail passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf.ext socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: mail master: path: /var/run/dovecot/auth-master mode: 432 user: postfix group: mail plugin: home: /var/local/vmail/%u sieve: /var/local/vmail/%n/.dovecot.sieve sieve_dir: /var/local/vmail/%n/sieve sieve_extensions: +notify +imapflags From jeetuindian at gmail.com Tue Mar 20 13:10:46 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Tue, 20 Mar 2012 16:40:46 +0530 Subject: [Dovecot] Dsync Dovecot Message-ID: Hey Frnds, Could you tell me about the error : # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com doveadm(root): Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, home=/home/example1.com/jitendra.b doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:/home/ example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b doveadm(jitendra.b at example.com): Debug: fs: root=/home/ example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ example.com/jitendra.b, alt= dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 doveadm(jitendra.b): Fatal: User doesn't exist dsync-local(jitendra.b at example.com): Error: read() from worker server failed: EOF -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From andreas.a.lamprecht at atos.net Tue Mar 20 13:16:33 2012 From: andreas.a.lamprecht at atos.net (Lamprecht, Andreas) Date: Tue, 20 Mar 2012 11:16:33 +0000 Subject: [Dovecot] IMAP and POP3 per SSL Message-ID: Hi! I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set SSLHonorCipherOrder On in apache config. This results in the following C-Code being executed: SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. Is there a way to implement such a setting into Dovecot, too? I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 *************** *** 924,930 **** X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); --- 924,930 ---- X509_STORE *store; STACK_OF(X509_NAME) *xnames = NULL; ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); if (*set->ssl_ca != '\0') { /* set trusted CA certs */ store = SSL_CTX_get_cert_store(ssl_ctx); Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... So, maybe you have the time to look over it and implement a final solution for the BEAST problem. Greetings Andreas lamprecht From support at palatineweb.com Tue Mar 20 13:26:56 2012 From: support at palatineweb.com (Palatine Support) Date: Tue, 20 Mar 2012 11:26:56 +0000 Subject: [Dovecot] INBOX cant be created In-Reply-To: <4F67CCDF.2010309@ngtech.co.il> References: <4F67584E.7030309@filez.com> <4F675D54.4020203@filez.com> <4F677988.9080403@filez.com> <4F67CCDF.2010309@ngtech.co.il> Message-ID: <4F686980.5040600@palatineweb.com> I have tried to unsubscribe from this mailing list 10 times now. Remove my email address please asap. Thanks Paul On 20/03/2012 00:18, Eliezer Croitoru wrote: > On 19/03/2012 20:23, Radim Kolar wrote: >> >>> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied >>> >>> The INBOX exists but has a wrong owner. >> nope >> ponto# cd /var/mail >> ponto# mv admin/ admin.X >> ponto# doveadm mailbox create -u admin INBOX >> doveadm(admin): Error: Can't create mailbox INBOX: Permission denied > get into the maildir folder and use: > ls -la to see all the directories and permissions. > it might be with a starting "." what will make it "invisible" to > regular ls. > > Regards, > Eliezer > >> >> but it might be that ordinary user admin cant create directories in >> /var/mail >> message from IMAP reply is wrong for sure because mailbox does not >> exists: >> >> ponto# cd /var/mail >> ponto# mv admin admin.x >> ponto# telnet localhost imap >> 3 select inbox >> 3 NO Mailbox doesn't exist: INBOX >> 4 create INBOX >> 4 NO [ALREADYEXISTS] Mailbox already exists: INBOX > > From robert at schetterer.org Tue Mar 20 13:32:04 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:32:04 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: References: Message-ID: <4F686AB4.3070506@schetterer.org> Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: > Hi! > > I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. > > Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. > The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 > > "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set > > SSLHonorCipherOrder On > > in apache config. This results in the following C-Code being executed: > > SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); > > This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. > > Is there a way to implement such a setting into Dovecot, too? > > I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. > This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: > > *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 > --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 > *************** > *** 924,930 **** > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > --- 924,930 ---- > X509_STORE *store; > STACK_OF(X509_NAME) *xnames = NULL; > > ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); > if (*set->ssl_ca != '\0') { > /* set trusted CA certs */ > store = SSL_CTX_get_cert_store(ssl_ctx); > > > Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... > > So, maybe you have the time to look over it and implement a final solution for the BEAST problem. > > Greetings > Andreas lamprecht > perhaps look at http://wiki2.dovecot.org/SSL/DovecotConfiguration -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From robert at schetterer.org Tue Mar 20 13:34:58 2012 From: robert at schetterer.org (Robert Schetterer) Date: Tue, 20 Mar 2012 12:34:58 +0100 Subject: [Dovecot] IMAP and POP3 per SSL In-Reply-To: <4F686AB4.3070506@schetterer.org> References: <4F686AB4.3070506@schetterer.org> Message-ID: <4F686B62.2050205@schetterer.org> Am 20.03.2012 12:32, schrieb Robert Schetterer: > Am 20.03.2012 12:16, schrieb Lamprecht, Andreas: >> Hi! >> >> I'm new to this list and i could not find a way to search through the already posted articles, so please forgive me if this subject has been discussed before. >> >> Our security scanner stumbled over the IMAPs server i've set up recently using dovecot on a RedHat Enterprise 64bit Server. >> The security scanner found an error regarding a new SSL security leak named "BEAST". The exact error number is CVE-2011-3389. Details can be found here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3389 >> >> "The internet" has some workarounds for this problem. For example, in Apache webserver, you need to set >> >> SSLHonorCipherOrder On >> >> in apache config. This results in the following C-Code being executed: >> >> SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE); >> >> This setting tells OpenSSL not to honor the Ciper Order sent from the client, but prefer it's own configured set of CipherSuites. According to Qualis SSL Labs ( https://www.ssllabs.com/ssldb/index.html ), a webserver configured with this setting is not affected by that BEAST security leak. >> >> Is there a way to implement such a setting into Dovecot, too? >> >> I have created a very quick and dirty solution to avoid being listed on our internal security problem's list. >> This patch is for dovecot 2.0.9 which is included in Redhat Enterprise Linux 6.2: >> >> *** src/login-common/ssl-proxy-openssl.c 2010-12-30 10:42:54.000000000 +0100 >> --- src/login-common/ssl-proxy-openssl.c_1 2012-03-20 09:48:28.359508087 +0100 >> *************** >> *** 924,930 **** >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> --- 924,930 ---- >> X509_STORE *store; >> STACK_OF(X509_NAME) *xnames = NULL; >> >> ! SSL_CTX_set_options(ssl_ctx, SSL_OP_ALL | SSL_OP_NO_SSLv2 | SSL_OP_CIPHER_SERVER_PREFERENCE ); >> if (*set->ssl_ca != '\0') { >> /* set trusted CA certs */ >> store = SSL_CTX_get_cert_store(ssl_ctx); >> >> >> Of course there should be a way to switch this setting on or off, but my C programming skills are rather basic ... >> >> So, maybe you have the time to look over it and implement a final solution for the BEAST problem. >> >> Greetings >> Andreas lamprecht >> > > perhaps look at > > http://wiki2.dovecot.org/SSL/DovecotConfiguration > and perhaps have a look at http://hg.dovecot.org/dovecot-2.0/rev/e3d46fd04105 and upgrade your dove version to dovecot 2.0.18 -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria From CMarcus at Media-Brokers.com Tue Mar 20 14:22:59 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 20 Mar 2012 08:22:59 -0400 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F6876A3.2040409@Media-Brokers.com> On 2012-03-20 3:06 AM, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. Distro specific questions should be directed to your distro support lists. -- Best regards, Charles From nmilas at noa.gr Tue Mar 20 15:12:31 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 20 Mar 2012 15:12:31 +0200 Subject: [Dovecot] Dovecot with postfix setup In-Reply-To: References: Message-ID: <4F68823F.4040905@noa.gr> On 20/3/2012 9:06 ??, evolution age wrote: > Could you suggest me any ebook or documentation for the setup of dovecot > with postfix on centos 5.7 . I need it. You have not provided any info on your requirements, so it's hard to provide assistance. If you are only now starting the design (you should devote at least some time to it - design is the cornerstone) and you want LDAP, you may want to check projects like: GOsa (https://oss.gonicus.de/labs/gosa/) Or use a packaged solution, if it's OK with your requirements: http://www.iredmail.org/ (I prefer to install/control packages personally.) If you go manually, it shouldn't be difficult to find one of the many guides on the web. First, you need to find packages supporting the features you need (because the CentOS standard packages are very old). Of course you can start with CentOS standard Postfix package (supports ldap, pcre, SASL, TLS), but make sure you upgrade soon esp. if it's a production system! For Postfix, check that the package offers support for whatever you want (e.g. LDAP, mysql, SASL auth, TLS, pcre etc.). Otherwise, you may need to build your own RPM. You may want to read: http://tech.groups.yahoo.com/group/postfix-users/message/284530 http://tech.groups.yahoo.com/group/postfix-users/message/284359 Dovecot RPMs from here: http://packages.atrpms.net/dist/el5/dovecot/ are fine (I think they support anything you would possibly want). Then, find a guide like: http://www.howtoforge.com/linux_postfix_virtual_hosting or http://wanderingbarque.com/howtos/mailserver/mailserver.html depending on the type of setup you want (e.g. multiple domains, local vs virtual users, etc.) Good luck! Nick From nicolas.kowalski at gmail.com Tue Mar 20 15:55:12 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Tue, 20 Mar 2012 14:55:12 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? Message-ID: <20120320135512.GD28951@petole.demisel.net> Hello, Does dovecot support the subject Alternative Name email value [1] as ssl_cert_username_field? If so, how should it be specified in the configuration? Thanks. [1] http://www.openssl.org/docs/apps/x509v3_config.html#Subject_Alternative_Name_ -- Nicolas From stephan at rename-it.nl Tue Mar 20 16:16:21 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Tue, 20 Mar 2012 15:16:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> Message-ID: <4F689135.6020602@rename-it.nl> Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > Hi there ! > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to The vacation action will not send a response when the envelope-to address (in your case virtual_user at hostname) is not contained in the To: or Cc: headers of the message itself; the message needs to be explicitly addressed to the recipient. For the version you are using this needs to match the final recipient as passed to Dovecot. In newer versions of the Pigeonhole Sieve implementation the original SMTP envelope recipient (i.e. before local rewrites) can also be used instead. Alternatively, new versions allow disabling this behavior entirely, although this is not recommended. Regards, Stephan. From andrei at lctax.ro Tue Mar 20 16:46:58 2012 From: andrei at lctax.ro (Michescu Andrei) Date: Tue, 20 Mar 2012 10:46:58 -0400 Subject: [Dovecot] Dsync Dovecot In-Reply-To: References: Message-ID: Hello, As log as example.com resolves to 192.0.43.10 (which I suppose it is a host that you don't own) this will not work. dsync seems to be resolving example.com and trying to connect there via ssh using the current user. It is better to test on domains that you own, on on domains that don't exists and you adjust your hosts file accordingly. Also seems that you have some issues with the configuration file. Can you post your dovecot -n output. Thnx, Andrei > Hey Frnds, > > Could you tell me about the error : > > # dsync -Dv -u jitendra.b at example.com mirror jitendra.b at example.com > > > > doveadm(root): Debug: Loading modules from directory: > /usr/local/lib/dovecot/doveadm > doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: > undefined symbol: acl_user_module (this is usually intentional, so just > ignore this message) > doveadm(root): Debug: Skipping module doveadm_expire_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined > symbol: expire_set_lookup (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_quota_plugin, because > dlopen() failed: > /usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined > symbol: quota_user_module (this is usually intentional, so just ignore > this > message) > doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because > dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: > undefined symbol: i_stream_create_deflate (this is usually intentional, so > just ignore this message) > doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() > failed: /usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: > undefined symbol: fts_list_backend (this is usually intentional, so just > ignore this message) > doveadm(jitendra.b at example.com): Debug: Effective uid=3846, gid=3846, > home=/home/example1.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: Namespace inbox: type=private, > prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes > location=mbox:/home/ > example.com/jitendra.b/mail:INBOX=/var/spool/example.com/jitendra.b > doveadm(jitendra.b at example.com): Debug: fs: root=/home/ > example.com/jitendra.b/mail, index=, control=, inbox=/var/spool/ > example.com/jitendra.b, alt= > dsync-local(jitendra.b at example.com): Debug: Namespace : Using permissions > from /home/example.com/jitendra.b/mail: mode=0777 gid=-1 > doveadm(jitendra.b): Fatal: User doesn't exist > dsync-local(jitendra.b at example.com): Error: read() from worker server > failed: EOF > > > -- > * Thanks & Regards * > *Jitendra Kumar Bhaskar* > Cell:- +91 7306311531 > +91 8102997821 > > > !DSPAM:4f6865bf72822789337279! > From cjeanneret at internux.ch Tue Mar 20 16:48:21 2012 From: cjeanneret at internux.ch (Cedric Jeanneret) Date: Tue, 20 Mar 2012 15:48:21 +0100 Subject: [Dovecot] dovecot, sieve and vacation In-Reply-To: <4F689135.6020602@rename-it.nl> References: <20120320120528.7c3c2e3d@cholatse.wrk.lsn.camptocamp.com> <4F689135.6020602@rename-it.nl> Message-ID: <20120320154821.11c80a31@cholatse.wrk.lsn.camptocamp.com> On Tue, 20 Mar 2012 15:16:21 +0100 Stephan Bosch wrote: > Op 3/20/2012 12:05 PM, Cedric Jeanneret schreef: > > Hi there ! > > > > Have a small problem with sieve and vacation: it seems to descard the vacation filter I created instead of sending back an email: > > > > Mar 20 11:56:28 hostname dovecot: deliver(virtual_user): sieve: msgid=unspecified: discarding vacation response for message implicitly delivered to > > The vacation action will not send a response when the envelope-to > address (in your case virtual_user at hostname) is not contained in the To: > or Cc: headers of the message itself; the message needs to be explicitly > addressed to the recipient. For the version you are using this needs to > match the final recipient as passed to Dovecot. In newer versions of the > Pigeonhole Sieve implementation the original SMTP envelope recipient > (i.e. before local rewrites) can also be used instead. Alternatively, > new versions allow disabling this behavior entirely, although this is > not recommended. > > Regards, > > Stephan. Hello, thanks for the hint. In fact, I have to add the address aliases in the rule, and it works. Not really cool, but it works like that :). Cheers, C. From patrickdk at patrickdk.com Tue Mar 20 16:55:38 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Tue, 20 Mar 2012 10:55:38 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120319183547.GA28363@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> Message-ID: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> And found two more users with this issue, but while looking at it, I see another related issue, but it's not throwing an error. all email in the INBOX/new and /cur are correct but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa This is happening for all folder moves. the Sent folder isn't affected, but I assume cause an email wasn't moved in that case. Quoting Ralf Hildebrandt : > * Patrick Domack : >> I'm having this problem also, with a very very few users. >> >> But in my case the email isn't double gzip, just single like normal. >> >> Error: read(.../.Deleted >> Messages/cur/1331840112.M186676P27974.5013:2,) failed: Input/output >> error (uid=250) >> >> All I have to do is rename the file to add back the lost S= part and >> all is fine. >> This has happened in the inbox, deleted, and trash folders so far. >> and always after a change, the S= exists for new emails. It's like >> it's loosing it on adding the read flag, and mailbox moves > > Yes, I'm also seeing it now with mailboxes where no mail is doubly > gzipped. > > -- > Ralf Hildebrandt > Gesch?ftsbereich IT | Abteilung Netzwerk > Charit? - Universit?tsmedizin Berlin > Campus Benjamin Franklin > Hindenburgdamm 30 | D-12203 Berlin > Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 > ralf.hildebrandt at charite.de | http://www.charite.de From ka at pacific.net Tue Mar 20 17:26:01 2012 From: ka at pacific.net (Ken A) Date: Tue, 20 Mar 2012 10:26:01 -0500 Subject: [Dovecot] mdbox and pop3 locking Message-ID: <4F68A189.2010800@pacific.net> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Thanks, Ken -- Ken Anderson Pacific Internet - http://www.pacific.net From Ralf.Hildebrandt at charite.de Tue Mar 20 17:33:20 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Tue, 20 Mar 2012 16:33:20 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120320153320.GD26616@charite.de> * Patrick Domack : > And found two more users with this issue, but while looking at it, I > see another related issue, but it's not throwing an error. > > all email in the INBOX/new and /cur are correct > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have > double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Yes indeed: postamt:/home/h/a/happel/Maildir/.Trash/cur# ll total 16 -rw------- 1 happel users 7541 Mar 20 15:23 1332253428.M342974P5666.postamt.charite.de,S=37641,W=38197,S=37641,W=38197:2,Se -rw------- 1 happel users 6378 Mar 20 15:42 1332254568.M9552P591.postamt.charite.de,S=27486,W=28188,S=27486,W=28188:2,Se -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From micah at riseup.net Tue Mar 20 17:40:43 2012 From: micah at riseup.net (Micah Anderson) Date: Tue, 20 Mar 2012 11:40:43 -0400 Subject: [Dovecot] Antispam plugin not compatible with Dovecot 2.1 References: <4F155670.6010905@gmail.com> <1326897258.11500.53.camel@innu> <1326904309.11500.83.camel@innu> Message-ID: <877gyfp9fo.fsf@algae.riseup.net> "Eugene Paskevich" writes: > On Wed, 18 Jan 2012 18:31:49 +0200, Timo Sirainen wrote: > >> On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: >>> >> mailbox.c: In function 'antispam_save_begin': >>> >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >>> >> 'copying' >>> > >>> > The "copying" should be changed to "copying_via_save". >>> >>> Thank you, Timo. >>> Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? >> >> Where do you expect to find such macro? ;) Hm. Perhaps I should try to >> add one. > > Heh. That's Johannes' package private macro... :) I notice that Johannes hasn't made a 2.1 version of the anti-spam plugin, Eugene were you able to build one successfully? If so, would you be willing to share your changes that were required to make it work? thanks, micah From jernej.porenta at arnes.si Tue Mar 20 19:44:26 2012 From: jernej.porenta at arnes.si (Jernej Porenta) Date: Tue, 20 Mar 2012 18:44:26 +0100 Subject: [Dovecot] bug uni_utf8_str_is_valid(vname) In-Reply-To: <1332165220.26095.71.camel@innu> References: <1331735355.2081.140.camel@innu> <480E51CE-AEE4-4FD6-BB2D-6CEC6DE69E4F@arnes.si> <1B9DA585-B55B-4214-9823-3864B1A74CAD@iki.fi> <3974AB53-476A-4945-A828-11425C667165@arnes.si> <1332165220.26095.71.camel@innu> Message-ID: <79D375C1-1009-46B3-A383-A33DD0A699E8@arnes.si> On Mar 19, 2012, at 2:53 PM, Timo Sirainen wrote: > On Mon, 2012-03-19 at 14:27 +0100, Jernej Porenta wrote: >>> Mar 19 10:56:40 server dovecot: imap(user): Panic: file mail-storage.c: line 628 (mailbox_alloc): assertion failed: (uni_utf8_str_is_valid(vname)) >>> >>> It is the same. We will try 2.1.3 today and report the results... > >> The home directory of the username is tar.gzipped here: http://www2.arnes.si/~krklubsls13/username.tar.gz > > Thanks, fixed: http://hg.dovecot.org/dovecot-2.1/rev/c77fbfce438d > Confirmed working? Thank you again, cheers, Jernej From mjeghers at Brocade.com Tue Mar 20 20:29:56 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 11:29:56 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> All, Below is my config. When I run dovecot from xinetd, I get these errors in the log: Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ total 248656 drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root users 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim Any ideas what is wrong? What is different running under xinetd? All the process run under the same user ids... Thanks, /Mark My config --------------------------------------------------------------- # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.7.1.el6.centos.plus.i686 i686 CentOS release 6.2 (Final) ext4 auth_debug = yes auth_verbose = yes auth_verbose_passwords = plain disable_plaintext_auth = no doveadm_worker_count = 4 mail_debug = yes mail_gid = users mail_location = mbox:/var/spool/mailpop3:INBOX=/var/spool/mailpop3/%u mail_uid = root mbox_write_locks = fcntl passdb { args = /etc/passwd.dovecot driver = passwd-file } passdb { driver = shadow } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size from subject vsize flags mail_log_group_events = yes } protocols = pop3 ssl_cert = References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F6943D6.1000600@hardwarefreak.com> On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From mjeghers at Brocade.com Wed Mar 21 06:26:23 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Tue, 20 Mar 2012 21:26:23 -0700 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <4F6943D6.1000600@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Hi Stan Afraid it did not help. Here is what I got: *** entered into a telnet session... user ann +OK pass ******** -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Connection closed by foreign host. [root at t4pserver2 mailpop3]# *** resulted in maillog... Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 *** file permissions... [root at t4pserver2 mailpop3]# ls -al total 248652 drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. Any other ideas? /Mark -----Original Message----- From: dovecot-bounces at dovecot.org [mailto:dovecot-bounces at dovecot.org] On Behalf Of Stan Hoeppner Sent: Tuesday, March 20, 2012 7:59 PM To: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 1:29 PM, Mark Jeghers wrote: > All, > > Below is my config. When I run dovecot from xinetd, I get these errors in the log: > > Mar 20 11:13:39 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=11624, secured > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Effective uid=500, gid=100, home=/home/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/mark > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: stat(/var/spool/mailpop3/mark) failed: Permission denied > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 11:13:39] > Mar 20 11:13:39 t4pserver2 dovecot: pop3(mark): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > ...it acts as if it has no file permission, but it seems like it certainly should. Here is the files it is trying to access: > > [root at t4pserver2 ~]# ls -al /var/spool/mailpop3/ > total 248656 > drwxrwxrwx. 3 mail mail 4096 Mar 20 00:31 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann users 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone users 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 mail users 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues users 327563 Dec 3 14:38 crimsonblues > drwxrwxrwx. 3 mark users 4096 Mar 20 00:31 .imap > -rw-rw-r--. 1 mark users 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone users 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan users 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root users 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser users 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim users 16212 Mar 18 15:51 tim The group owner of these files is "users". Should probably be "mail". E.g. $ ls -la /var/spool/mail/ total 724K drwxrwsr-x 2 root mail 4.0K Jan 19 01:16 . drwxr-xr-x 14 root root 4.0K Jun 2 2011 .. -rw------- 1 stan mail 707K Mar 20 21:32 stan Ownership of /var/spool/mailpop3 should probably be root:mail instead of mail:mail. And given that 'mail' is a standard group name, it's probably not wise to have an actual user named 'mail', as you've done here. -- Stan From nicolas.kowalski at gmail.com Wed Mar 21 08:50:49 2012 From: nicolas.kowalski at gmail.com (Nicolas KOWALSKI) Date: Wed, 21 Mar 2012 07:50:49 +0100 Subject: [Dovecot] ssl_cert_username_field and subjectAltName? In-Reply-To: <20120320135512.GD28951@petole.demisel.net> References: <20120320135512.GD28951@petole.demisel.net> Message-ID: <20120321065049.GE28951@petole.demisel.net> On Tue, Mar 20, 2012 at 02:55:12PM +0100, Nicolas KOWALSKI wrote: > Does dovecot support the subject Alternative Name email value [1] as > ssl_cert_username_field? If so, how should it be specified in the > configuration? Well, I just found the wiki states no: "The text is looked up from subject DN's specified field" (http://wiki2.dovecot.org/SSL/DovecotConfiguration) Sorry for the noise, -- Nicolas From stan at hardwarefreak.com Wed Mar 21 10:41:39 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 03:41:39 -0500 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> Message-ID: <4F699443.1090704@hardwarefreak.com> On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From nmilas at noa.gr Wed Mar 21 11:00:10 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Wed, 21 Mar 2012 11:00:10 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 Message-ID: <4F69989A.3000106@noa.gr> Hi, I've upgraded from 2.0.13 to 2.1.1 and when I started the service, I got the following warning: Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) I didn't see such warnings in 2.0.13. I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? (I could also set "userdb_warning_disable=yes" as advised, but I'm trying to figure out what's the real cause of the warning.) The config follows below. Thanks, Nick ============================================================= protocols = imap pop3 mail_location = maildir:~/Maildir/ mail_gid = 502 mail_uid = 502 auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes disable_plaintext_auth = no mail_plugins = quota protocol imap { imap_client_workarounds = "delay-newmail " mail_plugins = quota imap_quota } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota postmaster_address = sysadmin at example.com sendmail_path = /usr/lib/sendmail } userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap } passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u } service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root } service imap-login { service_count = 1 vsz_limit = 64 M } service pop3-login { service_count = 1 vsz_limit = 64 M } ssl_ca = References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> Message-ID: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> On 19.3.2012, at 21.16, Alex Ha wrote: >>>> dovecot: auth: Error: BUG: Authentication client gave a PID 7542 of >>>> existing connection >>> >>> Oh, right, PIDs of course aren't unique when you're using multiple servers. Try if the attached patch fixes your troubles. If it does, I'll commit it to hg. >>> >> >> Thanks Timo! I will try the patch and report to you. >> > > Hi Timo! > > I tried the patch with 2.0.19 and the dovecot error messages disappeared. OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever gets released). > I still get a lot of this postfix warnings: > > SASL LOGIN authentication failed: Connection lost to authentication server > > but only for ips which tried a sasl brute force attack. > > "Connection lost to authentication server" could this be because of > the dovecot auth penalties? > so far i did not get any complaints from users. The auth penalties wait for max. 17 seconds I think. Looks like Postfix has a timeout of 10 seconds. You could disable auth penalties, or perhaps Postfix should use 20 second limit. From CMarcus at Media-Brokers.com Wed Mar 21 13:55:19 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 07:55:19 -0400 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> Message-ID: <4F69C1A7.2040601@Media-Brokers.com> On 2012-03-21 7:48 AM, Timo Sirainen wrote: > On 19.3.2012, at 21.16, Alex Ha wrote: >>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>> 7542 of existing connection >>>> Oh, right, PIDs of course aren't unique when you're using >>>> mulitiple servers. Try if the attached patch fixes your >>>> troubles. If it does, I'll commit it to hg. >>> Thanks Timo! I will try the patch and report to you. >> I tried the patch with 2.0.19 and the dovecot error messages >> disappeared. > OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever > gets released). Presumably you mean 2.1.4 (since 2.1.3 is already released)? -- Best regards, Charles From tss at iki.fi Wed Mar 21 13:57:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:57:45 +0200 Subject: [Dovecot] dovecot runs from shell, but not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> Message-ID: On 20.3.2012, at 20.29, Mark Jeghers wrote: > Below is my config. When I run dovecot from xinetd, I get these errors in the log: You can't run Dovecot v2.x via inetd. You could run it via systemd though. From tss at iki.fi Wed Mar 21 13:58:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:58:09 +0200 Subject: [Dovecot] auth tcp socket, Authentication client gave a PID 7542 of existing connection In-Reply-To: <4F69C1A7.2040601@Media-Brokers.com> References: <2BEB9B38-182D-4DF4-A26E-9B13B2BF23F8@iki.fi> <69491648-0BF6-4415-BFB8-3916A912501E@iki.fi> <4F69C1A7.2040601@Media-Brokers.com> Message-ID: <38A53BE8-A53F-4906-996F-6CC863E537CC@iki.fi> On 21.3.2012, at 13.55, Charles Marcus wrote: > On 2012-03-21 7:48 AM, Timo Sirainen wrote: >> On 19.3.2012, at 21.16, Alex Ha wrote: >>>>>> dovecot: auth: Error: BUG: Authentication client gave a PID >>>>>> 7542 of existing connection > >>>>> Oh, right, PIDs of course aren't unique when you're using >>>>> mulitiple servers. Try if the attached patch fixes your >>>>> troubles. If it does, I'll commit it to hg. > >>>> Thanks Timo! I will try the patch and report to you. > >>> I tried the patch with 2.0.19 and the dovecot error messages >>> disappeared. > >> OK, it's going to be included in v2.1.3 and v2.0.20 (if that ever >> gets released). > > Presumably you mean 2.1.4 (since 2.1.3 is already released)? Ah, yes. :) From tss at iki.fi Wed Mar 21 13:59:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 13:59:50 +0200 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <4F68A189.2010800@pacific.net> References: <4F68A189.2010800@pacific.net> Message-ID: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> On 20.3.2012, at 17.26, Ken A wrote: > With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? > > Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. From tss at iki.fi Wed Mar 21 14:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:06:53 +0200 Subject: [Dovecot] ldap userdb warning in v2.1.1 In-Reply-To: <4F69989A.3000106@noa.gr> References: <4F69989A.3000106@noa.gr> Message-ID: On 21.3.2012, at 11.00, Nikolaos Milas wrote: > Mar 21 10:07:23 imapserver dovecot: master: Dovecot v2.1.1 starting up (core dumps disabled) > Mar 21 10:08:17 imapserverdovecot: auth: Warning: ldap: Ignoring changed user_attrs in /etc/dovecot/dovecot-passdb-ldap.conf, because userdb ldap not used. (If this is intentional, set userdb_warning_disable=yes) > > I didn't see such warnings in 2.0.13. > > I guess I should/could remove the "user_attrs" line from dovecot-passdb-ldap.conf because it's not needed? Hmm. Yes, if dovecot-usrdb-ldap.conf is a separate file from dovecot-passdb-ldap.conf you can just remove it. But this reminds me that in several places I've suggested to make one of them a symlink to the other, and you can't really do it then. Perhaps I'll need to remove this warning, or maybe make it recognize the symlink case. Anyway I added it for both LDAP and SQL hoping that it would reduce questions like: "I changed user_attrs, but it doesn't do anything!" From tss at iki.fi Wed Mar 21 14:26:05 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 14:26:05 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: On 20.3.2012, at 16.55, Patrick Domack wrote: > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > This is happening for all folder moves. Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From patrickdk at patrickdk.com Wed Mar 21 14:47:56 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 08:47:56 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321084756.Horde.xxWqdZLnE6FPac38iyLGWYA@kishi.patrickdk.com> Thanks, applied it to 2.1.3 and going to test. You didn't even give me enough time to look at the source myself to find the issue. Quoting Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > >> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have >> double S and W tags. >> >> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >> >> This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 From noel.butler at ausics.net Wed Mar 21 15:26:19 2012 From: noel.butler at ausics.net (Noel Butler) Date: Wed, 21 Mar 2012 23:26:19 +1000 Subject: [Dovecot] sysconfdir depreacted Message-ID: <1332336379.10474.5.camel@tardis> The purpose of any build scripts --sysconfdir is to tell the configuration to build in a path for its binaries configuration file(s). Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, is this a bug? if not, then I see no point of sysconfdir any more and it should be removed, if dovecot deliberately ignores what it is told to use. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Wed Mar 21 15:46:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 15:46:44 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332336379.10474.5.camel@tardis> References: <1332336379.10474.5.camel@tardis> Message-ID: On 21.3.2012, at 15.26, Noel Butler wrote: > The purpose of any build scripts --sysconfdir is to tell the > configuration to build in a path for its binaries configuration file(s). > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > is this a bug? if not, then I see no point of sysconfdir any more and it > should be removed, if dovecot deliberately ignores what it is told to > use. --sysconfdir=/etc uses /etc/dovecot/ --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ There is now always the dovecot/ suffix, but the the /etc part is still configurable. From jtl+dovecot at uvm.edu Wed Mar 21 15:53:50 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 09:53:50 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) Message-ID: <4F69DD6E.1090502@uvm.edu> Had a user who couldn't access his INBOX: > Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file > mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai > led: (offset < (uint16_t)-1) > Mar 21 09:21:17 penguina dovecot: imap([USER]): Error: Raw backtrace: > /usr/lib/dovecot/libdovecot.so.0 [0x342683c660] -> /usr > /lib/dovecot/libdovecot.so.0 [0x342683c6b6] -> > /usr/lib/dovecot/libdovecot.so.0 [0x342683bb73] -> > /usr/lib/dovecot/libdovecot > -storage.so.0 [0x3426c966a8] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_ext_intro+0x240) > [0x3426c979c0] -> / > usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_record+0x401) > [0x3426c99151] -> /usr/lib/dovecot/libdovecot-storage.s > o.0(mail_index_sync_map+0x245) [0x3426c99c55] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_map+0x71b) > [0x3426c8afbb > ] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c85d8b] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_open+0x1c > e) [0x3426c8617e] -> > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_mailbox_open+0xb5) > [0x3426c4d865] -> /usr/lib/dov > ecot/libdovecot-storage.so.0 [0x3426c75eab] -> > /usr/lib/dovecot/libdovecot-storage.so.0 [0x3426c31006] -> > dovecot/imap [hdtod > d 10.245.30.58 SELECT](cmd_ Stack trace made it look like it was the INBOX, so I deleted the index files for his INBOX and everything was OK. doveconf -n: > # OS: Linux 2.6.18-274.18.1.el5 x86_64 Red Hat Enterprise Linux Server > release 5.8 (Tikanga) > auth_gssapi_hostname = penguina.uvm.edu > auth_krb5_keytab = /etc/krb5.keytab.dovecot > auth_master_user_separator = * > auth_mechanisms = plain login gssapi > base_dir = /var/run/dovecot/ > default_process_limit = 250 > first_valid_uid = 50 > lock_method = flock > login_trusted_networks = [REDACTED] > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%1u/%1.1u/%u > mail_max_lock_timeout = 30 secs > mail_max_userip_connections = 100 > mbox_read_locks = flock > mbox_write_locks = flock > mmap_disable = yes > namespace { > inbox = yes > location = > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > args = /etc/dovecot/passwd.masterusers > driver = passwd-file > master = yes > } > passdb { > driver = pam > } > service imap { > process_limit = 4096 > } > service lmtp { > client_limit = 1 > inet_listener lmtp { > port = 24 > } > } > ssl_cert = <[REDACTED] > ssl_key = < [REDACTED] > userdb { > driver = passwd > } > verbose_proctitle = yes Any questions/suggestions welcome. Jim From tss at iki.fi Wed Mar 21 16:02:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 16:02:47 +0200 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <4F69DD6E.1090502@uvm.edu> References: <4F69DD6E.1090502@uvm.edu> Message-ID: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> On 21.3.2012, at 15.53, Jim Lawson wrote: > Had a user who couldn't access his INBOX: > >> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >> led: (offset < (uint16_t)-1) I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. > Stack trace made it look like it was the INBOX, so I deleted the index > files for his INBOX and everything was OK. If it happens again, get a copy of the indexes. From CMarcus at Media-Brokers.com Wed Mar 21 16:26:29 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Wed, 21 Mar 2012 10:26:29 -0400 Subject: [Dovecot] squat not working in 2.1 In-Reply-To: <20120229143038.GX13045@charite.de> References: <6A93411B-4058-4A7D-9F94-452403AE83ED@iki.fi> <4F4DF7F7.8020405@in.tum.de> <20120229100957.GX13045@charite.de> <20120229102250.GY13045@charite.de> <20120229132718.GN13045@charite.de> <92D75C5F-46E8-4EE4-B43D-60A3261E071C@iki.fi> <46a03b5105c847df7f7491f0889ef7ec@imt-systems.com> <20120229135851.GU13045@charite.de> <1be342370509d17ae81682aede00f016@imt-systems.com> <5febb8861c0cc824b0446cb2fec98d19@imt-systems.com> <20120229143038.GX13045@charite.de> Message-ID: <4F69E515.9080904@Media-Brokers.com> On 2012-02-29 9:30 AM, Ralf Hildebrandt wrote: > * Morten Stevens: > >> This is a Fedora-specific problem, because clucene (build >> requirement) is not correctly packaged. > > Well, debian showed the same packaging (wrong place). I just attempted to update to 2.1.3 on gentoo and received the same error: /usr/include/CLucene/SharedHeader.h:18:36: fatal error: CLucene/clucene-config.h: No such file or directory So, is this also a packaging error that I need to report to gentoo? -- Best regards, Charles From Ralf.Hildebrandt at charite.de Wed Mar 21 16:33:48 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 15:33:48 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> Message-ID: <20120321143348.GR2789@charite.de> * Timo Sirainen : > On 20.3.2012, at 16.55, Patrick Domack wrote: > > > but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. > > > > 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa > > > > This is happening for all folder moves. > > Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 That doesn't seem to work: Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename has wrong S value, renamed the file from /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S to /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) failed: Input/output error (uid=5270) It's renaming itself to itself again? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From luca at lm-net.it Wed Mar 21 16:43:14 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Wed, 21 Mar 2012 15:43:14 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup Message-ID: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Hello list. I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. It would be desirable for multiple reasons: - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) - be ready to serve traffic over IPv6 without touching our precious mailbox servers - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). I'd like to know if that plan sounds reasonable or if there's something stupid in it. Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? thanks, -- Luca Lesinigo From tss at iki.fi Wed Mar 21 17:00:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:00:26 +0200 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <20120321143348.GR2789@charite.de> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: On 21.3.2012, at 16.33, Ralf Hildebrandt wrote: >>> but in .Trash/cur since I upgraded from 2.0.19 to 2.1 they have double S and W tags. >>> >>> 1331941500.M220929P17982.5013,S=24845,W=25526,S=24845,W=25526:2,Sa >>> >>> This is happening for all folder moves. >> >> Fixed: http://hg.dovecot.org/dovecot-2.1/rev/3599790da3d7 > > That doesn't seem to work: It fixed only the duplicate S= and W= values. > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: Maildir filename > has wrong S value, renamed the file from > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > to > /home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S > Mar 21 15:32:50 postamt dovecot: imap(jkamp): Error: read(/home/j/k/jkamp/Maildir/cur/1330501473.M742455P30506.postamt.charite.de,S=36307:2,S) > failed: Input/output error (uid=5270) > > It's renaming itself to itself again? Hmm. Yeah, this is a bit problematic for compressed mails. If the S=size isn't correct, Dovecot fixes it by stat()ing the file and using it as the size. And that's of course wrong. Also Dovecot can't simply remove the S=size, because the current Maildir code assumes that it always exists for compressed mails. There's no easy and efficient way to fix this.. Maybe you could just manually rename the files to have correct S=size? :) zcat file | wc should give the right size. From tss at iki.fi Wed Mar 21 17:17:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:17:56 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F684938.9000208@unict.it> References: <4F684938.9000208@unict.it> Message-ID: Hi, On 20.3.2012, at 11.09, Luca Palazzo wrote: > Hi Timo, hi all, > after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > > 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > 710 { > (gdb) bt > #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 526 bytes Desc: not available URL: From ka at pacific.net Wed Mar 21 17:19:16 2012 From: ka at pacific.net (Ken Anderson) Date: Wed, 21 Mar 2012 10:19:16 -0500 Subject: [Dovecot] mdbox and pop3 locking In-Reply-To: <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> References: <4F68A189.2010800@pacific.net> <16516B45-8722-4505-ADA8-3785AC7A0EC0@iki.fi> Message-ID: <4F69F174.9000501@pacific.net> On 3/21/2012 6:59 AM, Timo Sirainen wrote: > On 20.3.2012, at 17.26, Ken A wrote: > >> With mdbox, what does dovecot lock when "pop3_lock_session(pop3): yes"? >> >> Specifically, I'm wondering if Dovecot LDA is able to deliver mail when a session is locked, if using mdbox, or if it will tempfail until the session is unlocked? > > Unfortunately it will tempfail. This is something I'm planning on changing soon. There should be a separate POP3-only lock. > Awesome! I haven't migrated to mdbox yet, but in testing with it on a dev server, it looks like it will solve a huge problem. Users seem to want ever larger mailboxes, and mdbox gives them that, without asking more than additional disk space. Fixing the pop locking would be an additional benefit! Thanks, Ken Pacific.Net From tss at iki.fi Wed Mar 21 17:21:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:21:43 +0200 Subject: [Dovecot] issues migration from dovecot 1.2 to version 2 In-Reply-To: <.120.61.90.33.1332228956.squirrel@24x7server.net> References: <.120.61.90.33.1332228956.squirrel@24x7server.net> Message-ID: <7D494B74-E138-415F-8010-F1208604E246@iki.fi> On 20.3.2012, at 9.35, Rajesh M wrote: > i migrated my email server with around 5000 users from dovecot version 1.2 > to version 2 > > i have two separate 2 tb hdd's storing webmail data of these users. You mean you simply upgraded the Dovecot version, the server is exactly the same? > the load on the server goes very high over 100 during peak load times and > the imap connections get dropped frequently, webmail becomes very slow. There shouldn't be much performance difference between v1.2 and v2.x. > in the dovecot log file i get errors as such > > Warning: Maildir /homebackup/domains/xxxx/xxxx/Maildir/.ALL_INBOX MAIL: > Synchronization took 71 seconds (20 new msgs, 0 flag change attempts, 0 > expunge attempts) This simply means that the disk IO usage is very high. > i am a bit confused as to what settings are to be done for a very busy server Show dovecot -n output of the new server, and if you have the old configuration available that could be helpful also to compare their differences. From tss at iki.fi Wed Mar 21 17:25:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:25:46 +0200 Subject: [Dovecot] replication howto In-Reply-To: References: <3A0ACB56-CBBE-41AE-859C-95635FF3010F@odo.in-berlin.de> <7B247863-1773-4784-BFA9-5A5AB43802E0@iki.fi> Message-ID: On 19.3.2012, at 12.50, Matteo Cazzador wrote: > Hi, i've a simple question, what do you mean for dovecot director setup? > 'i've a doubt. > The solution that i'm testing is using 3 mail server in different > geoghrapic locations. > An user can travel in varius location, and i want his imap mail reside > on mail server in every locations. > S? i use you solution about replication. First server (by dns record) > that receive mail sync it on the other servers, and when > user consult is mail by imap protocol everything is sync on all servers. > Do you suggest to use a horizontal structure for it like i explain or > is better to have a single node external mail server > and customer locations server like slave? Dovecot director isn't really meant to be used for geographic user distribution. Also the replication doesn't yet support more than two servers. A master-slave setup wouldn't have the UID conflict problems that multi-master dsync replication has, but the UID conflicts probably won't be a big problem. Anyway, difficult to give recommendations about an unfinished feature.. From lukas.mueller at newmedia.ch Wed Mar 21 17:45:09 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Wed, 21 Mar 2012 15:45:09 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS Message-ID: Hi, I'm stuck with a problem we have with dovecot. My suspicion is, that it has to do with accessing the same mailbox/mail stored on a NFS-share from two machines at the same time. setup We have to mail servers running, both run a Ubuntu 10.04, Postfix 2.70 and Dovecot 1.2.9. The mailboxes are stored in maildir format on a NFS-Share. In front of those to mail servers we have a load balancer. Unfortunately it can't be set up to use the same server for each domain, but it uses the same server for the same source-ip for at least 1 hour. Here is the output of dovecot -n: # 1.2.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-33-server x86_64 Ubuntu 10.04.3 LTS nfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 pop3s ssl_ca_file: /etc/ssl/ca-bundle/SSL123_CA_Bundle.pem ssl_cert_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.crt ssl_key_file: /etc/ssl/mail.newmedia.ch/mail.newmedia.ch.key ssl_verify_client_cert: yes disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login mail_max_userip_connections: 25 mail_privileged_group: mail mail_location: maildir:/data/vmail/%d/%n:INDEX=/data/vmail/%d/%n/indexes mmap_disable: yes dotlock_use_excl: no mail_nfs_storage: yes mail_nfs_index: yes mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): auth default: passdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-mysql.conf plugin: quota: maildir:storage=409600 sieve_global_path: /data/vmail/globalsieverc dict: quotadict: mysql:/etc/dovecot-dict-quota.conf problem the problem happens with a client's mailbox that is used by multiple users. >From time to time he cannot see any Emails in the mailbox, neither with his mail clients (Apple Mail) nor with in the webmail (Roundcube). Around this time I get the following entries in the log files: Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:30 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 5: 1326705103.V15I90105M613353.mail01:2,Sad (uid 1523 -> 1599) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1524, file=1327500903.V15I5722c8M210039.mail01:2,Se) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f37602b8b5a] -> imap(+0xaebc7) [0x7f37602b8bc7] -> imap(+0xae238) [0x7f37602b8238] -> imap(+0x497d7) [0x7f37602537d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f37602545c2] -> imap(+0x4bb06) [0x7f3760255b06] -> imap(maildir_uidlist_sync_init+0x4d) [0x7f376025652d] -> imap(+0x46ed4) [0x7f3760250ed4] -> imap(maildir_storage_sync_init+0x147) [0x7f3760251557] -> imap(imap_sync_init+0x70) [0x7f376023b190] -> imap(+0x2411e) [0x7f376022e11e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f37602c069c] -> imap(io_loop_handler_run+0x60) [0x7f37602c1000] -> imap(io_loop_run+0x18) [0x7f37602c0448] -> imap(main+0x58e) [0x7f376023bc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f375f877c4d] -> imap(+0x21979) [0x7f376022b979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 16934 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 6: 1327500903.V15I5722c8M210039.mail01:2,Se (uid 1524 -> 1600) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7ff81b415b5a] -> imap(+0xaebc7) [0x7ff81b415bc7] -> imap(+0xae238) [0x7ff81b415238] -> imap(+0x497d7) [0x7ff81b3b07d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7ff81b3b15c2] -> imap(maildir_uidlist_sync_init+0x105) [0x7ff81b3b35e5] -> imap(+0x46ed4) [0x7ff81b3aded4] -> imap(maildir_storage_sync_force+0x52) [0x7ff81b3ae392] -> imap(maildir_file_do+0x99) [0x7ff81b3b3cb9] -> imap(+0x4d944) [0x7ff81b3b4944] -> imap(index_mail_set_seq+0x148) [0x7ff81b3c8ed8] -> imap(index_storage_search_next_nonblock+0x162) [0x7ff81b3cd622] -> imap(mailbox_search_next_nonblock+0x20) [0x7ff81b3db2c0] -> imap(mailbox_search_next+0x26) [0x7ff81b3db316] -> imap(imap_fetch_more+0x2bf) [0x7ff81b39295f] -> imap(cmd_fetch+0x36c) [0x7ff81b38a9ec] -> imap(+0x28fad) [0x7ff81b38ffad] -> imap(+0x2908d) [0x7ff81b39008d] -> imap(client_handle_input+0x135) [0x7ff81b3902c5] -> imap(client_input+0x5f) [0x7ff81b390baf] -> imap(io_loop_handler_run+0xbd) [0x7ff81b41e05d] -> imap(io_loop_run+0x18) [0x7ff81b41d448] -> Mar 6 08:42:31 mail02 dovecot: IMAP(user at example.com): imap(main+0x58e) [0x7ff81b398c5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7ff81a9d4c4d] -> imap(+0x21979) [0x7ff81b388979] Mar 6 08:42:31 mail02 dovecot: dovecot: child 13712 (imap) killed with signal 6 (core dumps disabled) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Mar 6 08:42:31 mail01 dovecot: IMAP(user at example.com): Raw backtrace: imap(+0xaeb5a) [0x7f4a311fcb5a] -> imap(+0xaebc7) [0x7f4a311fcbc7] -> imap(+0xae238) [0x7f4a311fc238] -> imap(+0x497d7) [0x7f4a311977d7] -> imap(maildir_uidlist_refresh+0x6f2) [0x7f4a311985c2] -> imap(+0x47023) [0x7f4a31195023] -> imap(maildir_storage_sync_init+0x147) [0x7f4a31195557] -> imap(imap_sync_init+0x70) [0x7f4a3117f190] -> imap(+0x2411e) [0x7f4a3117211e] -> imap(+0x64c0e) [0x7f4a311b2c0e] -> imap(io_loop_handle_timeouts+0xcc) [0x7f4a3120469c] -> imap(io_loop_handler_run+0x60) [0x7f4a31205000] -> imap(io_loop_run+0x18) [0x7f4a31204448] -> imap(main+0x58e) [0x7f4a3117fc5e] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f4a307bbc4d] -> imap(+0x21979) [0x7f4a3116f979] Mar 6 08:42:31 mail01 dovecot: dovecot: child 24257 (imap) killed with signal 6 (core dumps disabled) Note: the first part is on a mail server, while the smaller second part is on the other one. Unfortunately I'm not able to reproduce this error. My suspicion/speculation what happens is the following: Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. Somehow this leads to problems with Locks on NFS, which leads to the crash. I have no idea how to solve this problem and any help is greatly appreciated. If you need further information, please say so. Mit freundlichen Gr?ssen Lukas M?ller Systems Engineer _______________________________________________ NEWMEDIA S?dostschweiz Newmedia AG Kasernenstrasse 1 Postfach 508, CH-7007 Chur http://www.newmedia.ch _______________________________________________ TYPO3 & Drupal - Wir wissen wie. Ihre professionelle Web Agentur in Chur, Ilanz, Glarus und Z?rich. From tss at iki.fi Wed Mar 21 17:50:29 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 17:50:29 +0200 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: Message-ID: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> On 21.3.2012, at 17.45, M?ller Lukas wrote: > Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) > Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. > My suspicion/speculation what happens is the following: > Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. > Somehow this leads to problems with Locks on NFS, which leads to the crash. Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > I have no idea how to solve this problem and any help is greatly appreciated. The only way to fully fix this is: http://wiki2.dovecot.org/Director From Ralf.Hildebrandt at charite.de Wed Mar 21 17:52:45 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Wed, 21 Mar 2012 16:52:45 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> Message-ID: <20120321155245.GS2789@charite.de> * Timo Sirainen : > > It's renaming itself to itself again? > > Hmm. Yeah, this is a bit problematic for compressed mails. If the > S=size isn't correct, Dovecot fixes it by stat()ing the file and using > it as the size. And that's of course wrong. Also Dovecot can't simply > remove the S=size, because the current Maildir code assumes that it > always exists for compressed mails. There's no easy and efficient way > to fix this.. Maybe you could just manually rename the files to have > correct S=size? :) zcat file | wc should give the right size. Right now the whole system is down because nobody can acces his/her mails due to this. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From fxmulder at gmail.com Wed Mar 21 17:56:12 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 09:56:12 -0600 Subject: [Dovecot] distributed mdbox Message-ID: Anyone know how to setup dovecot with mdbox so that it can be used through shared storage from multiple hosts? I've setup a gluster volume and am sharing it between 2 test clients. I'm using postfix/dovecot LDA for delivery and I'm using postal to send mail between 40 users. In doing this, I'm seeing these errors in the logs Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count 272 -> 271 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=3768 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but next_uid = 517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=4220 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log synchronization error at seq=4,offset=5088 for /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update for invalid uid=517 Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index This is my dovecot config currently: jdevine at test-gluster-client2:~> dovecot -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 lock_method = dotlock mail_fsync = always mail_location = mdbox:~/mdbox mail_nfs_index = yes mail_nfs_storage = yes mmap_disable = yes passdb { driver = pam } protocols = " imap" ssl_cert = References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> Message-ID: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > * Timo Sirainen : > >>> It's renaming itself to itself again? >> >> Hmm. Yeah, this is a bit problematic for compressed mails. If the >> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >> it as the size. And that's of course wrong. Also Dovecot can't simply >> remove the S=size, because the current Maildir code assumes that it >> always exists for compressed mails. There's no easy and efficient way >> to fix this.. Maybe you could just manually rename the files to have >> correct S=size? :) zcat file | wc should give the right size. > > Right now the whole system is down because nobody can acces his/her > mails due to this. All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 385 bytes Desc: not available URL: From luca.palazzo at unict.it Wed Mar 21 18:04:00 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Wed, 21 Mar 2012 17:04:00 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: References: <4F684938.9000208@unict.it> Message-ID: <4F69FBF0.6090003@unict.it> It worked. We have no more sigsegv on *-login process. Thanks Luca Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > Hi, > > On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> Hi Timo, hi all, >> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >> >> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> 710 { >> (gdb) bt >> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > > Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > From tss at iki.fi Wed Mar 21 18:05:52 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:05:52 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> On 21.3.2012, at 17.56, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs Dovecot assumes that the filesystem behaves the same way as regular local filesystems. > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 Looks like gluster doesn't fit that assumption. So, the solution is the same as with NFS: http://wiki2.dovecot.org/Director From tss at iki.fi Wed Mar 21 18:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 18:08:07 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <4F69FBF0.6090003@unict.it> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> Message-ID: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e I'll figure out a proper fix soon. On 21.3.2012, at 18.04, Luca Palazzo wrote: > It worked. We have no more sigsegv on *-login process. > > Thanks > > Luca > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >> Hi, >> >> On 20.3.2012, at 11.09, Luca Palazzo wrote: >> >>> Hi Timo, hi all, >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>> >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> 710 { >>> (gdb) bt >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >> >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >> > From fxmulder at gmail.com Wed Mar 21 18:25:14 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:25:14 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > On 21.3.2012, at 17.56, James Devine wrote: > > > Anyone know how to setup dovecot with mdbox so that it can be used > through > > shared storage from multiple hosts? I've setup a gluster volume and am > > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > > delivery and I'm using postal to send mail between 40 users. In doing > > this, I'm seeing these errors in the logs > > Dovecot assumes that the filesystem behaves the same way as regular local > filesystems. > > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: > Fixed > > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: > messages_count > > 272 -> 271 > > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > > synchronization error at seq=4,offset=3768 for > > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > > next_uid = 517 > > Looks like gluster doesn't fit that assumption. So, the solution is the > same as with NFS: http://wiki2.dovecot.org/Director > > What filesystem mechanisms might not be working in this case? From fxmulder at gmail.com Wed Mar 21 18:47:53 2012 From: fxmulder at gmail.com (James Devine) Date: Wed, 21 Mar 2012 10:47:53 -0600 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: Also I don't seem to get these errors with a single dovecot machine using the shared storage and it looks like there are multiple simultaneous delivery processes running On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > > > On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: > >> On 21.3.2012, at 17.56, James Devine wrote: >> >> > Anyone know how to setup dovecot with mdbox so that it can be used >> through >> > shared storage from multiple hosts? I've setup a gluster volume and am >> > sharing it between 2 test clients. I'm using postfix/dovecot LDA for >> > delivery and I'm using postal to send mail between 40 users. In doing >> > this, I'm seeing these errors in the logs >> >> Dovecot assumes that the filesystem behaves the same way as regular local >> filesystems. >> >> > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >> Fixed >> > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >> messages_count >> > 272 -> 271 >> > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >> Log >> > synchronization error at seq=4,offset=3768 for >> > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >> but >> > next_uid = 517 >> >> Looks like gluster doesn't fit that assumption. So, the solution is the >> same as with NFS: http://wiki2.dovecot.org/Director >> >> > What filesystem mechanisms might not be working in this case? > From tss at iki.fi Wed Mar 21 19:04:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 19:04:36 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> Message-ID: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. On 21.3.2012, at 18.47, James Devine wrote: > Also I don't seem to get these errors with a single dovecot machine using > the shared storage and it looks like there are multiple simultaneous > delivery processes running > > On Wed, Mar 21, 2012 at 10:25 AM, James Devine wrote: > >> >> >> On Wed, Mar 21, 2012 at 10:05 AM, Timo Sirainen wrote: >> >>> On 21.3.2012, at 17.56, James Devine wrote: >>> >>>> Anyone know how to setup dovecot with mdbox so that it can be used >>> through >>>> shared storage from multiple hosts? I've setup a gluster volume and am >>>> sharing it between 2 test clients. I'm using postfix/dovecot LDA for >>>> delivery and I'm using postal to send mail between 40 users. In doing >>>> this, I'm seeing these errors in the logs >>> >>> Dovecot assumes that the filesystem behaves the same way as regular local >>> filesystems. >>> >>>> Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: >>> Fixed >>>> index file /mnt/testuser34/mdbox/storage/dovecot.map.index: >>> messages_count >>>> 272 -> 271 >>>> Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: >>> Log >>>> synchronization error at seq=4,offset=3768 for >>>> /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, >>> but >>>> next_uid = 517 >>> >>> Looks like gluster doesn't fit that assumption. So, the solution is the >>> same as with NFS: http://wiki2.dovecot.org/Director >>> >>> >> What filesystem mechanisms might not be working in this case? >> From jtl+dovecot at uvm.edu Wed Mar 21 20:19:19 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Wed, 21 Mar 2012 14:19:19 -0400 Subject: [Dovecot] dovecot 2.0.19 Panic: file mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion failed: (offset < (uint16_t)-1) In-Reply-To: <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> References: <4F69DD6E.1090502@uvm.edu> <94A1158A-2F42-406A-9212-C58A4D4FF879@iki.fi> Message-ID: <4F6A1BA7.7030208@uvm.edu> On 3/21/12 10:02 AM, Timo Sirainen wrote: > On 21.3.2012, at 15.53, Jim Lawson wrote: > >> Had a user who couldn't access his INBOX: >> >>> Mar 21 09:21:17 penguina dovecot: imap([USER]): Panic: file >>> mail-index-sync-ext.c: line 209 (sync_ext_reorder): assertion fai >>> led: (offset < (uint16_t)-1) > I kind of remember that this was fixed by http://hg.dovecot.org/dovecot-2.1/rev/b4d8e950eb9d but I'm not entirely sure. I guess I should have included in the commit the error message it fixed. This applies cleanly against 2.0.19; should I try it on that version, or not recommended? >> Stack trace made it look like it was the INBOX, so I deleted the index >> files for his INBOX and everything was OK. > If it happens again, get a copy of the indexes. > I sent them, encrypted, to your email address/GPG key 0x40558AC9. Jim From mjeghers at Brocade.com Wed Mar 21 20:59:39 2012 From: mjeghers at Brocade.com (Mark Jeghers) Date: Wed, 21 Mar 2012 11:59:39 -0700 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <4F699443.1090704@hardwarefreak.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> Message-ID: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> All, I was mistaken in how I described my problem, please forgive this dovecot newbie for describing the problem incorrectly! It is not under xinitd, it is trying to run as an init.d service. Ok, let's try again... I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? Thanks, hope this clarifies things, /Mark -----Original Message----- From: Stan Hoeppner [mailto:stan at hardwarefreak.com] Sent: Wednesday, March 21, 2012 1:42 AM To: Mark Jeghers Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot runs from shell, but not xinetd On 3/20/2012 11:26 PM, Mark Jeghers wrote: > Hi Stan > > Afraid it did not help. Here is what I got: > > *** entered into a telnet session... > user ann > +OK > pass ******** > -ERR [IN-USE] Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Connection closed by foreign host. > [root at t4pserver2 mailpop3]# > > *** resulted in maillog... > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd-file(ann,::1): lookup: user=ann file=/etc/passwd.dovecot > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: client out: OK#0112#011user=ann > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master in: REQUEST#0113180593153#01113546#0112#0116c9a0569dcd246a9f9e7a94dbe852843 > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: passwd(ann,::1): lookup > Mar 20 21:16:05 t4pserver2 dovecot: auth: Debug: master out: USER#0113180593153#011ann#011system_groups_user=ann#011uid=501#011gid=501#011home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3-login: Login: user=, method=PLAIN, rip=::1, lip=::1, mpid=13549, secured > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Effective uid=501, gid=501, home=/home/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: fs: root=/var/spool/mailpop3, index=, control=, inbox=/var/spool/mailpop3/ann > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: Couldn't open INBOX: Internal error occurred. Refer to server log for more information. [2012-03-20 21:16:05] > Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Couldn't open INBOX top=0/0, retr=0/0, del=0/0, size=0 > > *** file permissions... > [root at t4pserver2 mailpop3]# ls -al > total 248652 > drwxrwxrwx. 2 root mail 4096 Mar 20 21:11 . > drwxr-xr-x. 17 root root 4096 Mar 18 18:22 .. > -rw-rw-r--. 1 ann mail 58739 Mar 17 04:26 ann > -rw-rw-r--. 1 annphone mail 2708345 Mar 17 05:22 annphone > -rw-rw-r--. 1 root mail 127272960 Mar 18 18:28 backups.tar > -rw-rw-r--. 1 crimsonblues mail 327563 Dec 3 14:38 crimsonblues > -rw-rw-r--. 1 mark mail 0 Mar 18 13:09 mark > -rw-rw-r--. 1 markphone mail 124147068 Mar 18 04:21 markphone > -rw-rw-r--. 1 nathan mail 5119 Dec 22 18:52 nathan > -rw-rw-r--. 1 root mail 0 Mar 18 13:13 root > -rw-rw-r--. 1 testuser mail 58739 Mar 18 18:42 testuser > -rw-rw-r--. 1 tim mail 16212 Mar 18 15:51 tim > > My CentOS installation created a user "mail" so I am hesitant to remove it, but it is no longer in use here. > > Any other ideas? What user does dovecot run as in the shell? Under xinetd? -- Stan From tss at iki.fi Wed Mar 21 21:16:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Wed, 21 Mar 2012 21:16:51 +0200 Subject: [Dovecot] dovecot runs from shell, but not as "service" -- MY MISTAKE, not xinetd In-Reply-To: <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> References: <3F73AF37684DDD44903405EE90ADDCB001D6165B2FFE@HQ1-EXCH02.corp.brocade.com> <4F6943D6.1000600@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3189@HQ1-EXCH02.corp.brocade.com> <4F699443.1090704@hardwarefreak.com> <3F73AF37684DDD44903405EE90ADDCB001D6165B3256@HQ1-EXCH02.corp.brocade.com> Message-ID: On 21.3.2012, at 20.59, Mark Jeghers wrote: > I am able to run it from a root shell prompt, but the errors below occur if it was started as a SERVICE, e.g. from the init.d script. So now the question is: what is different in those two environments...? .. >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Error: stat(/var/spool/mailpop3/ann) failed: Permission denied >> Mar 20 21:16:05 t4pserver2 dovecot: pop3(ann): Debug: Namespace : Using permissions from /var/spool/mailpop3: mode=0777 gid=-1 Permission errors point to SELinux being the problem. Try disabling it. From bear at rwhartzell.net Wed Mar 21 22:54:09 2012 From: bear at rwhartzell.net (Robert Hartzell) Date: Wed, 21 Mar 2012 13:54:09 -0700 Subject: [Dovecot] Dovecot 2.1.3 on solaris with mysql - make fails Message-ID: <95DD93BE-F841-4BEB-A96C-059FFF0ACF2F@rwhartzell.net> I'm trying to build 2.1.3 on solaris 11 11/11 with gcc 4.5.2 & sun studio 12.2 & 12.3 CPPFLAGS="-I/opt/openssl/include -I/usr/mysql/include/mysql" \ LDFLAGS="-L/opt/openssl/lib -L/usr/mysql/lib/mysql -R/opt/openssl/lib:/usr/mysql/lib/mysql" \ ./configure --prefix=/opt/dovecot \ --sysconfdir=/etc/opt \ --with-ssl=openssl \ --with-mysql make fails with both solaris standard openssl and my build of openssl. I'm also getting the same error using sunstudio mysql version is 5.1.37 The relevant output of make is on pastebin http://pastebin.com/aALHG0yL I have seen some reference to this with google but nothing thats very recent and no solutions. Anyone know how to get past this? Any tips on building dovecot on solaris? Pointers would be much appreciated. -- Robert From patrickdk at patrickdk.com Wed Mar 21 23:52:39 2012 From: patrickdk at patrickdk.com (Patrick Domack) Date: Wed, 21 Mar 2012 17:52:39 -0400 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302102501.GZ11180@charite.de> <5CA8822A-A387-45E2-A463-C0A9A4B9B04B@iki.fi> <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120321175239.Horde.6cigCZLnE6FPak2nibEXzWA@kishi.patrickdk.com> Quoting Timo Sirainen : > On 21.3.2012, at 17.52, Ralf Hildebrandt wrote: > >> * Timo Sirainen : >> >>>> It's renaming itself to itself again? >>> >>> Hmm. Yeah, this is a bit problematic for compressed mails. If the >>> S=size isn't correct, Dovecot fixes it by stat()ing the file and using >>> it as the size. And that's of course wrong. Also Dovecot can't simply >>> remove the S=size, because the current Maildir code assumes that it >>> always exists for compressed mails. There's no easy and efficient way >>> to fix this.. Maybe you could just manually rename the files to have >>> correct S=size? :) zcat file | wc should give the right size. >> >> Right now the whole system is down because nobody can acces his/her >> mails due to this. > > All of your mails are compressed and have wrong S=size in the > filename? You can disable the check with the attached patch, but I'm > not sure if there are other places where it fails. At least quota > calculations won't be correct. The issue only started happening since I upgraded to 2.1.1, it didn't exist before then, I have check my system, and files before the date of upgrade are fine, only files/emails moved after upgrading to 2.1.1 have lost the S= value. I have made something that can pretty easily fix the issue, but it only stays fixed till another email gets moved and looses it's S= value. Sorry, I haven't had time to test out 2.1.3 yet. This will print out the commands needed to fix the files though. find . -name '*hostname:*' -exec 'gzip' '-l' '{}' ';' | awk '/hostname/ {for(x=4;x References: <1332336379.10474.5.camel@tardis> Message-ID: <1332381356.4112.9.camel@tardis> On Wed, 2012-03-21 at 15:46 +0200, Timo Sirainen wrote: > On 21.3.2012, at 15.26, Noel Butler wrote: > > > The purpose of any build scripts --sysconfdir is to tell the > > configuration to build in a path for its binaries configuration file(s). > > > > Dovecot 2.1.3, seems to insist that that directory is now /etc/dovecot/ > > ignoring --sysconfdir=/etc as in 1.2.x and previous majors before that, > > is this a bug? if not, then I see no point of sysconfdir any more and it > > should be removed, if dovecot deliberately ignores what it is told to > > use. > > > --sysconfdir=/etc uses /etc/dovecot/ > > --sysconfdir=/opt/dovecot/etc uses /opt/dovecot/etc/dovecot/ > > There is now always the dovecot/ suffix, but the the /etc part is still configurable. > perhaps it should be renamed then, given it violates the known normal for SYSCONF dir, you've just created another form of --datadir from gnu.org: "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) "datadir" The directory for installing idiosyncratic read-only architecture-independent data files for this program. This is usually the same place as ?datarootdir?, but we use the two separate variables so that you can move these program-specific files without altering the location for Info files, man pages, etc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From gedalya at gedalya.net Thu Mar 22 04:46:20 2012 From: gedalya at gedalya.net (Gedalya) Date: Wed, 21 Mar 2012 22:46:20 -0400 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A8BAC.4000002@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> Message-ID: <4F6A927C.6010003@gedalya.net> On 3/21/2012 10:17 PM, Martin Schitter wrote: > Am 16.3.2011 20:59, schrieb Gedalya: >>> >>>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at >>>> example.com -o imapc_password=**** backup -u jedi at example.com -R >>>> imapc: >>> >> Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current >> versions are putting the body of the last message in "Sent Items" in >> place of every single email in INBOX. >> In other words, for every email that sits in INBOX in the source, I get >> a copy of the last email in "Sent Items" instead. >> This happens for every account I try to migrate. >> Very strange. I noticed this only now, and the last package I have left >> in the local apt cache which still works is 2.1.rc7-0~auto+0. > > i see the same regression (2.1.3-0~auto+4) :( > > doveadm sync/backup via impac puts the same message all over the place... Thanks Martin, I've set up a test platform to investigate this further but I've been short on time... From stan at hardwarefreak.com Thu Mar 22 06:11:19 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 21 Mar 2012 23:11:19 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> Message-ID: <4F6AA667.1080908@hardwarefreak.com> On 3/21/2012 12:04 PM, Timo Sirainen wrote: > The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. > > With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could use a true cluster file system such as OCFS or GFS. Both will eliminate this problem, and without requiring Dovecot director. And you'll get better performance than with Gluster, which, BTW, isn't really suitable as a transactional filesystem, was not designed for such a use case. -- Stan From ruskie at codemages.net Thu Mar 22 08:28:40 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Thu, 22 Mar 2012 07:28:40 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332381356.4112.9.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: :2012-03-22T11:55:Noel Butler: > perhaps it should be renamed then, given it violates the known normal > for SYSCONF dir, you've just created another form of --datadir Not really. The way I see it works as expected. The sysconf dir is the root of the configuration dir. Then if the app so chooses uses it's own directory structure under that. Considering that by default dovecot uses dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Quis custodiet ipsos custodes? From noel.butler at ausics.net Thu Mar 22 10:30:27 2012 From: noel.butler at ausics.net (Noel Butler) Date: Thu, 22 Mar 2012 18:30:27 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> Message-ID: <1332405027.6792.7.camel@tardis> On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > :2012-03-22T11:55:Noel Butler: > > > perhaps it should be renamed then, given it violates the known normal > > for SYSCONF dir, you've just created another form of --datadir > > Not really. The way I see it works as expected. The sysconf dir is the Then you and I and a few other devs involved in other very well known bits of software that everyone likely uses, will have to agree to disagree "sysconfdir" The directory for installing read-only data files that pertain to a single machine?that is to say, files for configuring a host. Mailer and network configuration files, ?/etc/passwd?, and so forth belong here. All the files in this directory should be ordinary ASCII text files. This directory should normally be ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are using Autoconf, write it as ?@sysconfdir@?.) > root of the configuration dir. Then if the app so chooses uses it's own > directory structure under that. Considering that by default dovecot uses > dovecot/dovecot.conf and dovecot/conf.d I don't see anything wrong here. > By default as of only 2.something, not in 0.x not in 1.0.x not in 1.1.x and not in 1.2.x I've said all I'm going to say on the mater, I got three emails offlist from others here agreeing with me, shame they didn't do it on-list, but I respect their right to remain silent so as not to endure the wrath of Timo and certain other cretins well known for having nothing else better to do. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From Ralf.Hildebrandt at charite.de Thu Mar 22 10:46:10 2012 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 22 Mar 2012 09:46:10 +0100 Subject: [Dovecot] 2.1: Error: Maildir filename has wrong S value, renamed the file from In-Reply-To: <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> References: <20120302104333.GD11180@charite.de> <20120319123728.Horde.dhJ3AJLnE6FPZ2DIHN_F3XA@kishi.patrickdk.com> <20120319183547.GA28363@charite.de> <20120320105538.Horde.V7ADKpLnE6FPaJpq2JkBf2A@kishi.patrickdk.com> <20120321143348.GR2789@charite.de> <20120321155245.GS2789@charite.de> <7A90F6CB-0A48-4E69-A5BB-C0526B4DE1FD@iki.fi> Message-ID: <20120322084609.GF28323@charite.de> * Timo Sirainen : > > Right now the whole system is down because nobody can acces his/her > > mails due to this. > > All of your mails are compressed and have wrong S=size in the filename? You can disable the check with the attached patch, but I'm not sure if there are other places where it fails. At least quota calculations won't be correct. That patch totally saved my ass. I rolled it out today and the Mar 22 09:33:00 postamt dovecot: imap(stoffelm): Error: Maildir filename has wrong S value, renamed the file from /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd to /home/s/t/stoffelm/Maildir/.Deleted Messages/cur/1331891533.M93099P19536.postamt.charite.de,S=1860:2,Scd errors subsided. At the same time the users CAN access the affected folder. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From amateo at um.es Thu Mar 22 11:55:58 2012 From: amateo at um.es (Angel L. Mateo) Date: Thu, 22 Mar 2012 10:55:58 +0100 Subject: [Dovecot] dovecot-auth restaring and caching Message-ID: <4F6AF72E.9030206@um.es> Hello, I'm trying to configure dovecot (2.0.13) to cache user and pass dbs. This a mail server whose purpose is only to deliver messages through dovecot lda. My users are in a ldap server. So I have configure auth_cache_size (with 20MB) and auth_cache_ttl (with 1 day). I have checked that caching is being done, and it is. If a send a message to a user, dovecot looks for it in my ldap server. If then I send another, then it uses cache information. The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. This is the auth log for the first message: Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:29:41 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:29:41 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:29:41 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:29:41 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXX Mar 22 10:29:41 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXXX#011gid=XXX#011home=XXXXXXXXXXXX And this is the second one, just after a few minutes: Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 22 10:41:03 lynx10 dovecot: auth: Debug: Module loaded: /usr/lib/dovecot/modules/auth/libauthdb_ldap.so Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master in: USER#0111#011amateo#011service=lda Mar 22 10:41:03 lynx10 dovecot: auth: Debug: prefetch(amateo): passdb didn't return userdb entries, trying the next userdb Mar 22 10:41:03 lynx10 dovecot: auth: Debug: userdb-cache(amateo): miss Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): user search: fields=irisMailbox,homeDirectory,uidNumber,gidNumber Mar 22 10:41:03 lynx10 dovecot: auth: Debug: ldap(amateo): result: uidNumber(uid)=XXXXX gidNumber(gid)=XXX homeDirectory(home)=XXXXXXXXXXX Mar 22 10:41:03 lynx10 dovecot: auth: Debug: master out: USER#0111#011amateo#011uid=XXXXX#011gid=XXX#011home=XXXXXXXXXX This is my configuration: root at lynx10:/etc/dovecot/conf.d# doveconf -n # 2.0.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-4-amd64 x86_64 Ubuntu 10.04.4 LTS auth_cache_size = 20 M auth_cache_ttl = 1 days auth_debug = yes auth_verbose = yes hostname = lynx10 passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +imapflags sieve_max_redirects = 15 } postmaster_address = postmaster at um.es protocols = " imap lmtp pop3" service auth { unix_listener auth-userdb { mode = 0666 } } ssl_cert = References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> Message-ID: <4F6A8BAC.4000002@mur.at> Am 16.3.2011 20:59, schrieb Gedalya: >> >>> Starting program: /usr/bin/doveadm -o imapc_user=jedi at example.com -o imapc_password=**** backup -u jedi at example.com -R imapc: >> > Timo, we have a problem, somewhere between 2.1.rc7 and 2.1.1. Current > versions are putting the body of the last message in "Sent Items" in > place of every single email in INBOX. > In other words, for every email that sits in INBOX in the source, I get > a copy of the last email in "Sent Items" instead. > This happens for every account I try to migrate. > Very strange. I noticed this only now, and the last package I have left > in the local apt cache which still works is 2.1.rc7-0~auto+0. i see the same regression (2.1.3-0~auto+4) :( doveadm sync/backup via impac puts the same message all over the place... From tss at iki.fi Thu Mar 22 16:05:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 16:05:34 +0200 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> Message-ID: <1332425134.26095.88.camel@innu> Hi, These should fix it properly: http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: > The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e > > I'll figure out a proper fix soon. > > On 21.3.2012, at 18.04, Luca Palazzo wrote: > > > It worked. We have no more sigsegv on *-login process. > > > > Thanks > > > > Luca > > > > Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: > >> Hi, > >> > >> On 20.3.2012, at 11.09, Luca Palazzo wrote: > >> > >>> Hi Timo, hi all, > >>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. > >>> > >>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> 710 { > >>> (gdb) bt > >>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 > >>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 > >>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 > >>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 > >> > >> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. > >> > > > From luca.palazzo at unict.it Thu Mar 22 16:16:34 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Thu, 22 Mar 2012 15:16:34 +0100 Subject: [Dovecot] 2.1.2 (pop3|imap)-login crash In-Reply-To: <1332425134.26095.88.camel@innu> References: <4F684938.9000208@unict.it> <4F69FBF0.6090003@unict.it> <3A80E95B-932A-40C9-B58B-36C6DEE5DDA6@iki.fi> <1332425134.26095.88.camel@innu> Message-ID: <4F6B3442.8020000@unict.it> I've applied both and reverted previous one. Everything seems to run flowless. Thanks Luca On 03/22/2012 03:05 PM, Timo Sirainen wrote: > Hi, > > These should fix it properly: > > http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 > http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d > > On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote: >> The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e >> >> I'll figure out a proper fix soon. >> >> On 21.3.2012, at 18.04, Luca Palazzo wrote: >> >>> It worked. We have no more sigsegv on *-login process. >>> >>> Thanks >>> >>> Luca >>> >>> Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto: >>>> Hi, >>>> >>>> On 20.3.2012, at 11.09, Luca Palazzo wrote: >>>> >>>>> Hi Timo, hi all, >>>>> after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer. >>>>> >>>>> 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> 710 { >>>>> (gdb) bt >>>>> #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 >>>>> #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 >>>>> #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa at dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 >>>>> #3 0xb77c9a45 in login_proxy_free_reason (_proxy=, reason=0x804e248 "Disconnected by server") at login-proxy.c:373 >>>> >>>> Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch. >>>> >>> >> > > From micah at riseup.net Thu Mar 22 16:38:55 2012 From: micah at riseup.net (Micah Anderson) Date: Thu, 22 Mar 2012 10:38:55 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply Message-ID: <87bonon1j4.fsf@algae.riseup.net> I've been moving users from one system to another by doing a dsync mirror operation. The first dsync mirror takes some time, because of the number of users involved, so I am doing an initial sync, and then I direct the users to the new location and do a final 'freshening' sync to get any changes that happened during the longer sync. The problem seems to be with this freshening sync, it seems quite fragile breaking with things like: dsync-local(user at example.com): Error: msg-get failed: box=Spam uid=3034 guid=1ad456015ae9694f083b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69096 guid=c22b541a71e4694fc93700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69097 guid=4b6d6b13d0e9694f505700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69098 guid=175b1c2e4aea694fc97100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69099 guid=bfb08c1b3bee694f133e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69100 guid=fa5d630c17ef694fa75f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69101 guid=7ca96011dcef694f3f0400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69102 guid=ef547107eff1694ff96700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69103 guid=5597bc0519f2694f2e7000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69104 guid=8336a53a54f5694fb21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69105 guid=96169d13c8fd694f831800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69106 guid=af21a5183f036a4f263200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69107 guid=d0fde3348e036a4ff44000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69108 guid=4ce01d1a59056a4fee2200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69109 guid=497f96066e056a4f322700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69110 guid=ef34f505c0066a4fc26b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69111 guid=81adcb2c6e076a4f751100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69112 guid=a110841e8a076a4fa21500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69113 guid=60d8e70a970d6a4fae2100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69114 guid=7c6cb41572106a4ff13c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69115 guid=aaf4d32b2f126a4ff21000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69116 guid=ab52f43a58126a4ffd1800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69117 guid=eb543a2179186a4fe45800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69118 guid=cd7cb408a12a6a4f272100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69119 guid=2ec02e2ef2326a4f9e1100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69120 guid=e7a4552ff8336a4f7f6700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69121 guid=0724b023d33a6a4f1b3300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69122 guid=9985c91afe3b6a4f127100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69123 guid=9300751b913d6a4f7a4000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69124 guid=822ff806ae3f6a4f293b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69125 guid=eac8ed1f2b426a4f164200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69126 guid=4109561ae3426a4ff26700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69127 guid=30bc832e5e496a4f563600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69128 guid=c0b36410bd4b6a4f102b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69129 guid=38a9d41a534d6a4ff40200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69130 guid=b8e84d239b4d6a4fd11000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69131 guid=f060ef22154f6a4f2b5c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69132 guid=e2999c107c4f6a4f5a7600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69133 guid=8d09280aae506a4f073500001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69134 guid=43d7ec3aa6556a4f963a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69135 guid=c5800130d2556a4f594200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69136 guid=83a91e08b4566a4f197100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69137 guid=50cf9721f95e6a4f7e4400001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69138 guid=fda2a82886606a4f881700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69139 guid=97ee1d1ad1636a4fc94d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69140 guid=4f50671f85666a4f306100001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69141 guid=0fea590fb4666a4f7a6b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69142 guid=f3210b02a5676a4ffa1f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69143 guid=8d325a06686a6a4f2b3600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69144 guid=0cbf1839f1756a4f8f6800001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69145 guid=d6209a2898796a4f671a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69146 guid=576ade31da7d6a4f5f5700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69147 guid=70a15b34247e6a4f445a00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69148 guid=3ff92631cd886a4ffd6300001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69149 guid=3ff2081568916a4f134d00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69150 guid=d1a67b0907ab6a4f546000001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69151 guid=3d4cb1197ee96a4fbf5f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=INBOX uid=69152 guid=aae2542818266b4f7d1e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2077 guid=c5a0930248e8694fb77c00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2078 guid=dce63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2079 guid=dde63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2080 guid=dee63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2081 guid=dfe63e1a6bee694f470200001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2082 guid=a110b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2083 guid=a210b53585056a4fe81b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2084 guid=93afeb1f7d0b6a4f694e00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2085 guid=ef2bb1098a2e6a4fcf1b00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2086 guid=d8ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2087 guid=d9ba601bee2e6a4f982f00001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2088 guid=7e5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2089 guid=7f5b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2090 guid=b08c5205ba646a4f106700001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2091 guid=805b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: msg-get failed: box=sent-mail uid=2092 guid=815b483b433d6a4fd93600001d12b59f dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. those final "Unexpected finish reply" and "Unexpected reply from server" are a bit surprising results. Running dsync again seems to resolve things, but there seems to be a bug here that is causing unexpected results to leak through to the dsync process? thanks, micah From jtl+dovecot at uvm.edu Thu Mar 22 18:17:10 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Thu, 22 Mar 2012 12:17:10 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6AA667.1080908@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> Message-ID: <4F6B5086.4030001@uvm.edu> On 03/22/2012 12:11 AM, Stan Hoeppner wrote: > On 3/21/2012 12:04 PM, Timo Sirainen wrote: >> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >> >> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. > If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could > use a true cluster file system such as OCFS or GFS. Both will eliminate > this problem, and without requiring Dovecot director. And you'll get > better performance than with Gluster, which, BTW, isn't really suitable > as a transactional filesystem, was not designed for such a use case. Speaking as an admin who has run Dovecot on top of GFS both with and without the director, I would never go back to a cluster without the director. The cluster performs *so* much better when glocks can be cached on a single node, and this can't happen if a single user has IMAP processes on separate nodes. No, you don't strictly need the director if you have GFS, but if you can manage it, you'll be a lot happier. Jim From ms at mur.at Thu Mar 22 19:09:13 2012 From: ms at mur.at (Martin Schitter) Date: Thu, 22 Mar 2012 18:09:13 +0100 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6A927C.6010003@gedalya.net> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> Message-ID: <4F6B5CB9.9080204@mur.at> Am 2012-03-22 03:46, schrieb Gedalya: >> >> doveadm sync/backup via impac puts the same message all over the place... > > Thanks Martin, I've set up a test platform to investigate this further > but I've been short on time... after some debugging a few more remarks about this problem: the bug only appears on recursive folder hierarchies. if you specity option "-m INBOX" everything works fine. for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( has anyone a clue how to fix this problem in the source code? From tss at iki.fi Thu Mar 22 20:57:21 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 22 Mar 2012 20:57:21 +0200 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <4F6AF72E.9030206@um.es> References: <4F6AF72E.9030206@um.es> Message-ID: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> On 22.3.2012, at 11.55, Angel L. Mateo wrote: > The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. service auth { idle_kill = 0 } From ednitido at gmail.com Thu Mar 22 23:18:12 2012 From: ednitido at gmail.com (Ed Nitido) Date: Thu, 22 Mar 2012 17:18:12 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy Message-ID: Hey all, I've upgraded from a working Dovecot 2.0.17 Proxy with a master user setup to Dovecot 2.1.3 and I've merged my conf settings from 2.0.17 into 2.1.3. I'm able to start up dovecot proxy and telnet localhost, however it creates the users home director on the proxy server instead of going to the backend dovecot server (which has already been successfully upgraded to 2.1.3 from 2.0.17). In my old 2.0.17, I had the entire namespace section commented out in 10-mail.conf. To achieve the same in 2.1.3 I had to remove the file 15-mailboxes.conf, otherwise my error log would have the following if I just commented out the namespace section and left that file in place: *Error: user edward at dev.domain.com: Initialization failed: namespace configuration error: inbox=yes namespace missing* *Error: Invalid user settings. Refer to server log for more information.* So now I'm trying to figure out why the proxy doesn't get the mail server IP from director, when doveadm has the backend server listed *# doveadm director status* *mail server ip vhosts users* *192.168.12.205 100 0* Can anyone see something I've missed? Here's my doveconf output: # 2.1.3: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-openvz-amd64 i686 Debian 6.0.4 auth_debug = yes auth_debug_passwords = yes auth_master_user_separator = * auth_mechanisms = plain login auth_socket_path = /usr/local/var/run/dovecot/auth-master auth_verbose = yes debug_log_path = /var/log/dovecot-debug.log director_doveadm_port = 542 director_mail_servers = 192.168.12.205 director_servers = 192.168.12.209 disable_plaintext_auth = no info_log_path = /var/log/dovecot-info.log listen = * lmtp_proxy = yes log_path = /var/log/dovecot-err.log mail_debug = yes mail_gid = vmail mail_location = maildir:%h/Maildir mail_plugins = " quota" mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/home/%d/shared-mailboxes quota = maildir:User quota quota_rule = *:storage=1G sieve = %h/.dovecot.sieve sieve_dir = ~/sieve sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M sieve_quota_max_scripts = 0 sieve_quota_max_storage = 0 } postmaster_address = postmaster at dev.domain.com protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0666 user = dovecot } } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 541 } inet_listener { port = 542 } unix_listener director-userdb { mode = 0600 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { port = 24 } unix_listener /var/spool/postfix/private/dovecot-lmtp { user = postfix } } service managesieve-login { executable = managesieve-login director inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service pop3-login { executable = pop3-login director inet_listener pop3 { port = 110 } } ssl_cert = Hi all, We are currently using snapshots and rsync to backup a large mail server to a backup mail server. I have been looking into using dsync to replace rsync in hopes that it would make backups more efficient. I decided to test the performance using a single mailbox. Unfortunately dsync seems to run much slower than rsync. Rsync was able to sync the mailbox in 2 seconds. dsync took over a minute. The test was run so that the source and destination are on the same filesystem. We would like to using the new replication system, but that doesn't seem likely since the performance of the underlying dsync is so much slower than rsync. Even with the extra work that dsync is doing I can't believe the difference in performance would be that great. I realize that dsync is actively being worked on and I hope bringing attention to performance issue will provoke some ideas on how to improve it. Here is the output of the tests using dovecot 2.1.3: [root at n24 bu]# du -hs /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ 517M /home/10.0.1.101/1009/users/testuser%domain.com/Maildir/ [root at n24 bu]# time rsync -va /home/10.0.1.101/1009/users/testuser% domain.com/Maildir/ . sending incremental file list Maildir/ Maildir/dovecot-uidlist [ ... deleted cruft ... ] Maildir/cur/1332387577.M381054P27635.n24,S=14215502,W=14448554:2, Maildir/new/ Maildir/tmp/ sent 540927820 bytes received 1222 bytes 216371616.80 bytes/sec total size is 540855755 speedup is 1.00 real 0m2.677s user 0m3.184s sys 0m1.513s [root at n24 bu]# time dsync backup -u testuser at domain.com \ mdbox:/home/bu/testuser real 1m9.519s user 1m7.592s sys 0m1.126s [root at n24 bu]# time dsync backup -u testuser at domain.com \ sdbox:/home/bu/testuser2 real 1m2.164s user 1m0.882s sys 0m0.993s [root at n24 bu]# From list at airstreamcomm.net Fri Mar 23 04:36:44 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Thu, 22 Mar 2012 21:36:44 -0500 Subject: [Dovecot] Dovecot and scalable database storage Message-ID: I saw some interesting mails from TImo back in 2009 talking about the idea of using something like Cassandra db or similar as a storage platform for both email and index/logs. I was wondering if this has been discussed since then, and if there are any plans to support something like this in the future? I have been playing with Cassandra and found that their RackAwareStrategy gives you the ability to replicate writes to as many nodes as you would like, but more importantly what nodes and one of those nodes could be defined by what rack it lives in or what data center it lives in. This means multiple sites high available storage clusters, seemingly a system that dovecot could benefit from in terms of performance and redundancy and simplicity. Any takers? From stan at hardwarefreak.com Fri Mar 23 09:13:18 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 02:13:18 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6B5086.4030001@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> Message-ID: <4F6C228E.5060902@hardwarefreak.com> On 3/22/2012 11:17 AM, Jim Lawson wrote: > On 03/22/2012 12:11 AM, Stan Hoeppner wrote: >> On 3/21/2012 12:04 PM, Timo Sirainen wrote: >>> The problem is most likely the same as with NFS: Server A caches data -> server B modifies data -> server A modifies data using stale cached state -> corruption. Glusterfs works with FUSE, and FUSE has quite similar problems as NFS. >>> >>> With director you guarantee that the same mailbox isn't accessed simultaneously by multiple servers, so this problem goes away. >> If using "real" shared storage i.e. an FC or iSCSI SAN LUN, you could >> use a true cluster file system such as OCFS or GFS. Both will eliminate >> this problem, and without requiring Dovecot director. And you'll get >> better performance than with Gluster, which, BTW, isn't really suitable >> as a transactional filesystem, was not designed for such a use case. > > Speaking as an admin who has run Dovecot on top of GFS both with and > without the director, I would never go back to a cluster without the > director. The cluster performs *so* much better when glocks can be > cached on a single node, and this can't happen if a single user has IMAP > processes on separate nodes. > > No, you don't strictly need the director if you have GFS, but if you can > manage it, you'll be a lot happier. Did/do you see the Director/glock benefit with both maildir and mdbox Jim? Do you see any noteworthy performance differences between the two formats on GFS, with and without Director? BTW, are you hitting FC or iSCSI LUNs? -- Stan From tlx at leuxner.net Fri Mar 23 11:40:52 2012 From: tlx at leuxner.net (Thomas Leuxner) Date: Fri, 23 Mar 2012 10:40:52 +0100 Subject: [Dovecot] Dovecot v2.1.3 (f30437ed63dc) Auth/Login Issues Message-ID: <20120323094052.GA9851@nihlus.leuxner.net> Hi, some change between ff5c341f8838 and f30437ed63dc seems to have broken auth: => Bad Login Mar 23 09:01:46 spectre dovecot: master: Dovecot v2.1.3 (f30437ed63dc) starting up [...] Mar 23 10:25:44 spectre dovecot: auth: Debug: auth client connected (pid=7266) Mar 23 10:25:45 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=62388#011resp= Mar 23 10:25:45 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): hit: #011userdb_quota_rule=*:storage=5G#011userdb_acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:25:45 spectre dovecot: auth: Debug: master in: REQUEST#0113958898689#0117266#0111#011bfc44f32051961b909e2b458440d645f Mar 23 10:25:45 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): hit: tlx at leuxner.net#011uid=5000#011gid=5000#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: auth: Debug: master out: USER#0113958898689#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:25:45 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=7267, TLS Mar 23 10:25:45 spectre dovecot: imap(tlx at leuxner.net): Connection closed in=0 out=319uthentication/login: => Good Login Mar 23 10:26:37 spectre dovecot: master: Dovecot v2.1.3 (ff5c341f8838) starting up [...] Mar 23 10:27:18 spectre dovecot: auth: Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 10:27:18 spectre dovecot: auth: Debug: auth client connected (pid=9832) Mar 23 10:27:19 spectre dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011lip=188.138.0.199#011rip=80.187.102.243#011lport=143#011rport=51647#011resp= Mar 23 10:27:19 spectre dovecot: auth: Debug: cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file /var/vmail/auth.d/leuxner.net/passwd: Read 1 users in 0 secs Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: client out: OK#0111#011user=tlx at leuxner.net Mar 23 10:27:19 spectre dovecot: auth: Debug: master in: REQUEST#0113656384513#0119832#0111#0114782efcbd0324b228bb85aaae916cfe6 Mar 23 10:27:19 spectre dovecot: auth: Debug: userdb-cache(tlx at leuxner.net,80.187.102.243): miss Mar 23 10:27:19 spectre dovecot: auth: Debug: passwd-file(tlx at leuxner.net,80.187.102.243): lookup: user=tlx at leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd Mar 23 10:27:19 spectre dovecot: auth: Debug: master out: USER#0113656384513#011tlx at leuxner.net#011uid=xxx#011gid=xxx#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins Mar 23 10:27:19 spectre dovecot: imap-login: Login: user=, method=PLAIN, rip=80.187.102.243, lip=188.138.0.199, mpid=9835, TLS Regards Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From tss at iki.fi Fri Mar 23 11:51:43 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 11:51:43 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332405027.6792.7.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> Message-ID: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> On 22.3.2012, at 10.30, Noel Butler wrote: > On Thu, 2012-03-22 at 07:28 +0100, Andra? 'ruskie' Levstik wrote: > >> :2012-03-22T11:55:Noel Butler: >> >>> perhaps it should be renamed then, given it violates the known normal >>> for SYSCONF dir, you've just created another form of --datadir >> >> Not really. The way I see it works as expected. The sysconf dir is the > > > Then you and I and a few other devs involved in other very well known > bits of software that everyone likely uses, will have to agree to > disagree A ton of software installs into /etc// directory. Most Linux distributions installed Dovecot v1.x that way as well. And of course everyone expects configuration to be under /etc. The default of sysconfdir is PREFIX/etc/. Dovecot v2.0 really shouldn't install its stuff into PREFIX/etc/ but into PREFIX/etc/dovecot/. So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? From alain.defrance at univ-evry.fr Fri Mar 23 12:20:01 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:20:01 +0100 Subject: [Dovecot] quota ldap Message-ID: <4F6C4E51.7010603@univ-evry.fr> hello all, i'm using quota + ldap with dovecot 2 in dovecot-ldap.conf.ext file i have the line : user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid,quota=quota_rule=*:storage=%$B how can i add "Trash:storage= " to have more place for deleting messages like in 90-quota.conf file ? quota_rule2 = Trash:storage thanks for help regards -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From mlists at edicom.eu Fri Mar 23 12:38:18 2012 From: mlists at edicom.eu (Miguel Tormo) Date: Fri, 23 Mar 2012 11:38:18 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <201203231138.18338.mlists@edicom.eu> El Mi?rcoles, 21 de Marzo de 2012 15:43:14 Luca Lesinigo escribi?: > Hello list. Hello, > > I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. > I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice. > > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. > It would be desirable for multiple reasons: I have recently deployed a very similar setup: imap proxy, mailbox sharding... Although not exactly like yours. Comments below: > - graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes This is reasonable. For example, I did this to seamless migrate lots of users from one server to another, migrating just a few of them at a time. > - be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) You can do the exact same thing in the future, of course. > - be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) This is fairly easy to accomplish with imap proxying. > - be ready to serve traffic over IPv6 without touching our precious mailbox servers This is doable. > - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > > Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). I didn't implement a lemonade profile nor quotas in my setup. However, I can confirm you that IMAP IDLE does work with imap proxy. > > In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). No problem with that, but this is related to the MTA configuration, not dovecot. > > I'd like to know if that plan sounds reasonable or if there's something stupid in it. > Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. From dovecot-l at fu-berlin.de Fri Mar 23 12:44:32 2012 From: dovecot-l at fu-berlin.de (Heiko Schlichting) Date: Fri, 23 Mar 2012 11:44:32 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <20120323104432.GB1353054@CIS.FU-Berlin.DE> Timo wrote: > So the only way I can think of how to change this is to add another > option to optionally remove the dovecot/ suffix from the directory, but > is this really worth the trouble? I would appreciate such option too. For large dedicated installations other schemes than /etc/dovecot are common. See http://dovecot.org/list/dovecot/2009-January/036131.html Heiko Heiko Schlichting Freie Universit?t Berlin heiko.schlichting at fu-berlin.de Zentraleinrichtung f?r Datenverarbeitung Telefon +49 30 838-54327 Fabeckstra?e 32 Telefax +49 30 838454327 14195 Berlin From nmilas at noa.gr Fri Mar 23 12:48:18 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Fri, 23 Mar 2012 12:48:18 +0200 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C4E51.7010603@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> Message-ID: <4F6C54F2.7020203@noa.gr> On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > how can i add "Trash:storage= " to have more place for deleting > messages like in > See, for example, my setup: http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html I use a single conf file (because it's small and it's more intuitive to me). Details: http://wiki2.dovecot.org/Quota/Configuration Regards, Nick From tss at iki.fi Fri Mar 23 12:50:04 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:50:04 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> Message-ID: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> On 21.3.2012, at 16.43, Luca Lesinigo wrote: > The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. From tss at iki.fi Fri Mar 23 12:53:16 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 12:53:16 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <20120323104432.GB1353054@CIS.FU-Berlin.DE> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> Message-ID: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> On 23.3.2012, at 12.44, Heiko Schlichting wrote: > Timo wrote: >> So the only way I can think of how to change this is to add another >> option to optionally remove the dovecot/ suffix from the directory, but >> is this really worth the trouble? > > I would appreciate such option too. For large dedicated installations other > schemes than /etc/dovecot are common. > > See http://dovecot.org/list/dovecot/2009-January/036131.html Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). From alain.defrance at univ-evry.fr Fri Mar 23 12:58:09 2012 From: alain.defrance at univ-evry.fr (Alain DEFRANCE) Date: Fri, 23 Mar 2012 11:58:09 +0100 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C54F2.7020203@noa.gr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> Message-ID: <4F6C5741.3000408@univ-evry.fr> thanks Nick so if i understand correctly i can mix the 2 quota_rule ? the one who came from ldap user_attrs (quota_rule=*:bytes=%$) and the other which from quota_rule2 = Trash:storage=+3%% in your case you add 3% quota more for Trash ? Am i write ? regards > On 23/3/2012 12:20 ??, Alain DEFRANCE wrote: > >> how can i add "Trash:storage= " to have more place for deleting >> messages like in >> > > See, for example, my setup: > http://old.nabble.com/ldap-userdb-warning-in-v2.1.1-td33544211.html > > I use a single conf file (because it's small and it's more intuitive > to me). > > Details: http://wiki2.dovecot.org/Quota/Configuration > > Regards, > Nick > -- *Alain DEFRANCE* - Ing?nieur syst?mes et r?seaux Direction des syst?mes d'information (DiSI) Centre d'Exploitation des Infrastructures Informatiques (CEDII) Cellule R?seau et Expertise Syst?mes B?t Ile de France - RDC - Bureau 58 Universit? d'Evry Val d'Essonne 4, Bd F. Mitterrand - 91025 EVRY Cedex Tel : 01.69.47.80.69 - Fax : 01.69.47.80.24 Mail : alain.defrance at univ-evry.fr Site UEVE : http://www.univ-evry.fr From jtam.home at gmail.com Fri Mar 23 12:58:36 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Fri, 23 Mar 2012 03:58:36 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 Message-ID: I ran into two issues trying to upgrade our dovecot installation (Solaris 10). 1) Does not compile with OpenSSL 0.9.7 Not a big deal, as I was able to successfully against OpenSSL 0.9.8, but does dovecot require OpenSSL >= 0.9.8 now? libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib -I../../src/lib-test -std=gnu99 -O3 -fomit-frame-pointer -mcpu=ultrasparc -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -MT istream-openssl.lo -MD -MP -MF .deps/istream-openssl.Tpo -c istream-openssl.c -fPIC -DPIC -o .libs/istream-openssl.o iostream-openssl-context.c:9:28: openssl/engine.h: No such file or directory iostream-openssl-context.c: In function `ssl_iostream_deinit_global': iostream-openssl-context.c:431: warning: implicit declaration of function `ENGINE_finish' iostream-openssl-context.c:432: warning: implicit declaration of function `ENGINE_cleanup' ... 2) Dovecot's LDA does not work After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe exact same config file, local mail delivery tempfails: Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address Mar 23 02:51:51 server dovecot: auth: Error: userdb connection: Failed to get peer's credentials Mar 23 02:51:51 server dovecot: lda: Error: userdb lookup(j.tam): Disconnected unexpectedly Mar 23 02:51:51 server dovecot: lda: Fatal: Internal error occurred. Refer to server log for more information. # Sendmail reports stat=Deferred: local mailer (/var/dovecot/libexec/dovecot-lda) exited with EX_TEMPFAIL After seeing 2) in the logs, I had to revert back to 2.0.16. Any hints on what could be wrong? Joseph Tam # 2.0.16: /var/dovecot/etc/dovecot/dovecot.conf # OS: SunOS 5.10 sun4u nfs auth_cache_negative_ttl = 10 mins auth_cache_size = 64 k auth_cache_ttl = 1 days auth_failure_delay = 5 secs auth_master_user_separator = * auth_socket_path = /var/dovecot/run/auth-userdb auth_username_chars = abcdefghijklmnopqrstuvwxyz01234567890.-_ auth_worker_max_count = 1 base_dir = /var/dovecot/run default_vsz_limit = 64 M deliver_log_format = first_valid_gid = 10000 first_valid_uid = 10000 hostname = our.mail.domain last_valid_gid = 19999 last_valid_uid = 19999 lda_mailbox_autocreate = yes log_timestamp = login_greeting = Ready. mail_location = mbox:/nfs/home/%n/mail:INBOX=/nfs/mail/%n:INDEX=/data/dc-cache/%n mail_nfs_storage = yes mail_temp_dir = /var/tmp mbox_very_dirty_syncs = yes mbox_write_locks = dotlock_try fcntl namespace { inbox = yes location = prefix = separator = / } namespace { hidden = yes list = no location = prefix = / separator = / } namespace { hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace { hidden = yes list = no location = prefix = mail/ separator = / } passdb { args = /var/dovecot/etc/master-users driver = passwd-file master = yes pass = yes } passdb { args = /var/yp/etc/passwd driver = passwd-file } postmaster_address = MAILER-DAEMON at our.mail.domain protocols = imap pop3 sendmail_path = /usr/lib/sendmail service auth-worker { user = dovecot } service auth { idle_kill = 1 hours } service imap-login { process_limit = 2 service_count = 0 } service imap { process_limit = 512 } service pop3-login { process_limit = 1 service_count = 0 } service pop3 { process_limit = 64 } shutdown_clients = no ssl_cert = References: Message-ID: On Fri, 23 Mar 2012, dovecot-request at dovecot.org wrote: >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the > dovecot/ suffix from other directories as well. That might be > something worth doing (--without-package-suffix or something?). +1. I fake it now with symlinks (e.g. etc/dovecot -> .). Joseph Tam From rainer.frey at inxmail.de Fri Mar 23 13:19:45 2012 From: rainer.frey at inxmail.de (Rainer Frey) Date: Fri, 23 Mar 2012 12:19:45 +0100 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> Message-ID: <69027C8C-A95B-41C2-B06B-824345F738DA@inxmail.de> On Mar 23, 2012, at 10:51 AM, Timo Sirainen wrote: >>> :2012-03-22T11:55:Noel Butler: >>> >>>> perhaps it should be renamed then, given it violates the known normal >>>> for SYSCONF dir, you've just created another form of --datadir >>> >>> Not really. The way I see it works as expected. >> >> The directory for installing read-only data files that pertain >> to a single machine?that is to say, files for configuring a >> host. Mailer and network configuration files, ?/etc/passwd?, and >> so forth belong here. All the files in this directory should be >> ordinary ASCII text files. This directory should normally be >> ?/usr/local/etc?, but write it as ?$(prefix)/etc?. (If you are >> using Autoconf, write it as ?@sysconfdir@?.) Well, I don't see that that prevents organizing the files in sysconfdir into a subdirectory. > ton of software installs into /etc// directory. [...] > So the only way I can think of how to change this is to add another option to optionally remove the dovecot/ suffix from the directory, but is this really worth the trouble? I really don't think so. What for? Nobody has shown a real-world problem with that subdirectory. From tss at iki.fi Fri Mar 23 13:26:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 13:26:54 +0200 Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: References: Message-ID: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> On 23.3.2012, at 12.58, Joseph Tam wrote: > I ran into two issues trying to upgrade our dovecot installation (Solaris 10). > > 1) Does not compile with OpenSSL 0.9.7 > > Not a big deal, as I was able to successfully against OpenSSL 0.9.8, > but does dovecot require OpenSSL >= 0.9.8 now? Hm. Maybe it's time by now? :) It could be fixed with some more #ifdefs but those make code more unreadable. > 2) Dovecot's LDA does not work > > After stopping the the old dovecot, and starting dovecot 2.1.3 using tghe > exact same config file, local mail delivery tempfails: > > Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? From hsn at filez.com Fri Mar 23 13:41:24 2012 From: hsn at filez.com (Radim Kolar) Date: Fri, 23 Mar 2012 12:41:24 +0100 Subject: [Dovecot] delivering with maildrop Message-ID: <4F6C6164.2050506@filez.com> Can somebody provide maildrop syntax for using deliver-lda as final delivery program during sorting mail in user mailfilter? i mean replacement for "to" statement if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) { to $MAIL/.dovecot/ } From jtl+dovecot at uvm.edu Fri Mar 23 14:13:21 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 08:13:21 -0400 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C228E.5060902@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> Message-ID: <4F6C68E1.4030400@uvm.edu> On 3/23/12 3:13 AM, Stan Hoeppner wrote: >> Speaking as an admin who has run Dovecot on top of GFS both with and >> without the director, I would never go back to a cluster without the >> director. The cluster performs *so* much better when glocks can be >> cached on a single node, and this can't happen if a single user has IMAP >> processes on separate nodes. >> >> No, you don't strictly need the director if you have GFS, but if you can >> manage it, you'll be a lot happier. > Did/do you see the Director/glock benefit with both maildir and mdbox > Jim? Do you see any noteworthy performance differences between the two > formats on GFS, with and without Director? BTW, are you hitting FC or > iSCSI LUNs? > Actually, we're all mbox. This primarily has to do with how users do self-service mail recovery from backup: one folder = one file. I'd like to move to mdbox, but it would mean the recovery scripts will need to understand which files are associated with which folders, as well as restoring the associated index files. That's a to-do. We're using fibrechannel (IBM v7000) storage, but I would expect to see the same thing with iSCSI. It's mostly about different nodes contending over locks on the same files (although I'm sure cache locality helps a great deal, too.) If you end up with imap processes for the same folder on different nodes, or mail delivery happening on one node and imap on the other, you will feel the lag in your IMAP client. "Oh, my INBOX has been unresponsive for 10 seconds, I must be getting a lot of mail right now!" That's an exaggeration, but not by much. Jim From amateo at um.es Fri Mar 23 14:15:40 2012 From: amateo at um.es (Angel L. Mateo) Date: Fri, 23 Mar 2012 13:15:40 +0100 Subject: [Dovecot] dovecot-auth restaring and caching In-Reply-To: <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> References: <4F6AF72E.9030206@um.es> <7B9D052D-5864-42A9-AE9A-6FCE858F48C0@iki.fi> Message-ID: <4F6C696C.5030900@um.es> El 22/03/12 19:57, Timo Sirainen escribi?: > On 22.3.2012, at 11.55, Angel L. Mateo wrote: > >> The problem I'm having is that if I have no activity in the server, dovecot stops its auth process and when another message is received, it restarted it, but with an empty cache. > > service auth { > idle_kill = 0 > } > In a test server I have, this have solved the problem. In my productions servers it is still being restarted. Could it be another parameter involve in this? service_count is set to 0. I have also seen that, whenever dovecot/auth is restarted, dovecot/config has also been restarted. Could be related? My config related with this service auth is: service auth { chroot = client_limit = 4096 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } -- Angel L. Mateo Mart?nez Secci?n de Telem?tica ?rea de Tecnolog?as de la Informaci?n _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 From list at airstreamcomm.net Fri Mar 23 15:39:07 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 08:39:07 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: Message-ID: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> On Wed, 21 Mar 2012 09:56:12 -0600, James Devine wrote: > Anyone know how to setup dovecot with mdbox so that it can be used through > shared storage from multiple hosts? I've setup a gluster volume and am > sharing it between 2 test clients. I'm using postfix/dovecot LDA for > delivery and I'm using postal to send mail between 40 users. In doing > this, I'm seeing these errors in the logs > > Mar 21 09:36:29 test-gluster-client2 dovecot: lda(testuser34): Error: Fixed > index file /mnt/testuser34/mdbox/storage/dovecot.map.index: messages_count > 272 -> 271 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=3768 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Append with UID 516, but > next_uid = 517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=4220 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Error: Log > synchronization error at seq=4,offset=5088 for > /mnt/testuser28/mdbox/storage/dovecot.map.index: Extension record update > for invalid uid=517 > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser28): Warning: > fscking index file /mnt/testuser28/mdbox/storage/dovecot.map.index > Mar 21 09:36:30 test-gluster-client2 dovecot: lda(testuser34): Warning: > fscking index file /mnt/testuser34/mdbox/storage/dovecot.map.index > > > This is my dovecot config currently: > > jdevine at test-gluster-client2:~> dovecot -n > # 2.0.13: /etc/dovecot/dovecot.conf > # OS: Linux 3.0.0-13-server x86_64 Ubuntu 11.10 > lock_method = dotlock > mail_fsync = always > mail_location = mdbox:~/mdbox > mail_nfs_index = yes > mail_nfs_storage = yes > mmap_disable = yes > passdb { > driver = pam > } > protocols = " imap" > ssl_cert = ssl_key = userdb { > driver = passwd > } I was able to get dovecot working across a gluster cluster a few weeks ago and it worked just fine. I would recommend using the native gluster mount option (need to install gluster software on clients), and using distributed replicated as your replication mechanism. If you're running two gluster servers you should have a replica count of two with distributed replicated. You should test first to make sure you can create a file in both mounts and see it from every mount point in the cluster, as well as interact with it. It's also very important to make sure your servers are running with synchronized clocks from an NTP server. Very bad things happen to a (dovecot or gluster) cluster out of sync with NTP. From eliezer at ngtech.co.il Fri Mar 23 15:57:30 2012 From: eliezer at ngtech.co.il (Eliezer Croitoru) Date: Fri, 23 Mar 2012 15:57:30 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <4F6C814A.2050803@ngtech.co.il> On 23/03/2012 12:53, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > >> Timo wrote: >>> So the only way I can think of how to change this is to add another >>> option to optionally remove the dovecot/ suffix from the directory, but >>> is this really worth the trouble? >> >> I would appreciate such option too. For large dedicated installations other >> schemes than /etc/dovecot are common. >> >> See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > well squid is using another way such as the directory you specify and without the /dovecot (squid) suffix. it's not that important. if you do change the config directory you know where you are putting it. i,m using the /opt/(service name) to install most of my self complied software so idont really care about it. but if the sysconfig directory as a directive it should be the default. Regards, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il From stan at hardwarefreak.com Fri Mar 23 16:02:41 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 09:02:41 -0500 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C6164.2050506@filez.com> References: <4F6C6164.2050506@filez.com> Message-ID: <4F6C8281.10906@hardwarefreak.com> On 3/23/2012 6:41 AM, Radim Kolar wrote: > Can somebody provide maildrop syntax for using deliver-lda as final > delivery program during sorting mail in user mailfilter? > > i mean replacement for "to" statement > > if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) > { > to $MAIL/.dovecot/ > } Dovecot's local delivery agent uses the Sieve language: http://wiki.dovecot.org/LDA/Sieve The syntax is quite different than maildrop or procmail. -- Stan From tss at iki.fi Fri Mar 23 16:06:25 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 16:06:25 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On 23.3.2012, at 15.39, wrote: > I was able to get dovecot working across a gluster cluster a few weeks ago > and it worked just fine. I would recommend using the native gluster mount > option (need to install gluster software on clients), and using distributed > replicated as your replication mechanism. Have you tried stress testing it with imaptest? Run in parallel for both servers: imaptest host=gluster1 user=testuser pass=testpass imaptest host=gluster2 user=testuser pass=testpass http://imapwiki.org/ImapTest And see if Dovecot logs any errors. From micah at riseup.net Fri Mar 23 17:52:02 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 11:52:02 -0400 Subject: [Dovecot] dovecot 2.1.3 dsync Unexpected finish reply References: <87bonon1j4.fsf@algae.riseup.net> Message-ID: <87ty1fl3h9.fsf@algae.riseup.net> Micah Anderson writes: > dsync-local(user at example.com): Error: Unexpected finish reply: by ims-d13.mx.aol.com (8.14.1/8.14.1) with ESMTP id q2LEhqXZ017169; > dsync-local(user at example.com): Error: Unexpected reply from server: Wed, 21 Mar 2012 10:43:52 -0400 > dsync-local(user at example.com): Warning: Mailbox changes caused a desync. You may want to run dsync again. I'm also getting similar strange results with my regular dsync backup: dsync-local(user at example.com): Error: Unexpected reply from server: 0 23bdce147b43674f8e2700002c449efa 1242 146 \Recent 1332335848 this is with 2.1.3. micah From micah at riseup.net Fri Mar 23 18:25:27 2012 From: micah at riseup.net (Micah Anderson) Date: Fri, 23 Mar 2012 12:25:27 -0400 Subject: [Dovecot] doveadm user -f index Message-ID: <87pqc3l1xk.fsf@algae.riseup.net> I've configured my mail_location to have a different location for performance reasons so they aren't in the same location as the mail_location. The 'doveadm user -f home' is useful to find where a user's home directory is for various scripting purposes, but I can't seem to find a way to determine the location of the user's indexes. I can do something with the output of dovecot -a to find the mail_location and then look for a configured INDEX, but then I don't have a good way of translating the %d/%1n/%n type string formatters into their values for a user. thanks for any suggestions! micah -- From stan at hardwarefreak.com Fri Mar 23 19:11:49 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Fri, 23 Mar 2012 12:11:49 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6C68E1.4030400@uvm.edu> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> Message-ID: <4F6CAED5.4000206@hardwarefreak.com> On 3/23/2012 7:13 AM, Jim Lawson wrote: > On 3/23/12 3:13 AM, Stan Hoeppner wrote: > >>> Speaking as an admin who has run Dovecot on top of GFS both with and >>> without the director, I would never go back to a cluster without the >>> director. The cluster performs *so* much better when glocks can be >>> cached on a single node, and this can't happen if a single user has IMAP >>> processes on separate nodes. >>> >>> No, you don't strictly need the director if you have GFS, but if you can >>> manage it, you'll be a lot happier. >> Did/do you see the Director/glock benefit with both maildir and mdbox >> Jim? Do you see any noteworthy performance differences between the two >> formats on GFS, with and without Director? BTW, are you hitting FC or >> iSCSI LUNs? >> > > Actually, we're all mbox. This primarily has to do with how users do > self-service mail recovery from backup: one folder = one file. Yeah, mbox isn't as dead as some people contend, but it just doesn't have legs for newer deployment architectures. > I'd like to move to mdbox, but it would mean the recovery scripts will > need to understand which files are associated with which folders, as > well as restoring the associated index files. That's a to-do. That's an easy weekend project. ;) > We're using fibrechannel (IBM v7000) storage, but I would expect to see > the same thing with iSCSI. It's mostly about different nodes contending > over locks on the same files (although I'm sure cache locality helps a > great deal, too.) If you end up with imap processes for the same folder > on different nodes, or mail delivery happening on one node and imap on > the other, you will feel the lag in your IMAP client. "Oh, my INBOX has > been unresponsive for 10 seconds, I must be getting a lot of mail right > now!" That's an exaggeration, but not by much. I was asking about your SAN storage unrelated to the locking issue. Just a curiosity thing. Note my email domain. ;) I'm an FC fan but iSCSI seems to be more popular in many circles, actually pretty much market wide these days. So when I come across another SAN user I'm naturally curious as to what hardware they use. Just so nobody gets the wrong idea, I wasn't advocating against Director earlier in the thread. I think it's fantastic and solves some critical scalability problems. As in your case, it allows one to use his mail storage format of choice with a cluster filesystem while mostly avoiding the locking headaches. In the past one pretty much had to use maildir with a cluster FS to avoid the locking performance killed. But one had to suffer the higher IOPS load on the storage. Not always a good tradeoff, especially for busy mail systems. I assume you do still have some minor locking/performance issues with the INBOX, even with Director, when LDA and the user MUA are both hitting the INBOX index and mbox files. You'll still see this with mdbox, but probably to a lesser degree if you use a smallish mdbox_rotate_size value. To mitigate this INBOX locking you could go with a dual namespaces, using maildir or sdbox for the INBOX and mdbox for the other user mail folders. -- Stan From tss at iki.fi Fri Mar 23 19:19:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 19:19:26 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <3845B569-3CE5-4C0C-BB60-B9CA91FF8B56@iki.fi> On 23.3.2012, at 19.11, Stan Hoeppner wrote: > I assume you do still have some minor locking/performance issues with > the INBOX, even with Director, when LDA and the user MUA are both > hitting the INBOX index and mbox files. You'll still see this with > mdbox, but probably to a lesser degree if you use a smallish > mdbox_rotate_size value. To mitigate this INBOX locking you could go > with a dual namespaces, using maildir or sdbox for the INBOX and mdbox > for the other user mail folders. The biggest difference is that mbox requires read locks, mdbox doesn't. mdbox lock waits are very similar to maildir's. Of course, I don't know about the cluster filesystems' internal locking, but I thought it was even worse with Maildir than with mbox because it had to get a read lock for each read file, but I guess this depends on the filesystem. From ruskie at codemages.net Fri Mar 23 19:22:11 2012 From: ruskie at codemages.net (=?UTF-8?Q?Andra=C5=BE_'ruskie'_Levstik?=) Date: Fri, 23 Mar 2012 18:22:11 +0100 (CET) Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: :2012-03-23T12:53:Timo Sirainen: > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). I would suggest to have a --layout=gnu|opt That would either do what it currently does(gnu) and opt to install everything into a single dir i.e.: /opt/dovecot/ With subdirs under there. -- Andra? 'ruskie' Levstik Source Mage GNU/Linux Games/Xorg grimoire guru Re-Alpine Coordinator http://sourceforge.net/projects/re-alpine/ Geek/Hacker/Tinker Be advised: causing a disturbance may result in fines, detainment, bodily harm, or death. Enjoy your stay. From jtl+dovecot at uvm.edu Fri Mar 23 19:33:42 2012 From: jtl+dovecot at uvm.edu (Jim Lawson) Date: Fri, 23 Mar 2012 13:33:42 -0400 Subject: [Dovecot] recovery of mdbox folders (was: Re: distributed mdbox) In-Reply-To: <4F6CAED5.4000206@hardwarefreak.com> References: <41204A29-D6F2-453F-938F-D50B72A56710@iki.fi> <29E412BF-B6FC-4432-B6F5-78164C273FE1@iki.fi> <4F6AA667.1080908@hardwarefreak.com> <4F6B5086.4030001@uvm.edu> <4F6C228E.5060902@hardwarefreak.com> <4F6C68E1.4030400@uvm.edu> <4F6CAED5.4000206@hardwarefreak.com> Message-ID: <4F6CB3F6.5010006@uvm.edu> On 3/23/12 1:11 PM, Stan Hoeppner wrote: > On 3/23/2012 7:13 AM, Jim Lawson wrote: > > >> I'd like to move to mdbox, but it would mean the recovery scripts will >> need to understand which files are associated with which folders, as >> well as restoring the associated index files. That's a to-do. > That's an easy weekend project. ;) > Out of curiosity, does anyone do self-service restoration of individual mdbox folders? If I'm going to write a script to do it, it'd be nice to avoid any pitfalls someone else has already run into. :-) We're already backing up from snapshots, so the synchronization issues are solved (at least at backup time...) Jim From list at airstreamcomm.net Fri Mar 23 19:43:34 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 12:43:34 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: On Fri, 23 Mar 2012 16:06:25 +0200, Timo Sirainen wrote: > On 23.3.2012, at 15.39, > wrote: > >> I was able to get dovecot working across a gluster cluster a few weeks >> ago >> and it worked just fine. I would recommend using the native gluster >> mount >> option (need to install gluster software on clients), and using >> distributed >> replicated as your replication mechanism. > > Have you tried stress testing it with imaptest? Run in parallel for both > servers: > > imaptest host=gluster1 user=testuser pass=testpass > imaptest host=gluster2 user=testuser pass=testpass > > http://imapwiki.org/ImapTest > > And see if Dovecot logs any errors. I did stress test it, but we have developed a "mail bot net" tool for the purpose. I should mention this was tested using dovecot 1.2, as this is our current production version (hopefully will be upgrading soon). Its comprised of a control server that starts a bot network of client machines that creates pop/imap connections (smtp as well) on our test cluster of dovecot (and postfix) servers. In my test I distributed the load across a two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN storage attached to it. I actually didn't change my configuration from when I had a test NFS server connected to the test servers (mmap disabled, fcntl locking, etc), because glusterfs was an afterthought when we were stress testing our new netapp system using NFS. We have everything in VMware, including the glusterfs servers. Using five bot servers and connecting 7 times a second from each server (35 connections per second) for both pop and imap (70 total connections per second) split between two dovecot servers I was not seeing any big issues. The load average was low, and there were no errors to speak of in dovecot (or postfix). I was mounting the storage with the glusterfs native client, not using NFS (which I have not tested). I would like to do a more thorough test of glusterfs using Dovecot 2.0 on some dedicated hardware and see how much further I can push the system. From busseniu at in.tum.de Fri Mar 23 20:02:10 2012 From: busseniu at in.tum.de (=?UTF-8?B?Q2hyaXN0b3BoIEJ1w59lbml1cw==?=) Date: Fri, 23 Mar 2012 19:02:10 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: <4F6CBAA2.5020409@in.tum.de> Hi, maybe try "dsync -o mail_fsync=never". Cheers, Christoph -- Christoph Bu?enius Rechnerbetriebsgruppe der Fakult?ten Informatik und Mathematik Technische Universit?t M?nchen +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From luca at lm-net.it Fri Mar 23 20:12:56 2012 From: luca at lm-net.it (Luca Lesinigo) Date: Fri, 23 Mar 2012 19:12:56 +0100 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> Message-ID: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: > Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... >> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) > Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... > I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. > However, I can confirm you that IMAP IDLE does work with imap proxy. That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. > You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. > You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). -- Luca Lesinigo From gedalya at gedalya.net Fri Mar 23 20:24:11 2012 From: gedalya at gedalya.net (Gedalya) Date: Fri, 23 Mar 2012 14:24:11 -0400 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> Message-ID: <4F6CBFCB.60209@gedalya.net> On 03/23/2012 02:12 PM, Luca Lesinigo wrote: > Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. > I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. > I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. >>> We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) >> Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. > Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :) > > Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto: >>> - isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... >> I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. > Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it. The same proxy_maybe (dummy proxy) setup works great for POP3 too. Very simple to set up, works like a charm. Nothing much to think about. > >> However, I can confirm you that IMAP IDLE does work with imap proxy. > That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go. It will work as well as it was working with your existing courier server. But it will work great for accounts migrated to native dovecot. >> You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. > Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things. It's a very good idea to have completely separate machines for outgoing mail. Once you have imap-only boxes, you can eliminate the need for an MTA by using the dovecot LMTP server. Your postfix transport map can send mail to either smtp:imap.yourdomain.com:25 or lmtp:imap.yourdomain.com:2525 on a per account basis, and you can get rid of the MTA in due time. >> You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. > On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external). Since you're sending everything back to the MX, you might as well have your MX use LMTP, looking up the correct protocol and host from the database, and spend the next couple of years telling your customers to change their mail client configuration to use a dedicated outgoing mail server. It's worth the trouble. > > Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients). > > -- > Luca Lesinigo From ednitido at gmail.com Fri Mar 23 21:44:23 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 15:44:23 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are the same Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 From tss at iki.fi Fri Mar 23 21:46:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:46:53 +0200 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: Message-ID: On 23.3.2012, at 21.44, Ed Nitido wrote: > I've compared doveconf -n from both Dovecot 2.0.17 and 2.1.3 and they are > the same > > Everything works when I go back to 2.0.17, but doesn't when I use 2.1.3 Set auth_debug=yes. What does it log with v2.1.3? Also what's in your dovecot-ldap.conf.ext? From tss at iki.fi Fri Mar 23 21:49:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 21:49:38 +0200 Subject: [Dovecot] Advice for new dovecot / imap proxy? setup In-Reply-To: <4F6CBFCB.60209@gedalya.net> References: <63125D76-583A-49E2-A482-25661EA755F3@lm-net.it> <4A3F213B-67F3-468B-881E-E866CD768FE6@iki.fi> <44513908-C2F7-496D-B4FF-4644CD0ACE48@lm-net.it> <4F6CBFCB.60209@gedalya.net> Message-ID: On 23.3.2012, at 20.24, Gedalya wrote: > On 03/23/2012 02:12 PM, Luca Lesinigo wrote: >> Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto: >>> Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. >> I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. >> I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... > I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems. Right, a lot of people have done migration from Courier -> Dovecot using the dummy proxying. Since v2.0 the proxying automatically handles any CAPABILITY string issues. From ednitido at gmail.com Fri Mar 23 22:26:46 2012 From: ednitido at gmail.com (Ed Nitido) Date: Fri, 23 Mar 2012 16:26:46 -0400 Subject: [Dovecot] Dovecot 2.1.3 Proxy creates mailbox on proxy In-Reply-To: References: <950E30E6-38A5-4F5F-B2D6-B12C810AB439@iki.fi> Message-ID: Ooops, didn't email the list... it working now thanks to Timo, solution below On Fri, Mar 23, 2012 at 4:14 PM, Timo Sirainen wrote: > >> On 23.3.2012, at 22.01, Ed Nitido wrote: >> >> > pass_attrs = >> uid=user,userPassword=password,=proxy,=master=doveadmin,=pass=xxxxxx >> >> I guess it doesn't like the "=proxy" part. I guess I should fix it. For >> now just set "=proxy=y". >> > > From ncjeffgus at zimage.com Fri Mar 23 22:42:23 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Fri, 23 Mar 2012 13:42:23 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6CBAA2.5020409@in.tum.de> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> Message-ID: <1332535343.5601.6.camel@sally> On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > Hi, > > maybe try "dsync -o mail_fsync=never". That didn't seem to make much of a difference. On a 3.1GB backup it shaved off 5 seconds. dsync's time was over 6 minutes with or without the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. It seems to me that dsync *should* be able to be just as fast, but it currently is spending way too much time doing something. What is it? ...Jeff From post at michael-neubert.de Fri Mar 23 22:57:28 2012 From: post at michael-neubert.de (Michael Neubert) Date: Fri, 23 Mar 2012 21:57:28 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 Message-ID: <4F6CE3B8.7020507@michael-neubert.de> Hello, I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". The config was not touched but now IMAP connections are not possible anymore (LMTP works fine). When I try to connect to a mailbox, the connect fails. Some log entries: ############################################################################################################### Mar 23 21:45:28 imap-login: Warning: SSL: where=0x10, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: before/accept initialization [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv2/v3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: auth client connected (pid=3431) Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server hello A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write server done A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=-1: SSLv3 read client certificate A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read client key exchange A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 read finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write session ticket A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write change cipher spec A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 write finished A [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2001, ret=1: SSLv3 flush data [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x20, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap-login: Warning: SSL: where=0x2002, ret=1: SSL negotiation finished successfully [xxx.xxx.xxx.xxx] Mar 23 21:45:28 auth: Debug: client in: AUTH 1 PLAIN service=imap secured lip=yyy.yyy.yyy.yyy rip=xxx.xxx.xxx.xxx lport=993 rport=51379 Mar 23 21:45:28 auth: Debug: client out: CONT 1 Mar 23 21:45:28 auth: Debug: client in: CONT 1 AG5lbWlAdmlzaXQtd29ybGQuZGUAUHJvNDUwLnN1 Mar 23 21:45:28 auth-worker(3433): Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth Mar 23 21:45:28 auth-worker(3433): Debug: Module loaded: /usr/lib/dovecot/modules/auth/libdriver_mysql.so Mar 23 21:45:28 auth-worker(3433): Info: mysql(zzz.zzz.zzz.zzz): Connected to database dovecot Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): query: SELECT password, 'directory' AS userdb_home, 'mail' AS userdb_uid, 'mail' AS userdb_gid FROM users WHERE username = 'username' AND domain = 'domain' AND active = 'Y' Mar 23 21:45:28 auth: Debug: client out: OK 1 user=username Mar 23 21:45:28 auth: Debug: master in: REQUEST 2286813185 3394 1 4727968fd3514dd45f623ad9f944e305 Mar 23 21:45:28 auth-worker(3433): Debug: sql(username,xxx.xxx.xxx.xxx): SELECT home, uid, gid FROM users WHERE username = 'username' AND domain = 'domain' Mar 23 21:45:28 auth: Debug: master out: USER 2286813185 username home=directory uid=8 gid=8 Mar 23 21:45:28 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=3434, TLS Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 23 21:45:28 imap(username): Info: Connection closed in=0 out=303 ############################################################################################################### The MySQL authentification seems to work fine, but after this the connection is closed with the SSL alert. In Dovecot 2.1.2 everything worked fine. The SSL certifcate is also correct. Any hints are welcome to identify the problem. Thanks in advance. Beste wishes Michael From tss at iki.fi Fri Mar 23 23:03:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:03:01 +0200 Subject: [Dovecot] distributed mdbox In-Reply-To: References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> Message-ID: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> On 23.3.2012, at 19.43, wrote: >> Have you tried stress testing it with imaptest? Run in parallel for both >> servers: > I did stress test it, but we have developed a "mail bot net" tool for the > purpose. I should mention this was tested using dovecot 1.2, as this is > our current production version (hopefully will be upgrading soon). Its > comprised of a control server that starts a bot network of client machines > that creates pop/imap connections (smtp as well) on our test cluster of > dovecot (and postfix) servers. In my test I distributed the load across a > two node dovecot (/postfix) cluster back ended by glusterfs, which has SAN > storage attached to it. I actually didn't change my configuration from > when I had a test NFS server connected to the test servers (mmap disabled, > fcntl locking, etc), because glusterfs was an afterthought when we were > stress testing our new netapp system using NFS. We have everything in > VMware, including the glusterfs servers. Using five bot servers and > connecting 7 times a second from each server (35 connections per second) > for both pop and imap (70 total connections per second) split between two > dovecot servers I was not seeing any big issues. The load average was low, > and there were no errors to speak of in dovecot (or postfix). I was > mounting the storage with the glusterfs native client, not using NFS (which > I have not tested). I would like to do a more thorough test of glusterfs > using Dovecot 2.0 on some dedicated hardware and see how much further I can > push the system. What did the bots do? Add messages and delete messages as fast as they could? I guess that's mostly enough to see if things work. imaptest anyway hammers the server as fast as it can with all kinds of commands. From tss at iki.fi Fri Mar 23 23:25:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 23 Mar 2012 23:25:28 +0200 Subject: [Dovecot] dsync redesign Message-ID: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: http://dovecot.org/tmp/dsync-redesign.txt and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: http://dovecot.org/tmp/dsync-redesign-problem.txt It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. From list at airstreamcomm.net Sat Mar 24 01:39:11 2012 From: list at airstreamcomm.net (list at airstreamcomm.net) Date: Fri, 23 Mar 2012 18:39:11 -0500 Subject: [Dovecot] distributed mdbox In-Reply-To: <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> References: <1ff190f691892c01a980927422f38753@mail.airstreamcomm.net> <1B3CA373-6DC2-4CAD-A4E8-2B3E9A181473@iki.fi> Message-ID: <7e40b18742c5053948aeaaa51d41ceca@mail.airstreamcomm.net> On Fri, 23 Mar 2012 23:03:01 +0200, Timo Sirainen wrote: > On 23.3.2012, at 19.43, > wrote: > >>> Have you tried stress testing it with imaptest? Run in parallel for both >>> servers: >> I did stress test it, but we have developed a "mail bot net" tool for the >> purpose. I should mention this was tested using dovecot 1.2, as this is >> our current production version (hopefully will be upgrading soon). Its >> comprised of a control server that starts a bot network of client >> machines >> that creates pop/imap connections (smtp as well) on our test cluster of >> dovecot (and postfix) servers. In my test I distributed the load across >> a >> two node dovecot (/postfix) cluster back ended by glusterfs, which has >> SAN >> storage attached to it. I actually didn't change my configuration from >> when I had a test NFS server connected to the test servers (mmap >> disabled, >> fcntl locking, etc), because glusterfs was an afterthought when we were >> stress testing our new netapp system using NFS. We have everything in >> VMware, including the glusterfs servers. Using five bot servers and >> connecting 7 times a second from each server (35 connections per second) >> for both pop and imap (70 total connections per second) split between two >> dovecot servers I was not seeing any big issues. The load average was >> low, >> and there were no errors to speak of in dovecot (or postfix). I was >> mounting the storage with the glusterfs native client, not using NFS >> (which >> I have not tested). I would like to do a more thorough test of glusterfs >> using Dovecot 2.0 on some dedicated hardware and see how much further I >> can >> push the system. > > What did the bots do? Add messages and delete messages as fast as they > could? I guess that's mostly enough to see if things work. imaptest anyway > hammers the server as fast as it can with all kinds of commands. We created two python scripts on the bots that listed all the messages in the inbox then deleted all the messages in the inbox, one script doing pop and the other doing imap. The bots were also sending messages to the server simultaneously to repopulate inboxes. I didn't know about imaptest, thanks! From noel.butler at ausics.net Sat Mar 24 03:19:50 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sat, 24 Mar 2012 11:19:50 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <1332551990.11835.19.camel@tardis> On Fri, 2012-03-23 at 12:53 +0200, Timo Sirainen wrote: > On 23.3.2012, at 12.44, Heiko Schlichting wrote: > > > Timo wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't take much effort at all to change that to look for the long time default of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ No-one is suggesting putting all the individual conf files in /etc, only for existence of dovecot.conf itself. There are plenty of linux and unix systems that have been using /etc for as long as I can recall (even early redhat did), its only certain distros that build as /etc/foo/ the ones that use rpms or debs are obviously not running anything special (we all know no build config process will suite all operations) there are a large number i'm sure who use source (besides, with debian and redhat, who knows WHAT butchering they've done to upstreams code)... Which brings up another question, may I ask why some of the options to disable some passwd types were removed from build process? Systems that dont use system password files (amongst other formats) dont need to build them, that's not a criticism, 'just sayin'. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From tss at iki.fi Sat Mar 24 03:50:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:50:07 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <1332551990.11835.19.camel@tardis> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: On 24.3.2012, at 3.19, Noel Butler wrote: >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > it is very easy to have a search path for config file, it shouldn't > take much effort at all to change that to look for the long time default > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > No-one is suggesting putting all the individual conf files in /etc, only > for existence of dovecot.conf itself. So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Which brings up another question, may I ask why some of the options to > disable some passwd types were removed from build process? Systems that > dont use system password files (amongst other formats) dont need to > build them, that's not a criticism, 'just sayin'. There's also no harm in having that code included. They add no extra library dependencies. The only thing they do is to use a few kilobytes of more disk space, and possibly a few kilobytes of more memory (even that isn't certain). All options just increase the number of combinations that can cause things to go wrong. If I add some code to be compiled optionally, it just adds more combinations that should be tested together to see if the code still even compiles. Previously I've broken SSL code many times by not testing if after changes Dovecot builds without OpenSSL. So the less options there are, the more robust Dovecot is, and the less work I have to do to keep it working when adding new features. So I add an option only when there is a good use case for it and I expect more than one person to use it. From tss at iki.fi Sat Mar 24 03:52:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 03:52:56 +0200 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: On 23.3.2012, at 19.22, Andra? 'ruskie' Levstik wrote: > :2012-03-23T12:53:Timo Sirainen: > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > I would suggest to have a --layout=gnu|opt > > That would either do what it currently does(gnu) and opt to install > everything into a single dir i.e.: > /opt/dovecot/ > > With subdirs under there. Yes, --with-layout=gnu|opt could be useful. Anyone want to volunteer to implement it? :) From dovecot at tlinx.org Sat Mar 24 08:12:44 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:12:44 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6D65DC.7030304@tlinx.org> Jeff Gustafson wrote: > On Fri, 2012-03-23 at 19:02 +0100, Christoph Bu?enius wrote: > >> Hi, >> >> maybe try "dsync -o mail_fsync=never". >> > > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? > ...Jeff > --- Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can use large buffers (~16MB gives best results on my local Gbit network), but it misses all those pesky acls and extended attrs, not to mention file perms...*sigh* Compare that to the I/O done 4k at a time by many older utils... If I'm writing to the LOCAL HD, instead of the network, then a 1GB-4GB buffer size gives best results (1GB/s raid5). Small buffers are such a PITA! From dovecot at tlinx.org Sat Mar 24 08:16:08 2012 From: dovecot at tlinx.org (Linda Walsh) Date: Fri, 23 Mar 2012 23:16:08 -0700 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? Message-ID: <4F6D66A8.3050208@tlinx.org> Is this redhat's version of the kernel only? Or does it apply to other linux kernels and other distros? Any idea what linux kernel versions might cause this? (from main dovecot webpage news) Thu Mar 22 14:38:53 EET 2012 Red Hat/CentOS users: A recent kernel update causes Dovecot to start failing after it has reached 1000 child processes. To fix this, downgrade your kernel until Red Hat releases a fixed kernel. From bra at fsn.hu Sat Mar 24 09:19:48 2012 From: bra at fsn.hu (Attila Nagy) Date: Sat, 24 Mar 2012 08:19:48 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <4F6D7594.10800@fsn.hu> On 03/23/12 22:25, Timo Sirainen wrote: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) It would bring Dovecot into the class of open source mail servers where there are currently no contenders. BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. For example, messages are immutable, so you won't face this issue in this area. And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). And of course there is always the problem of designing a good, consistent method which is also efficient. From jtam.home at gmail.com Sat Mar 24 11:36:33 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sat, 24 Mar 2012 02:36:33 -0700 (PDT) Subject: [Dovecot] Problems with upgrade 2.0.16 -> 2.1.3 In-Reply-To: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> References: <36F866F4-C128-4D05-8B05-C485BE9F9795@iki.fi> Message-ID: On Fri, 23 Mar 2012, Timo Sirainen wrote: > On 23.3.2012, at 12.58, Joseph Tam wrote: > >> I ran into two issues trying to upgrade our dovecot installation (Solaris 10). >> >> 1) Does not compile with OpenSSL 0.9.7 >> >> Not a big deal, as I was able to successfully against OpenSSL 0.9.8, >> but does dovecot require OpenSSL >= 0.9.8 now? > > Hm. Maybe it's time by now? :) It could be fixed with some more > #ifdefs but those make code more unreadable. It might still compile with OpenSSL 0.9.7 if it is built with engine support (the default), but yeah, it's time to move to 0.9.8 or 1.0.0. >> 2) Dovecot's LDA does not work >> >> After stopping the the old dovecot, and starting dovecot 2.1.3 using the >> exact same config file, local mail delivery tempfails: >> >> Mar 23 02:51:51 server dovecot: auth: Error: getpeerucred() failed: Bad address > > http://hg.dovecot.org/dovecot-2.1/rev/98fd46f8d1ab fixes this? Spot on, as usual. Thanks. Joseph Tam From janfrode at tanso.net Sat Mar 24 12:04:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 24 Mar 2012 11:04:07 +0100 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <20120324100407.GB31829@dibs.tanso.net> On Sat, Mar 24, 2012 at 08:19:48AM +0100, Attila Nagy wrote: > On 03/23/12 22:25, Timo Sirainen wrote: > > > Well, dsync is a very useful tool, but with continuous replication > it tries to solve a problem which should be handled -at least > partially- elsewhere. Storing stuff in plain file systems and > duplicating them to another one just doesn't scale. I don't see why this shouldn't scale. Mailboxes are after all changed relatively infrequently. One idea for making it more scalable might be to treat indexes/metadata and messages differently. Make index/metadata updates synchronous over the clusters/locations (with re-sync capability in case of lost synchronisation), while messages are store in one "altstorage" per cluster/location. For a two-location solution, message-data should be stored in: mail_location = mdbox:~/mdbox ALTcache=mdbox:~/mdbox-remoteip-cache ALT=dfetch://remoteip/ <-- new protocol If a message is in the index, look for it in that order: local mdbox ALTcache ALT if it finds the message in ALT, make a copy into ALTcache (or local mdbox?). Syncronizing messages could be a very low frequency job, and could be handled by simple rsync of ALT to ALTcache. No need for specialized tool for this job. Syncronizing ALTcache to local mdbox could be done with a reversed doveadm-altmove, but might not be necessary. Of course this is probably all very naive.. but you get the idea :-) -jf From stan at hardwarefreak.com Sat Mar 24 12:06:25 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Sat, 24 Mar 2012 05:06:25 -0500 Subject: [Dovecot] kernel problem in RedHat? -- RH specific, or what linux kernels does this affect? In-Reply-To: <4F6D66A8.3050208@tlinx.org> References: <4F6D66A8.3050208@tlinx.org> Message-ID: <4F6D9CA1.9050008@hardwarefreak.com> On 3/24/2012 1:16 AM, Linda Walsh wrote: > Is this redhat's version of the kernel only? Or does it apply to other > linux kernels and other distros? > > Any idea what linux kernel versions might cause this? > > (from main dovecot webpage news) > > Thu Mar 22 14:38:53 EET 2012 > > Red Hat/CentOS users: A recent kernel update > causes Dovecot to > start failing after it has reached 1000 child processes. To fix this, > downgrade your kernel until Red Hat releases a fixed kernel. It appears to be a Red Hat centric regression. They added a patch to fix one thing and broke other things, Dovecot, in the process, because the Red Hat programmer made an incorrect assumption about what real world applications were doing, apparently without investigating such first. Note that one won't see this problem on their REHL/CentOS system if they never hit 1000 child processes. And as Timo states in the bug report it's *possible* Postfix could suffer the same problem as it uses the same pipe/epoll system. However nobody runs 1000 Postfix smtp[d]s. Few, if any, run over 200. The ones that do usually don't know how to properly tune Postfix, and they use a high smtp[d] daemon count to compensate for suboptimal configuration elsewhere in the system. A properly setup Postfix server can handle 200-300 msgs/second with the default 100 smtp[d] processes. 1000 smtp[d]s would suggest a message rate 10x that, or 2000-3000 msgs/second. The server plus disk subsystem required to queue that kind of message rate would be impressive, and expensive, for a mail server. This same message rate can typically be achieved by a much less expensive scale out farm. If anyone on the planet is running a properly tuned 1000 process Postfix server, I'd love to read about it. -- Stan From tss at iki.fi Sat Mar 24 13:49:36 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 13:49:36 +0200 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <0B23962D-A067-4A71-9A10-067FCA76B06D@iki.fi> On 24.3.2012, at 9.19, Attila Nagy wrote: > Well, dsync is a very useful tool, but with continuous replication it tries to solve a problem which should be handled -at least partially- elsewhere. Storing stuff in plain file systems and duplicating them to another one just doesn't scale. dsync solves several other problems besides replication. Even if Dovecot had a super efficient replicated storage, dsync would still exist for doing things like: - migrating between mailbox formats - migrating from other imap/pop3 servers - creating (incremental) backups - the redesign works great for super-high latency replication (USB sticks, cross-planet replication :) - and when you really just don't want any kind of a complex replicated database, just something simple So I'll need to get this working well in any case. And with the redesign the replication should be efficient enough to scale pretty well. > I personally think that Dovecot could gain much more if the amount of work going into fixing or improving dsync would go into making Dovecot to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences compared to the "low latency" and consistency problem free local file systems, but its fruits are also sweeter for the long term. :) Yes, I'm also planning on implementing that, but not yet. > It would bring Dovecot into the class of open source mail servers where there are currently no contenders. > > BTW, for the previous question in this topic (are there any nosql dbs supporting application-level conflict resolution?), there are similar solutions (like CouchDB, but having some experiences with it, I wouldn't recommend it for massive mail storage -at least the plain CouchDB product), but I guess you would be better off with designing a schema which doesn't need it at the first time. > For example, messages are immutable, so you won't face this issue in this area. > And for metadata, maybe the solution is not to store "digested" snapshots of the current metadata (folders, flags, message links for folders etc), but to store the changes happening on the user's mailbox and occasionally aggregate them into a last known good and consistent state. My plan was to create similar index files as currently exists in filesystem. It would work pretty much the same as you described: There's a "log" where changes are appended, and once in a while the changes are written into an "index" snapshot. When reading you first read the snapshot and then apply new changes from the log. The conflict resolution if DB supports it would work by reading the two logs in parallel and figure out a way to merge them consistently, similar to how dsync does pretty much the same thing. Hmm. Perhaps the metadata log could exist exactly as the dsync data format and have dsync code do the merging?.. > Also, there are other interesting ideas, maybe with real single instance store (splitting mime parts? Storing attachments in plain binary form? This always brings up the question of whether the mail server should modify the mails, can be pretty bad for encrypted/signed stuff). This is already optionally done in v2.0+dbox. MIME attachments can be stored in plain binary form if they can be reconstructed back into their original form. It doesn't break any signed stuff. From CMarcus at Media-Brokers.com Sat Mar 24 14:01:07 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:01:07 -0400 Subject: [Dovecot] SIS and restoring from backups Message-ID: <4F6DB783.3050808@Media-Brokers.com> On 2012-03-24 7:49 AM, Timo Sirainen wrote: > This is already optionally done in v2.0+dbox. MIME attachments can be > stored in plain binary form if they can be reconstructed back into > their original form. It doesn't break any signed stuff. Hey Timo, Splitting this off into a separate thread... On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? Or maybe I misunderstood the problem? -- Best regards, Charles From tss at iki.fi Sat Mar 24 14:08:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 14:08:13 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DB783.3050808@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> Message-ID: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> On 24.3.2012, at 14.01, Charles Marcus wrote: > On the question of the existing SIS capability for attachments... have you given any thought as to how to solve the problem of restoring from backups when SIS is used? I was planning on using it initially, until I read on list that restoring from (normal disk-to-disk) backups would not work when SIS was enabled - this is obviously a deal breaker for anyone who relies on backups - which I would think would be almost everyone? > > Or maybe I misunderstood the problem? You can do full backups from a filesystem snapshot, which works "well enough" (might leave some unused attachments lying around in some rare cases, but that can also happen if Dovecot crashes/dies). The other possibility is to already use dsync (doveadm backup) to do full backups. With the redesigned dsync you would be able to do incremental backups also. In any case the solution involves de-SISing mails for backup. From CMarcus at Media-Brokers.com Sat Mar 24 14:54:14 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sat, 24 Mar 2012 08:54:14 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> Message-ID: <4F6DC3F6.70306@Media-Brokers.com> On 2012-03-24 8:08 AM, Timo Sirainen wrote: > You can do full backups from a filesystem snapshot, which works > "well enough" (might leave some unused attachments lying around in > some rare cases, but that can also happen if Dovecot crashes/dies). But the problem isn't with backups, but with restores, right? > The other possibility is to already use dsync (doveadm backup) to do > full backups. With the redesigned dsync you would be able to do > incremental backups also. In any case the solution involves > de-SISing mails for backup. So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Doesn't sound ideal... I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. Am I correct that enabling SIS as it is currently implemented would break this backup tool? I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... -- Best regards, Charles From post at michael-neubert.de Sat Mar 24 15:04:55 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 14:04:55 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6DC677.1000100@michael-neubert.de> The problem starts just after authorization: Console: ################################################################### openssl s_client -connect mailserver.com:993 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. a login "username" "password" closed ################################################################### Here are the logs of this moment: ################################################################### Mar 24 13:48:46 imap-login: Info: Login: user=, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy, mpid=10662, TLS Mar 24 13:48:46 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Mar 24 13:48:46 imap(username): Info: Connection closed in=0 out=303 ################################################################### So just after sucussful login with correct username / password the connection is closed. From tss at iki.fi Sat Mar 24 15:16:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:16:38 +0200 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6DC3F6.70306@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: On 24.3.2012, at 14.54, Charles Marcus wrote: > On 2012-03-24 8:08 AM, Timo Sirainen wrote: >> You can do full backups from a filesystem snapshot, which works >> "well enough" (might leave some unused attachments lying around in >> some rare cases, but that can also happen if Dovecot crashes/dies). > > But the problem isn't with backups, but with restores, right? Ah, right. Then it gets tricky. >> The other possibility is to already use dsync (doveadm backup) to do >> full backups. With the redesigned dsync you would be able to do >> incremental backups also. In any case the solution involves >> de-SISing mails for backup. > > So, this would make the backup storage requirements larger - maybe dramatically larger for sites that have a lot of large attachments? Some backup systems can do internal deduplication. > I currently use rsnapshot to keep many multiple (daily, weekly, and monthly) hardlinked snapshots, each of which consumes only a tiny fraction of extra storage over and above the first/main snapshot. > > Am I correct that enabling SIS as it is currently implemented would break this backup tool? I'm not sure. Are you running rsnapshot on live filesystem or on a snapshot? On live filesystem there would be race conditions. > I was also thinking of asking about how to provide read-only access to these backup snapshots to the users in some kind of special namespace, so that they could all essentially go 'back in time' to grab any emails that they may have inadvertently deleted... This should be possible, just point the namespace to such snapshot. You may need to point CONTROL dir to some temporary directory and index dir as well to either temp or to memory. From tss at iki.fi Sat Mar 24 15:17:28 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 15:17:28 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6DC677.1000100@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> Message-ID: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> On 24.3.2012, at 15.04, Michael Neubert wrote: > openssl s_client -connect mailserver.com:993 > * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. > a login "username" "password" > closed And what happens without SSL? e.g. telnet localhost 143 From mcbdovecot at robuust.nl Sat Mar 24 15:21:51 2012 From: mcbdovecot at robuust.nl (Maarten Bezemer) Date: Sat, 24 Mar 2012 14:21:51 +0100 (CET) Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332535343.5601.6.camel@sally> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: On Fri, 23 Mar 2012, Jeff Gustafson wrote: > That didn't seem to make much of a difference. On a 3.1GB backup it > shaved off 5 seconds. dsync's time was over 6 minutes with or without > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > It seems to me that dsync *should* be able to be just as fast, but it > currently is spending way too much time doing something. What is it? Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per second. Depending on the harddisks used, that would be quite a challenge. If you use rsync to only transfer the files that changed (based on file modification time) you may or may not miss files that have changed but still have the same time stamp. I assume you didn't use the --checksum parameter to rsync, right? dsync does so much more than simply copy some files... -- Maarten From post at michael-neubert.de Sat Mar 24 18:00:13 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:00:13 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <4F6DC677.1000100@michael-neubert.de> <56A5CC15-B5C7-4F72-9A99-C186A696C4DC@iki.fi> Message-ID: <4F6DEF8D.7090309@michael-neubert.de> > And what happens without SSL? e.g. telnet localhost 143 Without SSL it is no problem: ############################################## telnet imap-server 143 Trying xxx.xxx.xxx.xxx... Connected to imap-server. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. a login "username" "password" a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE QUOTA] Logged in b select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft nonjunk $Forwarded \*)] Flags permitted [...] ############################################## From tss at iki.fi Sat Mar 24 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 18:02:59 +0200 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". So what exactly is this version? dovecot --version? From post at michael-neubert.de Sat Mar 24 18:09:43 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sat, 24 Mar 2012 17:09:43 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> References: <4F6CE3B8.7020507@michael-neubert.de> <96C50F22-1D98-4663-AE7F-3140F3881CAA@iki.fi> Message-ID: <4F6DF1C7.1020306@michael-neubert.de> >> I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using >> Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > So what exactly is this version? dovecot --version? At the moment the version is "2.1.3-0~auto+6" from rename-it.nl. dovecot -n: # 2.1.3 (4ae85f573c93): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.4 ocfs2 From jackie.craig.sparks at live.com Sat Mar 24 22:34:49 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 16:34:49 -0400 Subject: [Dovecot] dovecot and cloudfile systems Message-ID: I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. I cant lock files, This is accessed using the fuse library. I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. From tss at iki.fi Sat Mar 24 22:43:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 22:43:34 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: Message-ID: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> On 24.3.2012, at 22.34, jackie sparks wrote: > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > I cant lock files, > > This is accessed using the fuse library. > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From jackie.craig.sparks at live.com Sat Mar 24 23:15:58 2012 From: jackie.craig.sparks at live.com (jackie sparks) Date: Sat, 24 Mar 2012 17:15:58 -0400 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= This e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is confidential, and is intended solely for the use of the individuals or entities to whom it is addressed. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail and any file attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify me by email at jackie.craig.sparks at live.com. You must destroy the original transmission and its contents. > From: tss at iki.fi > Date: Sat, 24 Mar 2012 22:43:34 +0200 > To: jackie.craig.sparks at live.com > CC: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot and cloudfile systems > > > On 24.3.2012, at 22.34, jackie sparks wrote: > > > > > I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. > > I cant lock files, > > > > This is accessed using the fuse library. > > I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. > > I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. > > http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. From tss at iki.fi Sat Mar 24 23:27:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Sat, 24 Mar 2012 23:27:47 +0200 Subject: [Dovecot] dovecot and cloudfile systems In-Reply-To: References: , <1763F751-855D-42A8-A085-A218D70D6F05@iki.fi> Message-ID: <00827B46-E6B3-4DD8-A035-421A522F489F@iki.fi> What you're trying to do is quite unlikely to work with any IMAP server / cloud filesystem combination. And if it does work, the performance will most likely be horrible. Of course, if it does work with any kind of a combination I'm interested in knowing about it. On 24.3.2012, at 23.15, jackie sparks wrote: > > This would be great if I wasn't trying to store mailboxes on the cloudfiles and had the mailboxes stored among-st the cluster but I wan't the maildirs on cloudfiles so they can be mounted between all the servers. then load balance imap, smtp and pop . I think I will just try on the amazon cloud, see if the "buckets" have the same problems, everything else is near done its just this mail problem I am having. Rackspaces solution is to pay them 1250 dollars for a minimum of 5 hours of development and this type of job hasn't even been quoted from them. > It just makes me think that Rackspace is a open source supporter but at the same time it just boils down to money. Buy up businesses that support that development so they can keep the good in house and release just enough so they can profit from the development. Then again they are backed by at&t and then with at&t you are dealing with the government. The government loves things that are in development as long as they don't develop. lol > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=-= > This e-mail (including attachments) is covered by the Electronic > Communications Privacy Act, 18 U.S.C. Sections 2510-2521, is > confidential, and is intended solely for the use of the individuals or > entities to whom it is addressed. If you are not the intended > recipient or the person responsible for delivering the e-mail to the > intended recipient, be advised that you have received this e-mail in > error and that any use, dissemination, forwarding, printing, or > copying of this e-mail and any file attachments is strictly > prohibited. If you have received this e-mail in error, please > immediately notify me by email at jackie.craig.sparks at live.com. You must destroy > the original transmission and its contents. > > >> From: tss at iki.fi >> Date: Sat, 24 Mar 2012 22:43:34 +0200 >> To: jackie.craig.sparks at live.com >> CC: dovecot at dovecot.org >> Subject: Re: [Dovecot] dovecot and cloudfile systems >> >> >> On 24.3.2012, at 22.34, jackie sparks wrote: >> >>> >>> I'm trying to store mailboxes in a cloudfile system and I am running into alot of problems using courier, between time skew, file locking and cache creation problems. I was wondering if dovecot has any problems using maildirs across a limited fuse file system. >>> I cant lock files, >>> >>> This is accessed using the fuse library. >>> I thought taking out the standard checking for return values would be a work around because everything imap daemon is doing can be done from a shell by hand except filelocking and setting times on files. >>> I am doing this on Rack Spaces cloud files. This is something that not many if any are doing. >> >> http://wiki2.dovecot.org/Director will probably make it work okay. Otherwise if even Courier doesn't work, Dovecot won't work either. > From post at michael-neubert.de Sun Mar 25 01:00:27 2012 From: post at michael-neubert.de (Michael Neubert) Date: Sun, 25 Mar 2012 00:00:27 +0100 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: <4F6E520B.4060303@michael-neubert.de> I just did some more tests with different binaries. The problem occurs since: Dovecot 2.1.3-0~auto+5 dovecot --version 2.1.3 (f30437ed63dc) Dovecot 2.1.3-0~auto+4 works fine dovecot --version 2.1.3 (ff5c341f8838) So my title is wrong. The problem only affects people "who like to live on the edge" of 2.1.3 release ;) The stable Dovecot 2.1.3 release http://dovecot.org/list/dovecot-news/2012-March/000219.html is not affected. From noel.butler at ausics.net Sun Mar 25 05:48:36 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:48:36 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332643716.4515.23.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > > > No-one is suggesting putting all the individual conf files in /etc, only > > for existence of dovecot.conf itself. > > So you don't want to remove dovecot/ suffix from all the other dirs (lib, libexec, etc.) only from etc? The only way I can think of how to do that is to add a special option just for it, and more options is generally bad: > Not at all, I'm suggesting that in search for dovecot.conf file only, the search path be preferenced by @sysconfdir@/dovecot.conf such as --sysconfdir=/etc it looks for /etc/dovecot.conf, if not found, the config file location search continues on to look for /etc/dovecot/dovecot.conf I might be wrong, there might only be a handful of people annoyed by this change, but as more and more using custom builds test out moving from 1.2 to 2.x, well, more and more might be caught out, wouldn't it be better to, as you said previously, " avoid emails". As I'm sure you got better things to do than read any tripe I post :) Anyway I have made my suggestion, nothing more I think I can say on the matter that I haven't said already, so I'll leave it there, if it's implemented, great, if not, well, it's not... Cheers -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From noel.butler at ausics.net Sun Mar 25 05:56:06 2012 From: noel.butler at ausics.net (Noel Butler) Date: Sun, 25 Mar 2012 12:56:06 +1000 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> <1332551990.11835.19.camel@tardis> Message-ID: <1332644166.4515.30.camel@tardis> On Sat, 2012-03-24 at 03:50 +0200, Timo Sirainen wrote: > On 24.3.2012, at 3.19, Noel Butler wrote: > > >> Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > > it is very easy to have a search path for config file, it shouldn't > > take much effort at all to change that to look for the long time default > > of /etc/dovecot.conf first, then if not there, look in /etc/dovecot/ > > Technically it's easy, but the result will be that more people will be confused. I'll get an increase of emails about "I changed dovecot.conf, but nothing happens?!?" My goal is to reduce the number of emails I get, not increase them. > grrr meant to comment on this too, umm since a default custom build doesnt install any config files, this would only become a confusion if one were using say an RPM package, and then decided to custom install, but IIRC, RPM renames the old config anyway, least it used to in some packages, dont know about .deb stuff though, it lacks a lot of intelligence so probably not (/me starts flamewars) Now I've said my bit.. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From stsiol at yahoo.co.uk Sun Mar 25 10:24:56 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 08:24:56 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot Message-ID: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Hello all, I want to ask about something I never did before. I have a dovecot/XMail/LAMP/Horde installation on a CentOS 5.5 32-bit system with two domains : domainA and domainB All the users used to have their mailboxes on domainA. However the personell dept decided that 90% of the users will have to have their mailboxes set to domainB and the other 10% will stay at domainA. So, I need to migrate those mailboxes from domainA to domainB. The only thing is I haven't done that ever. The directory structure is : "/var/MailRoot/domains/domainX/Username/Maildir/" and under there the usual suspects : cur (directory) dovecot.index.cache (file) dovecot-keywords (file) dovecot-uidvalidity (file) ????????? new (directory) tmp (directory) dovecot.index (file) dovecot.index.log (file) dovecot-uidlist (file) dovecot-uidvalidity.4cc055c6 (file) subscriptions (file) Using dovecot v1.2.15 Any help would be appreciated. Iam supposed to do this today !! :-) Thank you all, s. ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis? From jtam.home at gmail.com Sun Mar 25 10:46:25 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Sun, 25 Mar 2012 00:46:25 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value Message-ID: Subject: Different user messages clustered around the same date.saved value After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." to iterate through all user trash folders and expunge old messages. However, I noticed a strange thing: querying what would have been deleted doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d showed many date.saved values are clustered around the same timestamp, even among different user's Trash mailbox. One user's trash mailbox having the same date.saved is explained by a user deleting a lot of message at one time, but I can't explain why many different users would have messages with the same (or closeby) date.saved value. For example, the output of the above query on my system showed the 10s window /2012-03-05 18:08:0[0-9]/ matched 7658 messages among 22 different user Trash mailboxes, which is statistically unlikely. I did't see anything special in the dovecot logs at this time to explain this. What would cause this? Joseph Tam From gedalya at gedalya.net Sun Mar 25 11:23:50 2012 From: gedalya at gedalya.net (Gedalya) Date: Sun, 25 Mar 2012 04:23:50 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> Message-ID: <4F6ED616.5090504@gedalya.net> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: > The directory structure is : > > > "/var/MailRoot/domains/domainX/Username/Maildir/" You can probably just: 1. Do something to prevent the user from logging in, and any deliveries from happening, e.g. delete the user. 2. Kick any existing connections. 3. Just move the Username directory from domainA to domainB. 4. Create the new user in the new domain Test the procedure first. Don't let a client log in to a mailbox and see something he's not supposed to see, like an empty mailbox. It can cause the client to drop its local cache and possibly other data. From lists at wildgooses.com Sun Mar 25 14:16:55 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:16:55 +0100 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <4F6EFEA7.3030406@wildgooses.com> On 24/03/2012 13:21, Maarten Bezemer wrote: > > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > >> That didn't seem to make much of a difference. On a 3.1GB backup it >> shaved off 5 seconds. dsync's time was over 6 minutes with or without >> the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. >> It seems to me that dsync *should* be able to be just as fast, >> but it >> currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB > per second. Depending on the harddisks used, that would be quite a > challenge. rsync is only going to transfer files it believes has changed, so the transfer bandwidth will likely be lower > If you use rsync to only transfer the files that changed (based on > file modification time) you may or may not miss files that have > changed but still have the same time stamp. I assume you didn't use > the --checksum parameter to rsync, right? Dovecot is not very resiliant to files changing under it, but without the filename changing. I have no idea if it's supposed to work at all, but you might at least expect to see problems if you start doing this? > dsync does so much more than simply copy some files... Quite probably, but I don't think your expose above illustrates this? Regards Ed W From lists at wildgooses.com Sun Mar 25 14:41:35 2012 From: lists at wildgooses.com (Ed W) Date: Sun, 25 Mar 2012 12:41:35 +0100 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6C8281.10906@hardwarefreak.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> Message-ID: <4F6F046F.1070003@wildgooses.com> On 23/03/2012 14:02, Stan Hoeppner wrote: > On 3/23/2012 6:41 AM, Radim Kolar wrote: >> Can somebody provide maildrop syntax for using deliver-lda as final >> delivery program during sorting mail in user mailfilter? >> >> i mean replacement for "to" statement >> >> if ( /^(To|Cc):.*dovecot at dovecot.org/:h ) >> { >> to $MAIL/.dovecot/ >> } > Dovecot's local delivery agent uses the Sieve language: > http://wiki.dovecot.org/LDA/Sieve > > The syntax is quite different than maildrop or procmail. > I think that's why he asked the question? I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? In answer to the OP: read the maildropex man pages, but you have several options, eg: to "| someprogram" or: xfilter someprogram `someprogram` However, almost certainly I think you want the top option? Good luck Ed W From fumiyas at osstech.jp Sun Mar 25 15:17:36 2012 From: fumiyas at osstech.jp (SATOH Fumiyasu) Date: Sun, 25 Mar 2012 21:17:36 +0900 Subject: [Dovecot] sysconfdir depreacted In-Reply-To: <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> References: <1332336379.10474.5.camel@tardis> <1332381356.4112.9.camel@tardis> <1332405027.6792.7.camel@tardis> <4F93BC71-46CE-40A6-99EB-268562BE4CC1@iki.fi> <20120323104432.GB1353054@CIS.FU-Berlin.DE> <8614D3AE-5430-4FB6-A906-DC1C1E901792@iki.fi> Message-ID: <87zkb4j2n3.wl%fumiyas@osstech.jp> At Fri, 23 Mar 2012 12:53:16 +0200, Timo Sirainen wrote: > >> So the only way I can think of how to change this is to add another > >> option to optionally remove the dovecot/ suffix from the directory, but > >> is this really worth the trouble? > > > > I would appreciate such option too. For large dedicated installations other > > schemes than /etc/dovecot are common. > > > > See http://dovecot.org/list/dovecot/2009-January/036131.html > > Yes, I was also thinking about that, but it's about removing the dovecot/ suffix from other directories as well. That might be something worth doing (--without-package-suffix or something?). > OpenLDAP's (and smbldap-tools's) configure script has --with-subdir option. This option defaults to "/openldap". (/smbldap-tools) $ grep subdir configure.in build/top.mk configure.in:dnl --with-subdir configure.in:ldap_subdir="/openldap" configure.in:AC_ARG_WITH(subdir, configure.in:[ --with-subdir=DIR change default subdirectory used for installs], configure.in: no) ldap_subdir="" configure.in: ldap_subdir="$withval" configure.in: ldap_subdir="/$withval" configure.in:AC_SUBST(ldap_subdir)dnl build/top.mk:ldap_subdir = @ldap_subdir@ build/top.mk:datadir = @datadir@$(ldap_subdir) build/top.mk:moduledir = @libexecdir@$(ldap_subdir) build/top.mk:sysconfdir = @sysconfdir@$(ldap_subdir) -- -- Name: SATOH Fumiyasu (fumiyas @ osstech co jp) -- Business Home: http://www.OSSTech.co.jp/ -- Personal Home: https://github.com/fumiyas/ From dovecot at vosslamber.nl Sun Mar 25 15:53:16 2012 From: dovecot at vosslamber.nl (Luuk@dovecot) Date: Sun, 25 Mar 2012 14:53:16 +0200 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6ED616.5090504@gedalya.net> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> Message-ID: <4F6F153C.2020005@vosslamber.nl> On 25-03-2012 10:23, Gedalya wrote: > On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >> The directory structure is : >> >> >> "/var/MailRoot/domains/domainX/Username/Maildir/" > > You can probably just: > 1. Do something to prevent the user from logging in, and any deliveries > from happening, e.g. delete the user. > 2. Kick any existing connections. > 3. Just move the Username directory from domainA to domainB. > 4. Create the new user in the new domain > > Test the procedure first. > > Don't let a client log in to a mailbox and see something he's not > supposed to see, like an empty mailbox. It can cause the client to drop > its local cache and possibly other data. > > i would also setup a forward from domainA to domainB for all the users that have moved, at least until most people who do send email know the 'old'address has changed. From stsiol at yahoo.co.uk Sun Mar 25 16:04:33 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 14:04:33 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <1332680673.52988.YahooMailNeo@web132203.mail.ird.yahoo.com> >On 25-03-2012 10:23, Gedalya wrote: >> On 3/25/2012 3:24 AM, Spyros Tsiolis wrote: >>> The directory structure is : >>> >>> >>> "/var/MailRoot/domains/domainX/Username/Maildir/" >>? >> You can probably just: >> 1. Do something to prevent the user from logging in, and any deliveries >> from happening, e.g. delete the user. >> 2. Kick any existing connections. >> 3. Just move the Username directory from domainA to domainB. >> 4. Create the new user in the new domain >>? >> Test the procedure first. >>? >> Don't let a client log in to a mailbox and see something he's not >> supposed to see, like an empty mailbox. It can cause the client to drop >> its local cache and possibly other data. >>? >>? > >i would also setup a forward from domainA to domainB for all the users >that have moved, at least until most people who do send email know the >'old'address has changed. Hi chaps and thank you for your replies, Sorry forgot to mention. Most of the users use thunderbird. And yes, thank you for the forwarding issue of old e-mail accounts to the new ones. However, that is not my problem. Let me suggest something . . . : What if : 1. I setup the new accounts 2. Users log on normally from thunderbird 3. While they are using thunderbird, I get rid of their old e-mail address, create the new one (in thunderbird) 4. I make sure that the newly created address syncs with dovecot Wouldn't that be enough to migrate "on-the-fly" so to speak their existing directory tree structure and related messages to their new e-mail account ? Any ideas ? Cheers, spyros ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis From CMarcus at Media-Brokers.com Sun Mar 25 18:01:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:01:39 -0400 Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F153C.2020005@vosslamber.nl> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> Message-ID: <4F6F3353.6000700@Media-Brokers.com> On 2012-03-25 8:53 AM, Luuk at dovecot wrote: > i would also setup a forward from domainA to domainB for all the users > that have moved, at least until most people who do send email know the > 'old'address has changed. I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. -- Best regards, Charles From CMarcus at Media-Brokers.com Sun Mar 25 18:12:58 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Sun, 25 Mar 2012 11:12:58 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> Message-ID: <4F6F35FA.6050207@Media-Brokers.com> On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: > >> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>> You can do full backups from a filesystem snapshot, which works >>> "well enough" (might leave some unused attachments lying around in >>> some rare cases, but that can also happen if Dovecot crashes/dies). >> >> But the problem isn't with backups, but with restores, right? > > Ah, right. Then it gets tricky. Yeah, I seem to remember it was a comment like that that scared me about enabling it... Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) >> So, this would make the backup storage requirements larger - maybe >> dramatically larger for sites that have a lot of large >> attachments? > Some backup systems can do internal deduplication. Hmmm... and actually, rsnapshot (which uses rsync) does just that, which is *why* each additional snapshot only requires a small fraction of additional disk space (compared to the first main/full snapshot). >> Am I correct that enabling SIS as it is currently implemented would >> break this backup tool? > I'm not sure. Are you running rsnapshot on live filesystem or on a > snapshot? On live filesystem there would be race conditions. I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] but the rsnapshot guys assured me this will and does not cause any real problems, other than those files don't get backed up. I am however looking forward to migrating this to a VM so I can do snapshot for backups to get consistent point-in-time backups. >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. This is great news! I'm looking forward to getting this all working. -- Best regards, Charles From stsiol at yahoo.co.uk Sun Mar 25 18:40:08 2012 From: stsiol at yahoo.co.uk (Spyros Tsiolis) Date: Sun, 25 Mar 2012 16:40:08 +0100 (BST) Subject: [Dovecot] migrating mailboxes on dovecot In-Reply-To: <4F6F3353.6000700@Media-Brokers.com> References: <1332660296.94261.YahooMailNeo@web132204.mail.ird.yahoo.com> <4F6ED616.5090504@gedalya.net> <4F6F153C.2020005@vosslamber.nl> <4F6F3353.6000700@Media-Brokers.com> Message-ID: <1332690008.99877.YahooMailNeo@web132204.mail.ird.yahoo.com> Thanks Charles, s. ? ---- "I merely function as a channel that filters music through the chaos of noise" - Vangelis >________________________________ > From: Charles Marcus >To: dovecot at dovecot.org >Sent: Sunday, 25 March 2012, 17:01 >Subject: Re: [Dovecot] migrating mailboxes on dovecot > >On 2012-03-25 8:53 AM, Luuk at dovecot wrote: >> i would also setup a forward from domainA to domainB for all the users >> that have moved, at least until most people who do send email know the >> 'old'address has changed. > >I would only do that for a few days at most, otherwise it just turns into a crutch that will 'enable' lazy people to wait 'forever' until they change their address book. > >What I do is set up the alais for a few days, then convert it to a custom reject, informing the sender of the new email address. > >-- >Best regards, > >Charles > > > From hsn at filez.com Mon Mar 26 12:25:47 2012 From: hsn at filez.com (Radim Kolar) Date: Mon, 26 Mar 2012 11:25:47 +0200 Subject: [Dovecot] delivering with maildrop In-Reply-To: <4F6F046F.1070003@wildgooses.com> References: <4F6C6164.2050506@filez.com> <4F6C8281.10906@hardwarefreak.com> <4F6F046F.1070003@wildgooses.com> Message-ID: <4F70361B.1070304@filez.com> I presume he wants to filter first with maildir, then actually deliver using the dovecot delivery agent? yes > In answer to the OP: read the maildropex man pages, but you have > several options, eg: > Yes found that dovecot-lda -m will do it nicely. echo "mail message Test " | /usr/local/libexec/dovecot/dovecot-lda -m dovecot just user agents are not able to display message with 0 headers. That confused me. From jeetuindian at gmail.com Mon Mar 26 12:51:00 2012 From: jeetuindian at gmail.com (Jitendra Bhaskar) Date: Mon, 26 Mar 2012 15:21:00 +0530 Subject: [Dovecot] dovecot.log warning Message-ID: Hi Guys, Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data i.e home directory and mail data from previous server which was on dovecot 1.2.8 to new one. Every thing is working fine. Mails are going and coming. But in dovecot.log file I an getting like imap(user at example.com):Warning: fscking index file /home/ example.com/user/mail/.imap/VISA/dovecot.index Can any one specify why its coming and how can I fix it ? -- * Thanks & Regards * *Jitendra Kumar Bhaskar* Cell:- +91 7306311531 +91 8102997821 From Attila.Sipos at netcall.com Mon Mar 26 16:28:31 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 14:28:31 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied Message-ID: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Hi, When I issue an IMAP copy command using the wrong UID, the server gives an "OK No messages copied" response. This seems like the wrong response to me. If the UID doesn't exist, then it should respond with a "No" response - maybe something like "NO - copy error: bad UID" I believe "OK No messages copied" would only be a suitable response if the email with the supplied UID had already been known to be copied successfully. I am using dovecot 1.2.9 - has this been fixed in newer versions of dovecot? Regards Attila Attila Sipos Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From bob at db.org Mon Mar 26 16:29:08 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Mon, 26 Mar 2012 15:29:08 +0200 Subject: [Dovecot] fts-solr not indexing body content Message-ID: This could easily be me missing something, but I can't seem to get the fts-solr plugin to index message bodies. Tcpdump shows me the following being sent from Dovecot to Solr as a messages is indexed: 6549fde08816e80d6b4f26650000b5f0b4b2user6549/fde08816e80d6b4f26650000b5f0b4b2/user Return-path: ... As you can see, the -node contains just an empty line. The above is just a snippet to illustrate the problem. A full dump can be found at the following URL: http://db.org/temp/solr.xml.txt Full text search is configures such: plugin { fts = solr fts_solr = break-imap-search url=http://127.0.0.1:8080/solr/ } So; am I missing something, or is this a Dovecot problem? dovecot.conf: http://db.org/temp/dovecot.conf Regards, B?rd Johannessen From tss at iki.fi Mon Mar 26 17:45:45 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:45:45 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332773145.26095.121.camel@innu> On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server gives > an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response - > maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. From tss at iki.fi Mon Mar 26 17:47:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 17:47:15 +0300 Subject: [Dovecot] dovecot.log warning In-Reply-To: References: Message-ID: <1332773235.26095.122.camel@innu> On Mon, 2012-03-26 at 15:21 +0530, Jitendra Bhaskar wrote: > Hi Guys, > > Just I installed dovecot-2.1.0 in centos 5.7. and did copy of all user data > i.e home directory and mail data from previous server which was on dovecot > 1.2.8 to new one. Every thing is working fine. Mails are going and coming. > But in dovecot.log file I an getting like > > imap(user at example.com):Warning: fscking index file /home/ > example.com/user/mail/.imap/VISA/dovecot.index > > Can any one specify why its coming and how can I fix it ? This warning should not exist alone. Isn't there anything else logged? From tomislav.mihalicek at gmail.com Mon Mar 26 17:48:13 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 07:48:13 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap Message-ID: <33544743.post@talk.nabble.com> Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't create namespace 'Share/' for user miha-share at example.com: userdb didn't return a home directory, but location used it (%h): maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u Where is the problem, the same config worked with Dovecot 1.2.x -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544743.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Mon Mar 26 18:01:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:01:18 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <1332774078.26095.124.camel@innu> On Sun, 2012-03-25 at 00:46 -0700, Joseph Tam wrote: > Subject: Different user messages clustered around the same date.saved value > > After updating dovecot to 2.1.3, I can now use "doveadm expunge -A ..." > to iterate through all user trash folders and expunge old messages. > > However, I noticed a strange thing: querying what would have been deleted > > doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d > > showed many date.saved values are clustered around the same > timestamp, even among different user's Trash mailbox. One user's trash > mailbox having the same date.saved is explained by a user deleting a > lot of message at one time, but I can't explain why many different users > would have messages with the same (or closeby) date.saved value. Which mailbox format? With Maildir the date.saved is taken from dovecot.index.cache file, and in some cases that might get dropped. If it does, then it fallbacks to using the file's ctime. From tss at iki.fi Mon Mar 26 18:02:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:02:59 +0300 Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <1332774179.26095.125.camel@innu> On Mon, 2012-03-26 at 07:48 -0700, Tomislav Mihalicek wrote: > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x I'm guessing it didn't work properly with v1.2. Anyway, these would help giving suggestions: 1. dovecot -n output 2. Logs with auth_debug=yes and mail_debug=yes enabled From tss at iki.fi Mon Mar 26 18:06:58 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:06:58 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: Message-ID: <1332774418.26095.126.camel@innu> On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > This could easily be me missing something, but I can't seem to get the > fts-solr plugin to index message bodies. What Dovecot version? From Attila.Sipos at netcall.com Mon Mar 26 18:13:29 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:13:29 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332773145.26095.121.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 15:46 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > When I issue an IMAP copy command using the wrong UID, the server > gives an "OK No messages copied" response. > > This seems like the wrong response to me. > If the UID doesn't exist, then it should respond with a "No" response > - maybe something like "NO - copy error: bad UID" > > I believe "OK No messages copied" would only be a suitable response if > the email with the supplied UID had already been known to be copied > successfully. > I am using dovecot 1.2.9 - has this been fixed in newer versions of > dovecot? Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:18:46 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:18:46 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332774418.26095.126.camel@innu> References: <1332774418.26095.126.camel@innu> Message-ID: <1332775126.26095.127.camel@innu> On Mon, 2012-03-26 at 18:06 +0300, Timo Sirainen wrote: > On Mon, 2012-03-26 at 15:29 +0200, B?rd Johannessen wrote: > > This could easily be me missing something, but I can't seem to get the > > fts-solr plugin to index message bodies. > > What Dovecot version? Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This should fix it: http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 From tss at iki.fi Mon Mar 26 18:22:59 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:22:59 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332775379.26095.130.camel@innu> It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response if > > the email with the supplied UID had already been known to be copied > > successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From lukas.mueller at newmedia.ch Mon Mar 26 18:25:54 2012 From: lukas.mueller at newmedia.ch (=?iso-8859-1?Q?M=FCller_Lukas?=) Date: Mon, 26 Mar 2012 15:25:54 +0000 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: Thanks for the quick answer. I realised, that the error didn't occur since quite a while, opposed to what our client suggested. Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. Is it possible that those could have improved the situation? For now I don't have a way of reproducing the problem, so I will have to wait for an error to happen. Until then I will consider the problem as "Solved until happens again ;-) ", since the last error occured a while back (as mentioned above). >> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) .. >> My suspicion/speculation what happens is the following: >> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >> Somehow this leads to problems with Locks on NFS, which leads to the crash. >Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). I had a quick look a tour NFS (NetApp), but didn't find anything useful. In case the problem persists, I will check with the coworker responsible for NetApp. I will check what sort of locking is used by postfix, since I'm not sure if postfix and dovecot are configured to use the same mechanisms. If not, I think it could be part of the Problem. >> I have no idea how to solve this problem and any help is greatly appreciated. >The only way to fully fix this is: http://wiki2.dovecot.org/Director Unfortunately that is not an option right now, but I will keep it in mind. Thanks again. From Attila.Sipos at netcall.com Mon Mar 26 18:30:24 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:30:24 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332775379.26095.130.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> OK, clearly I am not experienced enough in IMAP to argue. I am sure the imap-protocol people will tell me to get lost! Thanks for your time. If you could possibly tell me how to know if an IMAP "UID COPY" is successful, I would appreciate it. Basically I'm moving a message from one folder to another. I thought I could issue a COPY command, check for success, then delete the email from the source folder. Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:23 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied It might be more useful from your point of view, but it might not be from from someone else's point of view. If you want this changed, see if you can convince other people in imap-protocol list. All of the widely used IMAP servers behave the way Dovecot does. On Mon, 2012-03-26 at 16:13 +0100, Attila Sipos wrote: > I can understand that if using a set of UIDS or a UID range it would be complicated to return a fully-descriptive result and from what I can see, the IMAP RFC author was trying to avoid this complexity. > > However, it someone specifies JUST ONE UID and that UID is non-existent, then a NO response could be a more useful response. I know the spec allows a response of OK but it is possible that this was not the intention for a single non-existent UID. > > Regards > > Attila > > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 15:46 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 14:28 +0100, Attila Sipos wrote: > > > When I issue an IMAP copy command using the wrong UID, the server > > gives an "OK No messages copied" response. > > > > This seems like the wrong response to me. > > If the UID doesn't exist, then it should respond with a "No" > > response > > - maybe something like "NO - copy error: bad UID" > > > > I believe "OK No messages copied" would only be a suitable response > > if the email with the supplied UID had already been known to be > > copied successfully. > > I am using dovecot 1.2.9 - has this been fixed in newer versions of > > dovecot? > > Dovecot's behavior is correct. This (or things related to this) has been discussed in IMAP protocol mailing list. Basically it's not an error to use nonexistent UIDs. If you want details, ask in IMAP protocol ml and someone will probably explain. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : > 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 > 1BB > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 18:38:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 18:38:26 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332776306.26095.139.camel@innu> On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. From Attila.Sipos at netcall.com Mon Mar 26 18:51:19 2012 From: Attila.Sipos at netcall.com (Attila Sipos) Date: Mon, 26 Mar 2012 16:51:19 +0100 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <1332776306.26095.139.camel@innu> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> Message-ID: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> thanks. I find it odd that the 2nd copy attempt returns OK. I would've thought the "expungeissued" reason would still stand. For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? Regards Attila -----Original Message----- From: Timo Sirainen [mailto:tss at iki.fi] Sent: 26 March 2012 16:38 To: Attila Sipos Cc: dovecot at dovecot.org Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > Thanks for your time. If you could possibly tell me how to know if an > IMAP "UID COPY" is successful, I would appreciate it. > Basically I'm moving a message from one folder to another. I thought > I could issue a COPY command, check for success, then delete the email > from the source folder. What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. If the client sees that some UID exists, but another session deletes it, the COPY will fail: a fetch 1 uid * 1 FETCH (UID 820) a OK Fetch completed. b uid copy 820 Trash * 1 EXPUNGE b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. c uid copy 820 Trash c OK No messages copied. Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB From tss at iki.fi Mon Mar 26 19:08:44 2012 From: tss at iki.fi (Timo Sirainen) Date: Mon, 26 Mar 2012 19:08:44 +0300 Subject: [Dovecot] dovecot-1.2.9: OK No messages copied In-Reply-To: <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> References: <71D2E0122074C64AB6574C4702126BB3C412E4@Exchange.hemel.telephonetics.co.uk> <1332773145.26095.121.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F0@Exchange.hemel.telephonetics.co.uk> <1332775379.26095.130.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F3@Exchange.hemel.telephonetics.co.uk> <1332776306.26095.139.camel@innu> <71D2E0122074C64AB6574C4702126BB3C412F7@Exchange.hemel.telephonetics.co.uk> Message-ID: <1332778124.26095.141.camel@innu> Note how Dovecot sent the client EXPUNGE notification. Prior to that client knew that the message existed. After that client knows that the message no longer exists. It was only during the COPY command that client didn't know that the message had already been expunged. On Mon, 2012-03-26 at 16:51 +0100, Attila Sipos wrote: > thanks. > I find it odd that the 2nd copy attempt returns OK. > I would've thought the "expungeissued" reason would still stand. > > For how long does the reason persist? I suppose it only persists for enough time to issue a "NO" response? > > Regards > Attila > > -----Original Message----- > From: Timo Sirainen [mailto:tss at iki.fi] > Sent: 26 March 2012 16:38 > To: Attila Sipos > Cc: dovecot at dovecot.org > Subject: Re: [Dovecot] dovecot-1.2.9: OK No messages copied > > On Mon, 2012-03-26 at 16:30 +0100, Attila Sipos wrote: > > Thanks for your time. If you could possibly tell me how to know if an > > IMAP "UID COPY" is successful, I would appreciate it. > > Basically I'm moving a message from one folder to another. I thought > > I could issue a COPY command, check for success, then delete the email > > from the source folder. > > What kind of an application are you building? Most IMAP clients would track the state of the mailbox, so they would already know if the UIDs exist or no before they do a COPY. And that's really the only solution for this. > > If the client sees that some UID exists, but another session deletes it, the COPY will fail: > > a fetch 1 uid > * 1 FETCH (UID 820) > a OK Fetch completed. > b uid copy 820 Trash > * 1 EXPUNGE > b NO [EXPUNGEISSUED] Some of the requested messages no longer exist. > c uid copy 820 Trash > c OK No messages copied. > > > > Netcall Telecom Ltd Registered in England 2831215. Registered Office : 3rd Floor, Hamilton House, 111 Marlowes, Hemel Hempstead, Herts, HP1 1BB > From ncjeffgus at zimage.com Mon Mar 26 22:11:40 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:11:40 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <4F6D65DC.7030304@tlinx.org> References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> <4F6D65DC.7030304@tlinx.org> Message-ID: <1332789100.28702.7.camel@sally> On Fri, 2012-03-23 at 23:12 -0700, Linda Walsh wrote: > Next -- bench "cp -ax", against rsync -axHAX when it has to copy >75% of > the data (cp ~6-8x speed). But for file speed, 'dd' is king, as it can > use large buffers (~16MB gives best results on my local Gbit network), > but it > misses all those pesky acls and extended attrs, not to mention file > perms...*sigh* Compare that to the I/O done 4k at a time by many older > utils... cp -ax: real 0m3.088s user 0m0.034s sys 0m3.054s rsync -axHAX real 0m15.850s user 0m19.314s sys 0m8.816s dsync's time was over six minutes. Each time I cleared out the destination folder. dsync is doing something that is taking much, much, much longer to do. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:25:28 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:25:28 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> <4F6CBAA2.5020409@in.tum.de> <1332535343.5601.6.camel@sally> Message-ID: <1332789928.28702.16.camel@sally> On Sat, 2012-03-24 at 14:21 +0100, Maarten Bezemer wrote: > On Fri, 23 Mar 2012, Jeff Gustafson wrote: > > > That didn't seem to make much of a difference. On a 3.1GB backup it > > shaved off 5 seconds. dsync's time was over 6 minutes with or without > > the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds. > > It seems to me that dsync *should* be able to be just as fast, but it > > currently is spending way too much time doing something. What is it? > > Syncing 3.1GB in 15 seconds would require a speed of more than 200MB per > second. Depending on the harddisks used, that would be quite a challenge. > If you use rsync to only transfer the files that changed (based on file > modification time) you may or may not miss files that have changed but > still have the same time stamp. I assume you didn't use the --checksum > parameter to rsync, right? The destination directory was empty. I was doing a full backup. > dsync does so much more than simply copy some files... I realize that. I am hoping that the extra data that dsync has available to it would improve the speed of syncing backups. My baseline testing of simply backing up a mailbox to an empty directory shows that dsync is takes way too long to backup a single mailbox. I have over a terabyte of data to backup. I'm currently using rsync and it must traverse tens of thousands of files and check the time information. It works, but I was hoping dsync would be a better solution. dsync should be able to sync faster, by gulping in the index information for each mailbox. I haven't even moved to the point of sync'ing since the baseline test of simply exporting a mailbox is so slow. ...Jeff From ncjeffgus at zimage.com Mon Mar 26 22:34:50 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Mon, 26 Mar 2012 12:34:50 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F6D7594.10800@fsn.hu> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> Message-ID: <1332790490.28702.23.camel@sally> On Sat, 2012-03-24 at 08:19 +0100, Attila Nagy wrote: > > I personally think that Dovecot could gain much more if the amount of > work going into fixing or improving dsync would go into making Dovecot > to (be able of) use a high scale, distributed storage backend. > I know it's much harder, because there are several major differences > compared to the "low latency" and consistency problem free local file > systems, but its fruits are also sweeter for the long term. :) Do you have any suggestions for a distributed replicated filesystem that works well with dovecot? I've looked into glusterfs, but the latency is way too high for lots of small files. They claim this problem is fixed in glusterfs 3.3. NFS too slow for my installation so I don't see how any of the distributed filesystems would help me. I've also tried out ZFS, but it appears to have issues with metadata look ups with directories that have tens or hundreds of thousands of files in them. For me, the best filesystem is straight up ext4 running on locally attached storage. I think a solid, fast dsync implementation would be very useful for a large installation. ...Jeff From tomislav.mihalicek at gmail.com Tue Mar 27 00:08:36 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Mon, 26 Mar 2012 14:08:36 -0700 (PDT) Subject: [Dovecot] Error: Couldn't create namespace 'Share/' Dovecot 2.1.3 ldap In-Reply-To: <33544743.post@talk.nabble.com> References: <33544743.post@talk.nabble.com> Message-ID: <33544762.post@talk.nabble.com> When i put service=lib-storage to users in ldap everything works. Is this a bug? cartman dovecot: auth: Debug: master in: USER 1 user at example.net service=lib-storage Tomislav Mihalicek wrote: > > Mar 26 16:38:58 cartman dovecot: imap(miha at example.com): Error: Couldn't > create namespace 'Share/' for user miha-share at example.com: userdb didn't > return a home directory, but location used it (%h): > maildir:%%h/Maildir:INDEX=%h/index/Shared/%%u > > Where is the problem, the same config worked with Dovecot 1.2.x > -- View this message in context: http://old.nabble.com/Error%3A-Couldn%27t-create-namespace-%27Share-%27-Dovecot-2.1.3-ldap-tp33544743p33544762.html Sent from the Dovecot mailing list archive at Nabble.com. From tss at iki.fi Tue Mar 27 00:14:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Tue, 27 Mar 2012 00:14:17 +0300 Subject: [Dovecot] Dovecot IMAP is broken after upgrade from 2.1.2 to 2.1.3 In-Reply-To: <4F6CE3B8.7020507@michael-neubert.de> References: <4F6CE3B8.7020507@michael-neubert.de> Message-ID: On 23.3.2012, at 22.57, Michael Neubert wrote: > I just upgraded my servers from Dovecot 2.1.2 to Dovecot 2.1.3-0~auto+5 by using > Debian binaries from "http://xi.rename-it.nl/debian/ stable-auto/dovecot-2.1 main". > > Mar 23 21:45:28 imap-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [xxx.xxx.xxx.xxx] Fixed: http://hg.dovecot.org/dovecot-2.1/rev/339b1337aab0 From andrei.michescu at miau.ca Tue Mar 27 01:14:22 2012 From: andrei.michescu at miau.ca (Michescu Andrei) Date: Mon, 26 Mar 2012 18:14:22 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Hello Timo, Thank you very much for planning a redesign of the dsycn and for opening this discussion. As I can see from the replies that came until now everybody misses the main point of IMAP: IMAP has been designed to work as a disconnected, high-latency data store. To make this more clear: once and IMAP client finishes the synchronization with the server, both have client and server have a consistent state of the mailbox. After this both the "client" and the "server" act like master for their own local copy (on the "server" new emails get created etc, on the "client" existing emails get changed (flags) and moved, and new emails appear (sent items)). So the protocol is designed, originally, to handle the master-master replication. And as this it make sense a deployment global-wide, where servers work independently and from time to time they "merge" the changes. This being said and acknowledged here are my 2 cents: I think that the current '1 brain / 2 workers' seems to be the correct model. The "the client" connects to the "server" and pushes the local changes and after retrieves the updated/new items from the "server". "The brain" considers first server as the "local storage" and the second server as "server storage". For the split design, "come to the same conclusion of the state" is very race-condition prone. As long as the algorithm is kept as you described it in the original document then the backups should really be incremental (because you only do the changes since last sync). As the most changes are "metadata-only" the sync can be pretty fast by merging indexes. Thank you, Andrei > In case anyone is interested in reading (and maybe helping!) with a dsync > redesign that's intended to fix all of its current problems, here are some > possibly incoherent ramblings about it: > > http://dovecot.org/tmp/dsync-redesign.txt > > and even if you don't understand that, here's another document disguising > as an algorithm class problem :) If anyone has thoughts on how to solve > it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but > those should be much simpler. > > > !DSPAM:4f6cea4c260302917022693! > > From abruce at tumnus.co.nz Tue Mar 27 03:57:04 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Tue, 27 Mar 2012 13:57:04 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage Message-ID: Hi there, We're setting up a Dovecot virtual email setup - we've got everything working perfect with LDAP logins authenticating against AD and so forth, but we're having issues with retrieving the maxStorage value from AD (this is a pre-setup field in AD that we'd like to use to set per user quotas). In our LDAP lookup, we have the maxStorage entry listed under user_attrs for the quota (user_attrs = maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see it trying to get the entry, but it fails with: Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=maxStorage Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no fields returned by the server At this point, we then see the default quota applied. If we change the name of the field from maxStorage to instanceType we see the value show up in the logs and passed through to the quota system and applied successfully: Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user search: base=dc=site,dc=local scope=subtree filter=(&(objectClass=person)(| (userPrincipalName=username at site) (|(mail=username at site)(samAccountName=username at site)))) fields=instanceType Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: instanceType(quota_rule=*:storage=%$M)=*:storage=4M Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 username at site quota_rule=*:storage=4M Which seems a bit weird. If we use ldapsearch and pass it the same search string and look for the field maxStorage, we clearly see the field and the value being returned. The result looks the same if we also lookup instanceType. We're using Dovecot 2.0.9. Does anyone have any idea as to why we can't use this field? Thanks, Andrew From jtam.home at gmail.com Tue Mar 27 04:16:24 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Mon, 26 Mar 2012 18:16:24 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >> However, I noticed a strange thing: querying what would have been >> deleted >> >> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >> >> showed many date.saved values are clustered around the same timestamp, >> even among different user's Trash mailbox. >> ... >> I can't explain why many different users would have messages with the >> same (or closeby) date.saved value. > > Which mailbox format? With Maildir the date.saved is taken from > dovecot.index.cache file, and in some cases that might get dropped. If > it does, then it fallbacks to using the file's ctime. mbox. A further look into this reveals that the clustered date.saved values are the earliest values for every mailbox in the system. This timestamp is close to the time I was testing "doveadm ... -A", so the likely explanation is that I accidentally deleted/updated these values using some variation of doveadm, even though I remember confining my testing to query/search/fetch. This appears to be a case of PEBKAC. These "wrong" values shouldn't cause problems with expunge queries since they err on the side of safety. Thanks for the insight though. Joseph Tam From koshikov at gmail.com Tue Mar 27 09:14:25 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Tue, 27 Mar 2012 09:14:25 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: Message-ID: <20120327091425.73963576@jimbo> On Tue, 27 Mar 2012 13:57:04 +1300 Bruce, Andrew wrote: > Hi there, > > We're setting up a Dovecot virtual email setup - we've got everything > working perfect with LDAP logins authenticating against AD and so > forth, but we're having issues with retrieving the maxStorage value > from AD (this is a pre-setup field in AD that we'd like to use to set > per user quotas). > > In our LDAP lookup, we have the maxStorage entry listed under > user_attrs for the quota (user_attrs = > maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > it trying to get the entry, but it fails with: > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=maxStorage > Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > fields returned by the server > > At this point, we then see the default quota applied. > Try to change your quota rule to be like: maxStorage=quota_rule=*:bytes=%$ ^^^^^^^^^ And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > > If we change the name of the field from maxStorage to instanceType we > see the value show up in the logs and passed through to the quota > system and applied successfully: > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > search: base=dc=site,dc=local scope=subtree > filter=(&(objectClass=person)(| (userPrincipalName=username at site) > (|(mail=username at site)(samAccountName=username at site)))) > fields=instanceType > Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > instanceType(quota_rule=*:storage=%$M)=*:storage=4M > Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 > username at site quota_rule=*:storage=4M > > > Which seems a bit weird. > > If we use ldapsearch and pass it the same search string and look for > the field maxStorage, we clearly see the field and the value being > returned. The result looks the same if we also lookup instanceType. > > We're using Dovecot 2.0.9. > > Does anyone have any idea as to why we can't use this field? > > Thanks, > > Andrew From luca.palazzo at unict.it Tue Mar 27 09:57:32 2012 From: luca.palazzo at unict.it (Luca Palazzo) Date: Tue, 27 Mar 2012 08:57:32 +0200 Subject: [Dovecot] 2.1.2 Corrupted squat uidlist Message-ID: <4F7164DC.7010706@unict.it> Hi Timo and All, after upgrading to 2.1.2 i'm getting a lot of these messages: Error: Corrupted squat uidlist file XXXXXX wrong indexid I did not have them before. Ideas? Luca From bob at db.org Tue Mar 27 10:12:42 2012 From: bob at db.org (=?UTF-8?Q?B=C3=A5rd_Johannessen?=) Date: Tue, 27 Mar 2012 09:12:42 +0200 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: <1332775126.26095.127.camel@innu> References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: 2012/3/26 Timo Sirainen : > Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This > should fix it: > > http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 Nope; exactly same result; body field contains just the empty line. -- B?rd Johannessen From nmilas at noa.gr Tue Mar 27 11:13:35 2012 From: nmilas at noa.gr (Nikolaos Milas) Date: Tue, 27 Mar 2012 11:13:35 +0300 Subject: [Dovecot] quota ldap In-Reply-To: <4F6C5741.3000408@univ-evry.fr> References: <4F6C4E51.7010603@univ-evry.fr> <4F6C54F2.7020203@noa.gr> <4F6C5741.3000408@univ-evry.fr> Message-ID: <4F7176AF.6000607@noa.gr> On 23/3/2012 12:58 ??, Alain DEFRANCE wrote: > so if i understand correctly i can mix the 2 quota_rule ? > the one who came from ldap user_attrs (quota_rule=*:bytes=%$) > and the other which from quota_rule2 = Trash:storage=+3%% Actually, in user_attrs you define the applicable ldap attributes and associated info. Rules are specified in the plugin {} section, but quota values may be overridden by ldap attribute values (but not for Trash). > in your case you add 3% quota more for Trash ? > Am i write ? Yes. Nick From janfrode at tanso.net Tue Mar 27 12:47:10 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 27 Mar 2012 11:47:10 +0200 Subject: [Dovecot] doveadm purge on clusterfs Message-ID: <20120327094710.GA10878@dibs.tanso.net> Since doveadm service proxying apparently doesn't work with dovecot v2.0, we need to find a way to safely run doveadm purge on the host the user is logged into. Would it be OK to run purge in the pop/imap postlogin scripts? We already do a conditional: test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today then touch /var/log/activemailaccounts/imap/$USER fi so adding a: doveadm purge -u $USER in this section would make it run once every day the users that log in. Does that sound like an OK solution? -jf From pw at wk-serv.de Tue Mar 27 13:11:59 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Tue, 27 Mar 2012 12:11:59 +0200 Subject: [Dovecot] Merge mails from two mail_locations Message-ID: <4F71926F.30500@wk-serv.de> Hi guys, recently I had some trouble with my ocfs2 cluster and it unmounted itself from /var/mail. Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. Regards Patrick From jacek at hapay.pl Tue Mar 27 14:20:14 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:20:14 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time Message-ID: <4F71A26E.5030400@hapay.pl> Hi all, I want to start new server with Postfix (I still have qmail ) and I think I have a problem with authentication in dovecot - it takes 3 seconds. Is this normal time? My configuration: Usernames: from MySQL Passwords: from Active Directory dovecot -n # 1.1.20: /etc/dovecot.conf # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 log_path: /var/log/dovecot.log protocols: pop3 pop3s imap imaps listen: * ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem ssl_key_file: /etc/pki/tls/private/iRedMail.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_uid: 500 mail_gid: 500 mail_location: maildir:/xxx/%u:INDEX=/xxx/%u mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota zlib mail_plugins(imap): quota imap_quota zlib mail_plugins(pop3): quota zlib mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 lda: postmaster_address: postmaster at domain.com mail_plugins: cmusieve quota log_path: /var/log/sieve.log auth default: mechanisms: plain login default_realm: infor.pl username_format: %Lu debug: yes debug_passwords: yes passdb: driver: pam passdb: driver: sql args: /etc/dovecot-ldap.conf userdb: driver: sql args: /etc/dovecot-mysql.conf socket: type: listen client: path: /var/spool/postfix/dovecot-auth mode: 438 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 438 user: vmail group: vmail plugin: quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 quota: maildir quota_rule: *:storage=300M quota_rule2: *:messages=0 expire: Trash 7 Trash/* 7 Junk 30 expire_dict: proxy::expire auth_socket_path: /var/run/dovecot/auth-master sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve dict: expire: db:/xxx/expire.db grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf driver = mysql default_pass_scheme = CRYPT connect = host=localhost dbname=xxx user=xxx password=xxx #password_query = SELECT password FROM mailbox WHERE username='%u' AND active='1' user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' postfix: postfix-2.5.9-5.ired dovecot: dovecot-1.1.20-1_98.el5 mysql: mysql-server-5.0.77-4.el5_6.6 This is not a TCP connection problem, because i have results from tcpdump and wireshark. There is a information that Active directory is answering with password in 0,2 second. Regards Jacek From jacek at hapay.pl Tue Mar 27 14:39:32 2012 From: jacek at hapay.pl (Jacek Kowalski) Date: Tue, 27 Mar 2012 13:39:32 +0200 Subject: [Dovecot] Problem with DOVECOT - long authentication time [SOLVED] In-Reply-To: <4F71A26E.5030400@hapay.pl> References: <4F71A26E.5030400@hapay.pl> Message-ID: <4F71A6F4.7030203@hapay.pl> W dniu 27.03.2012 13:20, Jacek Kowalski pisze: > Hi all, > > I want to start new server with Postfix (I still have qmail ) and I > think I have a problem with authentication in dovecot - it takes 3 > seconds. Is this normal time? > > My configuration: > Usernames: from MySQL > Passwords: from Active Directory > > > dovecot -n > > # 1.1.20: /etc/dovecot.conf > # OS: Linux 2.6.18-274.7.1.el5 i686 CentOS release 5.7 (Final) ext3 > log_path: /var/log/dovecot.log > protocols: pop3 pop3s imap imaps > listen: * > ssl_cert_file: /etc/pki/tls/certs/iRedMail_CA.pem > ssl_key_file: /etc/pki/tls/private/iRedMail.key > login_dir: /var/run/dovecot/login > login_executable(default): /usr/libexec/dovecot/imap-login > login_executable(imap): /usr/libexec/dovecot/imap-login > login_executable(pop3): /usr/libexec/dovecot/pop3-login > mail_uid: 500 > mail_gid: 500 > mail_location: maildir:/xxx/%u:INDEX=/xxx/%u > mail_executable(default): /usr/libexec/dovecot/imap > mail_executable(imap): /usr/libexec/dovecot/imap > mail_executable(pop3): /usr/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota zlib > mail_plugins(imap): quota imap_quota zlib > mail_plugins(pop3): quota zlib > mail_plugin_dir(default): /usr/lib/dovecot/imap > mail_plugin_dir(imap): /usr/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 > lda: > postmaster_address: postmaster at domain.com > mail_plugins: cmusieve quota > log_path: /var/log/sieve.log > auth default: > mechanisms: plain login > default_realm: infor.pl > username_format: %Lu > debug: yes > debug_passwords: yes > passdb: > driver: pam > passdb: > driver: sql > args: /etc/dovecot-ldap.conf > userdb: > driver: sql > args: /etc/dovecot-mysql.conf > socket: > type: listen > client: > path: /var/spool/postfix/dovecot-auth > mode: 438 > user: postfix > group: postfix > master: > path: /var/run/dovecot/auth-master > mode: 438 > user: vmail > group: vmail > plugin: > quota_warning: storage=85%% /usr/local/bin/dovecot-quota-warning.sh 85 > quota_warning2: storage=90%% /usr/local/bin/dovecot-quota-warning.sh 90 > quota_warning3: storage=95%% /usr/local/bin/dovecot-quota-warning.sh 95 > quota: maildir > quota_rule: *:storage=300M > quota_rule2: *:messages=0 > expire: Trash 7 Trash/* 7 Junk 30 > expire_dict: proxy::expire > auth_socket_path: /var/run/dovecot/auth-master > sieve: /xxx/sieve/%Ld/%Ln/dovecot.sieve > dict: > expire: db:/xxx/expire.db > > > > grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf > > driver = mysql > default_pass_scheme = CRYPT > connect = host=localhost dbname=xxx user=xxx password=xxx > #password_query = SELECT password FROM mailbox WHERE username='%u' AND > active='1' > user_query = SELECT CONCAT(storagebasedirectory, '/', storagenode, > '/', maildir) AS home, CONCAT('*:bytes=', quota*1048576) AS quota_rule > FROM mailbox WHERE username='%u' AND active='1' AND enable%Ls%Lc='1' > > > postfix: postfix-2.5.9-5.ired > dovecot: dovecot-1.1.20-1_98.el5 > mysql: mysql-server-5.0.77-4.el5_6.6 > > > > This is not a TCP connection problem, because i have results from > tcpdump and wireshark. There is a information that Active directory is > answering with password in 0,2 second. > > > Regards > > Jacek Ok. Problem Solved. It was "passdb pam" problem. Regards Jacek From campbell at cnpapers.com Tue Mar 27 17:40:11 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Tue, 27 Mar 2012 10:40:11 -0400 Subject: [Dovecot] Namespace, prefix questions Message-ID: <4F71D14B.2010301@cnpapers.com> We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) disable_plaintext_auth = no listen = * mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mbox_write_locks = fcntl namespace { hidden = yes inbox = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = separator = / type = private } namespace { hidden = yes list = no location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = "#mbox/" separator = / type = private } namespace { hidden = yes list = no location = prefix = mail/ separator = / type = private } namespace { hidden = yes list = yes location = mbox:~/mail:INBOX=/var/spool/mail/%u prefix = ~/mail/ separator = / type = private } namespace { hidden = yes list = no location = prefix = ~%u/mail/ separator = / type = private } passdb { driver = pam } protocols = pop3 imap ssl_cert = Message-ID: <87obrixcyp.fsf@algae.riseup.net> Timo Sirainen writes: > In case anyone is interested in reading (and maybe helping!) with a dsync redesign that's intended to fix all of its current problems, here are some possibly incoherent ramblings about it: thank you for opening this discussion about dsync! besides the problems I've encountered with dsync, there are a couple things that I think would be great to build into the new vision of the protocol. One would be the ability to perform *intelligent* incremental/rotated backups. I can do this now by running a dsync backup operation and then doing manual hardlinking or moving of the backup directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it would be more intelligent if this were baked into the backup process. Secondly, being able to filter out mailboxes could result in much more efficient syncing. Now there is the capability to operate on only specific mailboxes, but this doesn't scale well when I am trying to backup thousands of users and I want to omit the Spam and Trash folders from the sync. I would have to get a mailbox list of each user, and then iterate over each mailbox for each user, skipping the Spam and Trash folders, forking a new 'dsync backup' for each of their mailboxes, for each user. Lastly, there isn't a good method for restoring backups. I can reverse the backup process, onto the user's "live" mailbox, but that brings the user into an undesirable state (eg. their mailbox state one day ago). Better would be if their backup could be restored in such a way that the user can resolve the missing pieces manually, as they know best. thanks again for your work on this, from my position dovecot is an amazing piece of software, the only part that seems to have some issues is dsync and I applaud the effort to redesign to fix things! micah From me at benschumacher.com Tue Mar 27 19:49:50 2012 From: me at benschumacher.com (Ben Schumacher) Date: Tue, 27 Mar 2012 10:49:50 -0600 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: > the zlib_save question reminds me of a wish: > I think it's not possible to set zlib_save parameter per namespace (or even > mailbox). Per namespace would be something for the wish list to get rid of > the cron job method to compress archival mailboxes. > And maybe an option to add a "Z" flag to compressed maildir message files > as recommended in the wiki regarding compress crob job. +1 on this request. I have a slightly different use case -- I have both an dbox and Maildir. Incoming email goes to Maildir, but I archive off to dbox (using Thunderbird). After I archive my emails, compression seems like a reasonable choice. Any idea if this feature will be available at some point? Thanks, Ben From lists at wiesinger.com Tue Mar 27 20:28:56 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:28:56 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F8D8.6040700@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From gerhard at wiesinger.com Tue Mar 27 20:27:59 2012 From: gerhard at wiesinger.com (Gerhard Wiesinger) Date: Tue, 27 Mar 2012 19:27:59 +0200 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more Message-ID: <4F71F89F.9060903@wiesinger.com> Hello, After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. Relevant config dovecot.conf: mail_location = mbox:~:INBOX=/var/mail/%u Different LIST behaviour from rawlog: 1.2.x: with Prefix ~/Mail A002 LIST "" "~/Mail/sent" * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" A002 OK List completed. 2.0.x: with Prefix ~/Mail A0003 LIST "" ~/Mail/sent A0003 OK List completed.: Configured prefix for private spaces in roundcube mail is ~/Mail . Roundcube Webmail application checks for existence of the folder but dovecot doesn't return anything in 2.x.latest. Is this by design? Any ideas to fix it by configuration? (I tried prefix Mail/ in roundcube mail without success) Thunderbird works well. Thnx. Ciao, Gerhard From CMarcus at Media-Brokers.com Tue Mar 27 22:34:40 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Tue, 27 Mar 2012 15:34:40 -0400 Subject: [Dovecot] dsync redesign In-Reply-To: <87obrixcyp.fsf@algae.riseup.net> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <87obrixcyp.fsf@algae.riseup.net> Message-ID: <4F721650.4030901@Media-Brokers.com> On 2012-03-27 11:47 AM, Micah Anderson wrote: > One would be the ability to perform *intelligent* incremental / > rotated backups. I can do this now by running a dsync backup > operation and then doing manual hardlinking or moving of the backup > directories (daily.1, daily.2, weekly.1, monthly.1, etc.), but it > would be more intelligent if this were baked into the backup process. There are already numerous tools that do this flawlessly - I've been using rsnapshot (which uses rsync) for this for years. I don't know if Timo should be spending his time reinventing the wheel. I'm much more interested in dsync working flawlessly to keep one or more secondary servers in sync, and leave backups to backup software. > Lastly, there isn't a good method for restoring backups. I can reverse > the backup process, onto the user's "live" mailbox, but that brings the > user into an undesirable state (eg. their mailbox state one day > ago). Better would be if their backup could be restored in such a way > that the user can resolve the missing pieces manually, as they know > best. Again, best left to the backup software I think? Although, one interesting piece that I am hopeful I'll be able to implement soon (with Timo's professional help) is the ability to easily and automatically map my rsnapshot snapshots directory to a read-only 'Backups' namespace that automatically shows the snapshots by date and time as they are produced. This way users could 'go back in time' anytime they wanted without having to call me... :) > thanks again for your work on this, from my position dovecot is an > amazing piece of software, the only part that seems to have some issues > is dsync and I applaud the effort to redesign to fix things! Ditto all of that! :) -- Best regards, Charles From stan at hardwarefreak.com Tue Mar 27 23:09:44 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Tue, 27 Mar 2012 15:09:44 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332790490.28702.23.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> Message-ID: <4F721E88.8020309@hardwarefreak.com> On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > Do you have any suggestions for a distributed replicated filesystem > that works well with dovecot? I've looked into glusterfs, but the > latency is way too high for lots of small files. They claim this problem > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > see how any of the distributed filesystems would help me. I've also > tried out ZFS, but it appears to have issues with metadata look ups with > directories that have tens or hundreds of thousands of files in them. > For me, the best filesystem is straight up ext4 running on locally > attached storage. > I think a solid, fast dsync implementation would be very useful for a > large installation. It sounds like you're in need of a more robust and capable storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or incremental snapshot capability. Also, you speak of a very large maildir store, with hundreds of thousands of directories, obviously many millions of files, of 1TB total size. Thus I would assume you have many thousands of users, if not 10s of thousands. It's a bit hard to believe you're not running XFS on your storage, given your level of parallelism. You'd get much better performance using XFS vs EXT4. Especially with kernel 2.6.39 or later which includes the delayed logging patch. This patch increases metadata write throughput by a factor of 2-50+ depending on thread count, and decreases IOPS and MB/s hitting the storage by about the same factor, depending on thread count. Before this patch XFS sucked at the write portion of the maildir workload due to the extremely high IOPS and MB/s hitting just the log journal, not including the actual file writes. It's parallel maildir read performance was better than any other, but the write was so bad it bogged down the storage producing high latency for everything. With the delaylog patch, XFS now trounces every filesystem at medium to high parallelism levels. Delaylog was introduced in mid 2009, included in 2.6.35 as experimental, and is the default in 2.6.39 and later. If you're a Red Hat or CentOS user it's included in 6.2. This one patch, which was 5+ years in development, dramatically changed the character of XFS with this class of metadata intensive parallel workloads. Many people with such a workload who ran from XFS in the past, as if it were the Fukushima reactor, are now adopting it in droves. What a difference a few hundred lines of very creative code can make... -- Stan From abruce at tumnus.co.nz Tue Mar 27 23:39:37 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 09:39:37 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120327091425.73963576@jimbo> References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:36, Bruce, Andrew wrote: > On 27 March 2012 19:14, Nikita Koshikov wrote: >> On Tue, 27 Mar 2012 13:57:04 +1300 >> Bruce, Andrew wrote: >> >> Hi there, >> >> We're setting up a Dovecot virtual email setup - we've got everything >> working perfect with LDAP logins authenticating against AD and so >> forth, but we're having issues with retrieving the maxStorage value >> from AD (this is a pre-setup field in AD that we'd like to use to set >> per user quotas). >> >> In our LDAP lookup, we have the maxStorage entry listed under >> user_attrs for the quota (user_attrs = >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >> it trying to get the entry, but it fails with: >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=maxStorage >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >> fields returned by the server >> >> At this point, we then see the default quota applied. >> > Try to change your quota rule to be like: > maxStorage=quota_rule=*:bytes=%$ > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >> >> If we change the name of the field from maxStorage to instanceType we >> see the value show up in the logs and passed through to the quota >> system and applied successfully: >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >> search: base=dc=site,dc=local scope=subtree >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >> (|(mail=username at site)(samAccountName=username at site)))) >> fields=instanceType >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >> username at site ? ?quota_rule=*:storage=4M >> >> >> Which seems a bit weird. >> >> If we use ldapsearch and pass it the same search string and look for >> the field maxStorage, we clearly see the field and the value being >> returned. ?The result looks the same if we also lookup instanceType. >> >> We're using Dovecot 2.0.9. >> >> Does anyone have any idea as to why we can't use this field? >> >> Thanks, >> >> Andrew Tried your suggestion Nikita, no joy unfortunately. ?It still looks like the value never gets returned from the LDAP server to Dovecot. It definitely has something in the field (equivalent of 10GB, but in bytes as suggested) and I changed the user_attrs also, but still get the same "no fields returned by the server" error message. Modifying the user_attrs to lookup from a different field (instanceType) definitely works. What exact version are you using - perhaps it's a problem with our copy of 2.0.9. Thanks, Andrew From ncjeffgus at zimage.com Tue Mar 27 23:57:41 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Tue, 27 Mar 2012 13:57:41 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F721E88.8020309@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> Message-ID: <1332881861.29480.8.camel@sally> On Tue, 2012-03-27 at 15:09 -0500, Stan Hoeppner wrote: > On 3/26/2012 2:34 PM, Jeff Gustafson wrote: > > > Do you have any suggestions for a distributed replicated filesystem > > that works well with dovecot? I've looked into glusterfs, but the > > latency is way too high for lots of small files. They claim this problem > > is fixed in glusterfs 3.3. NFS too slow for my installation so I don't > > see how any of the distributed filesystems would help me. I've also > > tried out ZFS, but it appears to have issues with metadata look ups with > > directories that have tens or hundreds of thousands of files in them. > > For me, the best filesystem is straight up ext4 running on locally > > attached storage. > > It sounds like you're in need of a more robust and capable > storage/backup solution, such as an FC/iSCSI SAN array with PIT and/or > incremental snapshot capability. We do have a FC system that another department is using. The company dropped quite a bit of cash on it for a specific purpose. Our department does not have access it to. People are somewhat afraid of iSCSI around here because they believe it will add too much latency to the overall IO performance. They're a big believer in locally attached disks. Less features, but very good performance. We thought ZFS would provide us with a nice snapshot and backup system (with zfs send). We never got that far once we discovered that ZFS doesn't work very well in this context. Running rsync on it gave us terrible performance. > Also, you speak of a very large maildir store, with hundreds of > thousands of directories, obviously many millions of files, of 1TB total > size. Thus I would assume you have many thousands of users, if not 10s > of thousands. > > It's a bit hard to believe you're not running XFS on your storage, given > your level of parallelism. You'd get much better performance using XFS > vs EXT4. Especially with kernel 2.6.39 or later which includes the > delayed logging patch. This patch increases metadata write throughput > by a factor of 2-50+ depending on thread count, and decreases IOPS and > MB/s hitting the storage by about the same factor, depending on thread > count. I've relatively new here, but I'll ask around about XFS and see if anyone had tested it in the development environment. ...Jeff From abruce at tumnus.co.nz Wed Mar 28 00:06:55 2012 From: abruce at tumnus.co.nz (Bruce, Andrew) Date: Wed, 28 Mar 2012 10:06:55 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28 March 2012 09:39, Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: >> On 27 March 2012 19:14, Nikita Koshikov wrote: >>> On Tue, 27 Mar 2012 13:57:04 +1300 >>> Bruce, Andrew wrote: >>> >>> Hi there, >>> >>> We're setting up a Dovecot virtual email setup - we've got everything >>> working perfect with LDAP logins authenticating against AD and so >>> forth, but we're having issues with retrieving the maxStorage value >>> from AD (this is a pre-setup field in AD that we'd like to use to set >>> per user quotas). >>> >>> In our LDAP lookup, we have the maxStorage entry listed under >>> user_attrs for the quota (user_attrs = >>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>> it trying to get the entry, but it fails with: >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=maxStorage >>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>> fields returned by the server >>> >>> At this point, we then see the default quota applied. >>> >> Try to change your quota rule to be like: >> maxStorage=quota_rule=*:bytes=%$ >> ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ >> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >> >> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>> >>> If we change the name of the field from maxStorage to instanceType we >>> see the value show up in the logs and passed through to the quota >>> system and applied successfully: >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>> search: base=dc=site,dc=local scope=subtree >>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>> (|(mail=username at site)(samAccountName=username at site)))) >>> fields=instanceType >>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 >>> username at site ? ?quota_rule=*:storage=4M >>> >>> >>> Which seems a bit weird. >>> >>> If we use ldapsearch and pass it the same search string and look for >>> the field maxStorage, we clearly see the field and the value being >>> returned. ?The result looks the same if we also lookup instanceType. >>> >>> We're using Dovecot 2.0.9. >>> >>> Does anyone have any idea as to why we can't use this field? >>> >>> Thanks, >>> >>> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Further investigation shows that there are a few other fields that we can't retrieve in Dovecot, but can using the same search string and lookup user with ldapsearch. maxStorage is obviously one, but I tried a couple of other fields of varying types: mobile - Octet String and logonCount - Integer. Doesn't seem to be the type that restricts the search, just some fields won't return. From sorr at rightnow.com Wed Mar 28 00:38:19 2012 From: sorr at rightnow.com (Orr, Steve) Date: Tue, 27 Mar 2012 21:38:19 +0000 Subject: [Dovecot] Using getmail with sieve Message-ID: <1AA32A754D17E9478500E421F4099F9D1D13A5E8@IS-BOZ-MB02.corp.rightnow.com> I'm trying to setup a personal "mailmover" where I use getmail to retrieve remote IMAP server mail and load it into my local Dovecot then filter all email on the Dovecot server side with sieve. (I'm using Dovecot v. 2.0.9 with IMAP/Maildir.) 1) From the docs I gather that Dovecot sieve will not work as mail is loaded by getmail and I need to refilter the email after it has been loaded with getmail, right? 2) In http://wiki2.dovecot.org/HowTo/RefilterMail it says, "This HOWTO helps you create a folder for mail that needs refiltering..." But contrary to the doc I don't see any actual instructions about creating said folder. The HOWTO assumes mail already exists in a folder called "REFILTER' but doesn't say how this folder was populated. "How to?" 3) As a test I manually created the "REFILTER" folder and put my inbox mail in it (with Thunderbird) then following the HOWTO I ran my getmail script and my sieve script performed as expected. 4) How do I automate this? I tried a new getmail script to run against the inbox and tag new email so a subsequent getmail script could move it into the REFILTER folder for later sieve processing. This seems like way too much work. The HOWTO doc seems incomplete or presumes other knowledge, especially for an IMAP server and email sieve nubie. TIA, D. B. ---------------------------------------------- $ dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-131.0.15.el6.i686 i686 Red Hat Enterprise Linux Server release 6.1 (Santiago) log_path = /var/log/dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = test at host1 protocols = imap lmtp service lmtp { user = test } ssl_cert = I looked around the 'Net to see if there might be a custom program for offline Maildir to mdbox conversion. So far I haven't turned up anything. The problem for us is that the dsync program simply takes a lot of time to convert mailboxes. I wonder if time could be saved with a program that is optimized to convert mailboxes without the fancy locking that dsync needs to do. Does have (or seen) a tool that could do this? We're hoping that converting away from Maildir will help us speed up the backup processes by reducing the number of files to process. ...Jeff From stonegate at stonegate.homeip.net Wed Mar 28 01:24:59 2012 From: stonegate at stonegate.homeip.net (stonegate) Date: Tue, 27 Mar 2012 15:24:59 -0700 (PDT) Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook Message-ID: <33544803.post@talk.nabble.com> Hi, i use dovecot 2.1.3 on a gentoo system. Before i installed my new imap server box, everything worked. Now with Dovecot it does not. Problem: When i receive a new email, it does not appear in my Outlook unless i have the IMAP Inbox Folder open (highlighted selection). Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize it unless i click on the inbox folder. Before that problem occured on my old system ( i think it was dovecot as well ) the inbox folder refreshed automatically and i instantly knew when i had new emails. It was kind of like with my exchange account. Does anyone have a clue what might be wrong ? Since i did not re-install outlook or something it cannot be a client thing since no settings changed on the client side. Maybe i have to do something with dovecots config file? I think it should keep the connection to the server open or something. I?d appreciate any kind of help. Thanks alot Stoney Dovecot is capable of the following things: telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. Thats my dovecot.conf: # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 3.2.1-gentoo-r2 x86_64 Gentoo Base System release 2.0.3 auth_mechanisms = plain login listen = * disable_plaintext_auth = no mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } #passdb { # args = * # driver = pam #} #passdb { # args = /etc/dovecot/dovecot-sql.conf.ext # driver = sql #} passdb { driver = shadow } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = References: <1332888019.29480.17.camel@sally> Message-ID: <4F727ECE.4050305@r.paypc.com> On 3/27/2012 3:40 PM, Jeff Gustafson wrote: > I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. Is it slower than doing an IMAP APPEND over an authenticated dovecot connection? I've used a simple PERL script based on Mail::IMAPClient and Mail::Box to import 180,000+ mailboxes into dovecot's mdbox at fairly high speed, and all it does is IMAP APPENDs. (I had to shard the mailboxes because these PERL based tools exhaust RAM when run with mailboxes larger than about 600MB). On my development VM test box (32 bit Slack 13.37, 2G/2G split kernel, no RAID, Q6600 with only two cores allocated to the VM) and 8GB of DDR2 RAM does Emails=180,044 real 237m28.485s (12.5 emails/second) user 94m50.425s sys 10m09.389s 21,984,824 /mail/home I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox "transcoding" tool, since none appear to exist. To keep it simple, I/O to/from "remote" mailbox (connections) are not pipelined. It won't require more than MAXEMAILSIZE's worth of RAM (if one of the directions involves a remote connection), and so far when processing MIX, Maildir, and Mbox files, it's extremely fast. Adding support for [sm]dbox wouldn't appear to be problematic. At the moment, it supports everything Panda's c-client supports plus Maildir/Maildir++ (including Panda's "MIX"). Write support for Maildir's extremely UNDER-tested so far, as I've mainly used it to import Maildir hives. I've experimented with Maildir as a format, and while the one email to a file model seems like a sensible idea, it seems to simply transfer stress from one part of the system to another, mainly filesystems, and not many of those are really up for handling that many files in one directory very efficiently. None of my users have mailboxes with fewer than 100K emails in them, some have more than a million. =R= From koshikov at gmail.com Wed Mar 28 09:25:34 2012 From: koshikov at gmail.com (Nikita Koshikov) Date: Wed, 28 Mar 2012 09:25:34 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: <20120328092534.5690fa40@jimbo> On Wed, 28 Mar 2012 09:39:37 +1300 Bruce, Andrew wrote: > On 28 March 2012 09:36, Bruce, Andrew wrote: > > On 27 March 2012 19:14, Nikita Koshikov wrote: > >> On Tue, 27 Mar 2012 13:57:04 +1300 > >> Bruce, Andrew wrote: > >> > >> Hi there, > >> > >> We're setting up a Dovecot virtual email setup - we've got everything > >> working perfect with LDAP logins authenticating against AD and so > >> forth, but we're having issues with retrieving the maxStorage value > >> from AD (this is a pre-setup field in AD that we'd like to use to set > >> per user quotas). > >> > >> In our LDAP lookup, we have the maxStorage entry listed under > >> user_attrs for the quota (user_attrs = > >> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see > >> it trying to get the entry, but it fails with: > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=maxStorage > >> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no > >> fields returned by the server > >> > >> At this point, we then see the default quota applied. > >> > > Try to change your quota rule to be like: > > maxStorage=quota_rule=*:bytes=%$ > > ? ? ? ? ? ? ? ? ? ? ? ?^^^^^^^^^ > > And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. > > > > PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x > >> > >> If we change the name of the field from maxStorage to instanceType we > >> see the value show up in the logs and passed through to the quota > >> system and applied successfully: > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user > >> search: base=dc=site,dc=local scope=subtree > >> filter=(&(objectClass=person)(| (userPrincipalName=username at site) > >> (|(mail=username at site)(samAccountName=username at site)))) > >> fields=instanceType > >> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: > >> instanceType(quota_rule=*:storage=%$M)=*:storage=4M > >> Mar 27 11:09:01 auth: Debug: master out: USER ? 3901227009 > >> username at site ? ?quota_rule=*:storage=4M > >> > >> > >> Which seems a bit weird. > >> > >> If we use ldapsearch and pass it the same search string and look for > >> the field maxStorage, we clearly see the field and the value being > >> returned. ?The result looks the same if we also lookup instanceType. > >> > >> We're using Dovecot 2.0.9. > >> > >> Does anyone have any idea as to why we can't use this field? > >> > >> Thanks, > >> > >> Andrew > > Tried your suggestion Nikita, no joy unfortunately. ?It still looks > like the value never gets returned from the LDAP server to Dovecot. > It definitely has something in the field (equivalent of 10GB, but in > bytes as suggested) and I changed the user_attrs also, but still get > the same "no fields returned by the server" error message. > > Modifying the user_attrs to lookup from a different field > (instanceType) definitely works. > > What exact version are you using - perhaps it's a problem with our > copy of 2.0.9. > > Thanks, > > Andrew Show your full dovecot-ldap.conf file, also what port do you using ? maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) And show exact result of ldapsearch tool, binding under user from dovecot-ldap.conf + debug for this user when it trying to login and 'doveadm -D quota get -u $user' for this one. Also ensure that your search query returns only 1 result. We are using dovecot 2.0.19 now, but all versions of dovecot 2.0 branch was there in the past. I'm updating server since version 2.0.1 - no problem found. From janfrode at tanso.net Wed Mar 28 10:24:07 2012 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 28 Mar 2012 09:24:07 +0200 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <1332888019.29480.17.camel@sally> References: <1332888019.29480.17.camel@sally> Message-ID: On Wed, Mar 28, 2012 at 12:40 AM, Jeff Gustafson wrote: > ? ? ? ?I looked around the 'Net to see if there might be a custom program for > offline Maildir to mdbox conversion. So far I haven't turned up > anything. The problem for us is that the dsync program simply takes a > lot of time to convert mailboxes. I wonder if time could be saved with a > program that is optimized to convert mailboxes without the fancy locking > that dsync needs to do. Does have (or seen) a tool that could do this? Why is it a problem that dsync takes a long time, when it can be done without downtime for the users? I just started our maildir->mdbox convertion yesterday, using the attached script. I only converted a little over 10000 easy accounts (accounts with simple folder names, as I expect to run into problems once we start hitting accounts with trailing dot or broken latin1/utf8 characters in the folder names). I might agree it wasn't quick, but that really doesn't matter as the only downtime for the user is that he's potentially kicked out during the userdb update. -jf > ? ? ? ?We're hoping that converting away from Maildir will help us speed up > the backup processes by reducing the number of files to process. > -------------- next part -------------- A non-text attachment was scrubbed... Name: migrer-til-mdbox.sh Type: application/x-sh Size: 2131 bytes Desc: not available URL: From tomislav.mihalicek at gmail.com Wed Mar 28 11:57:07 2012 From: tomislav.mihalicek at gmail.com (Tomislav Mihalicek) Date: Wed, 28 Mar 2012 01:57:07 -0700 (PDT) Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage Message-ID: <33544816.post@talk.nabble.com> Hi Could someone explain what this strings mean in dovecot 2.1.3 debug log? Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 test1 at example.net service=lib-storage Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 test2 at example.net service=lib-storage -- View this message in context: http://old.nabble.com/Shared-mailboxes-with-dovecot-problem-service%3Dlib-storage-tp33544816p33544816.html Sent from the Dovecot mailing list archive at Nabble.com. From mafonso at hangas.net Wed Mar 28 13:13:17 2012 From: mafonso at hangas.net (Hangas) Date: Wed, 28 Mar 2012 10:13:17 +0000 (UTC) Subject: [Dovecot] dbox vs. mdbox References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: Timo Sirainen iki.fi> writes: > > 4. Are there real-world benchmarks showing measurable differences between > >maildir, sdbox mdbox? > > Not that I'm aware of. So far everyone I've tried to ask have replaced their > whole mail system and their storage, so the before/after numbers can't be > compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. The environment will be virtually the same as this is running in a virtualized environment. I'm keeping the same storage and storage network, same host hardware and about the same VM specs. Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information. My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly From campbell at cnpapers.com Wed Mar 28 17:10:32 2012 From: campbell at cnpapers.com (Steve Campbell) Date: Wed, 28 Mar 2012 10:10:32 -0400 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: <4F731BD8.8050307@cnpapers.com> On 3/27/2012 10:40 AM, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The > client allows me to specify the root folder, but not a prefix or > namespace. I'm still struggling with some users on our new server that > have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix > definitions on the imap server? > > On some clients, like Thunderbird, I have the option of specifying > namespace OR prefix. How do these differ? I thought that the prefix > was the "name" of the namespace. > > It appears that I have to delete and re-create the account on these OE > 6 clients to make the list of folders show properly. Does that sound > right? > > This all came about because one of these OE 6 users was not able to > use their imap folders (server errors). Turns out it was one of the > users that had their folders directly under ~. So I moved them to > ~/mail, created a .subscriptions file from their .mailboxlist file and > tried everything in the world to get the folders to list properly. > Only after specifying the root folder as ~/mail after recreating the > account and restarting OE did it show properly and the folders > remained listed. My default config has this setup as the > "mail_location" parm, but blanks as the root folder don't seem to work > in this situation. I'm also wondering where I specify the "list", > "hidden" and other parms that are usually set in namespace blocks. > > dovecot -n > # 2.0.9: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 CentOS release 6.2 (Final) > disable_plaintext_auth = no > listen = * > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } > namespace { > hidden = yes > list = no > location = > prefix = ~%u/mail/ > separator = / > type = private > } > passdb { > driver = pam > } > protocols = pop3 imap > ssl_cert = ssl_key = userdb { > driver = passwd > } > protocol pop3 { > pop3_uidl_format = %08Xu%08Xv > } > > > Thanks > > steve campbell > > > > Thanks > > steve campbell > > After googling a bit, it seems that all 3 can come into play in the same or different meanings. Seems that prefix and namespace mean the same thing. Root folder can mean the same as above, but can also stand alone as an individual pointer to a personal folder that differs from from what the imap server uses. It's still not clear to me, but at least I'm getting an idea of what may or may not work. Still not sure why the null or blank prefixed namespace doesn't take precedence when nothing is set in the client. steve From kiwi at oav.net Wed Mar 28 18:50:54 2012 From: kiwi at oav.net (Xavier Beaudouin) Date: Wed, 28 Mar 2012 17:50:54 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. Message-ID: <4F73335E.2070800@oav.net> Hi there, I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. One thing used by some of powerusers are archiving mail automatically with autocreated folders based on year + month. Is there any good way to make that with sieve... One example require "fileinto"; if address :is ["From", "To"] "dovecot at dovecot.org" { fileinto "INBOX.mls.%Y.%m.dovecot"; } This will fill any mails into INBOX.mls.2012.03.dovecot uppon receiving... I don't know if some sieve guru can tell me how to do that... ? Kind regards, Xavier From stephan at rename-it.nl Wed Mar 28 19:04:48 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:04:48 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F7336A0.3070202@rename-it.nl> Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? require ["variables","date","fileinto","mailbox"]; # Extract date info if currentdate :matches "year" "*" { set "year" "${1}"; } if currentdate :matches "month" "*" { set "month" "${1}"; } # Archive Dovecot mailing list items by year and month. # Create folder when it does not exist. if header :is "list-id" "dovecot.dovecot.org" { fileinto :create "INBOX.mls.${year}.${month}.dovecot"; } The above also uses a more reliable way to detect the Dovecot mailinglist. Regards, Stephan From stan at hardwarefreak.com Wed Mar 28 19:07:59 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Wed, 28 Mar 2012 11:07:59 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332881861.29480.8.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> Message-ID: <4F73375F.3070200@hardwarefreak.com> On 3/27/2012 3:57 PM, Jeff Gustafson wrote: > We do have a FC system that another department is using. The company > dropped quite a bit of cash on it for a specific purpose. Our department > does not have access it to. People are somewhat afraid of iSCSI around > here because they believe it will add too much latency to the overall IO > performance. They're a big believer in locally attached disks. Less > features, but very good performance. If you use a software iSCSI initiator with standard GbE ports, block IO latency can become a problem, but basically in only 3 scenarios: 1. Slow CPUs or not enough CPUs/cores. This is unlikely to be a problem in 2012, given the throughput of today's multi-core CPUs. Low CPU throughput hasn't generally been the cause of software iSCSI initiator latency problems since pre-2007/8 with most applications. I'm sure some science/sim apps that tax both CPU and IO may have still had issues. Those would be prime candidates for iSCSI HBAs. 2. An old OS kernel that doesn't thread IP stack, SCSI encapsulation, and/or hardware interrupt processing amongst all cores. Recent Linux kernels do this rather well, especially with MSI-X enabled, older ones not so well. I don't know about FreeBSD, Solaris, AIX, HP-UX, Windows, etc. 3. System under sufficiently high CPU load to slow IP stack and iSCSI encapsulation processing, and or interrupt handling. Again, with today's multi-core fast CPUs this probably isn't going to be an issue, especially given that POP/IMAP are IO latency bound, not CPU bound. Most people running Dovecot today are going to have plenty of idle CPU cycles to perform the additional iSCSI initiator and TCP stack processing without introducing undue block IO latency effects. As always, YMMV. The simply path is to acquire your iSCSI SAN array and use software initiators on client hosts. In the unlikely event you do run into block IO latency issues, you simply drop an iSCSI HBA into each host suffering the latency. They run ~$700-900 USD each for single port models, and they eliminate block IO latency completely, which is one reason they cost so much. They have an onboard RISC chip and memory doing the TCP and SCSI encapsulation processing. They also give you the ability to boot diskless servers from LUNs on the SAN array. This is very popular with blade server systems, and I've done this many times myself, albeit with fibre channel HBAs/SANs, not iSCSI. Locally attached/internal/JBOD storage typically offers the best application performance per dollar spent, until you get to things like backup scenarios, where off node network throughput is very low, and your backup software may suffer performance deficiencies, as is the issue titling this thread. Shipping full or incremental file backups across ethernet is extremely inefficient, especially with very large filesystems. This is where SAN arrays with snapshot capability come in really handy. The snap takes place wholly within the array and is very fast, without the problems you see with host based snapshots such as with Linux LVM, where you must first freeze the filesystem, wait for the snapshot to complete, which could be a very long time with a 1TB FS. While this occurs your clients must wait or timeout while trying to access mailboxes. With a SAN array snapshot system this isn't an issue as the snap is transparent to hosts with little or no performance degradation during the snap. Two relatively inexpensive units that have such snapshot capability are: http://www.equallogic.com/products/default.aspx?id=10613 http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or more host/network connection ports when equipped with dual controllers. There are many other vendors with similar models/capabilities. I mention these simply because Dell/HP are very popular and many OPs are already familiar with their servers and other products. > We thought ZFS would provide us with a nice snapshot and backup system > (with zfs send). We never got that far once we discovered that ZFS > doesn't work very well in this context. Running rsync on it gave us > terrible performance. There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, Linux FUSE. Which were you using? If the last, that would fully explain the suck. >> Also, you speak of a very large maildir store, with hundreds of >> thousands of directories, obviously many millions of files, of 1TB total >> size. Thus I would assume you have many thousands of users, if not 10s >> of thousands. >> >> It's a bit hard to believe you're not running XFS on your storage, given >> your level of parallelism. You'd get much better performance using XFS >> vs EXT4. Especially with kernel 2.6.39 or later which includes the >> delayed logging patch. This patch increases metadata write throughput >> by a factor of 2-50+ depending on thread count, and decreases IOPS and >> MB/s hitting the storage by about the same factor, depending on thread >> count. > > I've relatively new here, but I'll ask around about XFS and see if > anyone had tested it in the development environment. If they'd tested it properly, and relatively recently, I would think they'd have already replaced EXT4 on your Dovecot server. Unless others factors prevented such a migration. Or unless I've misunderstood the size of your maildir workload. -- Stan From gfinch at ldmltd.ca Wed Mar 28 19:08:07 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:08:07 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <4F733767.60003@ldmltd.ca> On 2012-03-28 8:50 AM, Xavier Beaudouin wrote: > Hi there, > > I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. > > One thing used by some of powerusers are archiving mail automatically > with autocreated folders based on year + month. > > Is there any good way to make that with sieve... > > One example > > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... > > I don't know if some sieve guru can tell me how to do that... ? > > Kind regards, > > Xavier I don't remember where I found out how to do this, but the following is what I use: require ["fileinto", "imap4flags", "date", "variables"]; if currentdate :matches "month" "*" { set "month" "${1}"; } if currentdate :matches "year" "*" { set "year" "${1}"; } fileinto :flags "\\seen" "${year}-${month}"; Deliver/lmtp is set to allow creation of folders. -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From gfinch at ldmltd.ca Wed Mar 28 19:26:25 2012 From: gfinch at ldmltd.ca (Gregory Finch) Date: Wed, 28 Mar 2012 09:26:25 -0700 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F7336A0.3070202@rename-it.nl> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> Message-ID: <4F733BB1.5060804@ldmltd.ca> On 2012-03-28 9:04 AM, Stephan Bosch wrote: > Op 3/28/2012 5:50 PM, Xavier Beaudouin schreef: >> Hi there, >> >> I am trying to move from lmtpd (lmtpd.sf.net) to dovecot sieve. >> >> One thing used by some of powerusers are archiving mail automatically >> with autocreated folders based on year + month. >> >> Is there any good way to make that with sieve... >> >> One example >> >> require "fileinto"; >> >> if address :is ["From", "To"] "dovecot at dovecot.org" { >> fileinto "INBOX.mls.%Y.%m.dovecot"; >> } >> >> This will fill any mails into INBOX.mls.2012.03.dovecot uppon >> receiving... >> >> I don't know if some sieve guru can tell me how to do that... ? > > require ["variables","date","fileinto","mailbox"]; > > # Extract date info > if currentdate :matches "year" "*" { set "year" "${1}"; } > if currentdate :matches "month" "*" { set "month" "${1}"; } > > # Archive Dovecot mailing list items by year and month. > # Create folder when it does not exist. > if header :is "list-id" "dovecot.dovecot.org" { > fileinto :create "INBOX.mls.${year}.${month}.dovecot"; > } > > > The above also uses a more reliable way to detect the Dovecot > mailinglist. > > > Regards, > > Stephan Stephan, Is the "mailbox" extension the one that lets "fileinto" use ":create"? I've had a hard time trying to find a useful sieve reference that I can understand. Thank you, -Greg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From stephan at rename-it.nl Wed Mar 28 19:31:31 2012 From: stephan at rename-it.nl (Stephan Bosch) Date: Wed, 28 Mar 2012 18:31:31 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F733BB1.5060804@ldmltd.ca> References: <4F73335E.2070800@oav.net> <4F7336A0.3070202@rename-it.nl> <4F733BB1.5060804@ldmltd.ca> Message-ID: <4F733CE3.4050101@rename-it.nl> Op 3/28/2012 6:26 PM, Gregory Finch schreef: > On 2012-03-28 9:04 AM, Stephan Bosch wrote: >> >> require ["variables","date","fileinto","mailbox"]; >> >> # Extract date info >> if currentdate :matches "year" "*" { set "year" "${1}"; } >> if currentdate :matches "month" "*" { set "month" "${1}"; } >> >> # Archive Dovecot mailing list items by year and month. >> # Create folder when it does not exist. >> if header :is "list-id" "dovecot.dovecot.org" { >> fileinto :create "INBOX.mls.${year}.${month}.dovecot"; >> } >> >> >> The above also uses a more reliable way to detect the Dovecot >> mailinglist. > Stephan, > > Is the "mailbox" extension the one that lets "fileinto" use ":create"? > I've had a hard time trying to find a useful sieve reference that I can > understand. > > Thank you, Yes: http://tools.ietf.org/html/rfc5490#section-3.2 You can find links to specifications of the various Sieve extensions implemented for Pigeonhole here: http://pigeonhole.dovecot.org Regards, Stephan. From ncjeffgus at zimage.com Wed Mar 28 23:54:01 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:54:01 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F73375F.3070200@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> Message-ID: <1332968041.26122.19.camel@sally> On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > Locally attached/internal/JBOD storage typically offers the best > application performance per dollar spent, until you get to things like > backup scenarios, where off node network throughput is very low, and > your backup software may suffer performance deficiencies, as is the > issue titling this thread. Shipping full or incremental file backups > across ethernet is extremely inefficient, especially with very large > filesystems. This is where SAN arrays with snapshot capability come in > really handy. I'm a new employee at the company. I was a bit surprised they were not using iSCSI. They claim they just can't risk the extra latency. I believe that you are right. It seems to me that offloading snapshots and backups to an iSCSI SAN would improve things. The problem is that this company has been burned on storage solutions more than once and they are a little skeptical that a product can scale to what they need. There are some SAN vendor names that are a four letter word here. So far, their newest FC SAN is performing well. I think having more, small, iSCSI boxes would be a good solution. One problem I've seen with smaller iSCSI products is that feature sets like snapshotting are not the best implementation. It works, but doing any sort of automation can be painful. > The snap takes place wholly within the array and is very fast, without > the problems you see with host based snapshots such as with Linux LVM, > where you must first freeze the filesystem, wait for the snapshot to > complete, which could be a very long time with a 1TB FS. While this > occurs your clients must wait or timeout while trying to access > mailboxes. With a SAN array snapshot system this isn't an issue as the > snap is transparent to hosts with little or no performance degradation > during the snap. Two relatively inexpensive units that have such > snapshot capability are: How does this work? I've always had Linux create a snapshot. Would the SAN doing a snapshot without any OS buy-in cause the filesystem to be saved in an inconsistent state? I know that ext4 is pretty good at logging, but still, wouldn't this be a problem? > > http://www.equallogic.com/products/default.aspx?id=10613 > > http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html > > The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be > had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or > more host/network connection ports when equipped with dual controllers. > There are many other vendors with similar models/capabilities. I > mention these simply because Dell/HP are very popular and many OPs are > already familiar with their servers and other products. I will take a look. I might have some convincing to do. > There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, > Linux FUSE. Which were you using? If the last, that would fully > explain the suck. There is one more that I had never used before coming on board here: ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My understanding is that they were using it on the backup machines with the front end dovecot machines using ext4. I'm told the metadata issue is a ZFS thing and they have the same problem on Solaris/Nexenta. > > I've relatively new here, but I'll ask around about XFS and see if > > anyone had tested it in the development environment. > > If they'd tested it properly, and relatively recently, I would think > they'd have already replaced EXT4 on your Dovecot server. Unless others > factors prevented such a migration. Or unless I've misunderstood the > size of your maildir workload. I don't know the entire history of things. I think they really wanted to use ZFS for everything and then fell back to ext4 because it performed well enough in the cluster. Performance becomes an issue with backups using rsync. Rsync is faster than Dovecot's native dsync by a very large margin. I know that dsync is doing more than rsync, but still, seconds compared to over five minutes? That is a significant difference. The problem is that rsync can't get a perfect backup. ...Jeff From ncjeffgus at zimage.com Wed Mar 28 23:58:38 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 13:58:38 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: References: <1332888019.29480.17.camel@sally> Message-ID: <1332968318.26122.22.camel@sally> On Wed, 2012-03-28 at 09:24 +0200, Jan-Frode Myklebust wrote: > Why is it a problem that dsync takes a long time, when it can be done > without downtime for the users? > > I just started our maildir->mdbox convertion yesterday, using the > attached script. I only converted a little over 10000 easy accounts > (accounts with simple folder names, as I expect to run into problems > once we start hitting accounts with trailing dot or broken latin1/utf8 > characters in the folder names). I might agree it wasn't quick, but > that really doesn't matter as the only downtime for the user is that > he's potentially kicked out during the userdb update. I looked over your script. I plan on doing some trial runs with it. I think the trick where you re-run the sync and then boot the user off the connection should work pretty well. I hadn't totally fleshed out the scripting on the conversion since there is a lot more I need to do with the database and configuration files first. It appears I can use your script as a starting point for our configuration. ...Jeff > > > -jf > > > We're hoping that converting away from Maildir will help us speed up > > the backup processes by reducing the number of files to process. > > From ncjeffgus at zimage.com Thu Mar 29 00:01:19 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 14:01:19 -0700 Subject: [Dovecot] Need fast Maildir to mdbox conversion In-Reply-To: <4F727ECE.4050305@r.paypc.com> References: <1332888019.29480.17.camel@sally> <4F727ECE.4050305@r.paypc.com> Message-ID: <1332968479.26122.24.camel@sally> On Tue, 2012-03-27 at 20:00 -0700, Robin wrote: > I'm writing a swiss-army (C-based, no bytecode crap languages) mailbox > "transcoding" tool, since none appear to exist. To keep it simple, I/O > to/from "remote" mailbox (connections) are not pipelined. It won't > require more than MAXEMAILSIZE's worth of RAM (if one of the directions > involves a remote connection), and so far when processing MIX, Maildir, > and Mbox files, it's extremely fast. This sounds interesting. If it could so [sm]dbox, it would be very, very useful to large installations. ...Jeff From tss at iki.fi Thu Mar 29 01:30:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:30:34 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <466fcdec099fca4dbdb5b1ce4e40fa49.squirrel@web.miau.ca> Message-ID: On 27.3.2012, at 1.14, Michescu Andrei wrote: > This being said and acknowledged here are my 2 cents: > > I think that the current '1 brain / 2 workers' seems to be the correct > model. The "the client" connects to the "server" and pushes the local > changes and after retrieves the updated/new items from the "server". "The > brain" considers first server as the "local storage" and the second server > as "server storage". This design makes it too easy to design it in a way that adds extra roundtrips = extra latency. It also kind of hides other problems as well. For example now dsync can way too easily just fail if something unexpected happens during dsync (e.g. mailbox gets renamed/deleted). And there are of course some bugs that I don't really understand why some people are seeing them at all. > For the split design, "come to the same conclusion of the state" is very > race-condition prone. It's race-condition prone with the brain design as well. dsync can't just lock the mailbox during its sync, since the sync can take a long time. With a "brainless" design it's clear from the beginning that there are race conditions and they need to be dealt with. From tss at iki.fi Thu Mar 29 01:43:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:43:07 +0300 Subject: [Dovecot] dsync redesign In-Reply-To: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> Message-ID: On 23.3.2012, at 23.25, Timo Sirainen wrote: > and even if you don't understand that, here's another document disguising as an algorithm class problem :) If anyone has thoughts on how to solve it, would be great: > > http://dovecot.org/tmp/dsync-redesign-problem.txt > > It only deals with saving new messages, not expunges/flag changes/etc, but those should be much simpler. Step #3 was more difficult than I first realized. I spent last two days figuring out a way to make it work, and looks like I finally did. I didn't update the document yet, but I wrote a test program: http://dovecot.org/tmp/test-dsync.c Step #2 should be easy enough. Step #4 I think I'll forget about and just implement a per-mailbox dsync lock. The main reason I wanted to get rid of locks was because a per-user lock can't work with shared mailboxes. But a per-mailbox lock is okay enough. Note that #3 allows the two dsyncs to run in parallel and send duplicate changes, just not modifying the same mailbox at the same time (which would duplicate mails due to two transactions adding the same mails). From tss at iki.fi Thu Mar 29 01:52:38 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:52:38 +0300 Subject: [Dovecot] Merge mails from two mail_locations In-Reply-To: <4F71926F.30500@wk-serv.de> References: <4F71926F.30500@wk-serv.de> Message-ID: On 27.3.2012, at 13.11, Patrick Westenberg wrote: > recently I had some trouble with my ocfs2 cluster and it unmounted > itself from /var/mail. > > Unfortunately I received mails while my mailstore was unmounted and some mails are stored in /var/mail on the hosts local harddisk. > > Now I need to merge/move these locally stored mails to my ocfs2 mailstore but I don't know how to do this. You can use "doveadm import" to copy mails from one location to another. From tss at iki.fi Thu Mar 29 01:59:15 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 01:59:15 +0300 Subject: [Dovecot] fts-solr not indexing body content In-Reply-To: References: <1332774418.26095.126.camel@innu> <1332775126.26095.127.camel@innu> Message-ID: <8C621D75-BC36-4C85-B80A-28473279DFAB@iki.fi> On 27.3.2012, at 10.12, B?rd Johannessen wrote: > 2012/3/26 Timo Sirainen : >> Yeah, looks no one has tried to use Solr with Dovecot v2.1 before. This >> should fix it: >> >> http://hg.dovecot.org/dovecot-2.1/rev/bcc5e71650b9 > > Nope; exactly same result; body field contains just the empty line. Always? It worked in my tests, and I don't see anything else wrong in the code.. From tss at iki.fi Thu Mar 29 02:12:13 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:12:13 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332451538.8339.17.camel@sally> References: <1332451538.8339.17.camel@sally> Message-ID: On 22.3.2012, at 23.25, Jeff Gustafson wrote: > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > mdbox:/home/bu/testuser > > real 1m9.519s > user 1m7.592s > sys 0m1.126s Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: - mbox:/tmp/mbox - mbox:/tmp/mbox:INDEX=MEMORY - sdbox:/tmp/sdbox From tss at iki.fi Thu Mar 29 02:25:57 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:25:57 +0300 Subject: [Dovecot] Namespace, prefix questions In-Reply-To: <4F71D14B.2010301@cnpapers.com> References: <4F71D14B.2010301@cnpapers.com> Message-ID: On 27.3.2012, at 17.40, Steve Campbell wrote: > We've got some users who are using Outlook Express version 6. The client allows me to specify the root folder, but not a prefix or namespace. I'm still struggling with some users on our new server that have crazy imap folder layouts, so I've got a few questions. > > When I specify the root folder, does that bypass any namespace/prefix definitions on the imap server? Like you noticed, these are pretty much the same things, since most clients don't understand about namespaces. And Dovecot doesn't "select" a namespace for clients. They mostly affect mailbox listing.. Like when a client asks Dovecot to list mailboxes under foo/, then Dovecot checks if a foo/ namespace exists. > On some clients, like Thunderbird, I have the option of specifying namespace OR prefix. How do these differ? I thought that the prefix was the "name" of the namespace. I have no idea how Thunderbird handles them differently. > It appears that I have to delete and re-create the account on these OE 6 clients to make the list of folders show properly. Does that sound right? Not really. > This all came about because one of these OE 6 users was not able to use their imap folders (server errors). Turns out it was one of the users that had their folders directly under ~. So I moved them to ~/mail, created a .subscriptions file from their .mailboxlist file and tried everything in the world to get the folders to list properly. Only after specifying the root folder as ~/mail after recreating the account and restarting OE did it show properly and the folders remained listed. My default config has this setup as the "mail_location" parm, but blanks as the root folder don't seem to work in this situation. I'm also wondering where I specify the "list", "hidden" and other parms that are usually set in namespace blocks. .. > mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = yes > inbox = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = > separator = / > type = private > } All of your namespaces have hidden=yes. There should be (at least) one with hidden=no (this prefix= namespace in your case). I guess I'll need to add a check to have Dovecot fail if there aren't any. Also it's not necessary to duplicate the "location" setting, since it defaults to the global mail_location. > namespace { > hidden = yes > list = no > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = "#mbox/" > separator = / > type = private > } This namespace is unlikely to be useful to you, unless you actually have some IMAP client configured to use #mbox/ prefix. > namespace { > hidden = yes > list = yes > location = mbox:~/mail:INBOX=/var/spool/mail/%u > prefix = ~/mail/ > separator = / > type = private > } This shouldn't have list=yes .. I'm not sure what even happens with it. I guess if client lists all mailboxes from root you'll get a mailbox named "~", which has a "mail" child, which has all of your mailboxes duplicated. From tss at iki.fi Thu Mar 29 02:30:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:30:51 +0300 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On 28.3.2012, at 13.13, Hangas wrote: > Timo Sirainen iki.fi> writes: > >>> 4. Are there real-world benchmarks showing measurable differences between >>> maildir, sdbox mdbox? >> >> Not that I'm aware of. So far everyone I've tried to ask have replaced their >> whole mail system and their storage, so the before/after numbers can't be >> compared. I'm very interested in knowing myself too. > > I think I can give my contribution here. I'm planning to migrate from dovecot > 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at > the time, but now I'm trying to decide the mailbox format for a 2.x > fresh install. .. > My ideia is to install a fresh server and replicate the production maildir on it > to build a test "source disk" that I'll use then to experiment the conversions > to sdbox and mdbox. > I then plan to test the performance of the dbox formats, but I can include > Maildir measurements just for the record. > > I'm open to suggestions on how to test this properly The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much. From tss at iki.fi Thu Mar 29 02:32:01 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:32:01 +0300 Subject: [Dovecot] Shared mailboxes with dovecot problem service=lib-storage In-Reply-To: <33544816.post@talk.nabble.com> References: <33544816.post@talk.nabble.com> Message-ID: On 28.3.2012, at 11.57, Tomislav Mihalicek wrote: > Could someone explain what this strings mean in dovecot 2.1.3 debug log? > > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 1 > test1 at example.net service=lib-storage > Mar 27 11:18:11 cartman dovecot: auth: Debug: master in: USER 2 > test2 at example.net service=lib-storage Dovecot is asking a user's home directory via userdb lookup. Looks like your userdb isn't returning a home directory. There should be an error message about it? From tss at iki.fi Thu Mar 29 02:34:54 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:34:54 +0300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: References: <20120327091425.73963576@jimbo> Message-ID: On 28.3.2012, at 0.06, Bruce, Andrew wrote: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server .. > Further investigation shows that there are a few other fields that we > can't retrieve in Dovecot, Looks to me like you can't retrieve any fields from LDAP, possibly because the dn user doesn't have access to the information or some other reason. From tss at iki.fi Thu Mar 29 02:37:03 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:37:03 +0300 Subject: [Dovecot] zlib_save per namespace/mailbox? In-Reply-To: References: Message-ID: <28C4EF51-12FB-4B11-A3FB-54949CAF0444@iki.fi> On 27.3.2012, at 19.49, Ben Schumacher wrote: > On Thu, Sep 22, 2011 at 8:44 AM, Lutz Pre?ler wrote: >> the zlib_save question reminds me of a wish: >> I think it's not possible to set zlib_save parameter per namespace (or even >> mailbox). Per namespace would be something for the wish list to get rid of >> the cron job method to compress archival mailboxes. >> And maybe an option to add a "Z" flag to compressed maildir message files >> as recommended in the wiki regarding compress crob job. > > +1 on this request. I have a slightly different use case -- I have > both an dbox and Maildir. Incoming email goes to Maildir, but I > archive off to dbox (using Thunderbird). After I archive my emails, > compression seems like a reasonable choice. > > Any idea if this feature will be available at some point? For v2.2 I'm hoping to have per-namespace mail settings, although I'm not sure if it actually gets implemented - so many other things to do as well. Then (among other things) you could do: namespace foo { plugin { zlib_save = gz } } From tss at iki.fi Thu Mar 29 02:42:50 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:42:50 +0300 Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <4F71F8D8.6040700@wiesinger.com> References: <4F71F8D8.6040700@wiesinger.com> Message-ID: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. doveconf -n output? > 2.0.x: with Prefix ~/Mail > A0003 LIST "" ~/Mail/sent > A0003 OK List completed.: Works with my v2.0.19 config: x list "" ~/Mail/sent * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" x OK List completed. From tss at iki.fi Thu Mar 29 02:46:17 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:46:17 +0300 Subject: [Dovecot] Dovecot / IMAP / New Mails are not shown unless you open the folder in Outlook In-Reply-To: <33544803.post@talk.nabble.com> References: <33544803.post@talk.nabble.com> Message-ID: <009AB5A8-303D-45F9-B1CF-33E26D859B0C@iki.fi> On 28.3.2012, at 1.24, stonegate wrote: > Problem: When i receive a new email, it does not appear in my Outlook unless > i have the IMAP Inbox Folder open (highlighted selection). > > Sometimes i have new mail in my inbox for over 15 Minutes and i dont realize > it unless i click on the inbox folder. > > Before that problem occured on my old system ( i think it was dovecot as > well ) the inbox folder refreshed automatically and i instantly knew when i > had new emails. It was kind of like with my exchange account. > > Does anyone have a clue what might be wrong ? Since i did not re-install > outlook or something it cannot be a client thing since no settings changed > on the client side. > > Maybe i have to do something with dovecots config file? I think it should > keep the connection to the server open or something. I?d appreciate any kind > of help. I'm not aware of any Dovecot change or any Dovecot setting that could change this behavior in Outlook. It's the client's choice how it monitors the mailboxes. From tss at iki.fi Thu Mar 29 02:48:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:48:53 +0300 Subject: [Dovecot] doveadm purge on clusterfs In-Reply-To: <20120327094710.GA10878@dibs.tanso.net> References: <20120327094710.GA10878@dibs.tanso.net> Message-ID: On 27.3.2012, at 12.47, Jan-Frode Myklebust wrote: > Would it be OK to run purge in the pop/imap postlogin scripts? We > already do a conditional: > > test /var/log/activemailaccounts/imap/$USER -ot /var/log/activemailaccounts/today > then > touch /var/log/activemailaccounts/imap/$USER > fi > > so adding a: > > doveadm purge -u $USER > > in this section would make it run once every day the users that log in. > Does that sound like an OK solution? Yeah, should work fine. Or you should make it run in background so user's login won't slow down because of the purging. I'm not sure if simply adding & at the end works (or if the process dies after login is finished, or what happens if it runs over 30 secs which is when normally post-login script is killed by SIGALRM). From tss at iki.fi Thu Mar 29 02:49:56 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:49:56 +0300 Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: <712362F5-EDFF-4BFA-B932-1C6E19855F7D@iki.fi> On 27.3.2012, at 4.16, Joseph Tam wrote: >>> However, I noticed a strange thing: querying what would have been >>> deleted >>> doveadm -ftab fetch -A "date.saved" mailbox Trash savedbefore 7d >>> showed many date.saved values are clustered around the same timestamp, >>> even among different user's Trash mailbox. >>> ... >>> I can't explain why many different users would have messages with the >>> same (or closeby) date.saved value. >> Which mailbox format? With Maildir the date.saved is taken from >> dovecot.index.cache file, and in some cases that might get dropped. If >> it does, then it fallbacks to using the file's ctime. > > mbox. Ah, with mbox there isn't any usable fallback for date.saved. If it's not in dovecot.index.cache, the current time is used. > These "wrong" values shouldn't cause problems with expunge queries since > they err on the side of safety. Right. From ncjeffgus at zimage.com Thu Mar 29 02:51:49 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 16:51:49 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: References: <1332451538.8339.17.camel@sally> Message-ID: <1332978709.26122.29.camel@sally> On Thu, 2012-03-29 at 02:12 +0300, Timo Sirainen wrote: > On 22.3.2012, at 23.25, Jeff Gustafson wrote: > > > [root at n24 bu]# time dsync backup -u testuser at domain.com \ > > mdbox:/home/bu/testuser > > > > real 1m9.519s > > user 1m7.592s > > sys 0m1.126s > > Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: > > - mbox:/tmp/mbox > - mbox:/tmp/mbox:INDEX=MEMORY > - sdbox:/tmp/sdbox My tests show that maildir to mdbox or sdbox backup/conversions take about the same length in time. I noticed maybe a second or two difference between mdbox and sdbox). On a 3.1GB mailbox either one took about 6 minutes. Rsync, on the other hand, took less than a minute. I will re-run the tests with a maildir to maildir backup and see how long it takes. ...Jeff From tss at iki.fi Thu Mar 29 02:53:35 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:53:35 +0300 Subject: [Dovecot] Dovecot 1.2.9 Crash, NFS In-Reply-To: References: <960699EB-4156-421E-8A4C-5FA3E1BC1B02@iki.fi> Message-ID: <2DF0AEB9-7410-473C-8404-1D270711A89B@iki.fi> On 26.3.2012, at 18.25, M?ller Lukas wrote: > Thanks for the quick answer. > > I realised, that the error didn't occur since quite a while, opposed to what our client suggested. > Back then I activated the two workarounds (imap_client_workarounds = outlook-idle delay-newmail) and increased mail_max_userip_connections for IMAP. > > Is it possible that those could have improved the situation? Unlikely. >>> Mar 6 08:26:31 mail02 dovecot: IMAP(user at example.com): fdatasync(/data/vmail/example.com/user/dovecot-uidlist) failed: Input/output error >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1522, file=1326961561.V15I4d8562M567017.mail02:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): Maildir /data/vmail/example.com/user: Expunged message reappeared, giving a new UID (old uid=1523, file=1326705103.V15I90105M613353.mail01:2,Sad) >>> Mar 6 08:42:29 mail02 dovecot: IMAP(user at example.com): /data/vmail/example.com/user/dovecot-uidlist: Duplicate file entry at line 4: 1326961561.V15I4d8562M567017.mail02:2,Sad (uid 1522 -> 1598) > .. > >>> My suspicion/speculation what happens is the following: >>> Multiple users are accessing the Mailbox from their offices (all on the same server), one (or more) uses the Webmail or accesses the Mailbox from a different IP. >>> Somehow this leads to problems with Locks on NFS, which leads to the crash. > >> Yes, most likely this is what's happening. Although your errors are more severe than what normally happens. I guess your NFS server is also partially to blame (microsecond resolution timestamps are at least helpful). > > I had a quick look a tour NFS (NetApp), but didn't find anything useful. > In case the problem persists, I will check with the coworker responsible for NetApp. I guess you just had very bad luck. NetApp has a good NFS implementation and normally works fine with Dovecot. From tss at iki.fi Thu Mar 29 02:57:22 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 02:57:22 +0300 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <4F6F35FA.6050207@Media-Brokers.com> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> Message-ID: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> On 25.3.2012, at 18.12, Charles Marcus wrote: > On 2012-03-24 9:16 AM, Timo Sirainen wrote: >> On 24.3.2012, at 14.54, Charles Marcus wrote: >> >>> On 2012-03-24 8:08 AM, Timo Sirainen wrote: >>>> You can do full backups from a filesystem snapshot, which works >>>> "well enough" (might leave some unused attachments lying around in >>>> some rare cases, but that can also happen if Dovecot crashes/dies). >>> >>> But the problem isn't with backups, but with restores, right? >> >> Ah, right. Then it gets tricky. > > Yeah, I seem to remember it was a comment like that that scared me about enabling it... > > Can you expand on what exactly is 'tricky' about it? Also, have you given any thought to how to eliminate the 'trickiness'? I'm of the old school and like for my backups to not have any 'trickiness' about them - including performing restores... ;) It's easy to restore a full backup. And it's easy to restore specific users if you have the full backup easily accessible (just run doveadm import with proper settings pointing to backup). What's difficult is if you just want to restore a specific user from the backup and can't easily do random access to all files. Then you'll first need to restore the user's dbox files and then somehow figure out which attachments to restore from the SIS directory. >>> Am I correct that enabling SIS as it is currently implemented would >>> break this backup tool? > >> I'm not sure. Are you running rsnapshot on live filesystem or on a >> snapshot? On live filesystem there would be race conditions. > > I've been running it on a live system for a long time, and never had a problem beyond occasional messages like this: > > file has vanished: "/var/vmail/example.com/username/cur/1332602593.Vfe02I9e7acdM308676.myhost.example.com:2," > rsync warning: some files vanished before they could be transferred (code 24) at main.c(1052) [sender=3.0.9] I'd guess that with rsnapshot + Maildir you can get duplicate Maildir files if the rsnapshot is accessing a large maildir at the same time as user is changing a message flag. Dovecot usually notices these duplicates and logs a warning about them. From tss at iki.fi Thu Mar 29 03:06:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 03:06:53 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332978709.26122.29.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> Message-ID: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> On 29.3.2012, at 2.51, Jeff Gustafson wrote: >> Most of the time is spent on usermode CPU code. I doubt the problem is dsync itself, most likely the problem is mdbox's saving code. Or possibly index/cache code. Try the same dsync backup for: >> >> - mbox:/tmp/mbox >> - mbox:/tmp/mbox:INDEX=MEMORY >> - sdbox:/tmp/sdbox > > My tests show that maildir to mdbox or sdbox backup/conversions take > about the same length in time. I noticed maybe a second or two > difference between mdbox and sdbox). On a 3.1GB mailbox either one took > about 6 minutes. Rsync, on the other hand, took less than a minute. I > will re-run the tests with a maildir to maildir backup and see how long > it takes. Try also with INDEX=MEMORY, since the problem may be related to updating the indexes. Another way to test if the problem is dsync or Dovecot's generic mail saving code is to run: time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all Or if it's the mail reading code: time doveadm fetch -u user at domain text all > /dev/null From ncjeffgus at zimage.com Thu Mar 29 03:48:06 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 17:48:06 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> Message-ID: <1332982086.26122.34.camel@sally> On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all This tried to write to /root for some reason and failed (dovecot 2.1.3): # time doveadm -o mail=maildir:/home/bu/test.mdbox import maildir:/home/users/user at domain.com/Maildir "" all doveadm(root): Error: chdir(/root/) failed: Permission denied (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not in group 0(root), dir owned by 0:0 mode=0550) doveadm(root): Error: chdir(/root) failed: Permission denied doveadm(root): Error: Can't find namespace for mailbox Trash doveadm(root): Error: Can't find namespace for mailbox test > Or if it's the mail reading code: > > time doveadm fetch -u user at domain text all > /dev/null This ran quicker than a full dsync. Only 40s for 3.1GB. rsync still beat it clocking in at 16s. I ran the fetch command twice figuring the files would get cached by the OS. ...Jeff From tss at iki.fi Thu Mar 29 04:07:51 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 04:07:51 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332982086.26122.34.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> Message-ID: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> On 29.3.2012, at 3.48, Jeff Gustafson wrote: > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > This tried to write to /root for some reason and failed (dovecot > 2.1.3): > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > maildir:/home/users/user at domain.com/Maildir "" all > doveadm(root): Error: chdir(/root/) failed: Permission denied > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > in group 0(root), dir owned by 0:0 mode=0550) > doveadm(root): Error: chdir(/root) failed: Permission denied > doveadm(root): Error: Can't find namespace for mailbox Trash > doveadm(root): Error: Can't find namespace for mailbox test Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. From ncjeffgus at zimage.com Thu Mar 29 05:07:13 2012 From: ncjeffgus at zimage.com (Jeff Gustafson) Date: Wed, 28 Mar 2012 19:07:13 -0700 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> Message-ID: <1332986833.26122.36.camel@sally> On Thu, 2012-03-29 at 04:07 +0300, Timo Sirainen wrote: > On 29.3.2012, at 3.48, Jeff Gustafson wrote: > > > On Thu, 2012-03-29 at 03:06 +0300, Timo Sirainen wrote: > > > >> time doveadm -o mail=mdbox:/tmp/mdbox import mdbox:/path/to/real/mdbox "" all > > > > This tried to write to /root for some reason and failed (dovecot > > 2.1.3): > > > > # time doveadm -o mail=maildir:/home/bu/test.mdbox import > > maildir:/home/users/user at domain.com/Maildir "" all > > doveadm(root): Error: chdir(/root/) failed: Permission denied > > (euid=10025(vmail) egid=10025(vmail) missing +x perm: /root, we're not > > in group 0(root), dir owned by 0:0 mode=0550) > > doveadm(root): Error: chdir(/root) failed: Permission denied > > doveadm(root): Error: Can't find namespace for mailbox Trash > > doveadm(root): Error: Can't find namespace for mailbox test > > > Maybe -o mail_home=/tmp parameter makes it happier? Or possibly it needs -u user at domain, but I'd test that first with a test account to make sure it doesn't break the mailbox in case the userdb lookup overrides some fields. That fixed some errors, but it still is having some sort of trouble with that command: # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash doveadm(user at domain.com): Error: Can't find namespace for mailbox test ...Jeff From jtam.home at gmail.com Thu Mar 29 05:41:16 2012 From: jtam.home at gmail.com (Joseph Tam) Date: Wed, 28 Mar 2012 19:41:16 -0700 (PDT) Subject: [Dovecot] Many messages clustered around the same date.saved value In-Reply-To: References: Message-ID: Timo Sirainen wrote: >>> Which mailbox format? With Maildir the date.saved is taken from >>> dovecot.index.cache file, and in some cases that might get dropped. If >>> it does, then it fallbacks to using the file's ctime. >> >> mbox. > > Ah, with mbox there isn't any usable fallback for date.saved. If it's > not in dovecot.index.cache, the current time is used. I'm a little confused as to why it needed a fallback. In other words, why wasn't date.saved put into the index as soon as the IMAP operation copied it into "Trash"? If this data isn't set at that time, when does it get instantiated? When I actually ask for it? Joseph Tam From tss at iki.fi Thu Mar 29 07:04:26 2012 From: tss at iki.fi (Timo Sirainen) Date: Thu, 29 Mar 2012 07:04:26 +0300 Subject: [Dovecot] dsync is SLOW compared to rsync In-Reply-To: <1332986833.26122.36.camel@sally> References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> Message-ID: <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> On 29.3.2012, at 5.07, Jeff Gustafson wrote: > That fixed some errors, but it still is having some sort of trouble > with that command: > > # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all > doveadm(user at domain.com): Error: Can't find namespace for mailbox Trash > doveadm(user at domain.com): Error: Can't find namespace for mailbox test Oh, you don't have prefix="" namespace? If you have e.g. prefix="INBOX." namespace then use: time doveadm -o mail=maildir:/home/bu/user.mdbox import -u user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all From lists at wiesinger.com Thu Mar 29 08:25:17 2012 From: lists at wiesinger.com (Gerhard Wiesinger) Date: Thu, 29 Mar 2012 07:25:17 +0200 (CEST) Subject: [Dovecot] Dovecot upgrade from 1.2.x to 2.0.x: roundcube/squirrelmail sent folder doesn't work any more In-Reply-To: <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> References: <4F71F8D8.6040700@wiesinger.com> <95C664AD-0532-4F3B-A2EB-2FD25B79187B@iki.fi> Message-ID: On Thu, 29 Mar 2012, Timo Sirainen wrote: > On 27.3.2012, at 20.28, Gerhard Wiesinger wrote: > >> After upgrading from 1.2.x to 2.0.x I'm having problems using sent folder in Webmail applications like roundcube mail and squirrelmail. > > doveconf -n output? > >> 2.0.x: with Prefix ~/Mail >> A0003 LIST "" ~/Mail/sent >> A0003 OK List completed.: > > Works with my v2.0.19 config: > > x list "" ~/Mail/sent > * LIST (\NoInferiors \UnMarked) "/" "~/Mail/sent" > x OK List completed. # 2.0.19: /etc/dovecot/dovecot.conf # OS: cutted for security reasons listen = * mail_full_filesystem_access = yes mail_location = mbox:~:INBOX=/var/mail/%u mbox_lazy_writes = no mbox_write_locks = fcntl passdb { driver = pam } protocols = imap service auth { unix_listener /var/run/dovecot-auth-master { group = users mode = 0660 } user = root } service imap { executable = imap postlogin } service postlogin { executable = script-login -d rawlog } ssl_cert = References: <1332451538.8339.17.camel@sally> <1332978709.26122.29.camel@sally> <5A878071-B180-4980-9B2C-698C50FBA6E5@iki.fi> <1332982086.26122.34.camel@sally> <6954F40E-1BDD-487C-B667-DF68F47E4AB2@iki.fi> <1332986833.26122.36.camel@sally> <69F562BD-91A5-4482-B735-EC0A3358C0E1@iki.fi> Message-ID: <405020e5dfb341332e535e905ff183c3@alpha.zimage.com> On Thu, 29 Mar 2012 07:04:26 +0300, Timo Sirainen wrote: > On 29.3.2012, at 5.07, Jeff Gustafson wrote: > >> That fixed some errors, but it still is having some sort of trouble >> with that command: >> >> # time doveadm -o mail=maildir:/home/bu/user.mdbox import -u >> user at domain.com maildir:/home/users/user%domain.com/Maildir/ "" all >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> Trash >> doveadm(user at domain.com): Error: Can't find namespace for mailbox >> test > > Oh, you don't have prefix="" namespace? If you have e.g. > prefix="INBOX." namespace then use: > > time doveadm -o mail=maildir:/home/bu/user.mdbox import -u > user at domain maildir:/home/users/user%domain.com/Maildir/ INBOX all Oh! I should have known that was the problem. This was very, very fast. This test is maildir to maildir: # time doveadm -o mail=maildir:/home/bu/test import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m0.412s user 0m0.036s sys 0m0.088s But it was just as slow to import into mdbox: # time doveadm -o mail=mdbox:/home/bu/test2 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 7m12.738s user 6m46.161s sys 0m7.046s mbox... still pretty fast: # time doveadm -o mail=mbox:/home/bu/test3 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 0m58.534s user 0m52.264s sys 0m5.762s sdbox seems a little on the slow side too: # time doveadm -o mail=sdbox:/home/bu/test4 import -u user at domain.com maildir:/home/users/user%domain.com/Maildir INBOX all real 6m11.616s user 6m6.924s sys 0m4.579s Does information help? It seems that [sm]dbox is on the slow side for the purpose of doing backups. ...Jeff From fabio.ferrari at unimore.it Thu Mar 29 11:15:32 2012 From: fabio.ferrari at unimore.it (FABIO FERRARI) Date: Thu, 29 Mar 2012 10:15:32 +0200 (CEST) Subject: [Dovecot] Problem about dovecot Panic Message-ID: Good morning, we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some mail services in it (postfix and dovecot 2). The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). >From last week we have this dovecot problem: suddenly dovecot doesn't accept any new connections, the dovecot.log file reports lines like these Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ libdovecot.so.0 [0x36ea4362e8] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) [0x36ea440b7f] -> /usr/li b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ imap [0x408179] Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 killed with signal 6 (core dumps disabled) Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command startup failed, throttling Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected too early Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: Invalid argument and the kern.log file reports Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI failure DID:010400 Status:x9/x32900 Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 can you help us? thanks in advance Fabio Ferrari From javierdemiguel at us.es Thu Mar 29 11:18:36 2012 From: javierdemiguel at us.es (=?UTF-8?Q?Javier_Miguel_Rodr=C3=ADguez?=) Date: Thu, 29 Mar 2012 10:18:36 +0200 Subject: [Dovecot] Problem about dovecot Panic In-Reply-To: References: Message-ID: <5ae4435a57981464edec3590216c7b41@us.es> We had the same problem. Reboot with an older kernel (2.6.18-274.17.1.el5 works for us). It is known bug of RHEL, see this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=681578 Regards Javier On Thu, 29 Mar 2012 10:15:32 +0200 (CEST), FABIO FERRARI wrote: > Good morning, > we have 2 Redhat Enterprise 5.7 machines, they are a cluster with some > mail services in it (postfix and dovecot 2). > > The version of dovecot is dovecot-2.0.1-1_118.el5 (installed via rpm). > > From last week we have this dovecot problem: suddenly dovecot doesn't > accept any new connections, the dovecot.log file reports lines like these > > Mar 15 12:38:54 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > Mar 15 12:38:54 secchia dovecot: imap: Error: Raw backtrace: > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436de0] -> > /usr/lib64/dovecot/libdovecot.so.0 [0x36ea436e3a] -> /usr/lib64/dovecot/ > libdovecot.so.0 [0x36ea4362e8] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_add+0x118) > [0x36ea441498] -> /usr/lib64/dovecot/libdovecot.so.0(io_add+0x8f) > [0x36ea440b7f] -> /usr/li > b64/dovecot/libdovecot.so.0(master_service_init_finish+0x1c6) > [0x36ea430c16] -> dovecot/imap(main+0x10a) [0x41773a] -> > /lib64/libc.so.6(__libc_start_main+0xf4) [0x36ea01d994] -> dovecot/ > imap [0x408179] > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): child 14514 > killed with signal 6 (core dumps disabled) > Mar 15 12:38:54 secchia dovecot: master: Error: service(imap): command > startup failed, throttling > Mar 15 12:39:50 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:51 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:52 secchia dovecot: imap-login: Error: master(imap): Auth > request timed out (received 0/12 bytes) > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:53 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap-login: Error: net_connect_unix(imap) > failed: Resource temporarily unavailable > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:54 secchia dovecot: imap: Error: Login client disconnected > too early > Mar 15 12:39:55 secchia dovecot: imap: Panic: epoll_ctl(add, 5) failed: > Invalid argument > > and the kern.log file reports > > Mar 15 12:38:52 secchia kernel: dlm: closing connection to node 1 > Mar 15 12:39:04 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:39:04 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:14 secchia kernel: lpfc 0000:03:00.0: 0:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:41:15 secchia kernel: lpfc 0000:83:00.0: 1:(0):2753 PLOGI > failure DID:010400 Status:x9/x32900 > Mar 15 12:42:11 secchia kernel: dlm: got connection from 1 > > can you help us? > > thanks in advance > > Fabio Ferrari From c at roessner-network-solutions.com Thu Mar 29 13:24:18 2012 From: c at roessner-network-solutions.com (=?iso-8859-1?Q?Christian_R=F6=DFner?=) Date: Thu, 29 Mar 2012 12:24:18 +0200 Subject: [Dovecot] File/folder permission issues in 2.1.3 Message-ID: <8B296F70-22B8-487B-AD7A-47BEB8C84F62@roessner-network-solutions.com> Hi, I figured out that Dovecot does not honer secondary groups with auth/auth-worker (??), if doing LDAP/TLS stuff. I had to use file system acls to add the user "vmail" to /etc/ssl/private and to the corresponding key file: doveconf -n # 2.1.3: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-40-generic-pae i686 Ubuntu 10.04.4 LTS auth_master_user_separator = * auth_mechanisms = plain login auth_verbose = yes hostname = mail.roessner-net.de lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_access_groups = vmail mail_gid = vmail mail_location = mdbox:~/mdbox mail_plugins = autocreate quota acl fts fts_solr zlib mail_log notify mail_privileged_group = mail mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = mdbox:%%h/mdbox prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox junkmail { special_use = \Junk } prefix = separator = / type = private } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_shared_dict = file:/var/mail/virtual/shared-mailboxes.db autocreate = Trash autocreate2 = Sent autocreate3 = Drafts autocreate4 = junkmail autosubscribe = Trash autosubscribe2 = Sent autosubscribe3 = Drafts autosubscribe4 = junkmail fts = solr fts_solr = break-imap-search url=http://localhost:8080/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:User quota::file:%h/mdbox/dovecot-quota quota_rule = *:storage=300M:messages=20000 quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve zlib_save = gz zlib_save_level = 6 } protocols = imap pop3 lmtp sieve service auth-worker { unix_listener auth-worker { user = vmail } user = vmail } service auth { unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = ::1 port = 24 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_ca = From mafonso at hangas.net Thu Mar 29 14:16:35 2012 From: mafonso at hangas.net (Miguel Afonso) Date: Thu, 29 Mar 2012 12:16:35 +0100 Subject: [Dovecot] dbox vs. mdbox In-Reply-To: References: <4D69FCB5.3090100@wildgooses.com> <58BED0A6-9EA0-4B4D-9065-69B22033D627@iki.fi> Message-ID: On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen wrote: > > The main problem is that it's difficult to do any "real world" tests with > IMAP, especially when users are using many different kinds of IMAP clients. > So I'm very interested in hearing some numbers (and disk IO graphs for a > few weeks would be great) before your migration and after your migration, > but the numbers for your tests might not mean all that much. I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days. I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards. From me at junc.org Thu Mar 29 14:57:39 2012 From: me at junc.org (Benny Pedersen) Date: Thu, 29 Mar 2012 13:57:39 +0200 Subject: [Dovecot] Sieve fileinto and year/month folders. In-Reply-To: <4F73335E.2070800@oav.net> References: <4F73335E.2070800@oav.net> Message-ID: <47266fb4a9b1a50c72ab892ac67d9744@junc.org> Den 2012-03-28 17:50, Xavier Beaudouin skrev: > require "fileinto"; > > if address :is ["From", "To"] "dovecot at dovecot.org" { > fileinto "INBOX.mls.%Y.%m.dovecot"; > } > is this valid sieve ? > This will fill any mails into INBOX.mls.2012.03.dovecot uppon > receiving... not all sieve have date support, and imho no one have macro supported > I don't know if some sieve guru can tell me how to do that... ? why not keep lmtp ? :) http://sieve.info From stan at hardwarefreak.com Thu Mar 29 15:24:05 2012 From: stan at hardwarefreak.com (Stan Hoeppner) Date: Thu, 29 Mar 2012 07:24:05 -0500 Subject: [Dovecot] dsync redesign In-Reply-To: <1332968041.26122.19.camel@sally> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> Message-ID: <4F745465.1030304@hardwarefreak.com> On 3/28/2012 3:54 PM, Jeff Gustafson wrote: > On Wed, 2012-03-28 at 11:07 -0500, Stan Hoeppner wrote: > >> Locally attached/internal/JBOD storage typically offers the best >> application performance per dollar spent, until you get to things like >> backup scenarios, where off node network throughput is very low, and >> your backup software may suffer performance deficiencies, as is the >> issue titling this thread. Shipping full or incremental file backups >> across ethernet is extremely inefficient, especially with very large >> filesystems. This is where SAN arrays with snapshot capability come in >> really handy. > > I'm a new employee at the company. I was a bit surprised they were not > using iSCSI. They claim they just can't risk the extra latency. I The tiny amount of extra latency using a software initiator is a non argument for a mail server workload, unless the server is undersized for the workload--high CPU load and low memory constantly. As I said, in that case you drop in an iSCSI HBA and eliminate any possibility of block latency. > believe that you are right. It seems to me that offloading snapshots and > backups to an iSCSI SAN would improve things. If you get the right unit you won't understand how you ever lived without it. The snaps complete transparently, and the data is on the snap LUN within a few minutes, depending on the priority you give to internal operations, snaps/rebuilds/etc, vs external IO requests. Depending on model > The problem is that this > company has been burned on storage solutions more than once and they are > a little skeptical that a product can scale to what they need. There are More than once? More than once?? Hmm... > some SAN vendor names that are a four letter word here. So far, their > newest FC SAN is performing well. Interesting. Care to name them (off list)? > I think having more, small, iSCSI boxes would be a good solution. One > problem I've seen with smaller iSCSI products is that feature sets like > snapshotting are not the best implementation. It works, but doing any > sort of automation can be painful. As is most often the case, you get what you pay for. >> The snap takes place wholly within the array and is very fast, without >> the problems you see with host based snapshots such as with Linux LVM, >> where you must first freeze the filesystem, wait for the snapshot to >> complete, which could be a very long time with a 1TB FS. While this >> occurs your clients must wait or timeout while trying to access >> mailboxes. With a SAN array snapshot system this isn't an issue as the >> snap is transparent to hosts with little or no performance degradation >> during the snap. Two relatively inexpensive units that have such >> snapshot capability are: > > How does this work? I've always had Linux create a snapshot. Would the > SAN doing a snapshot without any OS buy-in cause the filesystem to be > saved in an inconsistent state? I know that ext4 is pretty good at > logging, but still, wouldn't this be a problem? Instead of using "SAN" as a generic term for a "box", which it is not, please use the terms "SAN" for "storage area network", "SAN array" or "SAN controller" when talking about a box with or without disks that performs the block IO shipping and other storage functions, "SAN switch" for a fiber channel switch, or ethernet switch dedicated to the SAN infrastructure. The acronym "SAN" is an umbrella covering many different types of hardware and network topologies. It drives me nuts when people call a fiber channel or iSCSI disk array a "SAN". These can be part of a SAN, but are not themselves, a SAN. If they are direct connected to a single host they are simple disk arrays, and the word "SAN" isn't relevant. Only uneducated people, or those who simply don't care to be technically correct, call a single intelligent disk box a "SAN". Ok, end rant on "SAN". Read this primer from Dell: http://files.accord.com.au/EQL/Docs/CB109_Snapshot_Basic.pdf The snapshots occur entirely at the controller/disk level inside the box. This is true of all SAN units that offer snap ability. No host OS involvement at all in the snap. As I previously said, It's transparent. Snaps are filesystem independent, and are point-in-time, or PIT copies of one LUN to another. Read up on "LUN" if you're not familiar with the term. Everything in SAN storage is based on LUNs. Now, as the document above will tell you, array based snapshots may or may not be a total backup solution for your environment. You need to educate yourself and see if this technology is a feature that fits your file backup and disaster avoidance and recovery needs. >> http://www.equallogic.com/products/default.aspx?id=10613 >> >> http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241493.html >> >> The Equallogic units are 1/10 GbE iSCSI only IIRC, whereas the HP can be >> had in 8Gb FC, 1/10Gb iSCSI, or 6Gb direct attach SAS. Each offer 4 or >> more host/network connection ports when equipped with dual controllers. >> There are many other vendors with similar models/capabilities. I >> mention these simply because Dell/HP are very popular and many OPs are >> already familiar with their servers and other products. > > I will take a look. I might have some convincing to do. SAN array features/performance are an easy sell. Price not so much. Each fully loaded ~24 drive SAN array is going to run you between $15k-30k USD depending on the vendor and how many spindles you need for IOPS, disk size for total storage, snap/replication features you need, expandability, etc. >> There are 3 flavors of ZFS: native Oracle Solaris, native FreeBSD, >> Linux FUSE. Which were you using? If the last, that would fully >> explain the suck. > > There is one more that I had never used before coming on board here: > ZFSonLinux. ZFSonLinux is a real kernel level fs plugin. My It's a "roll your own" patch set not in mainline and not supported by any Linux distro/vendor, AFAIK. Which is why I didn't include it. > understanding is that they were using it on the backup machines with the > front end dovecot machines using ext4. I'm told the metadata issue is a > ZFS thing and they have the same problem on Solaris/Nexenta. I've never used ZFS, and don't plan to, so I can't really comment on this. That and I have no technical details of the problem. >>> I've relatively new here, but I'll ask around about XFS and see if >>> anyone had tested it in the development environment. >> >> If they'd tested it properly, and relatively recently, I would think >> they'd have already replaced EXT4 on your Dovecot server. Unless others >> factors prevented such a migration. Or unless I've misunderstood the >> size of your maildir workload. > > I don't know the entire history of things. I think they really wanted > to use ZFS for everything and then fell back to ext4 because it > performed well enough in the cluster. Performance becomes an issue with > backups using rsync. Rsync is faster than Dovecot's native dsync by a > very large margin. I know that dsync is doing more than rsync, but > still, seconds compared to over five minutes? That is a significant > difference. The problem is that rsync can't get a perfect backup. This happens with a lot of "fan boys". There was so much hype surrounding ZFS that even many logically thinking people were frothing at the mouth waiting to get their hands on it. Then, as with many/most things in the tech world, the goods didn't live up to the hype. XFS has been around since 1994, has never had hype surrounding it, has simply been steadily, substantially improved over time. It has been since day 1 the highest performance filesystem with parallel workloads, and finally overcame its last barrier preventing it from being suitable for just about any workload: metadata write performance. Which makes it faster than any FS with the maildir workload when sufficient parallelism/concurrency is present. Meaning servers with a few thousand active users will benefit. Those with 7 users won't. -- Stan From CMarcus at Media-Brokers.com Thu Mar 29 16:56:39 2012 From: CMarcus at Media-Brokers.com (Charles Marcus) Date: Thu, 29 Mar 2012 09:56:39 -0400 Subject: [Dovecot] SIS and restoring from backups In-Reply-To: <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> References: <4F6DB783.3050808@Media-Brokers.com> <99548DA7-2A46-47DC-92B8-6108765AB9A3@iki.fi> <4F6DC3F6.70306@Media-Brokers.com> <4F6F35FA.6050207@Media-Brokers.com> <964D8D2E-4667-4798-949D-FB9DF345F219@iki.fi> Message-ID: <4F746A17.8040500@Media-Brokers.com> On 2012-03-28 7:57 PM, Timo Sirainen wrote: > It's easy to restore a full backup. And it's easy to restore specific > users if you have the full backup easily accessible (just run doveadm > import with proper settings pointing to backup). What's difficult is > if you just want to restore a specific user from the backup and can't > easily do random access to all files. Then you'll first need to > restore the user's dbox files and then somehow figure out which > attachments to restore from the SIS directory. Well, I think I'm not going to worry about this, since you recently said: On 2012-03-24 9:16 AM, Timo Sirainen wrote: > On 24.3.2012, at 14.54, Charles Marcus wrote: >> I was also thinking of asking about how to provide read-only access >> to these backup snapshots to the users in some kind of special >> namespace, so that they could all essentially go 'back in time' to >> grab any emails that they may have inadvertently deleted... > This should be possible, just point the namespace to such snapshot. > You may need to point CONTROL dir to some temporary directory and > index dir as well to either temp or to memory. If we really can get these snapshots to automatically show up under a 'Backups' namespace, with each users folders under each snapshot showing by date, so they can easily 'go back in time' and retrieve anything they want from them, that totally eliminates any need for me to do individual restores... :) > I'd guess that with rsnapshot + Maildir you can get duplicate Maildir > files if the rsnapshot is accessing a large maildir at the same time > as user is changing a message flag. Dovecot usually notices these > duplicates and logs a warning about them. This won't be a problem wither, because our new system will be performing filesystem snapshots for rsnapshot to use as a source. Thanks again! -- Best regards, Charles From dovecot at r.paypc.com Thu Mar 29 21:13:58 2012 From: dovecot at r.paypc.com (Robin) Date: Thu, 29 Mar 2012 11:13:58 -0700 Subject: [Dovecot] dsync redesign In-Reply-To: <4F745465.1030304@hardwarefreak.com> References: <936E95DA-E032-4F8F-A323-DEBB7AAC0E1B@iki.fi> <4F6D7594.10800@fsn.hu> <1332790490.28702.23.camel@sally> <4F721E88.8020309@hardwarefreak.com> <1332881861.29480.8.camel@sally> <4F73375F.3070200@hardwarefreak.com> <1332968041.26122.19.camel@sally> <4F745465.1030304@hardwarefreak.com> Message-ID: <4F74A666.4000705@r.paypc.com> On 3/29/2012 5:24 AM, Stan Hoeppner wrote: > This happens with a lot of "fan boys". There was so much hype > surrounding ZFS that even many logically thinking people were frothing > at the mouth waiting to get their hands on it. Then, as with many/most > things in the tech world, the goods didn't live up to the hype. The problem with zfs especially is that there are so many different implementations, with only the commercial Sun, er, Oracle paid Solaris having ALL of the promised features and the bug-fixes to make them safely usable. For those users, with very large RAM-backed Sun, er, Oracle, hardware, it probably works well. FreeBSD and even the last versions of OpenSolaris lack fixes for some wickedly nasty box-bricking bugs in de-dup, as well as many of the "sexy" features in zpool that had people flocking to it in the first place. The bug database that used to be on the OpenSolaris portal by Sun's gone dark, but you may have some luck through archive.org. I know when I tried it out for myself using the "Community Edition" of Solaris, I did feel annoyed by the bait-and-switch, and the RAM requirements to run de-dupe with merely adequate performance were staggering if I wanted to have plenty of spare block cache left over for improving performance overall. Sun left some of the FOSS operating systems a poison pill with its CDDL licence, which is the main reason why the implementations of zfs on Linux are immature and is being "re-implemented" with US DOE sponsorship, ostensibly in a GNU compatible licence. zfs reminds me a great deal of TIFF - lots of great ideas in the "White Paper", but an elusive (or very very costly) white elephant to acquire. "Rapidly changing", "bleeding edge", and "hot & new" are not descriptors for filesystems I want to trust more than a token amount of data to. =R= From abruce at tumnus.co.nz Thu Mar 29 22:05:24 2012 From: abruce at tumnus.co.nz (Andrew Bruce) Date: Fri, 30 Mar 2012 08:05:24 +1300 Subject: [Dovecot] LDAP Lookup not returning value in maxStorage In-Reply-To: <20120328092534.5690fa40@jimbo> References: <20120327091425.73963576@jimbo> <20120328092534.5690fa40@jimbo> Message-ID: <4F74B274.8000600@tumnus.co.nz> On 28/03/2012 19:25, Nikita Koshikov wrote: > On Wed, 28 Mar 2012 09:39:37 +1300 > Bruce, Andrew wrote: > >> On 28 March 2012 09:36, Bruce, Andrew wrote: >>> On 27 March 2012 19:14, Nikita Koshikov wrote: >>>> On Tue, 27 Mar 2012 13:57:04 +1300 >>>> Bruce, Andrew wrote: >>>> >>>> Hi there, >>>> >>>> We're setting up a Dovecot virtual email setup - we've got everything >>>> working perfect with LDAP logins authenticating against AD and so >>>> forth, but we're having issues with retrieving the maxStorage value >>>> from AD (this is a pre-setup field in AD that we'd like to use to set >>>> per user quotas). >>>> >>>> In our LDAP lookup, we have the maxStorage entry listed under >>>> user_attrs for the quota (user_attrs = >>>> maxStorage=quota_rule=*:storage=%$M), and in the debug logs, can see >>>> it trying to get the entry, but it fails with: >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=maxStorage >>>> Mar 27 13:19:27 auth: Debug: ldap(username at site,192.168.1.5): no >>>> fields returned by the server >>>> >>>> At this point, we then see the default quota applied. >>>> >>> Try to change your quota rule to be like: >>> maxStorage=quota_rule=*:bytes=%$ >>> ^^^^^^^^^ >>> And put the value in bytes to maxStorage - if I remember correct - this is integer field and no K\M\G values is valid here. >>> >>> PS We successfully using maxStorage field to obtain non-default quota from AD, dovecot version 2.0.x >>>> If we change the name of the field from maxStorage to instanceType we >>>> see the value show up in the logs and passed through to the quota >>>> system and applied successfully: >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): user >>>> search: base=dc=site,dc=local scope=subtree >>>> filter=(&(objectClass=person)(| (userPrincipalName=username at site) >>>> (|(mail=username at site)(samAccountName=username at site)))) >>>> fields=instanceType >>>> Mar 27 11:09:01 auth: Debug: ldap(username at site,192.168.1.5): result: >>>> instanceType(quota_rule=*:storage=%$M)=*:storage=4M >>>> Mar 27 11:09:01 auth: Debug: master out: USER 3901227009 >>>> username at site quota_rule=*:storage=4M >>>> >>>> >>>> Which seems a bit weird. >>>> >>>> If we use ldapsearch and pass it the same search string and look for >>>> the field maxStorage, we clearly see the field and the value being >>>> returned. The result looks the same if we also lookup instanceType. >>>> >>>> We're using Dovecot 2.0.9. >>>> >>>> Does anyone have any idea as to why we can't use this field? >>>> >>>> Thanks, >>>> >>>> Andrew >> Tried your suggestion Nikita, no joy unfortunately. It still looks >> like the value never gets returned from the LDAP server to Dovecot. >> It definitely has something in the field (equivalent of 10GB, but in >> bytes as suggested) and I changed the user_attrs also, but still get >> the same "no fields returned by the server" error message. >> >> Modifying the user_attrs to lookup from a different field >> (instanceType) definitely works. >> >> What exact version are you using - perhaps it's a problem with our >> copy of 2.0.9. >> >> Thanks, >> >> Andrew > maybe you met restriction of ldap port 3268?(http://wiki2.dovecot.org/AuthDatabase/LDAP) > Dead on - it was a restriction of ldap port 3268 - as soon as we pointed ldapsearch at the same port, we got the same result - some of the fields were missing. It all makes perfect sense and I wish I noticed that earlier. Now need to work out why Dovecot can get the fields and username back from ldap on port 389, but it can't do the auth through it like it could with 3268. Thanks Nikita for your help. Andrew From tss at iki.fi Fri Mar 30 03:07:40 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 03:07:40 +0300 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: <4F6B5CB9.9080204@mur.at> References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: On 22.3.2012, at 19.09, Martin Schitter wrote: > Am 2012-03-22 03:46, schrieb Gedalya: >>> >>> doveadm sync/backup via impac puts the same message all over the place... >> >> Thanks Martin, I've set up a test platform to investigate this further >> but I've been short on time... > > after some debugging a few more remarks about this problem: > > the bug only appears on recursive folder hierarchies. > if you specity option "-m INBOX" everything works fine. > > for recursive hierarchies the rawlog (-o imapc_rawlog_dir=...) shows that "UID FETCH 1:* FLAGS" will be called for all folders but "UID FETCH NNN (INTERNALDATE)" and "UID FETCH NNN (BODY.PEEK[])" only happens for the messages in first found subfolder! the last message in this folder will substitute all other messages on the target side... :( > > has anyone a clue how to fix this problem in the source code? http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. From tss at iki.fi Fri Mar 30 05:27:09 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 05:27:09 +0300 Subject: [Dovecot] Dovecot migration from any IMAP/POP3 server Message-ID: With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: http://wiki2.dovecot.org/Migration/Dsync The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. From neuronetv at gmail.com Fri Mar 30 00:33:57 2012 From: neuronetv at gmail.com (neuronetv) Date: Thu, 29 Mar 2012 14:33:57 -0700 (PDT) Subject: [Dovecot] newbie: keep getting same emails in mail client Message-ID: <33544893.post@talk.nabble.com> dovecot-2.0.9-2.el6_1.1.i686 I've just set up dovecot in centos 6.2 (server install) and finally got it working (kind of). I set up a unix user (not a virtual user) sent a test email to this user but in my mail client I keep getting this test email over and over again. I don't think the fault is with the email client because other emails work fine and never duplicate and I've tweaked the account settings too, so it must be something I've done wrong in the dovecot setup. Here is my dovecot.conf file: # Dovecot configuration file protocols = pop3 imap disable_plaintext_auth = no mail_location = mbox:~/mail:INBOX=/var/spool/mail/unix-username ssl_cert = References: Message-ID: <4F7521D7.2080808@gedalya.net> On 3/29/2012 10:27 PM, Timo Sirainen wrote: > With the latest hg version / upcoming v2.1.4 you can do a perfect migration to Maildir using imapc/pop3c backends: > > http://wiki2.dovecot.org/Migration/Dsync > > The main new feature here is the pop3-migration plugin that matches messages from IMAP and POP3 servers together, so that when dsync needs to request POP3 UIDL for some IMAP message it's actually looked up from the POP3 server. > Bravo!! From joe at tao.org.uk Fri Mar 30 14:29:39 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 12:29:39 +0100 Subject: [Dovecot] How do I test if the anti spam plugin is working? Message-ID: <3779AD95-CA9E-484B-8B63-039F50B0426E@tao.org.uk> I've configured the dspam anti spam plugin, but it doesn't appear to be doing anything when I move mail between mailboxes. Can anyone help me determine what's going on? The plugin appears to be loading; at least if I don't define all the required configuration options I get a complaint in the log file. But beyond that I don't see any activity. Here's the config: babel# dovecot --version 2.1.3 The dovecot anti spam plugin config is: protocol imap { mail_plugins = $mail_plugins antispam } and plugin { antispam_debug_target = syslog antispam_verbose_debug = 1 antispam_backend = dspam antispam_signature = X-DSPAM-Signature antispam_signature_missing = error antispam_trash_pattern_ignorecase = trash;Deleted * antispam_spam_pattern = SPAM antispam_dspam_binary = /usr/local/bin/dspam antispam_dspam_args = --deliver=;--user;%n at _%d } I don't even appear to be seeing any log entries from the plugin. I've moving an email from my main mailbox into a mailbox called 'SPAM', which is how I thought that it was supposed to be triggered. Cheers, Joe From pw at wk-serv.de Fri Mar 30 14:31:58 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 13:31:58 +0200 Subject: [Dovecot] Hints for a NFS-Setup Message-ID: <4F7599AE.9080300@wk-serv.de> Hi everyone, as I have often trouble with OCFS2 I want to switch to NFS but I'm not sure how to rebuild my cluster with regard to locking and indexing problems. By now my I have a four server configuration (there are another 2 servers for outgoing mail but they can be ignored): MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 As far as I understood I will get poor performance if I'd just switch from OCFS2 to NFS (while keeping this configuration) with 4 hosts accessing the NFS-share and the index files on it and it is recommended to assign users to a specific host (http://wiki2.dovecot.org/NFS). I'm uncertain what's the meaning of "user" in this context. Is it an IMAP-User or every incoming mail? An IMAP-User assigned to a specific IMAP-Server is ok for me and I could store and profit from local index files. However, I want my incoming mailservers to be equally receiving mails. Both should accept mails for every mailbox but in this case I won't have local indexes. I would appreciate any hints. Patrick From busseniu at in.tum.de Fri Mar 30 14:37:44 2012 From: busseniu at in.tum.de (=?ISO-8859-1?Q?Christoph_Bu=DFenius?=) Date: Fri, 30 Mar 2012 13:37:44 +0200 Subject: [Dovecot] Dovecot allows creation of folders outside of a user's directory Message-ID: <4F759B08.1060603@in.tum.de> Hi, in our dovecot 2.0 setup with shared folders, users can make dovecot create directories outside their mail directory. Which is a bit scary imho. The following command: . create inbox.shared.abc123 or even . create "inbox.shared.strange &ANY- characters" -- even though it will fail with a "permission denied" error -- will create a directory like "/mail/users/strange &ANY- characters". That directory will only contain a subdirectory "Maildir" and therein dovecot-acl-list. I think basically the reason for this behaviour is that Dovecot checks whether the directory has enough ACLs for the user to access it, and auto-creates the directory in the process. Is there way to avoid this auto-creation - or maybe a way to make Dovecot check whether the directory name is an existing username? Here's a config to reproduce this: # 2.0.19: /usr/local/dovecot/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-35-server x86_64 Ubuntu 10.04.4 LTS auth_username_format = %Ln disable_plaintext_auth = no mail_gid = vmail mail_home = /mail/users/%u mail_location = maildir:~/Maildir mail_plugins = " acl" mail_uid = vmail maildir_very_dirty_syncs = yes namespace default { inbox = yes location = prefix = INBOX. separator = . type = private } namespace sharedns { inbox = no list = children location = maildir:/mail/users/%%u/Maildir prefix = INBOX.shared.%%u. separator = . subscriptions = no type = shared } passdb { args = /usr/local/dovecot/etc/dovecot/users driver = passwd-file } plugin { acl = vfile:/usr/local/dovecot/etc/dovecot/global-acls:cache_secs=300 acl_shared_dict = file:/mail/vmail/shared-mailboxes.db } service auth { unix_listener auth-userdb { group = vmail mode = 0660 } } ssl_cert = Raum 00.05.055 <> Boltzmannstr. 3 <> Garching From nick at mobilia.it Fri Mar 30 15:38:01 2012 From: nick at mobilia.it (Nick Warr) Date: Fri, 30 Mar 2012 14:38:01 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F7599AE.9080300@wk-serv.de> References: <4F7599AE.9080300@wk-serv.de> Message-ID: <4F75A929.9090902@mobilia.it> Il 30/03/2012 13.31, Patrick Westenberg ha scritto: > Hi everyone, > > as I have often trouble with OCFS2 I want to switch to NFS but > I'm not sure how to rebuild my cluster with regard to locking > and indexing problems. > > By now my I have a four server configuration (there are another 2 > servers for outgoing mail but they can be ignored): > > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > MTA(MX10) --(lmtp/socket)--> local dovecot --> iSCSI-LUN with OCFS2 > IMAP-User <--(imap)--> IMAP-Server1 (local dovecot) <--> iSCSI-LUN/OCFS2 > IMAP-User <--(imap)--> IMAP-Server2 (local dovecot) <--> iSCSI-LUN/OCFS2 > > As far as I understood I will get poor performance if I'd just switch > from OCFS2 to NFS (while keeping this configuration) with 4 hosts > accessing the NFS-share and the index files on it and it is recommended > to assign users to a specific host (http://wiki2.dovecot.org/NFS). > > I'm uncertain what's the meaning of "user" in this context. Is it an > IMAP-User or every incoming mail? > > An IMAP-User assigned to a specific IMAP-Server is ok for me and I > could store and profit from local index files. However, I want my > incoming mailservers to be equally receiving mails. Both should accept > mails for every mailbox but in this case I won't have local indexes. > > I would appreciate any hints. > > Patrick If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). SMTP wouldn't have to be balanced in the same way, you could just use round robin in that case.. I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. From andy at xecu.net Fri Mar 30 16:03:19 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:03:19 -0400 (EDT) Subject: [Dovecot] Multiple instances In-Reply-To: References: Message-ID: <20120330083210.L21999@shell.xecu.net> Sorry to respond to an old post, but I've just recently begun implementing multiple instances to facilitate our director proxies running along with our normal dovecot config on the same servers in the cluster. This is a VERY useful feature Timo, it may need just a little refinement. On Mon, 6 Feb 2012, Timo Sirainen wrote: > # doveadm instance remove proxy Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but when I do this, while it dissapears from doveadm, it still responds to pop/imap requests, and the process continues to run. Is remove supposed to be different than say, "stop"? > It would be possible to add commands to start/shutdown some/all > instances using doveadm, but is it all that useful? I'd guess people > would have their own init.d scripts anyway doing that. Eh, in a FreeBSD port-build environment, I have to hack something in place in the rc script that gets installed, and then make sure to duplicate it every time I upgrade dovecot...not ideal. So, if dovecot had some sort of mechanism in the main config file to alert it of the additional instances to start and their config files, that would be nice. Or, if you were to add a "instance_enable" switch in the config files and then have dovecot scan the /usr/local/etc/dovecot directory for appropriate config files to automatically parse. I dunno, it doesn't feel right to push the startup of the additional instances outside of dovecot. For example, consider postfix's master.cf file. BTW somebody needs to poke the dovecot port maintaner, he still has 2.0.18, I had to manually update the port to get 2.1.3. > Anything else that could be useful related to this? Yes...we should probably be able to start instances back up as well. Other then that, looks good. Definitely a great feature. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From andy at xecu.net Fri Mar 30 16:25:11 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 09:25:11 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides Message-ID: <20120330091204.B22325@shell.xecu.net> I've recently set up a director proxy environment on my test servers, with the intention of deploying on our cluster soon. One thing I found confusing in the proxying documentation [1] was the first bit about their being two ways to do the authentication...either you have the proxy forward the auth to the real server for authentication, or you have the proxy authenticate it and then login to the real server with a master password. Well, we use /bin/checkpassword authentication which hooks into a variety of subsytems for various specific customer needs, and sometimes we need to know the username AND password of the user in order to determine their home directory information. So, using a master password (which requires the back-end server not getting the user password) is out. However, when we have the front-end server do a static director proxy, the problem is that authentication failures are logged on the back-end server with a source IP of the proxy, and no authentication failure with the client IP address is logged on the proxy. So, fail2ban (which is a MUST these days, at least for us) will not be able to properly filter out the brute force attackers. My solution was an alternative: I authenticate with our /bin/checkpassword on the proxy, which authenticates the user and only at that point returns the proxy=y nopassword=y switch to proxy the connection and forward the authentication. As a result, we get logs on the proxy for failed attempts, and the full username and password is supplied to the back-end server for proper processing. Food for thought in case anybody else is implementing this. Thanks, Andy [1] http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From ms at mur.at Fri Mar 30 17:15:28 2012 From: ms at mur.at (Martin Schitter) Date: Fri, 30 Mar 2012 16:15:28 +0200 Subject: [Dovecot] doveadm sync impac problem In-Reply-To: References: <4F20D718.9010805@gedalya.net> <4F21E4CD.3070001@gedalya.net> <4F21E92C.4090509@gedalya.net> <3F3C09E9-1E8F-4243-BC39-BAEA38AF5300@iki.fi> <4F62815D.7020002@gedalya.net> <4F6A8BAC.4000002@mur.at> <4F6A927C.6010003@gedalya.net> <4F6B5CB9.9080204@mur.at> Message-ID: <4F75C000.2010201@mur.at> Am 2012-03-30 02:07, schrieb Timo Sirainen: >> has anyone a clue how to fix this problem in the source code? > > http://hg.dovecot.org/dovecot-2.1/rev/078697a32109 should fix it. thanks! -- now it works! :) From tss at iki.fi Fri Mar 30 17:30:12 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:30:12 +0300 Subject: [Dovecot] Multiple instances In-Reply-To: <20120330083210.L21999@shell.xecu.net> References: <20120330083210.L21999@shell.xecu.net> Message-ID: <1E4998FE-26F5-4F2C-A1DB-D4EF244A15A5@iki.fi> On 30.3.2012, at 16.03, Andy Dills wrote: > On Mon, 6 Feb 2012, Timo Sirainen wrote: > >> # doveadm instance remove proxy > > Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but > when I do this, while it dissapears from doveadm, it still responds to > pop/imap requests, and the process continues to run. > > Is remove supposed to be different than say, "stop"? Yes, the "remove" is meant to simply remove already stopped instances, e.g. some test instances. You can stop instances with "doveadm -i proxy stop". Dunno if there should be another "doveadm instance stop proxy" alias for that?.. >> It would be possible to add commands to start/shutdown some/all >> instances using doveadm, but is it all that useful? I'd guess people >> would have their own init.d scripts anyway doing that. > > Eh, in a FreeBSD port-build environment, I have to hack something in place > in the rc script that gets installed, and then make sure to duplicate it > every time I upgrade dovecot...not ideal. > > So, if dovecot had some sort of mechanism in the main config file to alert > it of the additional instances to start and their config files, that would > be nice. > > Or, if you were to add a "instance_enable" switch in the config files and > then have dovecot scan the /usr/local/etc/dovecot directory for > appropriate config files to automatically parse. Hmm. Perhaps a "doveadm instance auto yes|no" command to set which instances are started up automatically when Dovecot starts up. From tss at iki.fi Fri Mar 30 17:35:33 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:35:33 +0300 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: On 30.3.2012, at 15.38, Nick Warr wrote: >> As far as I understood I will get poor performance if I'd just switch >> from OCFS2 to NFS (while keeping this configuration) with 4 hosts >> accessing the NFS-share and the index files on it and it is recommended >> to assign users to a specific host (http://wiki2.dovecot.org/NFS). >> >> I'm uncertain what's the meaning of "user" in this context. Is it an >> IMAP-User or every incoming mail? IMAP, POP3, LDA so everything. > If you've got a load balancer, it should be fairly easy to do simple IP stickiness, with a long enough timeout, most IMAP and POP3 users will stay on the same server.. I'm sure there is some load balancing software that's also L7 aware, and could direct by username (though you'd probably have to have the LB terminate the SSL, not the server behind it). IP stickiness isn't enough if user uses more than one IMAP client, which is pretty common nowadays. And doesn't help at all with LDA. > I think some of the new Dovecot (director?) software is user aware, but I don't know if it's quite ready for production. Anything else except Dovecot director will cause corruption with NFS. Several really large sites already use director in production. From tss at iki.fi Fri Mar 30 17:39:34 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 17:39:34 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330091204.B22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> Message-ID: On 30.3.2012, at 16.25, Andy Dills wrote: > However, when we have the front-end server do a static director proxy, the > problem is that authentication failures are logged on the back-end server > with a source IP of the proxy, and no authentication failure with the > client IP address is logged on the proxy. So, fail2ban (which is a MUST > these days, at least for us) will not be able to properly filter out the > brute force attackers. This is a simple fix (and something you should do anyway): Add the proxy's IP/netmask to login_trusted_networks setting in the remote server. For this to work with POP3 you need v2.1.2+. > My solution was an alternative: I authenticate with our /bin/checkpassword > on the proxy, which authenticates the user and only at that point returns > the proxy=y nopassword=y switch to proxy the connection and forward the > authentication. Hm. Doesn't it do that even without nopassword=y? From andy at xecu.net Fri Mar 30 17:51:00 2012 From: andy at xecu.net (Andy Dills) Date: Fri, 30 Mar 2012 10:51:00 -0400 (EDT) Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: References: <20120330091204.B22325@shell.xecu.net> Message-ID: <20120330104543.N22325@shell.xecu.net> On Fri, 30 Mar 2012, Timo Sirainen wrote: > On 30.3.2012, at 16.25, Andy Dills wrote: > > > However, when we have the front-end server do a static director proxy, the > > problem is that authentication failures are logged on the back-end server > > with a source IP of the proxy, and no authentication failure with the > > client IP address is logged on the proxy. So, fail2ban (which is a MUST > > these days, at least for us) will not be able to properly filter out the > > brute force attackers. > > This is a simple fix (and something you should do anyway): Add the > proxy's IP/netmask to login_trusted_networks setting in the remote > server. For this to work with POP3 you need v2.1.2+. Well, the problem isn't that my proxies would be banned; the problem is I have no way of seeing the remote IP of the failed authentication so I can ban the people who should be banned. > > My solution was an alternative: I authenticate with our /bin/checkpassword > > on the proxy, which authenticates the user and only at that point returns > > the proxy=y nopassword=y switch to proxy the connection and forward the > > authentication. > > Hm. Doesn't it do that even without nopassword=y? Perhaps...I was going by the docs which seemed to suggest that nopassword=y was how you get the proxy to forward the users authentication credentials to the back-end server. I had been trying a lot of different things, and it was only when I realized I needed to not do a static passdb on the proxy, but instead do a full authentication so that the auth failure would be logged on the proxy for fail2ban, that things began to work the way I needed. It seems obvious in retrospect, but for whatever reason the way the docs were written made me feel like having the full authentication happen on both the proxy and the backend wasn't possible. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 --- From tss at iki.fi Fri Mar 30 18:03:00 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 18:03:00 +0300 Subject: [Dovecot] Proxying Authentication on both sides In-Reply-To: <20120330104543.N22325@shell.xecu.net> References: <20120330091204.B22325@shell.xecu.net> <20120330104543.N22325@shell.xecu.net> Message-ID: <0E6CE6EC-8682-4DF3-B983-DA0906B41BF2@iki.fi> On 30.3.2012, at 17.51, Andy Dills wrote: > On Fri, 30 Mar 2012, Timo Sirainen wrote: > >> On 30.3.2012, at 16.25, Andy Dills wrote: >> >>> However, when we have the front-end server do a static director proxy, the >>> problem is that authentication failures are logged on the back-end server >>> with a source IP of the proxy, and no authentication failure with the >>> client IP address is logged on the proxy. So, fail2ban (which is a MUST >>> these days, at least for us) will not be able to properly filter out the >>> brute force attackers. >> >> This is a simple fix (and something you should do anyway): Add the >> proxy's IP/netmask to login_trusted_networks setting in the remote >> server. For this to work with POP3 you need v2.1.2+. > > Well, the problem isn't that my proxies would be banned; the problem is I > have no way of seeing the remote IP of the failed authentication so I can > ban the people who should be banned. This is what the setting changes. The remote IP will be seen by the backends. > It seems obvious in retrospect, but for whatever reason the way the docs > were written made me feel like having the full authentication happen on > both the proxy and the backend wasn't possible. Oh. This is a pretty common configuration. I guess the docs could be clarified. From pw at wk-serv.de Fri Mar 30 22:37:05 2012 From: pw at wk-serv.de (Patrick Westenberg) Date: Fri, 30 Mar 2012 21:37:05 +0200 Subject: [Dovecot] Hints for a NFS-Setup In-Reply-To: <4F75A929.9090902@mobilia.it> References: <4F7599AE.9080300@wk-serv.de> <4F75A929.9090902@mobilia.it> Message-ID: <4F760B61.3090209@wk-serv.de> Nick Warr schrieb: > I think some of the new Dovecot (director?) software is user aware, but > I don't know if it's quite ready for production. Yes, with director it should be something like that: MTA --(lmtp)--\ /--(lmtp)--> backend1 --\ -- director -- -- NFS MTA --(lmtp)--/ \--(lmtp)--> backend2 --/ IMAP-User --> frontend1 --\ /--(imap)--> backend1 --\ -- director -- -- NFS IMAP-User --> frontend2 --/ \--(imap)--> backend2 --/ So what happens if user1 at example.tld receives a mail? - The director decides to connect to backend1 which in turn stores the mail on the NFS share and the index file locally? - Then, user1 at example.tld connects to one of the frontends. Does the director know that, earlier, this user received a mail and proxies him to backend1 too? From joe at tao.org.uk Fri Mar 30 22:44:08 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:44:08 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? Message-ID: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> I'm excited to hear that dovecot now supports imap as a mailbox type. I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. But, I can't find much documentation on imapc, other than it exists. Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? Thanks :), Joe From tss at iki.fi Fri Mar 30 22:52:18 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:52:18 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > I'm excited to hear that dovecot now supports imap as a mailbox type. > > I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. > > But, I can't find much documentation on imapc, other than it exists. > > Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. From joe at tao.org.uk Fri Mar 30 22:55:27 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 20:55:27 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> Message-ID: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> On 30 Mar 2012, at 20:52, Timo Sirainen wrote: > On 30.3.2012, at 22.44, Dr Josef Karthauser wrote: > >> I'm excited to hear that dovecot now supports imap as a mailbox type. >> >> I've got a mailbox over on another imap server, which I want to migrate to dovecot. I was wondering whether I could use 'doveadm backup' to setup a two way synchronisation process between the old imap mailbox and the new dovecot mailbox. That way I can do the migration, and can coordinate the mail clients changing where they pick up their email from later. >> >> But, I can't find much documentation on imapc, other than it exists. >> >> Is this kind of move possible now? Would someone mind giving me some hints as to how to make it work please? > > One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. > > http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? Joe From tss at iki.fi Fri Mar 30 22:59:47 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 22:59:47 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> Message-ID: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: >> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >> >> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >> > > But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. From joe at tao.org.uk Fri Mar 30 23:02:46 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:02:46 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: On 30 Mar 2012, at 20:59, Timo Sirainen wrote: > On 30.3.2012, at 22.55, Dr Josef Karthauser wrote: > >>> One-way sync will work fine. Two-way sync might be a bit troublesome. For redesigned dsync I've started thinking about kind of a 1,5-way sync. :) That would make sure that all messages from A are copied to B and no messages are deleted from B, but doesn't try to copy new messages from B to A. >>> >>> http://wiki2.dovecot.org/Migration/Dsync anyway has some docs. >>> >> >> But, 'dsync mirror' does a two way sync, doesn't it? Can't I just do that with imapc as the source mailbox? > > It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? Joe From tss at iki.fi Fri Mar 30 23:08:07 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:08:07 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> Message-ID: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: >> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. > > I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. From joe at tao.org.uk Fri Mar 30 23:26:24 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:26:24 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> Message-ID: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> On 30 Mar 2012, at 21:08, Timo Sirainen wrote: > On 30.3.2012, at 23.02, Dr Josef Karthauser wrote: > >>> It does, but the two way sync mirroring relies on messages having GUIDs. IMAP protocol doesn't have such a concept. I guess it could be kind of emulated by using e.g. GUID = sha1(message header). The pop3-replication plugin kind of does this already. But adding such code makes the regular "doveadm backup" slower since now it has to fetch first message headers and then message bodies. But I guess this could be an optional feature. Hmh. >> >> I have a need of it right now. If there's something quick and dirty that I can do, that would be great. It would take the risk out of migrating my users over to dovecot. :) How much code would what you propose be? > > I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. Do you think that this will reliably do the trick? Joe -------------- next part -------------- A non-text attachment was scrubbed... Name: imapc.patch Type: application/octet-stream Size: 2299 bytes Desc: not available URL: From tss at iki.fi Fri Mar 30 23:28:53 2012 From: tss at iki.fi (Timo Sirainen) Date: Fri, 30 Mar 2012 23:28:53 +0300 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> Message-ID: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: >> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. > > Do you think that this will reliably do the trick? > > If it works at all, I guess it should work reliably. :) From joe at tao.org.uk Fri Mar 30 23:42:13 2012 From: joe at tao.org.uk (Dr Josef Karthauser) Date: Fri, 30 Mar 2012 21:42:13 +0100 Subject: [Dovecot] Is it possible to migrating mail to dovecot using imapc? In-Reply-To: <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> References: <72230FBC-F639-415E-AD1D-7D49CE4C6287@tao.org.uk> <07E5D123-E233-4EB5-B5F8-99B90AF6A809@tao.org.uk> <76BAD212-0807-4EA6-9211-4C3D4F6518CC@iki.fi> <2F722E50-B9AC-4660-AAC2-39FBEEE14364@iki.fi> <25DE9E4A-478B-4D70-8B65-0A727B0DEFAF@tao.org.uk> <9F05AEAA-6D32-417E-85F2-240FF39D0426@iki.fi> Message-ID: <695D3A65-CFBF-4DC3-9DAC-E0C299ED0E6D@tao.org.uk> On 30 Mar 2012, at 21:28, Timo Sirainen wrote: > On 30.3.2012, at 23.26, Dr Josef Karthauser wrote: > >>> I suppose to do it quick and dirty you could just copy&paste the get_hdr_sha1() from http://hg.dovecot.org/dovecot-2.1/rev/78317179b4af to imapc code and have imapc_mail_get_special() use it for returning GUID. >> >> Do you think that this will reliably do the trick? >> >> > > If it works at all, I guess it should work reliably. :) Hmm. Didn't appear to: % dsync -Dv -u joe at local.com -o imapc_host=mail.remoteimap.com -o imapc_port=143 -o imapc_username=joe@ remoteimap.com -o imapc_password='somepass' mirror imapc: > & /tmp/output produced the following output. What do you think? Joe. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output2.txt URL: From ka at pacific.net Sat Mar 31 00:30:05 2012 From: ka at pacific.net (Ken A) Date: Fri, 30 Mar 2012 16:30:05 -0500 Subject: [Dovecot] Problem managing mbox In-Reply-To: <1332165600.26095.75.camel@innu> References: <4F64E923.5060401@gmail.com> <1332165600.26095.75.camel@innu> Message-ID: <4F7625DD.1050109@pacific.net> I have seen this error msg too. Deleting the index files 'fixes' the error messages in the log. The indexes seem to get corrupted sometimes, when the client is logged doing: copy dest=Trash, delete, expunge Ken On 3/19/2012 9:00 AM, Timo Sirainen wrote: > On Sat, 2012-03-17 at 20:42 +0100, PSTM wrote: >> Hello, >> >> I have a problem with dovecot. seems that do not erase mail that mail >> client request to be erased. > > Are you sure the clients have actually issued the EXPUNGE command, > rather than simply marked the mail with \Deleted flag? > >> And I have this errors: >>> Error: Next message unexpectedly corrupted in mbox file >> Info: > > mbox code isn't perfect, but if this doesn't happen often it shouldn't > matter much. doveconf -n output might have been helpful in giving more > suggestions. > > > -- Ken Anderson Pacific Internet - http://www.pacific.net Latest Pacific.Net Status - http://twitter.com/pacnetstatus From dm-list-email-dovecot at scs.stanford.edu Sat Mar 31 18:38:48 2012 From: dm-list-email-dovecot at scs.stanford.edu (dm-list-email-dovecot at scs.stanford.edu) Date: Sat, 31 Mar 2012 08:38:48 -0700 Subject: [Dovecot] dovecot 2.1 breaks FTS + pre-auth? Message-ID: <878vigok53.wl@ta.scs.stanford.edu> Hi. I use dovecot in the simplest possible way, as an IMAP server in pre-auth mode over ssh or just locally over a unix-domain socket (e.g., with offlineimap, which runs much faster using dovecot for the local message store). Ideally I would like to avoid running any extra daemons or setting up anything as root. Until recently, this has worked fine by just setting the CONFIG_FILE environment variable to something in my home directory. Here is my configuration: $ export CONFIG_FILE=$HOME/etc/dovecot.conf $ dovecot -n # 2.1.3: /home/dm/etc/dovecot.conf # OS: Linux 3.2.13-1-ARCH x86_64 mail_location = maildir:~/Mail/inbox mail_plugins = " fts fts_squat" plugin { fts = squat fts_squat = partial=4 full=10 } doveconf: Error: ssl enabled, but ssl_cert not set doveconf: Fatal: Error in configuration file /home/dm/etc/dovecot.conf: ssl enabled, but ssl_cert not set Full text search used to work just fine with this configuration, and still does on a machine I have running dovecot 2.0.13. However, on the machine with 2.1, I get errors about /var/run/dovecot/index not existing. $ printf "a select INBOX\nb search text xyzzy\nc logout\n" \ | /usr/lib/dovecot/imap * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE] Logged in as dm imap(dm): Error: net_connect_unix(/var/run/dovecot/indexer) failed: No such file or directory ... Needless to say, no dovecot.index.search or dovecot.index.search.uids file is created after this error. While I can't write /var/run/dovecot, this is not a permission issue. For example, adding base_dir=/home/dm (my home directory) to the configuration file yields the same error for /home/dm/indexer. I'm guessing something has changed where imap requires an indexer daemon and doesn't launch it in pre-auth mode any more, but I can't find anything about this in the documentation. In short, if anyone can tell me how to use FTS in conjunction with pre-auth mode or point me to a working example, I would appreciate it. From ghilt at shadowprojects.org Sat Mar 31 19:55:47 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 18:55:47 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 Message-ID: <4F773713.8030904@shadowprojects.org> Hello, Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. dovecot-antispam package is compiled for dovecot 2.0.15. So, i'm trying to compile a new dovecot antispam plugin. I followed dovecot wiki but i'm running into this error : Successfully compiled dspam.c (plugin). mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying_via_save' mailbox.c: In function 'antispam_save_finish': mailbox.c:174:12: error: 'struct mail_save_context' has no member named 'copying_via_save' Failed to compile mailbox.c (plugin)! Any hint please ? Regards, -- Guillaume Hilt From e-frog at gmx.de Sat Mar 31 20:01:53 2012 From: e-frog at gmx.de (e-frog) Date: Sat, 31 Mar 2012 19:01:53 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773713.8030904@shadowprojects.org> References: <4F773713.8030904@shadowprojects.org> Message-ID: <4F773881.10907@gmx.de> On 31.03.2012 18:55, wrote Guillaume Hilt: > Hello, > > Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. > dovecot-antispam package is compiled for dovecot 2.0.15. > > So, i'm trying to compile a new dovecot antispam plugin. > I followed dovecot wiki but i'm running into this error : > Successfully compiled dspam.c (plugin). > mailbox.c: In function 'antispam_save_begin': > mailbox.c:138:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > mailbox.c: In function 'antispam_save_finish': > mailbox.c:174:12: error: 'struct mail_save_context' has no member named > 'copying_via_save' > Failed to compile mailbox.c (plugin)! > > Any hint please ? > > Regards, > Revert this patch and it should work: http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 From ghilt at shadowprojects.org Sat Mar 31 20:13:55 2012 From: ghilt at shadowprojects.org (Guillaume Hilt) Date: Sat, 31 Mar 2012 19:13:55 +0200 Subject: [Dovecot] Problem compiling dovecot-antispam on Ubuntu 11.10 using Doveot 2.0.13 In-Reply-To: <4F773881.10907@gmx.de> References: <4F773713.8030904@shadowprojects.org> <4F773881.10907@gmx.de> Message-ID: <4F773B53.5070004@shadowprojects.org> Thanks, compilation works fine now. Guillaume Hilt Le 31/03/2012 19:01, e-frog a ?crit : > On 31.03.2012 18:55, wrote Guillaume Hilt: >> Hello, >> >> Last dovecot available version on Ubuntu 11.10 AMD64 is dovecot 2.0.13. >> dovecot-antispam package is compiled for dovecot 2.0.15. >> >> So, i'm trying to compile a new dovecot antispam plugin. >> I followed dovecot wiki but i'm running into this error : >> Successfully compiled dspam.c (plugin). >> mailbox.c: In function 'antispam_save_begin': >> mailbox.c:138:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> mailbox.c: In function 'antispam_save_finish': >> mailbox.c:174:12: error: 'struct mail_save_context' has no member named >> 'copying_via_save' >> Failed to compile mailbox.c (plugin)! >> >> Any hint please ? >> >> Regards, >> > > Revert this patch and it should work: > > http://hg.dovecot.org/dovecot-antispam-plugin/rev/5e8351bcfb29 > > > >