Re: Minor patches for builds against ancient platforms
"M. Balridge" dovecot@r.paypc.com writes:
I assume it's a rarely seen issue because few Dovecot users compile the software in caves on computers powered by horse-pulled generator wheels.
I resemble that remark.
Warning: Transaction log file /home/luser/mail/.imap/INBOX/dovecot.index.log was locked for 95 seconds (rotating while syncing)
Timo recently explained to me it's probably caused by slow I/O or processing. This explanation is consistent with my observation that the users who get these messages have jumbo mailboxes.
imap(luser): Error: fchown(/home/luser/mail/.subscriptions.lock, group=501(coregroup)) failed: Operation not permitted (egid=200(users), group based on /home/luser/mail - see http://wiki2.dovecot.org/Errors/ChgrpNoPerm)
imap(luser): Error: file_dotlock_open() failed with subscription file /home/luser/mail/.subscriptions: Operation not permitted
.subscriptions doesn't exist either as a file or a directory in the named directories.
Is there a "filter" against dot-files being opened within the bowels of dovecot?
You can disable dotclock altogether, but I don't think this is what you meant. You can use locking method "dotlock_try" rather than "dotlock" -- the former will ignore quota/permissions problems and plow on. (It still logs it.) You could also align luser's mail folder group with with luser's GID, which is usually what I do.
Maybe locks are created even when files don't exist as there may be a race condition where another process is creating/deleting it at the same time.
Joseph Tam jtam.home@gmail.com
Warning: Transaction log file /home/luser/mail/.imap/INBOX/dovecot.index.log was locked for 95 seconds (rotating while syncing)
Timo recently explained to me it's probably caused by slow I/O or processing. This explanation is consistent with my observation that the users who get these messages have jumbo mailboxes.
I do know that this little box of horrors has 200-300MB mbox INBOXes on an ext3 filesystem formatted in 2005. I am very nervous about converting them to Maildir at this point. If I could get someone (or something) to the site and replace it with something much more suitable, I could have these people join the 21st Century.
You can disable dotclock altogether, but I don't think this is what you meant. You can use locking method "dotlock_try" rather than "dotlock" -- the former will ignore quota/permissions problems and plow on. (It still logs it.) You could also align luser's mail folder group with with luser's GID, which is usually what I do.
What I meant was, are certain types of filenames "blocked" by policy from being created via IMAP commands? I'm sure I could run a few tests to answer this for myself, or better still, go through Timo's code.
Maybe locks are created even when files don't exist as there may be a race condition where another process is creating/deleting it at the same time.
Sure, I could see that. In reading through the locking code of sendmail, dovecot, UW-IMAPD, and procmail, it's clear that locking files under UNIX is chaotic and filled with no small amount of voodoo. And naturally, opinions and implementations radically disagree. (How sensible it was of Timo to make it a RUNTIME configuration option in dovecot.)
I appreciate your reply, Joseph.
48 hours after switching half of the userbase to dovecot, I am not seeing any serious problems, and already people are more often than not very pleased by the improved performance and responsiveness.
One last problem area is that many users have soft-links to mailboxes located on a second drive, but these never appear in folder enumeration lists or they appear grayed out in SeaMonkey/Thunderbird.
I've tried just symbolically linking to directories containing other mboxes, but sometimes it works, and sometimes it doesn't. I wonder if there's paranoia checking in the code that follows symbolic links to ensure that uids/gids of the "owning" directory and the linked-to directory (or files within it) are the same.
I'm still trying to absorb all of the documentation for dovecot.
=M=
On 06/09/2017 05:13 PM, M. Balridge wrote:
I do know that this little box of horrors has 200-300MB mbox INBOXes on an ext3 filesystem formatted in 2005. I am very nervous about converting them to Maildir at this point. If I could get someone (or something) to the site and replace it with something much more suitable, I could have these people join the 21st Century.
I for one am finding this thread extremely entertaining. I have to wonder how you'd sound if you came across a machine that was actually OLD. ;)
-Dave
-- Dave McGuire, AK4HZ New Kensington, PA
David "Show Me The Vintage!" McGuire wrote:
I for one am finding this thread extremely entertaining. I have to wonder how you'd sound if you came across a machine that was actually OLD. ;)
Well, I am fond of "old" hardware, which may still be on the wrong side of the New/Old divide for some of you: DECSYSTEM-20s and VAX 11/780s were the first "big" systems I ran across that I admired a great deal as an assembly language programmer and software engineer in general.
[ IBM System/3X0 systems (and EBCDIC) were horrors, though POWER & PowerPC were very interesting for me. ] My first assembly language was Z80, though, which spoiled me for the more primitive CPUs like 6502.
I still maintain some respect for the old iron that could seemingly gracefully handle 200+ users, many of them hammering the system with compile jobs, Mandlebrot set "renders", or other geek-driven nonsense, without going off into the weeds the way a 2017 system does when you do something trivial like enumerate a large directory of files/folders recursively.
Joseph of Tam (I am) wrote:
If all your concerned is dovecot dot-files, you can place the indices somewhere else other than the user's filesapace.
When I manually created the .subscriptions file(s) in the right places, the error message went away, and the functionality seems to work in SeaMonkey clients.
I wonder if it's a combination of permissions (even though the mail directories are all owned by their respective users), or dovecot settings... On that note, has anyone written a tool that "harmonises" users mail directories' permissions - ideally reading the dovecot configuration to assess where *THE* mail directories are actually used by dovecot? I was surprised by the pickiness of the group ownership/permissions issues, though reflecting on things, I can see why you'd at least want some logging by default for those conditions.
His Timoness boomed:
On 9 Jun 2017, at 5.03, M. Balridge dovecot@r.paypc.com wrote:
- In src/lib/compat.h there is a definition for p(read|write) that conflicts with the one in /usr/include/unistd.h [...]
I don't know about this. Anyway, can't apply this patch since it likely fails elsewhere.
Fair enough. I knew this was unlikely to be accepted for multiple reasons, never mind a ferociously high potential-pain:reward ratio.
I'm happy to help in my insignificant way, re: the second patch.
DO many people use filesystem quotas with dovecot much, you think?
I think it's just doing a lot of work on the mbox file itself (reading/writing/rewriting). Would be nice of course if it logged more information, but mbox format is a bit too legacy to spend much time on improving.
I suspect the (heavy) use of procmail on Herr Frankbox is contributing to either some lock "confusion" *OR* triggering dovecot to do "expensive" mbox re-read/syncs or something?
There are mail-mulching scriplets in the global procmail (tied to spamassassin results). Some daintily direct the dross to the /dev/null paradise in the sky. Some "consume" the mail and redirect them to one of two or three folders within mail/, while the "rest" allow procmail to append it to INBOX.
My question is:
Is there is smarter way to do the "delivery" so that the dovecot system is "informed" of an append (or excision), obviating or at least reducing the need to perform more costly re-syncs (or timeouts awaiting a lock break)?
I anticipate a thundering herd declaiming that procmail is the spawn of Satan, Hitler, and He-who-shall-not-be-named. As someone who was responsible for much of them (nearing 10 years ago, now), I don't disagree with that view.
I don't have the budget or mandate to bring slivers of Elysium to this downtrodden backwater of technology. I would expect that any use of procmail with dovecot's "special" mail storage formats would *REQUIRE* the use of "deliver" or some other tool to properly incorporate new mail into a dovecot hive.
My thanks as always, =M=
On 12 Jun 2017, at 2.09, M. Balridge dovecot@r.paypc.com wrote:
I think it's just doing a lot of work on the mbox file itself (reading/writing/rewriting). Would be nice of course if it logged more information, but mbox format is a bit too legacy to spend much time on improving.
I suspect the (heavy) use of procmail on Herr Frankbox is contributing to either some lock "confusion" *OR* triggering dovecot to do "expensive" mbox re-read/syncs or something?
Have you set mbox_very_dirty_syncs=yes? That should be helpful.
Timo Sirainen inscribed:
Have you set mbox_very_dirty_syncs=yes? That should be helpful.
Oh, that sounded like a risky option.
I do have mbox_dirty_syncs enabled.
Are there still "safety checks" with the extra down-and-dirty sync option?
Joseph Tam-a-lyne wrote:
doveadm user $user
which will supply the second half: it will spit out the UID, GID, home and mail directories of a user as specified by dovecot's configuration.
Yes, that outputs the UID/GID/location of user mail, which can feed a tool to audit and/or change directory permissions to conform to expectations.
This is a consequence of writing secure software: it employs least privilege so that a fault will not result in someone being able to mess around with someone else's mail (or indices). GID can also governaccess to shared mailboxes.
Sure, sure, I understand the notion, as I aspire towards "least privilege necessary" designs in my own software. In this case, it seemed that the software was throwing an error when it failed to do something most unprivileged processes cannot do: change the group ownership of an object to a group of which you're not a member.
I would certainly want log entries, sure... but an outright failure when ownership/u+ permissions are otherwise supportive of the operation in question?
I appreciate the fact my questions (and Piltdown Box) are probably noising up your list, and yet you're still both giving me the time of day.
My thanks, once again, =M=
participants (4)
-
Dave McGuire
-
Joseph Tam
-
M. Balridge
-
Timo Sirainen