[Dovecot] IMAP server <defunct> (beta 7)
Hi,
We've been using a 0.99 version of Dovecot for nearly a year without
problems, but due to a server death I had to reinstall with the same
configuration as before, but more recent versions of the software, on
different hardware.
This time though, every few days IMAP login and access fails, and the
output of ps ax shows entries such as [imap-login] <defunct> and
[imap] <defunct>, with the lovely brain-eating Zombie status. There
is nothing in the log file that suggests an issue.
We're using dovecot with authentication/etc being done against a
Postgresql database. I searched online for a solution, but the
nearest problem said it was a problem with PAM, which we are not
using with Dovecot. We're using Maildir as the message storage format.
Does anyone have any suggestions as to why this might be occurring?
If it will help to see the dovecot configuration files then I can
provide them.
The platform is Debian 64-bit running on a dual-core Opteron system.
Yes, it isn't Debian-stable!
Graham
Hi,
It happens in beta 8 as well, but I've narrowed it down to failing
when the system clock is changed back.
The server we're having problems on has a wayward clock that gains 30
minutes a week, and I've just realised that Dovecot zombifies itself
whenever I've corrected the time. I've now just installed openntpd to
keep it more in sync (and dovecot died upon the initial time
synchronisation), and I'm hoping that dovecot doesn't die on me every
time the time is corrected by the ntp client.
Graham
Begin forwarded message:
From: Graham Briggs <graham@historicalengineering.com> Date: 17 May 2006 14:36:10 BDT To: dovecot@dovecot.org Subject: [Dovecot] IMAP server <defunct> (beta 7)
Hi,
We've been using a 0.99 version of Dovecot for nearly a year
without problems, but due to a server death I had to reinstall with
the same configuration as before, but more recent versions of the
software, on different hardware.This time though, every few days IMAP login and access fails, and
the output of ps ax shows entries such as [imap-login] <defunct>
and [imap] <defunct>, with the lovely brain-eating Zombie status.
There is nothing in the log file that suggests an issue.We're using dovecot with authentication/etc being done against a
Postgresql database. I searched online for a solution, but the
nearest problem said it was a problem with PAM, which we are not
using with Dovecot. We're using Maildir as the message storage format.Does anyone have any suggestions as to why this might be occurring?
If it will help to see the dovecot configuration files then I can
provide them.The platform is Debian 64-bit running on a dual-core Opteron
system. Yes, it isn't Debian-stable!Graham
Am Montag, 22. Mai 2006 14:06 schrieb Graham Briggs:
when the system clock is changed back.
The system clock should never be stepped during normal operations, especially not backwards!
Install a time server which corrects the clock drift by dynamically adjusting the system clock's speed, continuously slowly speeding the clock up or slowing it down until any time offset is corrected.
I'm not sure if OpenNTPd does do this, but AFAIK at least xntpd and chronyd do.
Greetings,
Gunter
-- *** Powered by AudioScrobbler --> http://www.last.fm/user/Interneci/ *** 15:58 | Tristania - Crushed Dreams 15:53 | Tristania - World of Glass 15:46 | Tristania - Hatred Grows 15:40 | Tristania - Selling Out *** PGP-Verschlüsselung bei eMails erwünscht :-) *** PGP: 0x1128F25F ***
Hi,
On 22 May 2006, at 15:55, Gunter Ohrner wrote:
Am Montag, 22. Mai 2006 14:06 schrieb Graham Briggs:
when the system clock is changed back.
The system clock should never be stepped during normal operations, especially not backwards!
Install a time server which corrects the clock drift by dynamically adjusting the system clock's speed, continuously slowly speeding the clock up or slowing it down until any time offset is corrected.
Yes, I've installed OpenNTPd to fix this, but for some reason it is
failing to adjust the clock correctly (or the terrible hardware clock
on this server is gaining time faster than openntpd can slow it down
using adjtime()).
I just thought that having dovecot die horribly when the clock is
adjusted (using settimeofday() I assume) wasn't the most desirable
behaviour.
I'm not sure if OpenNTPd does do this, but AFAIK at least xntpd and chronyd do.
xntpd is no longer part of the Debian distribution as far as I can tell.
Greetings,
Gunter
Cheers,
Graham
-- *** Powered by AudioScrobbler --> http://www.last.fm/user/ Interneci/ *** 15:58 | Tristania - Crushed Dreams 15:53 | Tristania - World of Glass 15:46 | Tristania - Hatred Grows 15:40 | Tristania - Selling Out *** PGP-Verschlüsselung bei eMails erwünscht :-) *** PGP:
0x1128F25F ***
On 22-05-2006 16:17:58 +0100, Graham Briggs wrote:
On 22 May 2006, at 15:55, Gunter Ohrner wrote:
Install a time server which corrects the clock drift by dynamically adjusting the system clock's speed, continuously slowly speeding the clock up or slowing it down until any time offset is corrected.
Yes, I've installed OpenNTPd to fix this, but for some reason it is failing to adjust the clock correctly (or the terrible hardware clock on this server is gaining time faster than openntpd can slow it down using adjtime()).
Correcting giant leaps can take somewhat long, as ntpd has to ensure that the time is going forwards all the time, so usually during boot, a ntp-client call is done which just sets the time based on a known server. After that ntpd should run and keep the clock synchronised with 1 or more other servers. Are you sure the ntpd has a list of (valid and reachable!) ntp servers so it can properly adjust the time?
Regards
-- Fabian Groffen
Correcting giant leaps can take somewhat long, as ntpd has to ensure that the time is going forwards all the time, so usually during
boot, a ntp-client call is done which just sets the time based on a known server. After that ntpd should run and keep the clock synchronised
with 1 or more other servers. Are you sure the ntpd has a list of
(valid and reachable!) ntp servers so it can properly adjust the time?
From what I've read online, it's a creeping system clock issue in
the Linux kernel (the clock gains a minute every 2 or 3 hours, it
gains faster if there's more network usage, ntpd cannot compete with
that rate of clock skewing). We're going to try using noapic and
other parameters.
That, or run a script daily that shuts down dovecot, fixes the time,
then restarts dovecot!
ntpd is getting the time correctly. syslog is full of 'adjusting
local clock by -xyzs' messages anyway, byt xyz goes up - in 24 hours
it is now 909s!
Cheers for the advice though, I'll pursue the problem somewhere more
relevant from now on!
Graham
Graham Briggs wrote:
From what I've read online, it's a creeping system clock issue in the Linux kernel (the clock gains a minute every 2 or 3 hours, it gains faster if there's more network usage, ntpd cannot compete with that rate of clock skewing). We're going to try using noapic and other parameters.
You can also try running Dan Bernstein's clockspeed:
http://cr.yp.to/clockspeed.htmlusing this helper script:
http://foo42.de/devel/sysutils/clockspeed-conf/WARNING: DJB's software is tersely written and really intended for people who actually know what they are doing.
FWIW, I have not seen this issue at all with the clocks under my control (mostly SLES 9), which have clock drift that varies per machine. With clockspeed, however, I'm usually within a couple of 100ths of a second each time the script checks/adjusts it. YMMV
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4720 Boston Way Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
Graham Briggs wrote:
Correcting giant leaps can take somewhat long, as ntpd has to ensure that the time is going forwards all the time, so usually during
boot, a ntp-client call is done which just sets the time based on a known server. After that ntpd should run and keep the clock synchronised
with 1 or more other servers. Are you sure the ntpd has a list of
(valid and reachable!) ntp servers so it can properly adjust the time?From what I've read online, it's a creeping system clock issue in
the Linux kernel (the clock gains a minute every 2 or 3 hours, it
gains faster if there's more network usage, ntpd cannot compete with
that rate of clock skewing). We're going to try using noapic and
other parameters.That, or run a script daily that shuts down dovecot, fixes the time,
then restarts dovecot!ntpd is getting the time correctly. syslog is full of 'adjusting
local clock by -xyzs' messages anyway, byt xyz goes up - in 24 hours
it is now 909s!
The maximum drift Linux's adjtime() can correct for (in 2.6, anyway) is 500 PPM (1000 PPM if HZ=1000.) adjtimex() has a hard limit of 512 PPM. Stated another way, that means that the kernel clock can only smoothly gain about 500 seconds for every 1000000 seconds of real time.
Gaining 30 minutes every week works out to almost 3000 PPM, so ntpd alone isn't gonna do the trick. FWIW, I believe that the ntp.org ntpd (ntp-server in Debian, since you mentioned it) will step the time hard if it needs to, but this brings you back to square one -- the time isn't monotonic anymore, and dovecot gets angry about it.
There's a standalone adjtimex utility that's basically a command-line interface to the kernel call, and running adjtimex --tick 9970 ought to bring things into the range where ntpd can keep things in check (depending on how much of an estimate '30 minutes every week' is.)
Cheers for the advice though, I'll pursue the problem somewhere more
relevant from now on!Graham
Dovecot's not the only thing that gets upset when the system time isn't monotonic, but I agree that it'd be nice if it wasn't one of the things that did.
HTH,
Ben Winslow <rain@bluecherry.net>
On Mon, 22 May 2006, Graham Briggs wrote:
Yes, I've installed OpenNTPd to fix this, but for some reason it is failing to adjust the clock correctly (or the terrible hardware clock on this server is gaining time faster than openntpd can slow it down using adjtime()).
Huh, during runtime of my Debian servers the hardware clock plays no rule (the system time is saved to the hardware clock on shutdown and restored from it on startup), did you configured your local clock in NTP? If so, better drop it - or get a radio or GPS clock.
I just thought that having dovecot die horribly when the clock is adjusted (using settimeofday() I assume) wasn't the most desirable behaviour.
Well, although I agree with you, many unexpected things can happen (esp. in Unix) when you go into the past :-(
Bye,
-- Steffen Kaiser
On Tue, 2006-05-23 at 08:09 +0200, Steffen Kaiser wrote:
I just thought that having dovecot die horribly when the clock is adjusted (using settimeofday() I assume) wasn't the most desirable behaviour.
Well, although I agree with you, many unexpected things can happen (esp. in Unix) when you go into the past :-(
From programmer's point of view it can be difficult to try to correctly handle clock going backwards. I've never even bothered trying, since it really shouldn't happen. :)
Quoting Timo Sirainen <tss@iki.fi>:
On Tue, 2006-05-23 at 08:09 +0200, Steffen Kaiser wrote:
I just thought that having dovecot die horribly when the clock is adjusted (using settimeofday() I assume) wasn't the most desirable behaviour.
Well, although I agree with you, many unexpected things can happen (esp. in Unix) when you go into the past :-(
From programmer's point of view it can be difficult to try to correctly handle clock going backwards. I've never even bothered trying, since it really shouldn't happen. :)
My clocks go backwards from time to time. Most people at least go backwards once a year for daylight savings time (though there are a few lucky ones who don't observe DST changes).
Fast clocks, incorrect settings being fixed, DST, BIOS resets, mother board replacements, moving locations, etc. All kinds of things can make a clock go backwards (though jumping forward is more common).
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
On Tue, 2006-05-30 at 09:21 -0500, Eric Rostetter wrote:
Quoting Timo Sirainen <tss@iki.fi>:
On Tue, 2006-05-23 at 08:09 +0200, Steffen Kaiser wrote:
I just thought that having dovecot die horribly when the clock is adjusted (using settimeofday() I assume) wasn't the most desirable behaviour.
Well, although I agree with you, many unexpected things can happen (esp. in Unix) when you go into the past :-(
From programmer's point of view it can be difficult to try to correctly handle clock going backwards. I've never even bothered trying, since it really shouldn't happen. :)
My clocks go backwards from time to time. Most people at least go backwards once a year for daylight savings time (though there are a few lucky ones who don't observe DST changes).
No, daylight savings changes only the local time, not the UTC time which time() function returns (and what Dovecot uses).
Quoting Timo Sirainen <tss@iki.fi>:
My clocks go backwards from time to time. Most people at least go backwards once a year for daylight savings time (though there are a few lucky ones who don't observe DST changes).
No, daylight savings changes only the local time, not the UTC time which time() function returns (and what Dovecot uses).
Yeah, sorry, I wasn't following the thread, and didn't read the message carefully, and probably shouldn't have jumped in at all. While clocks may go backwards, it is none of my business since I hadn't really been following the thread anyway.
Sorry for the noise...
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Eric Rostetter wrote:
My clocks go backwards from time to time. Most people at least go backwards once a year for daylight savings time (though there are a few lucky ones who don't observe DST changes).
Umm, no, this has nothing to do with DST. Your *timezone* changes during DST (if you are in a location that observes DST), but your *clock* should always proceed monotonically forward. A well regulated server will never _step_ time backwards, but through the use of software like ntpd or clockspeed, a clock can be regulated via slowing or speeding it up to sync with an external source.
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
participants (8)
-
Ben Winslow
-
Eric Rostetter
-
Graham Briggs
-
Grobian
-
Gunter Ohrner
-
John Peacock
-
Steffen Kaiser
-
Timo Sirainen