Hi Timo,

From: "Timo Sirainen" tss@iki.fi

...

I suggest that Dovecot simply terminate the current connections (causing the client to reconnect) or -- if the time change is really that much of a problem -- to restart itself automatically.  I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me..

Nevertheless, it would be very nice if you could fix it. It's a fairly big availability problem (for us, at least). And after all, if we are terminating already, adding a simple spawn call before that should not take much time?

Best wishes Eugene

Charles Marcus wrote:

On 5/13/2008, Eugene (genie@geniechka.ru) wrote:

...

...

I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me..

...

Nevertheless, it would be very nice if you could fix it. It's a fairly big availability problem (for us, at least).

The problem is not so much how dovecot deals with this issue, the problem is, why is your server having such drastic problems keeping its time sane?

Fix that, and your problem disappears.

I would just like to mention a circumstance that happened to me this Sunday. We had a total power outage in our building, longer than our UPS's could last and we don't have a generator for servers (nor is it economical or needed). When the power came back on, my local NTP server came on at the same time as my mail servers, as well a majority of my other servers. My servers tried to step their time to be in sync with my local NTP server, which was still busy trying to sync itself with outside sources, which takes a while, so my mail servers did not get an answer. Later, dovecot died because the time finally synced, and I found out why pretty quick (have seen this before) but this was an unusual situation.

My point is, we had an unusual circumstance, and even though I've taken steps to have my mail servers sync their time at boot and run ntpd afterwards, there are some circumstances in which this is not enough, and dovecot still died. Its not always because someone was lazy about their time setup. But it doesn't cause me "big availability problems" since in general, my time is fine.

On 5/13/08 3:33 PM, Scott Silva wrote:

...

This would be a good case for running ntpdate on startup at least on the ntp server. Just point it to a reliable outside server. AFAIR RedHat and clones do this in the init script for ntpd. ...and how much more TIME shall we spend rechewing this non-dovecot issue ?

B. Bodger

On 15:48:42 2008-05-13 Bill Cole dovecot-20061108@billmail.scconsult.com wrote:

At 11:31 AM +0400 5/13/08, Eugene wrote:

...

Hi Timo,

From: "Timo Sirainen" tss@iki.fi

...

...

I suggest that Dovecot simply terminate the current connections (causing the client to reconnect) or -- if the time change is really that much of a problem -- to restart itself automatically.

...

I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me..

Nevertheless, it would be very nice if you could fix it. It's a fairly big availability problem (for us, at least).

Then you have a badly broken system. There is no explanation for time going backwards on a server on a frequent unplanned basis that is not reducible to administrative incompetence or malfunctioning hardware (and the latter as a chronic issue can be seen as just a special case of the former.)

Harsh...

...

And after all, if we are terminating already, adding a simple spawn call before that should not take much time?

A system clock that moves backwards is indicative of a problem. Having a service respawn itself as a response to a problem that is outside of its control (i.e. the respawn is not itself a fix) is begging for trouble, because that behavior has to be carefully controlled to prevent it from contributing to a cascading problem. On a system whose clock is untrustworthy, this is a significant challenge. The effort to do that sort of code correctly just to accommodate people with broken systems seems like a terrible waste.

On the other hand, writing a freestanding watchdog for a critical service is (or at least should be) something any good sysadmin can do. If you are stuck with hardware so broken that it jumps backwards in time without warning but not so broken that you can get it replaced, and it lives in a network or resource environment that prevents you from fixing the core problem, you can adapt to the breakage yourself.

I use monit for monitoring services... It so far has worked great... auto restarts etc... depneding on configuration along with a web based interface one can see a quick overview and control things(of course optional web interface)

-- Andraž "ruskie" Levstik Source Mage GNU/Linux Games grimoire guru Geek/Hacker/Tinker

Be sure brain is in gear before engaging mouth. Ryle hira.

Key id = F4C1F89C Key fingerprint = 6FF2 8F20 4C9D DB36 B5B6 F134 884D 72CC F4C1 F89C

At 3:58 PM +0200 5/13/08, AndraÏ 'ruskie' Levstik wrote:

On 15:48:42 2008-05-13 Bill Cole dovecot-20061108@billmail.scconsult.com wrote:

...

At 11:31 AM +0400 5/13/08, Eugene wrote:

...

Hi Timo,

From: "Timo Sirainen" tss@iki.fi

...

...

I suggest that Dovecot simply terminate the current connections (causing the client to reconnect) or -- if the time change is really that much of a problem -- to restart itself automatically.

...

I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me..

Nevertheless, it would be very nice if you could fix it. It's a fairly big availability problem (for us, at least).

Then you have a badly broken system. There is no explanation for time going backwards on a server on a frequent unplanned basis that is not reducible to administrative incompetence or malfunctioning hardware (and the latter as a chronic issue can be seen as just a special case of the former.)

Harsh...

I think it is not so harsh if you read what I wrote carefully.

Part of what I meant to convey was that the real circumstances of a clock jumping backwards ought to be rare and predictable, such as a long period unpowered, either long enough to drain the clock battery or just long enough for the system clock to drift more than 1/8 of a second. If your system clock doesn't stay pretty close across a regular reboot, you have a hardware problem (most likely a dead clock battery...)

I should probably also note that I did not use 'incompetence' as a generic term and it does not mean 'stupid' or 'bad' or anything else more general, vague, and pejorative.

-- Bill Cole bill@scconsult.com

On Tue, 2008-05-13 at 12:51 +0400, Anton Yuzhaninov wrote:

Timo Sirainen пишет:

...

On Tue, 2008-05-13 at 11:13 +0400, Eugene wrote:

...

I suggest that Dovecot simply terminate the current connections (causing the client to reconnect) or -- if the time change is really that much of a problem -- to restart itself automatically.  I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me..

IMHO more robust is to use clock_gettime(CLOCK_MONOTONIC, ..) for timeouts and just work fine even if time was changed via settimeofday().

Two problems with that:

1. clock_gettime() doesn't work everywhere (e.g. OSX).

2. With the current design gettimeofday() has to be called anyway to get the current real timestamp, causing extra unnecessary work.

Anyway that's not the main problem. I'm more concerned about timestamp comparison code where one or both of the timestamps come from filesystem. It might cause some corruption, such as dotlock being deleted while another process still holds it.

Hello,

From: "Quentin Garnier" cube@cubidou.net

...

4. In about a minute, NTP daemon feels confident about adjusting the system time. The admin should run ntpdate before launching ntpd and dovecot. ntpd will _never_ move time backwards under normal drifting conditions (it has other ways of coping with that).

Please read carefully. ntpd IS run before dovecot, but a change of time happens some time later. Of course, dovecot starting script can be hacked to sleep for some time, but it feels like a wrong way to solve a problem.

Best wishes Eugene

Hello,

From: "Quentin Garnier" cube@cubidou.net

I'm not the one having trouble reading, here. The proper way to start a system is to run ntp*date* (as early as possible) and then ntpd.

That's what you say, and it is far from being officially accepted. NTP project clearly deprecates ntpdate for several reasons. In addition, "the clock should not be stepped until a consistent offset has been observed for a sanity interval, currently 15 minutes". So ntpd may in principle step time again. http://www.ntp.org/ntpfaq/NTP-s-config.htm

Eugene

At 12:48 PM +0400 5/13/08, Eugene wrote:

Hello,

From: "Quentin Garnier" cube@cubidou.net

...

I'm not the one having trouble reading, here. The proper way to start a system is to run ntp*date* (as early as possible) and then ntpd.

That's what you say, and it is far from being officially accepted.

There is nothing 'official' in regards to how you start up your system that should carry more influence than having it start up correctly.

NTP project clearly deprecates ntpdate for several reasons.

I think that is an incorrect statement, and to the degree that it is correct, it still does not mean that it is reasonable to have an unstable clock.

The practice of running ntpdate as a cron job rather than a ntp daemon certainly is deprecated and always has been, but even that is not so much because of how well it works (on most systems it is perfectly functional) but rather because it does not scale across the net: the public NTP infrastructure gets what Dr. Mills calls "little fireballs of congestion" as the cron jobs fire, and that has bad impacts on everyone's NTP-based time accuracy.

On the other hand, ntpdate run once at boot still serves a purpose, even if you are running the latest ntpd with the initial stepping functionality rolled in and enabled, and it is helpful to note Per Hedeland's succinct and practical counterpoint in the FAQ immediately following Dr. Mills' long explanation of the various edge cases involved in choosing when to step and when not to.

In addition, "the clock should not be stepped until a consistent offset has been observed for a sanity interval, currently 15 minutes". So ntpd may in principle step time again. http://www.ntp.org/ntpfaq/NTP-s-config.htm

You are misreading and/or misrepresenting the context of the piece you quote. A good NTP daemon is properly quite conservative about stepping the clock at times *other than* at boot, and should take precautions against trusting clocks that seem wrong once it has what should be a trustworthy synchronization and characterization of its own local clock.

-- Bill Cole
bill@scconsult.com