[Dovecot] dovecot and ntp: Fatal: Time just moved backwards
Hallo,
I am running dovecot on a PC (a workstation) to have a mail client independent storage for my mails. Now I would like to have the system clock set correctly by using ntpd or ntpdate (using debian/sid).
The problem is, that the PC is not online at boot time, but is set online on demand manually using "pon" to start the pppd later. So ntpd cannot sync the time on boot time before dovecot gets started.
First I was using ntpdate that was started in ppp's if.up.d/ directory. This tool does set the time very hard instead of just slightly shifting some milliseconds. So dovecot did stop itself, reporting:
"Fatal: Time just moved backwards by 118 seconds. This might cause a lot of problems, so I'll just kill myself now."
So I have tried ntpd as I thought it shifts the time in smaller time deltas. But this takes much time (several seconds) when booting, giving up on all configured servers, because they are not reachable when booting (as going online later manually). This turns off ntp functionality because ntpd does think that all servers are unreachable for ever. (I have tried a command "dynamic" in ntp.conf but this did not change anything and ntpd reports it to be obsolete.)
My idea now is to not start ntpd on system boot, but only on if-up. But this brings up the same fatal error of dovecot as ntpd seems to hardly set the time, too. The only idea I have left is to stop dovecot, start ntpd and then start dovecot again on if-up.
Is there a more elegant way to use dovecot and ntpd on a manually dialed in PC?
Thanks, Arno
Arno Wald wrote:
Is there a more elegant way to use dovecot and ntpd on a manually dialed in PC?
2 things might help: 1, run ntpd in cron every 10 minutes or so. That should avoid the startup issue. 2, sync your hardware clock every time. That should keep your clock closer to accurate.
You may want to write a script that checks to see if the 'net is up, and only if it is does ntp.
j
John Gateley wrote:
2 things might help:
Thank you for your suggestions. This reminded me of an ntpdate option: ntpdate can be configured to change the time not in a big step on startup by using the option
-B Force the time to always be slewed using the adjtime() system call, even if the measured offset is greater than +-128 ms...
I try if this will help. (I just do not understand if ntpdate keeps changing the time until it is correct or if it only is changing it a bit each time it is executed.)
Arno
On Jun 6, 2009, at 11:19 AM, Arno Wald wrote:
My idea now is to not start ntpd on system boot, but only on if-up.
But this brings up the same fatal error of dovecot as ntpd seems to
hardly set the time, too. The only idea I have left is to stop
dovecot, start ntpd and then start dovecot again on if-up.Is there a more elegant way to use dovecot and ntpd on a manually
dialed in PC?
Wonder if http://cr.yp.to/clockspeed.html would work better than ntpd?
Timo Sirainen wrote:
Wonder if http://cr.yp.to/clockspeed.html would work better than ntpd?
This is interesting, thank you. I will give it a try if ntpd or ntpdate will be no solution for me. But as there is no package for this in debian I would like to use the ntp stuff instead first.
Arno
Have you seen http://support.ntp.org/Support ?
You said your clock is running fast, so it's not a clock interrupt issue.
If your OS supports it, and you have a *steady* problem with your clock, you might be able to correct this problem with the tickadj program and then ntpd should be able to keep the clock in sync.
H
On 06/06/2009 09:22 PM Arno Wald wrote:
This is interesting, thank you. I will give it a try if ntpd or ntpdate will be no solution for me. But as there is no package for this in debian I would like to use the ntp stuff instead first.
On Debian systems I'm very happy with the OpenBSD NTP daemon. Package: openntpd This ntpd adjusts the local time in little steps.
Regards, Pascal
The trapper recommends today: defaced.0915721@localdomain.org
Pascal Volk wrote:
On Debian systems I'm very happy with the OpenBSD NTP daemon. Package: openntpd This ntpd adjusts the local time in little steps.
Also on startup? ntpd uses little steps while running, too. But only at startup it seems to do a big step.
But as I have found in the Debian-Changelog ntpd should handle the unreachable server situation automatically in current versions. This is way the "dynamic" command is obsolete. Maybe I did not wait long enough when testing this. So I could try ntpd again. But good to know there is the alternative openntp
From the openntp package description: "Alternative packages which provide similar functionality are ntp and chrony."
And chrony does what clockspeed seems to do. ;) "`chronyd' determines the rate at which the computer gains or loses time, and compensates for this."
Arno
on 6-6-2009 12:52 PM Arno Wald spake the following:
Pascal Volk wrote:
On Debian systems I'm very happy with the OpenBSD NTP daemon. Package: openntpd This ntpd adjusts the local time in little steps.
Also on startup? ntpd uses little steps while running, too. But only at startup it seems to do a big step.
Many startup scripts for ntp make a call to ntpdate or an equivalent to get the time close on startup, then call ntp to keep the clock synced.
On Jun 8, 2009, at 11:28 AM, Scott Silva wrote:
Many startup scripts for ntp make a call to ntpdate or an equivalent
to get the time close on startup, then call ntp to keep the clock synced.
Hmm. I suppose I could change Dovecot master so that if no imap/pop3
processes have been created yet, it would silently ignore the clock
move.
Timo Sirainen wrote:
Hmm. I suppose I could change Dovecot master so that if no imap/pop3 processes have been created yet, it would silently ignore the clock move.
Also it might be an idea to just restart dovecot instead of completely stopping it. If this happens to often in a certain time interval it still could be stopped to avoid too many restarts in a loop if there is an error situation somehow.
Bye, Arno
On Jun 8, 2009, at 11:40 PM, Arno Wald wrote:
Timo Sirainen wrote:
Hmm. I suppose I could change Dovecot master so that if no imap/ pop3 processes have been created yet, it would silently ignore the
clock move.Also it might be an idea to just restart dovecot instead of
completely stopping it. If this happens to often in a certain time
interval it still could be stopped to avoid too many restarts in a
loop if there is an error situation somehow.
http://wiki.dovecot.org/TimeMovedBackwards bottom answers this. A
restart isn't much better than just ignoring the time change.
NTP comes with a script, ntp-wait, that is specifically designed to be used during the boot-sequence for the purpose of waiting until the clock is sync'd before starting time-sensitive applications.
See http://support.ntp.org/bin/view/Support/StartingNTP4 for more information.
BCP is:
- Start ntpd as early as possible during the boot sequence
- As late as possible during the boot sequence run ntp-wait, before starting things like Dovecot and database servers
H
Timo Sirainen wrote:
http://wiki.dovecot.org/TimeMovedBackwards bottom answers this. A restart isn't much better than just ignoring the time change.
In point 3 ("you can use clockspeed as well") you could mention "chrony", too:
Thank you, Arno
Hi Arno,
http://wiki.dovecot.org/TimeMovedBackwards bottom answers this. A
restart isn't much better than just ignoring the time change. In point 3 ("you can use clockspeed as well") you could mention
"chrony", too: That page has an edit button, so feel free!
Gr.
Matthijs
On Mon, 2009-06-08 at 16:09 -0700, Timo Sirainen wrote:
Hmm. I suppose I could change Dovecot master so that if no imap/pop3
processes have been created yet, it would silently ignore the clock
move.
Implemented to v2.0. Maybe I'll backport it to v1.2 some day. From http://wiki.dovecot.org/TimeMovedBackwards :
Dovecot v2.0 finally tries to handle this a bit more gracefully. Its behavior when time moves backwards is:
* Existing imap and pop3 processes either sleep or die, just like
with older versions
* Master process stops creating new processes until either the
original time is reached, or after a maximum wait of 3 minutes.
* Other processes log a warning, but do nothing else.
* Timeouts are updated so that the timeout is executed
approximately at the original intended time. Dovecot v2.0 also notices when time unexpectedly jumps forwards. In that situation it logs a warning and also updates timeouts.
The reason why imap/pop3 processes get killed and new ones can't be created for a while is to avoid problems related to timestamps. Some issues are:
* Uniqueness of Maildir filenames and dbox global unique
identifiers relies on a growing timestamp
* Dotlock files' staleness is detected by looking at its mtime.
* Timestamps are stored internally all around in memory (as well
as in index files) and compared to current time. Those checks
may or may not be buggy if current time shrinks. While killing mail processes doesn't fully solve any of those issues, they're at least less likely to happen then.
Sorry for the top post and lack of snipping this email down, I'm using
a mobile phone.
Can you explain why the system clock gets so far out of time? I
certainly struggle with crime drifting even on an always network
connected machine, but I'm not understanding why your machine can not
maintain time within a very small margin of error even if turned off
for months.
Scott Iphone says hello.
On Jun 6, 2009, at 11:19 AM, Arno Wald <arno.wald@netcologne.de> wrote:
Hallo,
I am running dovecot on a PC (a workstation) to have a mail client
independent storage for my mails. Now I would like to have the
system clock set correctly by using ntpd or ntpdate (using debian/ sid).The problem is, that the PC is not online at boot time, but is set
online on demand manually using "pon" to start the pppd later. So
ntpd cannot sync the time on boot time before dovecot gets started.First I was using ntpdate that was started in ppp's if.up.d/
directory. This tool does set the time very hard instead of just
slightly shifting some milliseconds. So dovecot did stop itself,
reporting:"Fatal: Time just moved backwards by 118 seconds. This might cause a
lot of problems, so I'll just kill myself now."So I have tried ntpd as I thought it shifts the time in smaller time
deltas. But this takes much time (several seconds) when booting,
giving up on all configured servers, because they are not reachable
when booting (as going online later manually). This turns off ntp
functionality because ntpd does think that all servers are
unreachable for ever. (I have tried a command "dynamic" in ntp.conf
but this did not change anything and ntpd reports it to be obsolete.)My idea now is to not start ntpd on system boot, but only on if-up.
But this brings up the same fatal error of dovecot as ntpd seems to
hardly set the time, too. The only idea I have left is to stop
dovecot, start ntpd and then start dovecot again on if-up.Is there a more elegant way to use dovecot and ntpd on a manually
dialed in PC?Thanks, Arno
Scott Haneda wrote:
Can you explain why the system clock gets so far out of time?
No, I cannot, I do not know. Is it possible that the clock is out of order?
I did compare the times with another PC that is ntpd controlled. And after 1 hour the times differ for 1 second again. It seems that my PC is too fast :)
After some days (I do not know when the last ntp-sync did really happen because of the problems with ntpd that I have described) my PC was about 2 minutes in the future.
--
on 6-6-2009 11:50 AM Scott Haneda spake the following:
Sorry for the top post and lack of snipping this email down, I'm using a mobile phone.
Can you explain why the system clock gets so far out of time? I certainly struggle with crime (time?) drifting even on an always network connected machine, but I'm not understanding why your machine can not maintain time within a very small margin of error even if turned off for months. Over time the crystal that controls the system clock will degrade. So it will oscillate at slightly different frequencies and affect the clock. The less expensive systems seem to degrade faster, as will a system that gets hot often. No simple electronic circuit can be totally accurate, it would be too expensive.
On Sat, Jun 06, 2009 at 08:19:14PM +0200, Arno Wald wrote:
Hallo,
I am running dovecot on a PC (a workstation) to have a mail client
independent storage for my mails. Now I would like to have the system
clock set correctly by using ntpd or ntpdate (using debian/sid).The problem is, that the PC is not online at boot time, but is set
online on demand manually using "pon" to start the pppd later. So ntpd
cannot sync the time on boot time before dovecot gets started.
Chrony is what you are looking for, see http://chrony.sunsite.dk/
Greetings Juergen
--
Juergen Daubert | mailto:jue@jue.li
Korb, Germany | http://jue.li/crux
Juergen Daubert wrote:
On Sat, Jun 06, 2009 at 08:19:14PM +0200, Arno Wald wrote:
I am running dovecot on a PC (a workstation) to have a mail client
independent storage for my mails. Now I would like to have the system
clock set correctly by using ntpd or ntpdate (using debian/sid).The problem is, that the PC is not online at boot time, but is set
online on demand manually using "pon" to start the pppd later. So ntpd
cannot sync the time on boot time before dovecot gets started.Chrony is what you are looking for, see http://chrony.sunsite.dk/
How will chrony help here if the PC is not online at boot time?
What will stop dovecot from aborting if the time gets set backwards when the PC goes online?
If you want to run ntpd, I recommend using its ntp-wait script to block starting dovecot until the clock is sync'd.
H
On Sat, Jun 06, 2009 at 04:17:39PM -0400, Harlan Stenn wrote:
Juergen Daubert wrote:
On Sat, Jun 06, 2009 at 08:19:14PM +0200, Arno Wald wrote:
I am running dovecot on a PC (a workstation) to have a mail client
independent storage for my mails. Now I would like to have the system
clock set correctly by using ntpd or ntpdate (using debian/sid).The problem is, that the PC is not online at boot time, but is set
online on demand manually using "pon" to start the pppd later. So ntpd
cannot sync the time on boot time before dovecot gets started.Chrony is what you are looking for, see http://chrony.sunsite.dk/
How will chrony help here if the PC is not online at boot time?
chronyd can perform usefully in an environment where access to the time reference is intermittent. chronyd estimates both the current time offset and the rate at which the computer's clock gains or loses time, and can use that rate estimate to trim the clock after the reference disappears.
chronyd provides support to work out the gain or loss rate of the `real-time clock', i.e. the clock that maintains the time when the computer is turned off. It can use this data when the system boots to set the system time from a corrected version of the real-time clock.
--
Juergen Daubert | mailto:jue@jue.li
Korb, Germany | http://jue.li/crux
Juergen wrote:
How will chrony help here if the PC is not online at boot time?
- chronyd can perform usefully in an environment where access to the time reference is intermittent. chronyd estimates both the current time offset and the rate at which the computer's clock gains or loses time, and can use that rate estimate to trim the clock after the reference disappears.
Doesn't apply to the use case. This is *before* the PC goes online.
- chronyd provides support to work out the gain or loss rate of the `real-time clock', i.e. the clock that maintains the time when the computer is turned off. It can use this data when the system boots to set the system time from a corrected version of the real-time clock.
There is no corrected version of the real-time clock before the PC goes online.
H
On Sat, Jun 06, 2009 at 05:06:50PM -0400, Harlan Stenn wrote:
Juergen wrote:
How will chrony help here if the PC is not online at boot time?
- chronyd can perform usefully in an environment where access to the time reference is intermittent. chronyd estimates both the current time offset and the rate at which the computer's clock gains or loses time, and can use that rate estimate to trim the clock after the reference disappears.
Doesn't apply to the use case. This is *before* the PC goes online.
- chronyd provides support to work out the gain or loss rate of the `real-time clock', i.e. the clock that maintains the time when the computer is turned off. It can use this data when the system boots to set the system time from a corrected version of the real-time clock.
There is no corrected version of the real-time clock before the PC goes online.
I'd suggest to read chrony's manual. Chrony stores the reference values collected while running online for further use after reboot, even if we have no online connection at that point. Of course, this doesn't work if you never have synced with an NTP server.
--
Juergen Daubert | mailto:jue@jue.li
Korb, Germany | http://jue.li/crux
Juergen wrote:
Harlan wrote:
There is no corrected version of the real-time clock before the PC goes online.
I'd suggest to read chrony's manual. Chrony stores the reference values collected while running online for further use after reboot, even if we have no online connection at that point.
I'm pretty familiar with ntp in particular and computer timekeeping in general.
I've also seen a fair number of situations where the hardware clock is "just wrong", especially after a reboot.
In my world, it's about getting things to work right in as many cases as possible.
Of course, this doesn't work if you never have synced with an NTP server.
And more often than one might think, if one has just rebooted a machine.
H
I have been running AIX, IBM's Unix on IBM hardware for years but am
being forced into Lintel, mostly because of IBM hollowing out support,
moving to a Fortune 500 only customer base and leaving the small shop
increasingly SOL. One of the realities of Lintel is that the hardware
is, compared to purpose built Unix hardware with a matching O/S, crap.
Partially because it's based on commodity hardware, 'little' matters
like time consistency don't necessarily work consistently. Another area
is storage corruption. Interesting piece here:
http://linuxmafia.com/faq/Filesystems/reiserfs.html
Juergen wrote:
I'm pretty familiar with ntp in particular and computer timekeeping in general.
I've also seen a fair number of situations where the hardware clock is "just wrong", especially after a reboot.
In my world, it's about getting things to work right in as many cases as possible.
-- ==== Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows.... ==== Stewart Dean, Unix System Admin, Bard College, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
on 6-6-2009 2:45 PM Harlan Stenn spake the following:
Juergen wrote:
Harlan wrote:
There is no corrected version of the real-time clock before the PC goes online. I'd suggest to read chrony's manual. Chrony stores the reference values collected while running online for further use after reboot, even if we have no online connection at that point.
I'm pretty familiar with ntp in particular and computer timekeeping in general.
I've also seen a fair number of situations where the hardware clock is "just wrong", especially after a reboot.
In my world, it's about getting things to work right in as many cases as possible.
Of course, this doesn't work if you never have synced with an NTP server.
And more often than one might think, if one has just rebooted a machine.
H
I just read the crony manual, and it seems it was written for a use like this. Over time it looks at the difference between the system clock and the hardware clock and keeps track of the drift. It does need to sync occasionally to build the accuracy. Then at startup, it refers to its drift per time calculation and sets the time based on this estimate. Hopefully it would get fairly close over a short period of running, and at least close enough to keep dovecot from erroring.
I suppose you could also write something that watches the logs for "time moved backwards", and run ntpdate followed by dovecot start.
Juergen Daubert wrote:
Chrony is what you are looking for, see http://chrony.sunsite.dk/
chrony does exactly look like what I need. But there is one big disadvantage when using it on manual dial up PCs: You have to configure the NTP-servers by IP-address instead of there names. I do not like to hardcode IP-addresses in a config file. (Also server-pools will not work (if I am right) where one server name returns different IP addresses each time for better load balancing or whatever.)
It would be much better if chrony would look up the server addresses again when it does recieve the "online" state command. It seems that there is no option to turn on such a functionality.
Thanks, Arno
Arno Wald wrote:
It would be much better if chrony would look up the server addresses again when it does recieve the "online" state command. It seems that there is no option to turn on such a functionality.
For completeness, even if it is getting OT in this mailing list (Sorry):
http://chrony.sunsite.dk/faq.php#question_5.2 "The problem is that chronyd (currently) isn't designed in a way that allows hostname->IP address lookups during normal operation. I hope to work on this problem very soon." (How old is this entry? ;) )
The solution is to restart chrony at ppp ip-up: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=268289
My /etc/ppp/ip-up.d/chrony now is:
#!/bin/sh
sleep 1 invoke-rc.d chrony restart sleep 1
... the original unmodified script follows here ...
Arno
participants (10)
-
Arno Wald
-
Harlan Stenn
-
John Gateley
-
Juergen Daubert
-
Matthijs Kooijman
-
Pascal Volk
-
Scott Haneda
-
Scott Silva
-
Stewart Dean
-
Timo Sirainen