hi,

On Mon, Oct 5, 2009 at 4:02 PM, Timo Sirainen <tss@iki.fi> wrote:

And no ntpd in your DomU?

nope.

service ntp status Checking for network time protocol daemon (NTPD): unused

...

any suggestions as to what/how to fix?

If no one here can give you a good answer, I'd try some Xen mailing list. I'm sure a lot of people are running Dovecot in Xen without time problems. If you do find out the problem, please let us know also.

i've been poring over the lists ... found nothing yet :-/

Hmm, I have been running dovecot inside xen for almost 3 years now
without any time issues. I checked my logs and I have no ntp time
reset messages for the last month.

I think it's more possible ntp is stepping the time instead of slewing
it (http://www.ntp.org/ntpfaq/NTP-s-algo.htm section 5.1.1.4), OR, the
ntp servers your using or network connection you have are giving you
lots of jitter. I personally try to pick 3 good low jitter, low
latency servers and 1 higher latency. Only running ntpd on the dom0
and nothing on the domU's.

Hmm, checking the lots on my home machine, even it doesn't have a time
reset log message, and it's network can be overloaded for hours at a
time.

Quoting Thierry DOSTES <tdostes@ibsm.cnrs-mrs.fr>:

Hi,

Here comes an extract from Debian Wiki about Xen to allow a domU to
keep its own time :

---

(...) your domU is likely using the xen clocksource instead of its
own clock ticks. In practice, this seems to be the cause of
infrequent lockups under load (and/or problems with suspending). A
workaround is to decouple the clock in the domU from the dom0:

In your dom0 and domU /etc/sysctl.conf add the line:
xen.independent_wallclock=1. On the dom0, edit the configuration
file of the domU (e.g. /etc/xen/foobar.cfg and add (or expand) the
extra-line: extra="clocksource=jiffies".

These settings can be activated without rebooting the domU. After
editing the configuration files, issue sysctl -p and echo "jiffies">
/sys/devices/system/clocksource/clocksource0/current_clocksource on
the domU prompt.

Because the clock won't be relying on the dom0 clock anymore, you
probably need to use ntp on the domU to synchronize it properly to
the world.

Hope this helps.

Thierry.

PGNet Dev a écrit :

...

hi,

On Mon, Oct 5, 2009 at 4:02 PM, Timo Sirainen <tss@iki.fi> wrote:

...

And no ntpd in your DomU?

nope.

service ntp status Checking for network time protocol daemon (NTPD): unused

...

...

any suggestions as to what/how to fix?

If no one here can give you a good answer, I'd try some Xen mailing list. I'm sure a lot of people are running Dovecot in Xen without time problems. If you do find out the problem, please let us know also.

i've been poring over the lists ... found nothing yet :-/

This reminds me of an odd issue I had also, where mine stepped at a
given amount per time too. In the datacenter one server was at limited
it to 10mbit half duplex, and I had endless ntp issues. I could only
replicate this offsite with the same server using 10mbit and fully
saturating the network. Switching to Full duplex almost solved the
issue.

But the real issue was the time clock chosen by the freebsd kernel in
this case, APCI, was unreliable on that motherboard. Switching it to a
different timing method fixed the issue (TSC in this case).

In freebsd (default): kern.timecounter.choice: TSC(-100) ACPI-safe(1000) i8254(0) dummy(-1000000) kern.timecounter.hardware: ACPI-safe

I am not sure what the commands are in linux. I haven't had ntp go
nuts on a linux system so far.

Quoting Rob Middleton <robm-dovecot@centenary.org.au>:

On 6/10/2009 12:54 PM, PGNet Dev wrote:

...

<snip - from dom0> looking at my ntp logs around the same time(s).

... 5 Oct 16:41:17 ntpd[5696]: synchronized to 64.125.78.85, stratum 1 5 Oct 16:51:38 ntpd[5696]: time reset -2.140133 s 5 Oct 16:56:40 ntpd[5696]: synchronized to 66.220.9.122, stratum 1 5 Oct 17:01:28 ntpd[5696]: synchronized to 64.125.78.85, stratum 1 5 Oct 17:07:20 ntpd[5696]: time reset -2.137760 s 5 Oct 17:11:49 ntpd[5696]: synchronized to 204.152.184.72, stratum 1

This indicates that ntpd is actually stepping the time 2 seconds
into the past approx every 900 seconds. So dovecot is correct that
time has moved backwards. You need to stop time moving backwards :-). [so not dovecot's fault, and likely not xen's fault either]

I'm no ntp expert, but I wonder if searching for 900s in the ntpd
man page might help (caught my eye due to the step every 15 minutes
the middle of an interrupt occasionally. Sorry - can't provide any

• network congestion and excessive jitter causing stepping)?
  Otherwise perhaps a problem with a bad hardware driver stalling in

further pointers. It is highly dependent on your hardware, kernel &
drivers. If you have any other physical servers and they are also
having 'time reset' error messages, then the problem is some odd
network configuration - partial drop-outs and/or high jitter.

Unfortunately -x will not be a solution here as slew cannot possibly
correct for a drift as big as 2 in every 900 seconds.

You may want to try just a single upstream ntp server as a debugging
step (identify it by IP, not by a pool DNS record) and/or use the
prefer keyword against your favourite.

Cheers, Rob Middleton.

and, of course, immediately after hitting 'Send', i see in logs,

Oct 06 11:22:08 dovecot: Error: Time just moved backwards by 1 seconds. I'll sleep now until we're back in present. http://wiki.dovecot.org/TimeMovedBackwards

Oct 6 11:22:07 mx ntpd[17697]: time reset -2.075483 s Oct 6 11:22:16 mx ntpd[17697]: synchronized to 128.2.1.21, stratum 2

:-(

The wiki page also suggests clockspeed or chrony if ntpd can't seem to keep the time correct. Maybe one of those helps. Hmm. The Chrony's web site seems to be gone, wonder if it has a new one somewhere..

sure, but with the _widespread_ use of ntp(d), this bears investigation.

and, unfortunately, at least on opensuse, both

http://software.opensuse.org/search?q=chrony http://software.opensuse.org/search?q=clockspeed

return empty. which means that a manual intervention -- certainly doable, but hardly 'mainstream' -- will be required.

atm, anyway, trying another approach. reading @,

http://lists.ntp.isc.org/pipermail/questions/2009-August/024110.html

changing,

@ Dom0 echo "1" > /proc/sys/xen/independent_wallclock echo "jiffies" > /sys/devices/system/clocksource/clocksource0/current_clocksource

@ DomU echo "0" > /proc/sys/xen/independent_wallclock echo "xen" > /sys/devices/system/clocksource/clocksource0/current_clocksource

i.e., Dom0 _not_ using xen timekeeping, rather 'traditional' ntpd service, and DomU (running Dovecot) depending on DomU _using_ the xen timesource drivers.

then, @ DomU

service ntp stop
service dovecot-custom restart

& watching,

tail -f /var/log/dovecot/*log /var/log/messages

returns,

Oct 06 11:41:53 dovecot: Info: Dovecot v1.2.6 starting up (core dumps disabled)
Oct 06 11:41:53 auth(default): Info: passwd-file

/data/mail/Data/USERS/imap_user_file: Read 2 users Oct 06 11:41:54 auth(default): Info: new auth connection: pid=18001 Oct 06 11:41:54 auth(default): Info: new auth connection: pid=18003 Oct 06 11:41:54 auth(default): Info: new auth connection: pid=18002 ...

with this approach, at least as of

 Tue Oct  6 12:05:40 PDT 2009

no further errors. a 'new record' at 24 minutes ... encouraging, but will keep an eye on it for awhile.

If you notice in your ntpq dumps you did, you have >400ms of jitter. That is a hell of alot.

I dunno if it makes a difference but you used 3 servers from the same
edu, and they have 90ms on them, shouldn't matter, if they where the
only ones with jitter I would replace them, but all 4 of your entries
have high jitter.

Jitter stops ntp from doing it's job properly. I'm not sure what is
causing you jitter to be so bad, but it's caused by the delay amount
changing from packet to packet. The delay should stay consistant (like
a ping time). If it keeps bouncing all over the place, ntp can't
figure out what time it really is, cause it doesn't know how long that
packet was on the network. You can safely ignore offset, it is just
how much different your clock is from what the other computer is.

Try adding some of the pool servers in there, like: server 0.us.pool.ntp.org server 1.us.pool.ntp.org server 2.us.pool.ntp.org

I assume your in the us atleast, if not you could change to eu or
something (check www.pool.ntp.org)

A few samples from my dom0's (and noticing lots of people changed from
st 2/3 up to 1 lately it seems)

  remote           refid      st t when poll reach   delay   offset  jitter

============================================================================== *18.26.4.105 .PPS. 1 u 813 1024 377 7.939 -2.960 0.937 +64.90.182.55 .ACTS. 1 u 487 1024 377 9.375 4.435 0.670 +204.152.184.72 .GPS. 1 u 339 1024 377 82.343 -6.345 0.956 -10.1.11.62 206.246.118.250 2 u 3 16 377 0.191 0.965 0.085 10.1.11.69 10.1.11.61 3 u 15 16 376 0.146 0.448 0.036

  remote           refid      st t when poll reach   delay   offset  jitter

============================================================================== *206.246.118.250 .ACTS. 1 u 642 1024 377 11.615 1.992 0.934 +209.51.161.238 .CDMA. 1 u 955 1024 377 9.701 -0.562 0.200 -128.105.39.11 128.105.201.11 2 u 665 1024 377 37.122 -2.228 0.634 -10.1.11.61 18.26.4.105 2 u 10 16 376 0.184 -0.916 0.247 +10.1.11.69 10.1.11.61 3 u 11 16 376 0.215 -0.542 0.023

  remote           refid      st t when poll reach   delay   offset  jitter

============================================================================== +128.59.16.20 204.123.2.5 2 u 40 64 377 1.873 -0.291 0.092 -198.82.1.203 198.82.247.164 2 u 32 64 357 16.517 -3.454 0.300 -128.2.129.21 69.10.36.2 3 u 41 64 377 28.699 4.447 0.094 -132.236.56.250 129.6.15.29 2 u 2 64 377 9.290 -2.892 0.295 *10.1.11.61 18.26.4.105 2 u 1 16 377 0.161 -0.455 0.051 +10.1.11.62 206.246.118.250 2 u 5 16 377 0.206 0.546 0.020

Quoting PGNet Dev <pgnet.dev+dovecot@gmail.com>:

just fwiw, as of 10/06/09 19:03:27 still no errors. apparently, time's moving forward again ...

so, it seems the config above works. why some others have NOT seen the same problems, remains for me a bit of a mystery.

On Tue, Oct 6, 2009 at 8:19 PM, Patrick Domack <patrickdk@patrickdk.com> wrote:

If you want to email a list that normally no traffic, but many people willing to help with ntp, try timekeepers@fortytwo.ch (not sure if you have to subscribe to send emails)

good reference, and good advice --> http://fortytwo.ch/mailman/pipermail/timekeepers/2009/004773.html

we'll see what comes of that, there.

thanks