From patchwork Sat May 27 03:33:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100585 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp536073qge; Fri, 26 May 2017 20:34:02 -0700 (PDT) X-Received: by 10.84.169.3 with SMTP id g3mr6084204plb.37.1495856042213; Fri, 26 May 2017 20:34:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495856042; cv=none; d=google.com; s=arc-20160816; b=oNC4pI4G/+8Sb0cXBAu5KuFBBO+s/6ZdNIwAWoS02cE50LUYWyhQ0JxeW1v9oMb/bK RABPkREpo224URM9zmT2ZCbCF9XoVw8/kyRob5lL2gzKAp4cx332M/Llr5FmWjuIQCD1 NYtLwkQLtb6FMjlMmCIHYzDhbI5mQVdtlvG7FFHUnJ9l1n6k+Ss1Pk3bpVimudPqFU1j wVx0/FzY+iJR5JiEfdWhUWGk8MOV02GD85Ah2WgS15lOCO8hrO7ksfWt1KU9KEGMkZm0 LfG301PRJPM0cVbveaUsM2isu/al7UUYVi6EhRMeT2Q45fOdJAN/YDmbRP0QgIxD97z9 I39A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=eSdmE153oTfW5bndjZq0gTghCKJMJhQKzVLNIb/AY/A=; b=E/3bQzsOCgq6XB3ezh4pE2RfJsYgCrZMz2ubG7w5SJhPO5m/fvCEbAA9G6k5svyFLv ScH/6z4b3JO1XJZmjaoitgrmvssSBuj1nP3FahJBh+ISMSLRomZgS/kbRG4Tu60ncfx6 8taLh3wKtdqMarng5tVsqXjkHpLmS2ODXWaqHsgMM5azSBGMtnk5pDtpnd3YhP9uCdRl SG4b7F7wvOPVw+uiwgQCUATUrk4JVNx6LKxmyrvh0Mut33bldcQB50l/vbGzZyDjx26J 7TwGE7j5Ib0rFApeJyM3VAvE0bRiFp9hYo+26HKnN8DpM5L9EPwBy4NdYTSqKCyEk/tE ZhSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x233.google.com (mail-pf0-x233.google.com. [2607:f8b0:400e:c00::233]) by mx.google.com with ESMTPS id x15si33756800plm.276.2017.05.26.20.34.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 May 2017 20:34:02 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) client-ip=2607:f8b0:400e:c00::233; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x233.google.com with SMTP id n23so25105563pfb.2 for ; Fri, 26 May 2017 20:34:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eSdmE153oTfW5bndjZq0gTghCKJMJhQKzVLNIb/AY/A=; b=FhptszYlEidQm5OOkM464E4/QFiO3tzct+SKa55jrhC0VNsNSZThdxuxKlbk9+G3nr 8ExqTBptcjmgHQANtn9xqae144hs8bq2TpF3ChJUk10AZbLR3Q1QzhYxvVcNZcxYd39z ILY8Z9SoSLBfKcQXdJB4Lp6GT1xwsM7QmGKtc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eSdmE153oTfW5bndjZq0gTghCKJMJhQKzVLNIb/AY/A=; b=FwYt+3FyvExvUKREvH9P/3C53DCcQOfLGsIfAxn4Ft9eN9Z4ttYalvQ5PNZqSOpj5z brCZ6MrpEUo3OMwy7IhFKqXCGLKqz7CqKFDHiD6MhD16HOmhqrk+f+rqqOSGrrS+/5A9 17Cnav6TUeK2QfBP7bVtc+P0ad90as1D1XjfxgAuw3gpZwrCeKEj6syhlLdTWdoc6apP AxcdB3OdS7bjv+kbj5jifwAiK/mihvpCc6q6yBir0+RRbbUlLBdZYwpiVCY3X2cEl+BX 6CdU3J8e5yMhbPHDnjBZv26cuqsfB3GnBwcVs1ICvyD7L3M4OSEDoqML6tMEeyC1osBQ LXLw== X-Gm-Message-State: AODbwcBfVw+3u+8Er/1o+JZM268/+DvRzzmeaHLjLFIQguJBHNBFPaiY xKsus+74KDIR7kSEp9Q= X-Received: by 10.84.137.1 with SMTP id 1mr63688636plm.128.1495856041914; Fri, 26 May 2017 20:34:01 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id d185sm3413255pgc.39.2017.05.26.20.34.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 May 2017 20:34:00 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Daniel Mentz Subject: [RFC][PATCH 1/4] time: Fix clock->read(clock) race around clocksource changes Date: Fri, 26 May 2017 20:33:52 -0700 Message-Id: <1495856035-6622-2-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> References: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> In some testing on arm64 platforms, I was seeing null ptr crashes in the kselftest/timers clocksource-switch test. This was happening in a read function like: u64 clocksource_mmio_readl_down(struct clocksource *c) { return ~(u64)readl_relaxed(to_mmio_clksrc(c)->reg) & c->mask; } Where the callers enter the seqlock, and then call something like: cycle_now = tkr->read(tkr->clock); The problem seeming to be that since the read and clock references are happening separately, its possible the clocksource change happens in between and we end up calling the old read with the new clocksource, (or vice-versa) which causes the to_mmio_clksrc() in the read function to run off into space. This patch tries to address the issue by providing a helper function that atomically reads the clock value and then calls the clock->read(clock) call so that we always call the read funciton with the appropriate clocksource and don't accidentally mix them. The one exception where this helper isn't necessary is for the fast-timekepers which use their own locking and update logic to the tkr structures. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Daniel Mentz Signed-off-by: John Stultz --- kernel/time/timekeeping.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) -- 2.7.4 diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 9652bc5..abc1968 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -118,6 +118,26 @@ static inline void tk_update_sleep_time(struct timekeeper *tk, ktime_t delta) tk->offs_boot = ktime_add(tk->offs_boot, delta); } +/* + * tk_clock_read - atomic clocksource read() helper + * + * This helper is necessary to use in the read paths as, while the seqlock + * ensures we don't return a bad value while structures are updated, it + * doesn't protect from potential crashes. There is the possibility that + * the tkr's clocksource may change between the read reference, and the + * clock refernce passed to the read function. This can cause crashes if + * the wrong clocksource is passed to the wrong read function. + * This isn't necessary to use when holding the timekeeper_lock or doing + * a read of the fast-timekeeper tkrs (which is protected by its own locking + * and update logic). + */ +static inline u64 tk_clock_read(struct tk_read_base *tkr) +{ + struct clocksource *clock = READ_ONCE(tkr->clock); + + return clock->read(clock); +} + #ifdef CONFIG_DEBUG_TIMEKEEPING #define WARNING_FREQ (HZ*300) /* 5 minute rate-limiting */ @@ -175,7 +195,7 @@ static inline u64 timekeeping_get_delta(struct tk_read_base *tkr) */ do { seq = read_seqcount_begin(&tk_core.seq); - now = tkr->read(tkr->clock); + now = tk_clock_read(tkr); last = tkr->cycle_last; mask = tkr->mask; max = tkr->clock->max_cycles; @@ -209,7 +229,7 @@ static inline u64 timekeeping_get_delta(struct tk_read_base *tkr) u64 cycle_now, delta; /* read clocksource */ - cycle_now = tkr->read(tkr->clock); + cycle_now = tk_clock_read(tkr); /* calculate the delta since the last update_wall_time */ delta = clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask); @@ -240,7 +260,7 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock) tk->tkr_mono.clock = clock; tk->tkr_mono.read = clock->read; tk->tkr_mono.mask = clock->mask; - tk->tkr_mono.cycle_last = tk->tkr_mono.read(clock); + tk->tkr_mono.cycle_last = tk_clock_read(&tk->tkr_mono); tk->tkr_raw.clock = clock; tk->tkr_raw.read = clock->read; @@ -477,7 +497,7 @@ static void halt_fast_timekeeper(struct timekeeper *tk) struct tk_read_base *tkr = &tk->tkr_mono; memcpy(&tkr_dummy, tkr, sizeof(tkr_dummy)); - cycles_at_suspend = tkr->read(tkr->clock); + cycles_at_suspend = tk_clock_read(tkr); tkr_dummy.read = dummy_clock_read; update_fast_timekeeper(&tkr_dummy, &tk_fast_mono); @@ -649,11 +669,10 @@ static void timekeeping_update(struct timekeeper *tk, unsigned int action) */ static void timekeeping_forward_now(struct timekeeper *tk) { - struct clocksource *clock = tk->tkr_mono.clock; u64 cycle_now, delta; u64 nsec; - cycle_now = tk->tkr_mono.read(clock); + cycle_now = tk_clock_read(&tk->tkr_mono); delta = clocksource_delta(cycle_now, tk->tkr_mono.cycle_last, tk->tkr_mono.mask); tk->tkr_mono.cycle_last = cycle_now; tk->tkr_raw.cycle_last = cycle_now; @@ -929,8 +948,7 @@ void ktime_get_snapshot(struct system_time_snapshot *systime_snapshot) do { seq = read_seqcount_begin(&tk_core.seq); - - now = tk->tkr_mono.read(tk->tkr_mono.clock); + now = tk_clock_read(&tk->tkr_mono); systime_snapshot->cs_was_changed_seq = tk->cs_was_changed_seq; systime_snapshot->clock_was_set_seq = tk->clock_was_set_seq; base_real = ktime_add(tk->tkr_mono.base, @@ -1108,7 +1126,7 @@ int get_device_system_crosststamp(int (*get_time_fn) * Check whether the system counter value provided by the * device driver is on the current timekeeping interval. */ - now = tk->tkr_mono.read(tk->tkr_mono.clock); + now = tk_clock_read(&tk->tkr_mono); interval_start = tk->tkr_mono.cycle_last; if (!cycle_between(interval_start, cycles, now)) { clock_was_set_seq = tk->clock_was_set_seq; @@ -1629,7 +1647,7 @@ void timekeeping_resume(void) * The less preferred source will only be tried if there is no better * usable source. The rtc part is handled separately in rtc core code. */ - cycle_now = tk->tkr_mono.read(clock); + cycle_now = tk_clock_read(&tk->tkr_mono); if ((clock->flags & CLOCK_SOURCE_SUSPEND_NONSTOP) && cycle_now > tk->tkr_mono.cycle_last) { u64 nsec, cyc_delta; @@ -2030,7 +2048,7 @@ void update_wall_time(void) #ifdef CONFIG_ARCH_USES_GETTIMEOFFSET offset = real_tk->cycle_interval; #else - offset = clocksource_delta(tk->tkr_mono.read(tk->tkr_mono.clock), + offset = clocksource_delta(tk_clock_read(&tk->tkr_mono), tk->tkr_mono.cycle_last, tk->tkr_mono.mask); #endif From patchwork Sat May 27 03:33:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100586 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp536079qge; Fri, 26 May 2017 20:34:04 -0700 (PDT) X-Received: by 10.99.2.68 with SMTP id 65mr6220000pgc.61.1495856044088; Fri, 26 May 2017 20:34:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495856044; cv=none; d=google.com; s=arc-20160816; b=yKwl07JUlaacP1iJAZbxnP77iG3JLVpAHP++4IDCd9sZdwlAkVHx5wNv7Ynj078msX opkcL37QXNg4s0XS1uZuQBgitqr/8qMwusAMDwPBgwgiYtbpCY8sBxr1xrrirF8wERUR Flbe56QwiP3HgbgDUUk1gpXSf8lLi4tsEv1OxkpfqNUfsjCMBeWM9i10whBUT6RljayN IaFmTVaIJIQn9S97ExlR2HDVSNcFuw/0q4WEh74HfSVCqj6eNN7xi+o7PaDWUqtxjLFy lRICCrgix4U9zkPJplMDVHRme080adVlH5kgxgjAc6FidwXB6pXiz+jU2RtFCkWxs6mj uO5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=/C8i3wvNUonK2NSxf19uz+zI5Pd7nUHG+q/zgHrdegY=; b=AVfA2+kJLM/69sXaXANGuNC6luCr6xQUyqlcQ5cxhZoOoU89DzEd+Hdus1HtDhu5Iy cXbnKZGX4K6m9MTM8eChPCxm0cbyIb6W62m0hEzIGtVLGp2N5uCK4RwEwuEgyQ/4elO1 ZQ/5pJQH/OfRw/ugz6lBfXMO1CEorlrXKKk/PZsSD8HwsKYV1/ZGgzn99rHX4j3kTd4R fK+kvvtWZrEd+eaS7JO5E20y5QnmK+PjwKwtfmMTnm4C2vDCfCRR97sxKfwEJOTtp+0q Cs1mrI+2h70A9pdM3agqngF45iKomvUnfFU8XtKG/56H7j/nyA/8EcO7ozMh8Yxb+eQ2 JE8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::230 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x230.google.com (mail-pf0-x230.google.com. [2607:f8b0:400e:c00::230]) by mx.google.com with ESMTPS id d19si2873896pgk.108.2017.05.26.20.34.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 May 2017 20:34:04 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::230 as permitted sender) client-ip=2607:f8b0:400e:c00::230; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::230 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x230.google.com with SMTP id 9so25093848pfj.1 for ; Fri, 26 May 2017 20:34:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/C8i3wvNUonK2NSxf19uz+zI5Pd7nUHG+q/zgHrdegY=; b=EUUMmVzzt0dwV2AdvVLXE1IOcP5EjfhHxwB1lr8kyNpuGSFeLE/b120Z1gCM6+dwcg jAjTRN467S3X9zYexPtqzjDA08Fi5LJ8dpfmQikBI1UJ4q0StdIV2En9EfCU4n9Til7g 1D5uHTYiePRp1hf2IrGl0ANYMfU2yvsC1IrQ4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/C8i3wvNUonK2NSxf19uz+zI5Pd7nUHG+q/zgHrdegY=; b=rnXIbg2b2sPdNR2VJ0IwIsQbnvK/fJ0njLP4Vg4h5HQFforKeB71H8MleJZgcoFr3v 89nt+DHac5jMrLHRZGq+SKHTOLKk9uaGlUO3M6wghwoFDv/Ri2DpwkMilWYAraT6upI0 +rq96gNPEp4LAd553dW03cGdAdPnaGwBsjkdnWvly1MJUSO4scDEOgrvOtOgFy4C1vBY 4U8EI82lcfrOZ8o2JKgcN7wRGiVym/kPgVZtGzOZAQll3BbEMC3UU7aIm4GrlUs4zgB5 20AeI3ODnmi6XbsDqzY9eIm2VL3tVKI6w57772B1qabqwrYp8919pInU1HVkS472We73 5c7Q== X-Gm-Message-State: AODbwcCMbEukQ0zZ8WDXbT1YtvF8TayCcZoLckQB8K69YjPyKxGdKu3w UVn+2gpUUkoPrjjJVjQ= X-Received: by 10.84.217.8 with SMTP id o8mr24946034pli.50.1495856043803; Fri, 26 May 2017 20:34:03 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id d185sm3413255pgc.39.2017.05.26.20.34.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 May 2017 20:34:02 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Kevin Brodsky , Will Deacon , Daniel Mentz Subject: [RFC][PATCH 2/4] time: Fix CLOCK_MONOTONIC_RAW sub-nanosecond accounting Date: Fri, 26 May 2017 20:33:53 -0700 Message-Id: <1495856035-6622-3-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> References: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> Due to how the MONOTONIC_RAW accumulation logic was handled, there is the potential for a 1ns discontinuity when we do accumulations. This small discontinuity has for the most part gone un-noticed, but since ARM64 enabled CLOCK_MONOTONIC_RAW in their vDSO clock_gettime implementation, we've seen failures with the inconsistency-check test in kselftest. This patch addresses the issue by using the same sub-ns accumulation handling that CLOCK_MONOTONIC uses, which avoids the issue for in-kernel users. Since the ARM64 vDSO implementation has its own clock_gettime calculation logic, this patch reduces the frequency of errors, but failures are still seen. The ARM64 vDSO will need to be updated to include the sub-nanosecond xtime_nsec values in its calculation for this issue to be completely fixed. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Kevin Brodsky Cc: Will Deacon Cc: Daniel Mentz Signed-off-by: John Stultz --- include/linux/timekeeper_internal.h | 4 ++-- kernel/time/timekeeping.c | 21 ++++++++++++--------- 2 files changed, 14 insertions(+), 11 deletions(-) -- 2.7.4 diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h index 110f453..528cc86 100644 --- a/include/linux/timekeeper_internal.h +++ b/include/linux/timekeeper_internal.h @@ -58,7 +58,7 @@ struct tk_read_base { * interval. * @xtime_remainder: Shifted nano seconds left over when rounding * @cycle_interval - * @raw_interval: Raw nano seconds accumulated per NTP interval. + * @raw_interval: Shifted raw nano seconds accumulated per NTP interval. * @ntp_error: Difference between accumulated time and NTP time in ntp * shifted nano seconds. * @ntp_error_shift: Shift conversion between clock shifted nano seconds and @@ -100,7 +100,7 @@ struct timekeeper { u64 cycle_interval; u64 xtime_interval; s64 xtime_remainder; - u32 raw_interval; + u64 raw_interval; /* The ntp_tick_length() value currently being used. * This cached copy ensures we consistently apply the tick * length for an entire tick, as ntp_tick_length may change diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index abc1968..35d3ba3 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -282,7 +282,7 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock) /* Go back from cycles -> shifted ns */ tk->xtime_interval = interval * clock->mult; tk->xtime_remainder = ntpinterval - tk->xtime_interval; - tk->raw_interval = (interval * clock->mult) >> clock->shift; + tk->raw_interval = interval * clock->mult; /* if changing clocks, convert xtime_nsec shift units */ if (old_clock) { @@ -1994,7 +1994,7 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset, u32 shift, unsigned int *clock_set) { u64 interval = tk->cycle_interval << shift; - u64 raw_nsecs; + u64 nsecps; /* If the offset is smaller than a shifted interval, do nothing */ if (offset < interval) @@ -2009,14 +2009,17 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset, *clock_set |= accumulate_nsecs_to_secs(tk); /* Accumulate raw time */ - raw_nsecs = (u64)tk->raw_interval << shift; - raw_nsecs += tk->raw_time.tv_nsec; - if (raw_nsecs >= NSEC_PER_SEC) { - u64 raw_secs = raw_nsecs; - raw_nsecs = do_div(raw_secs, NSEC_PER_SEC); - tk->raw_time.tv_sec += raw_secs; + tk->tkr_raw.xtime_nsec += (u64)tk->raw_time.tv_nsec + << tk->tkr_raw.shift; + tk->tkr_raw.xtime_nsec += tk->raw_interval << shift; + nsecps = (u64)NSEC_PER_SEC << tk->tkr_raw.shift; + while (tk->tkr_raw.xtime_nsec >= nsecps) { + tk->tkr_raw.xtime_nsec -= nsecps; + tk->raw_time.tv_sec++; } - tk->raw_time.tv_nsec = raw_nsecs; + tk->raw_time.tv_nsec = tk->tkr_raw.xtime_nsec >> tk->tkr_raw.shift; + tk->tkr_raw.xtime_nsec -= (u64)tk->raw_time.tv_nsec + << tk->tkr_raw.shift; /* Accumulate error between NTP and clock interval */ tk->ntp_error += tk->ntp_tick << shift; From patchwork Sat May 27 03:33:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100587 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp536084qge; Fri, 26 May 2017 20:34:05 -0700 (PDT) X-Received: by 10.98.224.1 with SMTP id f1mr6044025pfh.116.1495856045460; Fri, 26 May 2017 20:34:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495856045; cv=none; d=google.com; s=arc-20160816; b=F6oskR2E2ckL9Qgy0R5VVdz1jkqTUPLrJ5FHxYlNO3lEHBLffSVuH0jLBMrC/467oW X/uRIoh3P/DSK9oIbfuHBcP0zi+0oG4AGNIk5EqkGXKp4WmB+QoxGhLgHY56VhCxRpGr yD5BbLSZdgPdbvDKqf/6M27NrGEVh56U3YDrjqJKKbMqg7vB78rDY6OoFEa24z2zbzKY tE2HPGlFMnC+1NKCDkHOhnsfhtlsysJdEMyr+HdkPanGQj4wq/WSlZ6UJy/Xqi1E95Fm Si4E/G+jZM8CxubTHtvEWcAXkkvb0aZ9LcjtCjr/lVBpGQsct9T7vMZusxCDXGepeSB+ +8hA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=PZLGpQT7alfMqARS4XWpF1YHIS0cr9Gi5YPB8QvjAr0=; b=vA5I3gquDvdG76mc0t/EzDa4AWh8L87F9wG2QYmy4xVLV7/OMUhFLCG7xpXHY7ZJAx hUbN6eBgH7v3UldD6HFZHlJgwJwGTp4k2V9xgZIn4h3fFO405qoNoxiNuqmVyWPW8nIM I8aaHB38K0wPE+dknBQHpKolcKFJrzAE/KeKaTaTWFe1PypSLlH9mZZfzMHJE2gKlzEJ 0woEwG8tJDuePxMvEbxy+fpK2B6XHqOGnstfID/kF01QOrC5WRasML9/jpPCCmq899Ff BY0Oo2JkSYyzoS3vFCj7qL8g4g8kgYe/7vn7Tya1xNMagbfiij4hjDtA5I+BBgDd24+d 5efg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x22d.google.com (mail-pf0-x22d.google.com. [2607:f8b0:400e:c00::22d]) by mx.google.com with ESMTPS id 59si33776019pld.76.2017.05.26.20.34.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 May 2017 20:34:05 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) client-ip=2607:f8b0:400e:c00::22d; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x22d.google.com with SMTP id 9so25093983pfj.1 for ; Fri, 26 May 2017 20:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PZLGpQT7alfMqARS4XWpF1YHIS0cr9Gi5YPB8QvjAr0=; b=SFROtI5f6gT/IBcs1qD75L+DVM8XqW414egfZver05woXTV5lp1G6qk8SgIr3mmkRa t4HKAeKjnsEDM2KQ8cISRMBgGMIec74EVlEv/r4aeRgvdUDsHAXpSQGP0QdPG+3J+jXU mW+GebsfKwYA3p6OwKRnYJpW+C0vGAS4EAJSc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PZLGpQT7alfMqARS4XWpF1YHIS0cr9Gi5YPB8QvjAr0=; b=aBDgaMzuuN4Ig6L6ILCcMOmrqzrKkv/K/pwT8cHAFNnpwIR7d3Xya7BmnTYTSjSRlE LVykZb0jD12mm7UIXYKA99r/LDlO+pS+qmkRNxjHNINZu637iFoDm6dQXbeUQ39wJE/e hpiDUr5kKoueXSG7WZX4dKbolSzbwxixlCPTmBxT1PolDPX68/Qt+rFX1YNnv9dY0M7E uAnU56enXKESK83PDNNRE9kS7lk+S9s2oYqp+m4x864PPn8p3AGBoVLZLlKhG3ppuWgo Ejo+7lbWFC2RMith+T23asqrNRxp+Tl3WvjFwlUwPWzdfVY0bqhOxKC8OrJqe102oMf5 /Qxg== X-Gm-Message-State: AODbwcAwlDQoeWDH22gvdkWXe8MzPxjFsep4cG22P3Y1Z3XkxGD6c2xH 6p7Zz8w3YfGTnrJ3kxA= X-Received: by 10.84.248.73 with SMTP id e9mr62455946pln.76.1495856045188; Fri, 26 May 2017 20:34:05 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id d185sm3413255pgc.39.2017.05.26.20.34.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 May 2017 20:34:04 -0700 (PDT) From: John Stultz To: lkml Cc: Will Deacon , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Kevin Brodsky , Daniel Mentz , John Stultz Subject: [RFC][PATCH 3/4] arm64: vdso: Fix nsec handling for CLOCK_MONOTONIC_RAW Date: Fri, 26 May 2017 20:33:54 -0700 Message-Id: <1495856035-6622-4-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> References: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> From: Will Deacon Commit 45a7905fc48f ("arm64: vdso: defer shifting of nanosecond component of timespec") fixed sub-ns inaccuracies in our vDSO clock_gettime implementation by deferring the right-shift of the nanoseconds components until after the timespec addition, which operates on left-shifted values. That worked nicely until support for CLOCK_MONOTONIC_RAW was added in 49eea433b326 ("arm64: Add support for CLOCK_MONOTONIC_RAW in clock_gettime() vDSO"). Noticing that the core timekeeping code never set tkr_raw.xtime_nsec, the vDSO implementation didn't bother exposing it via the data page and instead took the unshifted tk->raw_time.tv_nsec value which was then immediately shifted left in the vDSO code. Now that the core code is actually setting tkr_raw.xtime_nsec, we need to take that into account in the vDSO by adding it to the shifted raw_time value. Rather than do that at each use (and expand the data page in the process), instead perform the shift/addition operation when populating the data page and remove the shift from the vDSO code entirely. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Kevin Brodsky Cc: Will Deacon Cc: Daniel Mentz Reported-by: John Stultz Acked-by: Acked-by: Kevin Brodsky Signed-off-by: Will Deacon [jstultz: minor whitespace tweak] Signed-off-by: John Stultz --- arch/arm64/kernel/vdso.c | 5 +++-- arch/arm64/kernel/vdso/gettimeofday.S | 1 - 2 files changed, 3 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index 41b6e31..d0cb007 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -221,10 +221,11 @@ void update_vsyscall(struct timekeeper *tk) /* tkr_mono.cycle_last == tkr_raw.cycle_last */ vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last; vdso_data->raw_time_sec = tk->raw_time.tv_sec; - vdso_data->raw_time_nsec = tk->raw_time.tv_nsec; + vdso_data->raw_time_nsec = (tk->raw_time.tv_nsec << + tk->tkr_raw.shift) + + tk->tkr_raw.xtime_nsec; vdso_data->xtime_clock_sec = tk->xtime_sec; vdso_data->xtime_clock_nsec = tk->tkr_mono.xtime_nsec; - /* tkr_raw.xtime_nsec == 0 */ vdso_data->cs_mono_mult = tk->tkr_mono.mult; vdso_data->cs_raw_mult = tk->tkr_raw.mult; /* tkr_mono.shift == tkr_raw.shift */ diff --git a/arch/arm64/kernel/vdso/gettimeofday.S b/arch/arm64/kernel/vdso/gettimeofday.S index e00b467..76320e9 100644 --- a/arch/arm64/kernel/vdso/gettimeofday.S +++ b/arch/arm64/kernel/vdso/gettimeofday.S @@ -256,7 +256,6 @@ monotonic_raw: seqcnt_check fail=monotonic_raw /* All computations are done with left-shifted nsecs. */ - lsl x14, x14, x12 get_nsec_per_sec res=x9 lsl x9, x9, x12 From patchwork Sat May 27 03:33:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100588 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp536090qge; Fri, 26 May 2017 20:34:07 -0700 (PDT) X-Received: by 10.98.137.93 with SMTP id v90mr5955717pfd.69.1495856046963; Fri, 26 May 2017 20:34:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495856046; cv=none; d=google.com; s=arc-20160816; b=bfYUDgataomODf6Cg+wRV33EEYCtUo7s4lHfy3Mj3ILTAZPMGWBytalqst8aP3BdqV bKGXB3f/Tiuyl3LxVH1JXJtPszfxWGfObt5iFDK6tMuUj3lCsupx6Sb09zplkk4WESVP bMYIVCzfOwxDePDDq0Znf8OETzXCOrTsibFbjSon+ZMdS862/luCBDkI+a1PlmICMVfv 7UYzH782FBDgOHGpC4pquGGeGA7IDrwR6RWajyXP+8HYsWa7xszDIQNAb7eZjaAZjQvB VGWfHCYBSvDOw9PaUbzufxIKEUEE/xXyJi+nrB54jHuYkWGYhjvOOo0hSiIvT3b/4zyi JNcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=uyFTj3Z57ciqpRAc66nOoXJn017zk8YMq1gIGvvmt4k=; b=Kem/Sgvnop1H4SA55hFChawYmFXOxiTv8S49SsrEXEWuMSOVKd0YjjDzTKVCFPb9MK xU1rup+MMEzo9tVerKxeXRfaF2lP0A3XnNkjEcuLRUfEpZJ357eOjQeW7tWIl08DVtQ4 EW+78M/ZpksR2TjPsnKPwTuLjylMgp96NiVpt3g+fqr19eDYywCqJMoJOHxNSoKyEa1C /nTOhEEyjg5eRQUQCJTcA2YY6dBQhqyup4RudW2zlObym+oDgs3CL9YTJnDUlgj2IF/4 1nCR75jp8gW/xwRH0HfUDZu0wpRLe6QZN9JaDbRy6F3qWW6RMFlvjDD20Xqh5kbDe53n G/hQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x22d.google.com (mail-pf0-x22d.google.com. [2607:f8b0:400e:c00::22d]) by mx.google.com with ESMTPS id n6si2934737pfb.286.2017.05.26.20.34.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 May 2017 20:34:06 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) client-ip=2607:f8b0:400e:c00::22d; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::22d as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x22d.google.com with SMTP id 9so25094143pfj.1 for ; Fri, 26 May 2017 20:34:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uyFTj3Z57ciqpRAc66nOoXJn017zk8YMq1gIGvvmt4k=; b=jZqxdGIrt/6b8u+KnQ4RkNmfVcUziJhluCGflJMGmbC/HmTJs8weHho00yWr3u324e okY979L3hmF7r2e67NwoIRDcOLR9sezSyYZ3bEjuQj30/VEd+m8SVj9AsVw+LgC0kmfZ z8TuCK+Tr02q21LMWjgPR3ZqAycnwhFB12gFc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uyFTj3Z57ciqpRAc66nOoXJn017zk8YMq1gIGvvmt4k=; b=r2DmdAyzSr/6j50KyNyDv8v9h8g6s2sWVxd2BRpDtOOV4aPc9ot8S7byUQc9BybIrk 7uESkzevwyw3iN+fQex7Fmpi9dWsSzcZvhThsLicKVPJUJ+0Wv4i9wSCDcAJ9pVocYd2 qyslIiRgbKX66wJqBN8MYdoeukYp7QY9oc79KkOdXUBkoiqTnF0TvWEw2yZZujCO734g MuVzMRipjXKOfpxKcqI2vV7DBqtUuY8L7T46DVRdqM2F4zf1JqLZ3kQ+H3EZ7Uw62Kkx wzrjPbEiO2Gp8W5OcDX53iIQyevb/jq2UEOfhSGV83bXBHQxY0ijzrggrBMSDLzi/lYU M7Ug== X-Gm-Message-State: AODbwcDijY95cm6Q0f3KKTZJXhbEnzGe8eoBK7M3wMwXjmfcsJSXRVjt n3wObwfrjEJEr0vS+oA= X-Received: by 10.84.224.4 with SMTP id r4mr62134317plj.173.1495856046672; Fri, 26 May 2017 20:34:06 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id d185sm3413255pgc.39.2017.05.26.20.34.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 May 2017 20:34:05 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Kevin Brodsky , Will Deacon , Daniel Mentz Subject: [RFC][PATCH 4/4] time: Clean up CLOCK_MONOTONIC_RAW time handling Date: Fri, 26 May 2017 20:33:55 -0700 Message-Id: <1495856035-6622-5-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> References: <1495856035-6622-1-git-send-email-john.stultz@linaro.org> Now that we fixed the sub-ns handling for CLOCK_MONOTONIC_RAW, remove the duplicitive tk->raw_time.tv_nsec, which can be stored in tk->tkr_raw.xtime_nsec (similarly to how its handled for monotonic time). Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Kevin Brodsky Cc: Will Deacon Cc: Daniel Mentz Signed-off-by: John Stultz --- arch/arm64/kernel/vdso.c | 6 ++--- include/linux/timekeeper_internal.h | 4 ++-- kernel/time/timekeeping.c | 47 ++++++++++++++++++++----------------- 3 files changed, 29 insertions(+), 28 deletions(-) -- 2.7.4 diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index d0cb007..7492d90 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -220,10 +220,8 @@ void update_vsyscall(struct timekeeper *tk) if (!use_syscall) { /* tkr_mono.cycle_last == tkr_raw.cycle_last */ vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last; - vdso_data->raw_time_sec = tk->raw_time.tv_sec; - vdso_data->raw_time_nsec = (tk->raw_time.tv_nsec << - tk->tkr_raw.shift) + - tk->tkr_raw.xtime_nsec; + vdso_data->raw_time_sec = tk->raw_sec; + vdso_data->raw_time_nsec = tk->tkr_raw.xtime_nsec; vdso_data->xtime_clock_sec = tk->xtime_sec; vdso_data->xtime_clock_nsec = tk->tkr_mono.xtime_nsec; vdso_data->cs_mono_mult = tk->tkr_mono.mult; diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h index 528cc86..8abf6df 100644 --- a/include/linux/timekeeper_internal.h +++ b/include/linux/timekeeper_internal.h @@ -52,7 +52,7 @@ struct tk_read_base { * @clock_was_set_seq: The sequence number of clock was set events * @cs_was_changed_seq: The sequence number of clocksource change events * @next_leap_ktime: CLOCK_MONOTONIC time value of a pending leap-second - * @raw_time: Monotonic raw base time in timespec64 format + * @raw_sec: CLOCK_MONOTONIC_RAW time in seconds * @cycle_interval: Number of clock cycles in one NTP interval * @xtime_interval: Number of clock shifted nano seconds in one NTP * interval. @@ -94,7 +94,7 @@ struct timekeeper { unsigned int clock_was_set_seq; u8 cs_was_changed_seq; ktime_t next_leap_ktime; - struct timespec64 raw_time; + u64 raw_sec; /* The following members are for timekeeping internal use */ u64 cycle_interval; diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 35d3ba3..0c3b8c1 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -72,6 +72,10 @@ static inline void tk_normalize_xtime(struct timekeeper *tk) tk->tkr_mono.xtime_nsec -= (u64)NSEC_PER_SEC << tk->tkr_mono.shift; tk->xtime_sec++; } + while (tk->tkr_raw.xtime_nsec >= ((u64)NSEC_PER_SEC << tk->tkr_raw.shift)) { + tk->tkr_raw.xtime_nsec -= (u64)NSEC_PER_SEC << tk->tkr_raw.shift; + tk->raw_sec++; + } } static inline struct timespec64 tk_xtime(struct timekeeper *tk) @@ -287,12 +291,14 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock) /* if changing clocks, convert xtime_nsec shift units */ if (old_clock) { int shift_change = clock->shift - old_clock->shift; - if (shift_change < 0) + if (shift_change < 0) { tk->tkr_mono.xtime_nsec >>= -shift_change; - else + tk->tkr_raw.xtime_nsec >>= -shift_change; + } else { tk->tkr_mono.xtime_nsec <<= shift_change; + tk->tkr_raw.xtime_nsec <<= shift_change; + } } - tk->tkr_raw.xtime_nsec = 0; tk->tkr_mono.shift = clock->shift; tk->tkr_raw.shift = clock->shift; @@ -617,9 +623,6 @@ static inline void tk_update_ktime_data(struct timekeeper *tk) nsec = (u32) tk->wall_to_monotonic.tv_nsec; tk->tkr_mono.base = ns_to_ktime(seconds * NSEC_PER_SEC + nsec); - /* Update the monotonic raw base */ - tk->tkr_raw.base = timespec64_to_ktime(tk->raw_time); - /* * The sum of the nanoseconds portions of xtime and * wall_to_monotonic can be greater/equal one second. Take @@ -629,6 +632,11 @@ static inline void tk_update_ktime_data(struct timekeeper *tk) if (nsec >= NSEC_PER_SEC) seconds++; tk->ktime_sec = seconds; + + /* Update the monotonic raw base */ + seconds = tk->raw_sec; + nsec = (u32)(tk->tkr_raw.xtime_nsec >> tk->tkr_raw.shift); + tk->tkr_raw.base = ns_to_ktime(seconds * NSEC_PER_SEC + nsec); } /* must hold timekeeper_lock */ @@ -670,7 +678,6 @@ static void timekeeping_update(struct timekeeper *tk, unsigned int action) static void timekeeping_forward_now(struct timekeeper *tk) { u64 cycle_now, delta; - u64 nsec; cycle_now = tk_clock_read(&tk->tkr_mono); delta = clocksource_delta(cycle_now, tk->tkr_mono.cycle_last, tk->tkr_mono.mask); @@ -682,10 +689,13 @@ static void timekeeping_forward_now(struct timekeeper *tk) /* If arch requires, add in get_arch_timeoffset() */ tk->tkr_mono.xtime_nsec += (u64)arch_gettimeoffset() << tk->tkr_mono.shift; - tk_normalize_xtime(tk); - nsec = clocksource_cyc2ns(delta, tk->tkr_raw.mult, tk->tkr_raw.shift); - timespec64_add_ns(&tk->raw_time, nsec); + tk->tkr_raw.xtime_nsec += delta * tk->tkr_raw.mult; + + /* If arch requires, add in get_arch_timeoffset() */ + tk->tkr_raw.xtime_nsec += (u64)arch_gettimeoffset() << tk->tkr_raw.shift; + + tk_normalize_xtime(tk); } /** @@ -1371,19 +1381,18 @@ int timekeeping_notify(struct clocksource *clock) void getrawmonotonic64(struct timespec64 *ts) { struct timekeeper *tk = &tk_core.timekeeper; - struct timespec64 ts64; unsigned long seq; u64 nsecs; do { seq = read_seqcount_begin(&tk_core.seq); + ts->tv_sec = tk->raw_sec; nsecs = timekeeping_get_ns(&tk->tkr_raw); - ts64 = tk->raw_time; } while (read_seqcount_retry(&tk_core.seq, seq)); - timespec64_add_ns(&ts64, nsecs); - *ts = ts64; + ts->tv_nsec = 0; + timespec64_add_ns(ts, nsecs); } EXPORT_SYMBOL(getrawmonotonic64); @@ -1507,8 +1516,7 @@ void __init timekeeping_init(void) tk_setup_internals(tk, clock); tk_set_xtime(tk, &now); - tk->raw_time.tv_sec = 0; - tk->raw_time.tv_nsec = 0; + tk->raw_sec = 0; if (boot.tv_sec == 0 && boot.tv_nsec == 0) boot = tk_xtime(tk); @@ -2009,17 +2017,12 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset, *clock_set |= accumulate_nsecs_to_secs(tk); /* Accumulate raw time */ - tk->tkr_raw.xtime_nsec += (u64)tk->raw_time.tv_nsec - << tk->tkr_raw.shift; tk->tkr_raw.xtime_nsec += tk->raw_interval << shift; nsecps = (u64)NSEC_PER_SEC << tk->tkr_raw.shift; while (tk->tkr_raw.xtime_nsec >= nsecps) { tk->tkr_raw.xtime_nsec -= nsecps; - tk->raw_time.tv_sec++; + tk->raw_sec++; } - tk->raw_time.tv_nsec = tk->tkr_raw.xtime_nsec >> tk->tkr_raw.shift; - tk->tkr_raw.xtime_nsec -= (u64)tk->raw_time.tv_nsec - << tk->tkr_raw.shift; /* Accumulate error between NTP and clock interval */ tk->ntp_error += tk->ntp_tick << shift;