From patchwork Fri Jun 8 12:09:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137959 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp793155lji; Fri, 8 Jun 2018 05:12:26 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJeBi23x+XsuZ4ukPGamG1al43y7uuEoXwzRDalf3ZOT1Nr7R2P5snz850bVmi615EpuMQJ X-Received: by 2002:a62:4b16:: with SMTP id y22-v6mr5817482pfa.214.1528459946384; Fri, 08 Jun 2018 05:12:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459946; cv=none; d=google.com; s=arc-20160816; b=h9Vz+v/hXHYPcTURfNgfmWCUq69jYy3qU9DfV78CSWJyNoBL74jHd4gzvoAKM1gKH1 XIEz3qU+/CKYHWqBrC+Kd6YxLUwfUIcSO5X19ztdxQmiM/sLi75QV+xQ673JjI16vm6P 3HcaS+PCj3Jo2ZG6FpR+XDHl7YZnZfcW03fenGFjPneKVZnyZoGOdIU7uA153FKes1I5 vAVD9kPmDatMzHem4+c0IB4mmxzHbee0LFgVeg2oSMS7FiSa+EpNC1WSDOUwMoDg0RoR 7XVSmM8Ls26gpom/1TN2Q7iHAtNe7FGkoJ7CjrWDOji2KyBNep/58ClK9A6MVCs1jINm DliA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ZBJMc4fq0Rd3S2EM03sm86Apr6TC+j3QAxXy6lZvtMs=; b=WBQgLWCGxjQKIolQgNXS7V5RqHQdjmNSY1Yxa0AWOQHo2pA7smad0ssK0cEldqnW5m eoJiUHGqBdO/u+FSnxv2b5Df/ZbHdVe7i7Y6tc7SkH/nnb8/GUQedo1kHZn+BOSvxVWU sKCRhSvtndCEb8fcBNDj7xKgdmNoAtQEx4vlHU797KCQ6vVKl+dr4bpRYoPw3vpJ+++3 ppjmAT/4su1DP2UPoNdQdxK6MNrW+imAZyxVh6BdEhUTPfw7SKEH1yLuMy8jvMQ2cSKC 5kiLP6xTZCJVs7tpHmsfOyv+H4GCb9FSgOmqlSaNf2zQ/ea6OKub8X1WJvV/0UxWEzTM iVqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eh0JlXKu; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x15-v6si10476277pgv.389.2018.06.08.05.12.26; Fri, 08 Jun 2018 05:12:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eh0JlXKu; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752769AbeFHMKT (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:19 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:40086 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751137AbeFHMKP (ORCPT ); Fri, 8 Jun 2018 08:10:15 -0400 Received: by mail-wm0-f66.google.com with SMTP id n5-v6so3263791wmc.5 for ; Fri, 08 Jun 2018 05:10:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZBJMc4fq0Rd3S2EM03sm86Apr6TC+j3QAxXy6lZvtMs=; b=eh0JlXKulTiQUuAd1/yJKDWMtNU4RNiwSgFC+g9DOtYoN+XW0CmEl+p7kSd+fw31E1 WqUz+xe2ABwFSGFoybZ8zRG5Xb7wrkgbpyAQcfH3N8Hjf6a+TxZMN7Vn8Qn476X2OzOR X2dUg7KnyVuiC1tEcQCF2iqWmbdxbTc1jmNHU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZBJMc4fq0Rd3S2EM03sm86Apr6TC+j3QAxXy6lZvtMs=; b=qIlGxd40/Yk4KJM9oo2fJbmLL9HdgYCLWPlvWMfOF71pKb6KbqbWpxplR1Rd9uOi4/ 5MWe8qDfUoXSynbI2XJti153FTr8p+Ejo2qEQia3jHjF0eDjKJaDmxteWZIs7mEJc0WT VYtP3XDtv7xd22V7OH7TkPZuh/D9RuzkTBgzVGhHpEMWavDpD+iIhZJq1P5xAE3Tv7UW 9ooVbrXDl5QiIatwlPyYWz2zQUPqcxV0kaNW6smzbCiIb/c3c+daA8rsdcbEPAK/0n9R +JlUlmJoui9DRh1s1wwQLY2Z1CIt1PwBvSvVERFYTUNavvGech8nP7gNttX6ZIrV+sYv JFcg== X-Gm-Message-State: APt69E3LKiE2DMAxUyiR58mEowvhktqKl84rV3zaZF4rS7waxj4dW2ec QDxnKjnzo9oenS+fsqezGoyIwQ== X-Received: by 2002:a1c:1c8f:: with SMTP id c137-v6mr1377757wmc.142.1528459813448; Fri, 08 Jun 2018 05:10:13 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:12 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 01/11] sched/pelt: Move pelt related code in a dedicated file Date: Fri, 8 Jun 2018 14:09:44 +0200 Message-Id: <1528459794-13066-2-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We want to track rt_rq's utilization as a part of the estimation of the whole rq's utilization. This is necessary because rt tasks can steal utilization to cfs tasks and make them lighter than they are. As we want to use the same load tracking mecanism for both and prevent useless dependency between cfs and rt code, pelt code is moved in a dedicated file. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/Makefile | 2 +- kernel/sched/fair.c | 333 +------------------------------------------------- kernel/sched/pelt.c | 311 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/pelt.h | 43 +++++++ kernel/sched/sched.h | 19 +++ 5 files changed, 375 insertions(+), 333 deletions(-) create mode 100644 kernel/sched/pelt.c create mode 100644 kernel/sched/pelt.h -- 2.7.4 diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile index d9a02b3..7fe1834 100644 --- a/kernel/sched/Makefile +++ b/kernel/sched/Makefile @@ -20,7 +20,7 @@ obj-y += core.o loadavg.o clock.o cputime.o obj-y += idle.o fair.o rt.o deadline.o obj-y += wait.o wait_bit.o swait.o completion.o -obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o +obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o pelt.o obj-$(CONFIG_SCHED_AUTOGROUP) += autogroup.o obj-$(CONFIG_SCHEDSTATS) += stats.o obj-$(CONFIG_SCHED_DEBUG) += debug.o diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e497c05..6390c66 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -255,9 +255,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return cfs_rq->rq; } -/* An entity is a task if it doesn't "own" a runqueue */ -#define entity_is_task(se) (!se->my_q) - static inline struct task_struct *task_of(struct sched_entity *se) { SCHED_WARN_ON(!entity_is_task(se)); @@ -419,7 +416,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return container_of(cfs_rq, struct rq, cfs); } -#define entity_is_task(se) 1 #define for_each_sched_entity(se) \ for (; se; se = NULL) @@ -692,7 +688,7 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se) } #ifdef CONFIG_SMP - +#include "pelt.h" #include "sched-pelt.h" static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu); @@ -2749,19 +2745,6 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) } while (0) #ifdef CONFIG_SMP -/* - * XXX we want to get rid of these helpers and use the full load resolution. - */ -static inline long se_weight(struct sched_entity *se) -{ - return scale_load_down(se->load.weight); -} - -static inline long se_runnable(struct sched_entity *se) -{ - return scale_load_down(se->runnable_weight); -} - static inline void enqueue_runnable_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { @@ -3062,314 +3045,6 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags) } #ifdef CONFIG_SMP -/* - * Approximate: - * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) - */ -static u64 decay_load(u64 val, u64 n) -{ - unsigned int local_n; - - if (unlikely(n > LOAD_AVG_PERIOD * 63)) - return 0; - - /* after bounds checking we can collapse to 32-bit */ - local_n = n; - - /* - * As y^PERIOD = 1/2, we can combine - * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) - * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { - val >>= local_n / LOAD_AVG_PERIOD; - local_n %= LOAD_AVG_PERIOD; - } - - val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); - return val; -} - -static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) -{ - u32 c1, c2, c3 = d3; /* y^0 == 1 */ - - /* - * c1 = d1 y^p - */ - c1 = decay_load((u64)d1, periods); - - /* - * p-1 - * c2 = 1024 \Sum y^n - * n=1 - * - * inf inf - * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) - * n=0 n=p - */ - c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; - - return c1 + c2 + c3; -} - -/* - * Accumulate the three separate parts of the sum; d1 the remainder - * of the last (incomplete) period, d2 the span of full periods and d3 - * the remainder of the (incomplete) current period. - * - * d1 d2 d3 - * ^ ^ ^ - * | | | - * |<->|<----------------->|<--->| - * ... |---x---|------| ... |------|-----x (now) - * - * p-1 - * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 - * n=1 - * - * = u y^p + (Step 1) - * - * p-1 - * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) - * n=1 - */ -static __always_inline u32 -accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - unsigned long scale_freq, scale_cpu; - u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ - u64 periods; - - scale_freq = arch_scale_freq_capacity(cpu); - scale_cpu = arch_scale_cpu_capacity(NULL, cpu); - - delta += sa->period_contrib; - periods = delta / 1024; /* A period is 1024us (~1ms) */ - - /* - * Step 1: decay old *_sum if we crossed period boundaries. - */ - if (periods) { - sa->load_sum = decay_load(sa->load_sum, periods); - sa->runnable_load_sum = - decay_load(sa->runnable_load_sum, periods); - sa->util_sum = decay_load((u64)(sa->util_sum), periods); - - /* - * Step 2 - */ - delta %= 1024; - contrib = __accumulate_pelt_segments(periods, - 1024 - sa->period_contrib, delta); - } - sa->period_contrib = delta; - - contrib = cap_scale(contrib, scale_freq); - if (load) - sa->load_sum += load * contrib; - if (runnable) - sa->runnable_load_sum += runnable * contrib; - if (running) - sa->util_sum += contrib * scale_cpu; - - return periods; -} - -/* - * We can represent the historical contribution to runnable average as the - * coefficients of a geometric series. To do this we sub-divide our runnable - * history into segments of approximately 1ms (1024us); label the segment that - * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. - * - * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... - * p0 p1 p2 - * (now) (~1ms ago) (~2ms ago) - * - * Let u_i denote the fraction of p_i that the entity was runnable. - * - * We then designate the fractions u_i as our co-efficients, yielding the - * following representation of historical load: - * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... - * - * We choose y based on the with of a reasonably scheduling period, fixing: - * y^32 = 0.5 - * - * This means that the contribution to load ~32ms ago (u_32) will be weighted - * approximately half as much as the contribution to load within the last ms - * (u_0). - * - * When a period "rolls over" and we have new u_0`, multiplying the previous - * sum again by y is sufficient to update: - * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) - * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] - */ -static __always_inline int -___update_load_sum(u64 now, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - u64 delta; - - delta = now - sa->last_update_time; - /* - * This should only happen when time goes backwards, which it - * unfortunately does during sched clock init when we swap over to TSC. - */ - if ((s64)delta < 0) { - sa->last_update_time = now; - return 0; - } - - /* - * Use 1024ns as the unit of measurement since it's a reasonable - * approximation of 1us and fast to compute. - */ - delta >>= 10; - if (!delta) - return 0; - - sa->last_update_time += delta << 10; - - /* - * running is a subset of runnable (weight) so running can't be set if - * runnable is clear. But there are some corner cases where the current - * se has been already dequeued but cfs_rq->curr still points to it. - * This means that weight will be 0 but not running for a sched_entity - * but also for a cfs_rq if the latter becomes idle. As an example, - * this happens during idle_balance() which calls - * update_blocked_averages() - */ - if (!load) - runnable = running = 0; - - /* - * Now we know we crossed measurement unit boundaries. The *_avg - * accrues by two steps: - * - * Step 1: accumulate *_sum since last_update_time. If we haven't - * crossed period boundaries, finish. - */ - if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) - return 0; - - return 1; -} - -static __always_inline void -___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) -{ - u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; - - /* - * Step 2: update *_avg. - */ - sa->load_avg = div_u64(load * sa->load_sum, divider); - sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); - sa->util_avg = sa->util_sum / divider; -} - -/* - * When a task is dequeued, its estimated utilization should not be update if - * its util_avg has not been updated at least once. - * This flag is used to synchronize util_avg updates with util_est updates. - * We map this information into the LSB bit of the utilization saved at - * dequeue time (i.e. util_est.dequeued). - */ -#define UTIL_AVG_UNCHANGED 0x1 - -static inline void cfs_se_util_change(struct sched_avg *avg) -{ - unsigned int enqueued; - - if (!sched_feat(UTIL_EST)) - return; - - /* Avoid store if the flag has been already set */ - enqueued = avg->util_est.enqueued; - if (!(enqueued & UTIL_AVG_UNCHANGED)) - return; - - /* Reset flag to report util_avg has been updated */ - enqueued &= ~UTIL_AVG_UNCHANGED; - WRITE_ONCE(avg->util_est.enqueued, enqueued); -} - -/* - * sched_entity: - * - * task: - * se_runnable() == se_weight() - * - * group: [ see update_cfs_group() ] - * se_weight() = tg->weight * grq->load_avg / tg->load_avg - * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg - * - * load_sum := runnable_sum - * load_avg = se_weight(se) * runnable_avg - * - * runnable_load_sum := runnable_sum - * runnable_load_avg = se_runnable(se) * runnable_avg - * - * XXX collapse load_sum and runnable_load_sum - * - * cfq_rs: - * - * load_sum = \Sum se_weight(se) * se->avg.load_sum - * load_avg = \Sum se->avg.load_avg - * - * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum - * runnable_load_avg = \Sum se->avg.runable_load_avg - */ - -static int -__update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - return 1; - } - - return 0; -} - -static int -__update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, - cfs_rq->curr == se)) { - - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - cfs_se_util_change(&se->avg); - return 1; - } - - return 0; -} - -static int -__update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) -{ - if (___update_load_sum(now, cpu, &cfs_rq->avg, - scale_load_down(cfs_rq->load.weight), - scale_load_down(cfs_rq->runnable_weight), - cfs_rq->curr != NULL)) { - - ___update_load_avg(&cfs_rq->avg, 1, 1); - return 1; - } - - return 0; -} - #ifdef CONFIG_FAIR_GROUP_SCHED /** * update_tg_load_avg - update the tg's load avg @@ -4045,12 +3720,6 @@ util_est_dequeue(struct cfs_rq *cfs_rq, struct task_struct *p, bool task_sleep) #else /* CONFIG_SMP */ -static inline int -update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) -{ - return 0; -} - #define UPDATE_TG 0x0 #define SKIP_AGE_LOAD 0x0 #define DO_ATTACH 0x0 diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c new file mode 100644 index 0000000..e6ecbb2 --- /dev/null +++ b/kernel/sched/pelt.c @@ -0,0 +1,311 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Per Entity Load Tracking + * + * Copyright (C) 2007 Red Hat, Inc., Ingo Molnar + * + * Interactivity improvements by Mike Galbraith + * (C) 2007 Mike Galbraith + * + * Various enhancements by Dmitry Adamushko. + * (C) 2007 Dmitry Adamushko + * + * Group scheduling enhancements by Srivatsa Vaddagiri + * Copyright IBM Corporation, 2007 + * Author: Srivatsa Vaddagiri + * + * Scaled math optimizations by Thomas Gleixner + * Copyright (C) 2007, Thomas Gleixner + * + * Adaptive scheduling granularity, math enhancements by Peter Zijlstra + * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra + * + * Move PELT related code from fair.c into this pelt.c file + * Author: Vincent Guittot + */ + +#include +#include "sched.h" +#include "sched-pelt.h" +#include "pelt.h" + +/* + * Approximate: + * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) + */ +static u64 decay_load(u64 val, u64 n) +{ + unsigned int local_n; + + if (unlikely(n > LOAD_AVG_PERIOD * 63)) + return 0; + + /* after bounds checking we can collapse to 32-bit */ + local_n = n; + + /* + * As y^PERIOD = 1/2, we can combine + * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) + * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { + val >>= local_n / LOAD_AVG_PERIOD; + local_n %= LOAD_AVG_PERIOD; + } + + val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); + return val; +} + +static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) +{ + u32 c1, c2, c3 = d3; /* y^0 == 1 */ + + /* + * c1 = d1 y^p + */ + c1 = decay_load((u64)d1, periods); + + /* + * p-1 + * c2 = 1024 \Sum y^n + * n=1 + * + * inf inf + * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) + * n=0 n=p + */ + c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; + + return c1 + c2 + c3; +} + +#define cap_scale(v, s) ((v)*(s) >> SCHED_CAPACITY_SHIFT) + +/* + * Accumulate the three separate parts of the sum; d1 the remainder + * of the last (incomplete) period, d2 the span of full periods and d3 + * the remainder of the (incomplete) current period. + * + * d1 d2 d3 + * ^ ^ ^ + * | | | + * |<->|<----------------->|<--->| + * ... |---x---|------| ... |------|-----x (now) + * + * p-1 + * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 + * n=1 + * + * = u y^p + (Step 1) + * + * p-1 + * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) + * n=1 + */ +static __always_inline u32 +accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + unsigned long scale_freq, scale_cpu; + u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ + u64 periods; + + scale_freq = arch_scale_freq_capacity(cpu); + scale_cpu = arch_scale_cpu_capacity(NULL, cpu); + + delta += sa->period_contrib; + periods = delta / 1024; /* A period is 1024us (~1ms) */ + + /* + * Step 1: decay old *_sum if we crossed period boundaries. + */ + if (periods) { + sa->load_sum = decay_load(sa->load_sum, periods); + sa->runnable_load_sum = + decay_load(sa->runnable_load_sum, periods); + sa->util_sum = decay_load((u64)(sa->util_sum), periods); + + /* + * Step 2 + */ + delta %= 1024; + contrib = __accumulate_pelt_segments(periods, + 1024 - sa->period_contrib, delta); + } + sa->period_contrib = delta; + + contrib = cap_scale(contrib, scale_freq); + if (load) + sa->load_sum += load * contrib; + if (runnable) + sa->runnable_load_sum += runnable * contrib; + if (running) + sa->util_sum += contrib * scale_cpu; + + return periods; +} + +/* + * We can represent the historical contribution to runnable average as the + * coefficients of a geometric series. To do this we sub-divide our runnable + * history into segments of approximately 1ms (1024us); label the segment that + * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. + * + * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... + * p0 p1 p2 + * (now) (~1ms ago) (~2ms ago) + * + * Let u_i denote the fraction of p_i that the entity was runnable. + * + * We then designate the fractions u_i as our co-efficients, yielding the + * following representation of historical load: + * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... + * + * We choose y based on the with of a reasonably scheduling period, fixing: + * y^32 = 0.5 + * + * This means that the contribution to load ~32ms ago (u_32) will be weighted + * approximately half as much as the contribution to load within the last ms + * (u_0). + * + * When a period "rolls over" and we have new u_0`, multiplying the previous + * sum again by y is sufficient to update: + * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) + * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] + */ +static __always_inline int +___update_load_sum(u64 now, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + u64 delta; + + delta = now - sa->last_update_time; + /* + * This should only happen when time goes backwards, which it + * unfortunately does during sched clock init when we swap over to TSC. + */ + if ((s64)delta < 0) { + sa->last_update_time = now; + return 0; + } + + /* + * Use 1024ns as the unit of measurement since it's a reasonable + * approximation of 1us and fast to compute. + */ + delta >>= 10; + if (!delta) + return 0; + + sa->last_update_time += delta << 10; + + /* + * running is a subset of runnable (weight) so running can't be set if + * runnable is clear. But there are some corner cases where the current + * se has been already dequeued but cfs_rq->curr still points to it. + * This means that weight will be 0 but not running for a sched_entity + * but also for a cfs_rq if the latter becomes idle. As an example, + * this happens during idle_balance() which calls + * update_blocked_averages() + */ + if (!load) + runnable = running = 0; + + /* + * Now we know we crossed measurement unit boundaries. The *_avg + * accrues by two steps: + * + * Step 1: accumulate *_sum since last_update_time. If we haven't + * crossed period boundaries, finish. + */ + if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) + return 0; + + return 1; +} + +static __always_inline void +___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) +{ + u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; + + /* + * Step 2: update *_avg. + */ + sa->load_avg = div_u64(load * sa->load_sum, divider); + sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); + sa->util_avg = sa->util_sum / divider; +} + +/* + * sched_entity: + * + * task: + * se_runnable() == se_weight() + * + * group: [ see update_cfs_group() ] + * se_weight() = tg->weight * grq->load_avg / tg->load_avg + * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg + * + * load_sum := runnable_sum + * load_avg = se_weight(se) * runnable_avg + * + * runnable_load_sum := runnable_sum + * runnable_load_avg = se_runnable(se) * runnable_avg + * + * XXX collapse load_sum and runnable_load_sum + * + * cfq_rq: + * + * load_sum = \Sum se_weight(se) * se->avg.load_sum + * load_avg = \Sum se->avg.load_avg + * + * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum + * runnable_load_avg = \Sum se->avg.runable_load_avg + */ + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + return 1; + } + + return 0; +} + +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, + cfs_rq->curr == se)) { + + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + cfs_se_util_change(&se->avg); + return 1; + } + + return 0; +} + +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) +{ + if (___update_load_sum(now, cpu, &cfs_rq->avg, + scale_load_down(cfs_rq->load.weight), + scale_load_down(cfs_rq->runnable_weight), + cfs_rq->curr != NULL)) { + + ___update_load_avg(&cfs_rq->avg, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h new file mode 100644 index 0000000..9cac73e --- /dev/null +++ b/kernel/sched/pelt.h @@ -0,0 +1,43 @@ +#ifdef CONFIG_SMP + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); + +/* + * When a task is dequeued, its estimated utilization should not be update if + * its util_avg has not been updated at least once. + * This flag is used to synchronize util_avg updates with util_est updates. + * We map this information into the LSB bit of the utilization saved at + * dequeue time (i.e. util_est.dequeued). + */ +#define UTIL_AVG_UNCHANGED 0x1 + +static inline void cfs_se_util_change(struct sched_avg *avg) +{ + unsigned int enqueued; + + if (!sched_feat(UTIL_EST)) + return; + + /* Avoid store if the flag has been already set */ + enqueued = avg->util_est.enqueued; + if (!(enqueued & UTIL_AVG_UNCHANGED)) + return; + + /* Reset flag to report util_avg has been updated */ + enqueued &= ~UTIL_AVG_UNCHANGED; + WRITE_ONCE(avg->util_est.enqueued, enqueued); +} + +#else + +static inline int +update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) +{ + return 0; +} + +#endif + + diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 67702b4..757a3ee 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -666,7 +666,26 @@ struct dl_rq { u64 bw_ratio; }; +#ifdef CONFIG_FAIR_GROUP_SCHED +/* An entity is a task if it doesn't "own" a runqueue */ +#define entity_is_task(se) (!se->my_q) +#else +#define entity_is_task(se) 1 +#endif + #ifdef CONFIG_SMP +/* + * XXX we want to get rid of these helpers and use the full load resolution. + */ +static inline long se_weight(struct sched_entity *se) +{ + return scale_load_down(se->load.weight); +} + +static inline long se_runnable(struct sched_entity *se) +{ + return scale_load_down(se->runnable_weight); +} static inline bool sched_asym_prefer(int a, int b) { From patchwork Fri Jun 8 12:09:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137958 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp792936lji; Fri, 8 Jun 2018 05:12:15 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLj6Pke4fzkA4hb8HaCbXN3z8RRysJKEo8pJHLOXR+KDEFOy5fajAw8+efu6ZNL/eRxC1of X-Received: by 2002:a62:b509:: with SMTP id y9-v6mr5793310pfe.121.1528459935798; Fri, 08 Jun 2018 05:12:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459935; cv=none; d=google.com; s=arc-20160816; b=RW2ctOllfHvGyKVPcdrfcuS7RNvZv+EPSd4VYx487pocZ53syEthT+SWXh/agwG/Jd iEJEM/9sdhWduOdRVa4QbA+JRS6NNlE0PSIZ3D0FAQeS5MVUPIczmbYOSP2y3EU9dYHI X0SGrNI7cMPks0w8eb5yspWaNqnd66UMp6nUOYbh0qagITynhOQcYLgAdCABztVlKLB6 MHVGIeEooT9VLtzLCytLP2+C1ouZUVOF01oHQLknf0Mp/RPB+ViX6SjF9jnLsFbqwWLO 9fsdHP9BHChj5BoSEmcChmliolTjPFif67GgZCjHewSsc54iE5FH6IvkcxmzGfpOJiNO qGHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=BBIn+dyfWNg2BP9P3LhrqhOynHzD7w3d3ReCisqZKME=; b=iAbe2IHnBZIxcRBi/pJi+ediBh8ywZc0RoRh1sqhp9heVxxw5NGC8JnJ2jDC+mVvXb lLSySlcwfRVfm5qNkJ8GXmzTpZTIZLZO9a+Y0QxBFhGG98qx9fZQVZvNSANCg05iR4FO QnJijLMaDxfGQC0y5/IQ/7+Vs/f02C+K8anwti3cpbQwhjHB3V77SUpXASHL6wQeZKux 8mi9DHCF68Ft8AyWVi0Ea3KqeiWGilswEXdLg/34E9ovMS638xcautiPZ0FQ7sw20i70 P6VnuvNEgqBxhZezNuydr35lw+8KqDRYzuLzLfX4vsZPXyIHQLPa9Xv9ysDXL1ea1WjU ueqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C9L61ZhJ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x15-v6si10476277pgv.389.2018.06.08.05.12.15; Fri, 08 Jun 2018 05:12:15 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C9L61ZhJ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752828AbeFHMKV (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:21 -0400 Received: from mail-wm0-f65.google.com ([74.125.82.65]:35959 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752356AbeFHMKQ (ORCPT ); Fri, 8 Jun 2018 08:10:16 -0400 Received: by mail-wm0-f65.google.com with SMTP id v131-v6so3297210wma.1 for ; Fri, 08 Jun 2018 05:10:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BBIn+dyfWNg2BP9P3LhrqhOynHzD7w3d3ReCisqZKME=; b=C9L61ZhJ1Sa5Y6q567YMSeE2NNegSM5WgObtDSG89B7saYRNQQQBxaq05wuD5FwLJT 1wQ4LJcx7GASQonynvBrEv1zfjIjHOeDO2A2oiSemnYd+l+OHxH3MaHBEilJjH6Ad5he Lu/TBZWniV7MUesLN7clXhKEgMu25rcHaKZmo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BBIn+dyfWNg2BP9P3LhrqhOynHzD7w3d3ReCisqZKME=; b=sIMqhBYuQXUY7MYaojtg92fkvwFmfLJ6SNGtRqvgzYfRzLqMxTXkqy+uh1RrYK3jOp DrbwxGo054qmYZQ7qR2sT0vHvOyuKAbgRV17wIPOV7xdZ06vFeiO/dgue3VAtBMWwv5F yeF8c37wRC+14xgNS+WL+esypa/bVhdfGUCAgFjO2+6cj5qKb3XSBG6jcAzOpBiNBAzG l8u+NomQY9nfgWHD7KcmzQINrsVhVYVMkbA/gm3lQUyeMWn/bahe+z0PPbflvXWizZd4 iz49v+LnQ+/3vQrnlsLZhJM42UkxLndXS1eWo+LcmZYI4Tx5o7qrl52XFrOPaeNfhPOi vo1w== X-Gm-Message-State: APt69E1D8HcRVN2LXCDgA99vAcujYQiNC7Gj9CyrO/1GpSbx8ws0frKs UOLNdOz+HyYREikPtbKM1/r9WQ== X-Received: by 2002:a1c:9c0b:: with SMTP id f11-v6mr1362187wme.148.1528459815720; Fri, 08 Jun 2018 05:10:15 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:15 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 02/11] sched/pelt: remove blank line Date: Fri, 8 Jun 2018 14:09:45 +0200 Message-Id: <1528459794-13066-3-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org remove some blank lines in __update_load_avg_se() and __update_load_avg_cfs_rq Cc: Ingo Molnar Cc: Peter Zijlstra Reported-by: Patrick Bellasi Signed-off-by: Vincent Guittot --- kernel/sched/pelt.c | 2 -- 1 file changed, 2 deletions(-) -- 2.7.4 diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index e6ecbb2..4174582 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -287,7 +287,6 @@ int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_e if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, cfs_rq->curr == se)) { - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); cfs_se_util_change(&se->avg); return 1; @@ -302,7 +301,6 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) scale_load_down(cfs_rq->load.weight), scale_load_down(cfs_rq->runnable_weight), cfs_rq->curr != NULL)) { - ___update_load_avg(&cfs_rq->avg, 1, 1); return 1; } From patchwork Fri Jun 8 12:09:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137949 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp790861lji; Fri, 8 Jun 2018 05:10:25 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIFUd6MeYTrA9RmkhmUGuTr1AvcREmHzXnbruhM8JnBCV6GTZuxq46sipLHk+Jkd8HepYeV X-Received: by 2002:a17:902:e00a:: with SMTP id ca10-v6mr6266278plb.224.1528459825309; Fri, 08 Jun 2018 05:10:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459825; cv=none; d=google.com; s=arc-20160816; b=T2hy6AWiIhffGLt7ic5Y7GWI4IOI1jH2DqkE2sVZ4hlejPrwPsslmdsbn2RXHvmKhF j9KBdOSb2S0dIHvhg4YXs7rNbF5mo+gMjc7l8jsQsfq0Y4AzxHdqVGAAatHyvv4QMEBv uqIK0HT6u8hxo++eo4MS71+vroC1jA+YEDLJPhrp5LBxHkW+hg64UMOs5jwI5cYNOATD WiR2oYHokqvSt1NZ74G1uyWW+OyjXqWLxPIYfgCpyVkGzwPk6pQB7WB8d4cygMiQ3Zf6 Qlaed7A4c1KU7/IWt2rSZBl8u33+dbWVXNrmnyDQu6tx1Mh2U2QqtdGin/TRArJAvb+D mBVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=E+UXXuIxLDd2ExlgZRDY0Sm8MkqDcF4HECh9MjCADl4=; b=AMSlGrZZ4T4yJvoIhZY5F5WOEiuqN9qNrggHwumLXVHxXBIJOWedTMZLXLFuI4d7Jz mDeTW8l9yjzN3HgqH1PHPxy4TZn77dLpPVPltX8ue/fGLY1+Ld8+hbdncmUTL+smNUOk +UiHoCVPpr4qcu8O9bEOhCmFKBulib5kxY8HmEcJcJ+7dEL89yrm8XvaF5gwDOiJ4YK5 zXpvg+NqIuMj4RFr6gevSkKPeDYWwDyJXRg1zOhrc0piIEVYbV9XdOz8zEXagRb/7EYY Xqnl5DmQX7P/CRGbsffH85wFC+BDeyV2d7Xf+6nZjTZJz0KHElFDrotKqLtKX4M23iJB L8ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Mfqu9C+o; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f13-v6si9823967pgo.265.2018.06.08.05.10.25; Fri, 08 Jun 2018 05:10:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Mfqu9C+o; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752852AbeFHMKX (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:23 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:33799 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752609AbeFHMKT (ORCPT ); Fri, 8 Jun 2018 08:10:19 -0400 Received: by mail-wr0-f193.google.com with SMTP id a12-v6so13170753wro.1 for ; Fri, 08 Jun 2018 05:10:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=E+UXXuIxLDd2ExlgZRDY0Sm8MkqDcF4HECh9MjCADl4=; b=Mfqu9C+oMzK8izfYwe5RwEWYqNAtjERs2X512exH3/fFYosiFTwNao2lFF3jYz4QFW LKB4YjUNAFuOyjLIwX0i40pErQQ2xnE8w3n9vKHKqVGCNKlPy+CRvziJzudaBl9HNrmg e9z0OBCPd4dD7Bq8m9aH/N1YH72y907PMLSko= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=E+UXXuIxLDd2ExlgZRDY0Sm8MkqDcF4HECh9MjCADl4=; b=OWN9kVbMJfdCD/7I76KI0RckrXG0BgbD4eYxDAJGR1ip4yeVLR8NrvrfxXmkNp2QAw t/SsKnI56aPnVlZvavpl0/xT4HYI1QQNk4qJY11cdzc1DLWBXhltnpTJOFOUyFWGBxLd +w0eUiLBLL3BBSBQCCZB7+ATL+KO73R73N6xAtTz671QsRdtOBw70lK2vjENRpLYy9uA MDDlqnZvOji1yLawy7+ltzh9vnaxI5RhlZeO9NmQgIq//JddwMoXS1q/Vbk5qlvmQzj7 sY/uygsseLt0aLgaEbg/jac9SFavL/7upuOgXVpGeMFscBXohq3ZdJDUv3aM2s6SN8ox fMRA== X-Gm-Message-State: APt69E1itx9aAR5fqVGRXipAWzzhM2OkIkwoS3L0e38B4MYLD1geTzAY DxCRcmqmaQ321GUc1Ukm8NxY8w== X-Received: by 2002:adf:b89b:: with SMTP id i27-v6mr4592332wrf.163.1528459817717; Fri, 08 Jun 2018 05:10:17 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:17 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 03/11] sched/rt: add rt_rq utilization tracking Date: Fri, 8 Jun 2018 14:09:46 +0200 Message-Id: <1528459794-13066-4-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org schedutil governor relies on cfs_rq's util_avg to choose the OPP when cfs tasks are running. When the CPU is overloaded by cfs and rt tasks, cfs tasks are preempted by rt tasks and in this case util_avg reflects the remaining capacity but not what cfs want to use. In such case, schedutil can select a lower OPP whereas the CPU is overloaded. In order to have a more accurate view of the utilization of the CPU, we track the utilization of rt tasks. rt_rq uses rq_clock_task and cfs_rq uses cfs_rq_clock_task but they are the same at the root group level, so the PELT windows of the util_sum are aligned. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 15 ++++++++++++++- kernel/sched/pelt.c | 22 ++++++++++++++++++++++ kernel/sched/pelt.h | 7 +++++++ kernel/sched/rt.c | 13 +++++++++++++ kernel/sched/sched.h | 7 +++++++ 5 files changed, 63 insertions(+), 1 deletion(-) -- 2.7.4 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6390c66..e471fae 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7290,6 +7290,14 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } +static inline bool rt_rq_has_blocked(struct rq *rq) +{ + if (READ_ONCE(rq->avg_rt.util_avg)) + return true; + + return false; +} + #ifdef CONFIG_FAIR_GROUP_SCHED static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) @@ -7349,6 +7357,10 @@ static void update_blocked_averages(int cpu) if (cfs_rq_has_blocked(cfs_rq)) done = false; } + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + /* Don't need periodic decay once load/util_avg are null */ + if (rt_rq_has_blocked(rq)) + done = false; #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; @@ -7414,9 +7426,10 @@ static inline void update_blocked_averages(int cpu) rq_lock_irqsave(rq, &rf); update_rq_clock(rq); update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !rt_rq_has_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index 4174582..81c0d7e 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -307,3 +307,25 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) return 0; } + +/* + * rt_rq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) +{ + if (___update_load_sum(now, rq->cpu, &rq->avg_rt, + running, + running, + running)) { + ___update_load_avg(&rq->avg_rt, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index 9cac73e..b2983b7 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -3,6 +3,7 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); +int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); /* * When a task is dequeued, its estimated utilization should not be update if @@ -38,6 +39,12 @@ update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) return 0; } +static inline int +update_rt_rq_load_avg(u64 now, struct rq *rq, int running) +{ + return 0; +} + #endif diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index ef3c4e6..e8c08a8 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -5,6 +5,8 @@ */ #include "sched.h" +#include "pelt.h" + int sched_rr_timeslice = RR_TIMESLICE; int sysctl_sched_rr_timeslice = (MSEC_PER_SEC / HZ) * RR_TIMESLICE; @@ -1572,6 +1574,14 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rt_queue_push_tasks(rq); + /* + * If prev task was rt, put_prev_task() has already updated the + * utilization. We only care of the case where we start to schedule a + * rt task + */ + if (rq->curr->sched_class != &rt_sched_class) + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + return p; } @@ -1579,6 +1589,8 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p) { update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); + /* * The previous task needs to be made eligible for pushing * if it is still active @@ -2308,6 +2320,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) struct sched_rt_entity *rt_se = &p->rt; update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); watchdog(rq, p); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 757a3ee..7a16de9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -592,6 +592,7 @@ struct rt_rq { unsigned long rt_nr_total; int overloaded; struct plist_head pushable_tasks; + #endif /* CONFIG_SMP */ int rt_queued; @@ -847,6 +848,7 @@ struct rq { u64 rt_avg; u64 age_stamp; + struct sched_avg avg_rt; u64 idle_stamp; u64 avg_idle; @@ -2205,4 +2207,9 @@ static inline unsigned long cpu_util_cfs(struct rq *rq) return util; } + +static inline unsigned long cpu_util_rt(struct rq *rq) +{ + return rq->avg_rt.util_avg; +} #endif From patchwork Fri Jun 8 12:09:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137950 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp790947lji; Fri, 8 Jun 2018 05:10:29 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLdf/ebICMb3NUFh3Npwk0tUi2T098ktZMDJgiD2Uxsdqb91Pu0qjxhFl5tTLmOKkcQIx7g X-Received: by 2002:a65:40c3:: with SMTP id u3-v6mr1559515pgp.252.1528459829607; Fri, 08 Jun 2018 05:10:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459829; cv=none; d=google.com; s=arc-20160816; b=cs365v6pcIzH34Z1PJpJegcqkLJfKc4qioKa4bMAHJm8noXV5cFQEb7y5U4Y36ZG5V t2HWLP0S8ZqIUX0UKe8pXUI13UpgvJAeuu3FVgTl1gHvLMY4SIjInJQoRYjmwPXbPqiL Pdrsrz5Zc39Ev7GOQPuIFdOGwoQ0gwzN4f4q7wagHp0vHTlgNFkFkzVWAyub4SNaiD3e qt2u/xhNCoIJ+rWi2PMWKSuP29gGMS6/xe49UQEE7Ui2054mqpcj/HaFDUgDpuWrJWgc aGCSUXosyHgF5rLWQwbFCNJFZyijBdQsS65huxwOlibYf8VCOCb1847YebuP5mORSwkB gALg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=rc6KcJA7VB4fL2hYqNhJEYV529lK7iqRkyH/j0r3A1A=; b=O8UCBiIffdLTVJNDWqQlufM/R6aI9SODKwF7gpE+dveaFgpFb2MU0mCTSkWKMpmgHB el8Zq3/8PwCs/HLQ2Dm4m2jpgUVZMpqTqDueQ/0W1A7jAbIXPi8eobCuCkZ/VzeJVC0K jHAkKB7UZ8z5yaDhQI4iGQLIOmAVYYZdylSeZIm2L+pTDjqAK5/PhX3HNZd7A5xk5Lle 0CnZrG2fPnhZZo4V0dZLij3/3etDGJ9LrURLMJBlb8GEiSYI2AQ8RiUHiT4Wvp74EDzS nI4tooN9sTw3S7INU//ByVfslt6FvEFvuCj6IzcLetmlLguUL4w9aJ3IhQaTOC5KlcB6 Z/xA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=T0goJCZ5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q4-v6si54794107plb.251.2018.06.08.05.10.29; Fri, 08 Jun 2018 05:10:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=T0goJCZ5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752885AbeFHMK0 (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:26 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:33736 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752853AbeFHMKX (ORCPT ); Fri, 8 Jun 2018 08:10:23 -0400 Received: by mail-wr0-f194.google.com with SMTP id k16-v6so13172803wro.0 for ; Fri, 08 Jun 2018 05:10:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rc6KcJA7VB4fL2hYqNhJEYV529lK7iqRkyH/j0r3A1A=; b=T0goJCZ5EZvTsMkH/XgMyLAtxlGKeXWxgi9Mx0LplKzg2SvBkW43t8XO4cg31ZzsnI bjqHn3fbpmNGgThluiBF0SgZvLuUCf7rNW4WHqxknZqN9y0/LGvXuemFwZPQIDojc6Ow Y5W7QtFo9zg9r1Y1MU8Kd4ekBjKYzXsgWDfrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rc6KcJA7VB4fL2hYqNhJEYV529lK7iqRkyH/j0r3A1A=; b=FqLXutOJfrKh214jVQTI6qQmiK60/hIv9qKzSHdSCZga8HlILL8jdAL9GrSiCtn0+M M/ziLRMfPVCCTmaHdzKXB0sTHyf5NRNOSzlbHI+NzGM1WNxXY5/C6srL+UF27LPbhnZa SKo/XYJQiSPE8M0+wbjg81HuPW4LdXs7xq0rCKjkW9YssfuP90Y68pdp/OC8cLBWz0qA iBolhoIfLNN0zaVMlfloekz05y8R1RsTFPOaD6iaaIrFX26Xc4vFE53Z1QqXyHWXy9Ao y8loAMDSs32m/iuoAeq22oNo7rbeGvAgUS3if70DdocFoPJ+m6qxWAAQxqOEODPNIHVj q3ww== X-Gm-Message-State: APt69E3ZVi/YshYHWSC28ewV73rJyIHCECKsXfGcO9JBNTbq+gL9JJ7M UYVsdWh4U0ucQ0GfxgDxD8tmBQ== X-Received: by 2002:adf:dc52:: with SMTP id m18-v6mr5154307wrj.84.1528459822438; Fri, 08 Jun 2018 05:10:22 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:21 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 04/11] cpufreq/schedutil: use rt utilization tracking Date: Fri, 8 Jun 2018 14:09:47 +0200 Message-Id: <1528459794-13066-5-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Take into account rt utilization when selecting an OPP for cfs tasks in order to reflect the utilization of the CPU. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) -- 2.7.4 diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 28592b6..32f97fb 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -56,6 +56,7 @@ struct sugov_cpu { /* The fields below are only needed when sharing a policy: */ unsigned long util_cfs; unsigned long util_dl; + unsigned long util_rt; unsigned long max; /* The field below is for single-CPU policies only: */ @@ -178,15 +179,21 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); sg_cpu->util_cfs = cpu_util_cfs(rq); sg_cpu->util_dl = cpu_util_dl(rq); + sg_cpu->util_rt = cpu_util_rt(rq); } static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) { struct rq *rq = cpu_rq(sg_cpu->cpu); + unsigned long util; if (rq->rt.rt_nr_running) return sg_cpu->max; + util = sg_cpu->util_dl; + util += sg_cpu->util_cfs; + util += sg_cpu->util_rt; + /* * Utilization required by DEADLINE must always be granted while, for * FAIR, we use blocked utilization of IDLE CPUs as a mechanism to @@ -197,7 +204,7 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) * util_cfs + util_dl as requested freq. However, cpufreq is not yet * ready for such an interface. So, we only do the latter for now. */ - return min(sg_cpu->max, (sg_cpu->util_dl + sg_cpu->util_cfs)); + return min(sg_cpu->max, util); } static void sugov_set_iowait_boost(struct sugov_cpu *sg_cpu, u64 time, unsigned int flags) From patchwork Fri Jun 8 12:09:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137951 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791039lji; Fri, 8 Jun 2018 05:10:34 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLtHQ6011J7AALgvs4ByQ6BZLI3PP/aX3rN+eg77ZItWhCFwy5e2LYW5Ck8IzRu34uyKFnA X-Received: by 2002:a65:41c6:: with SMTP id b6-v6mr5085766pgq.372.1528459834289; Fri, 08 Jun 2018 05:10:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459834; cv=none; d=google.com; s=arc-20160816; b=yB+KWpOm5LI1lwNXtlpKnmsGhJg/XBQRgUzJIAHsyC2KswhUn0rlczJ9gVw7GIuqBv DDPxVLMV6a28TcM8N0Fggu/eo/ndqgp8PuaWMqhKPyY5jJ8Ay0KJ8pZOpZxQztiiimM1 ThwJj2li+wI5VuArDFvWdIiNbg8OR7UVNyRDeAMONQr+o1ZcPqphflesj8g2/o1FK+3r ETAry/u/a0abCUroOV/PJAHtB63Nx7LvIdqQmqxkJlEgAdsnfIc5lRt/G6vElz+7HOu4 W5BXV6nSoMKlY+wvbcKYoIaWHbkzuQJIjW3H1chQoqXUBLo4wG+CN/gRFU2qLgMVZFDe 8bIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=tqETDECeHvrT5aJ8KzIJhpVxpXGSJNRMYuHT26YsoE8=; b=VIa0q3tM9kd6cxJsxyqXsBRiEpwq7vw+K0m910cCIV/iH4WbSa0faqNRS1wUaaAb0E A1B3R/LdSAh+t7Zxwk/uurTHPzBmJVAFnKl4PBWZ0ppnxP10vxJ6gtO6UnGwzXVtP0FX /Qy5tDvhtsLM7i6LtKVeC1qnrbeS57sNjDr8ITSHM1ccLFSzo2IkntcYA/t5pR7M8V7V uj9emg2KP9992/jl88EvJc4CApqOnr6RMQJYXCYhFBOI3uFfrVb+XVLDWhTKoKTd/Zw5 Ept6ysciVwOv3wainlZpGNATR5keRJnSN1lrpVFFdpPyhHMACy7ZUhpKgBKoR1mhf6gY Qo8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hCR4gIPY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q4-v6si54794107plb.251.2018.06.08.05.10.34; Fri, 08 Jun 2018 05:10:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hCR4gIPY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752945AbeFHMKb (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:31 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:55447 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752881AbeFHMK1 (ORCPT ); Fri, 8 Jun 2018 08:10:27 -0400 Received: by mail-wm0-f67.google.com with SMTP id v16-v6so2949536wmh.5 for ; Fri, 08 Jun 2018 05:10:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tqETDECeHvrT5aJ8KzIJhpVxpXGSJNRMYuHT26YsoE8=; b=hCR4gIPYlQq/CgpIVbuur9qfPRPpejaabh8BzCWYSOV9dTudICAVEcGhbUpsWWubj4 12LhSpuAGTAQZwwTgoBgKcUfF/z0RROIvoiSbjjIXcKHz1QATvR+quGiV2wqQBy9VL5N vQqBHL77whAICakoVZN3vpE+WEmLxQ2inKHdw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tqETDECeHvrT5aJ8KzIJhpVxpXGSJNRMYuHT26YsoE8=; b=hHCPAQIOp2gRRwNQXyDRsEtUE/6m2DD+OtZQois+2phS42/RlaIrg7pSdNShMrhoyX TjAjP4iHqkOrmnU1uoDabiNWyQ/OWzOBC300MOXS41+OK/Dbj1TN5Dvo4tq/GJaFkd3t qmZgoG7wg3uidb0cIL+A9TtYi5uSJhfzEpzeiFYcEk1rvXheC5gWQAtLUyqcHYrb7yoN 9Lfe8PPUe3NWovS31wE/J7xEnefRG9tpG3co3BA+MK9RVfm+PF2EzL9ruSOWcVBn+MUT Tm8bIwaLQancUiO8/KX41ftKBjlPSyc5mk4UxAJneTl+u/+pyYJ8gnALq2L0Zg4mEEmv cc+A== X-Gm-Message-State: APt69E1sYhPbBB8iXltWqXt17lTPUnj82uvr2M08TC/P2kpF+WsvdIve Vf09s9vgoZbRsWZCyWFlRFYRMXvwfOY= X-Received: by 2002:a1c:aa12:: with SMTP id t18-v6mr1321547wme.54.1528459825725; Fri, 08 Jun 2018 05:10:25 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:25 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 05/11] sched/dl: add dl_rq utilization tracking Date: Fri, 8 Jun 2018 14:09:48 +0200 Message-Id: <1528459794-13066-6-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similarly to what happens with rt tasks, cfs tasks can be preempted by dl tasks and the cfs's utilization might no longer describes the real utilization level. Current dl bandwidth reflects the requirements to meet deadline when tasks are enqueued but not the current utilization of the dl sched class. We track dl class utilization to help to estimate the system utilization. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/deadline.c | 6 ++++++ kernel/sched/fair.c | 11 ++++++++--- kernel/sched/pelt.c | 22 ++++++++++++++++++++++ kernel/sched/pelt.h | 6 ++++++ kernel/sched/sched.h | 1 + 5 files changed, 43 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 1356afd..596097f 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -16,6 +16,7 @@ * Fabio Checconi */ #include "sched.h" +#include "pelt.h" struct dl_bandwidth def_dl_bandwidth; @@ -1761,6 +1762,9 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) deadline_queue_push_tasks(rq); + if (rq->curr->sched_class != &dl_sched_class) + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + return p; } @@ -1768,6 +1772,7 @@ static void put_prev_task_dl(struct rq *rq, struct task_struct *p) { update_curr_dl(rq); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } @@ -1784,6 +1789,7 @@ static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued) { update_curr_dl(rq); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); /* * Even when we have runtime, update_curr_dl() might have resulted in us * not being the leftmost task anymore. In that case NEED_RESCHED will diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e471fae..71fe74a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7290,11 +7290,14 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } -static inline bool rt_rq_has_blocked(struct rq *rq) +static inline bool others_rqs_have_blocked(struct rq *rq) { if (READ_ONCE(rq->avg_rt.util_avg)) return true; + if (READ_ONCE(rq->avg_dl.util_avg)) + return true; + return false; } @@ -7358,8 +7361,9 @@ static void update_blocked_averages(int cpu) done = false; } update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); /* Don't need periodic decay once load/util_avg are null */ - if (rt_rq_has_blocked(rq)) + if (others_rqs_have_blocked(rq)) done = false; #ifdef CONFIG_NO_HZ_COMMON @@ -7427,9 +7431,10 @@ static inline void update_blocked_averages(int cpu) update_rq_clock(rq); update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq) && !rt_rq_has_blocked(rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !others_rqs_have_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index 81c0d7e..b86405e 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -329,3 +329,25 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } + +/* + * dl_rq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) +{ + if (___update_load_sum(now, rq->cpu, &rq->avg_dl, + running, + running, + running)) { + ___update_load_avg(&rq->avg_dl, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index b2983b7..0e4f912 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -4,6 +4,7 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); +int update_dl_rq_load_avg(u64 now, struct rq *rq, int running); /* * When a task is dequeued, its estimated utilization should not be update if @@ -45,6 +46,11 @@ update_rt_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } +static inline int +update_dl_rq_load_avg(u64 now, struct rq *rq, int running) +{ + return 0; +} #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 7a16de9..4526ba6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -849,6 +849,7 @@ struct rq { u64 rt_avg; u64 age_stamp; struct sched_avg avg_rt; + struct sched_avg avg_dl; u64 idle_stamp; u64 avg_idle; From patchwork Fri Jun 8 12:09:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137952 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791101lji; Fri, 8 Jun 2018 05:10:37 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL3BsJvS3KViwv6nJWlySGEhKecRMnFp1VJLORR0CbylWphq/JwpTD+9M7CyNQOai178N97 X-Received: by 2002:a17:902:903:: with SMTP id 3-v6mr6390351plm.377.1528459837700; Fri, 08 Jun 2018 05:10:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459837; cv=none; d=google.com; s=arc-20160816; b=oYxvvl3yrUCDQCb6Hir1thfqGgl6SXa/fCItlshZmZSuBTv+f02hS9BrAZ/b770fBW 4IxneD8QolHbb63PTW4OV9bl5iSb7d9TRA+tzREvkydrH4T43q1ycLpDT+bmHFuHsIW4 pkulhPxSjc3wqwKtpTaS2wzt2CQcP8dl51UOr9XW2x1p2XvD4nuHZb0dFUcx7AYUzYfk hcguNAlC7qmTPxk/Ix0Zrt+8AJSDkT0SsGIhM3D+UZhFMtaNFFdovALqva5WJ96NzIHh esYUHH2R3Wr7ZgVzsOeogOr+ltXL1qQ2KCg/LWF90YdDyY4JWUPtLQfDU4IWIqWYAHI7 ZG1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=gUNOi+06UjqIu3kZ8NcmOFtpA+W+JZqfpkKfqG940N0=; b=k8OgXSI0EtLj+D9LDdqHaRMzMx8R6Sr7AWt0oHJ8AvkHqpV843o+YvRwD5NF+Pttpq YbF7iITjAw9en89msdFblsBYuCnE4V3FI+OsC9PGVKqGBR2R3D0FZl9ANM83VpEiu5rO ZrMbJp/kcBhYqZIHdfH8pwu6v78pwPhPxEoHempyXAK3C7QjoODFXaR+uzEN15UiN7JM T9pkI4ydgpfxWx5OoWKYv7CJ03alATKIUjqEnzerKPi8zEqPL8PvavymwhRf5Txuhujv S8iabyjJ6s2QP7KIBu8tIwoSbgfhJLi/rNBAoUgKY/N41lmZWEbHMN8vmZdZ3GrZMmAV i5xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=dU1Piiot; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e2-v6si35109018pls.575.2018.06.08.05.10.37; Fri, 08 Jun 2018 05:10:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=dU1Piiot; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752985AbeFHMKf (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:35 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:38275 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752938AbeFHMKb (ORCPT ); Fri, 8 Jun 2018 08:10:31 -0400 Received: by mail-wm0-f66.google.com with SMTP id 69-v6so3287347wmf.3 for ; Fri, 08 Jun 2018 05:10:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gUNOi+06UjqIu3kZ8NcmOFtpA+W+JZqfpkKfqG940N0=; b=dU1PiiotISxwH0ggnI0s9vTAO0eAhEgbarb6Cnz2QFt0T1NY/2Urm0UHgQmKaj2KQ6 py0rdflU/NpFecvQ2yYJRYS5sdg+VeTK/1PBnSV8SzsYaqdJXna3f/DCzU/k2tR7TgVv BHm09w4tVGE91+93NyJxJTssysMTBjZT7YgbE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gUNOi+06UjqIu3kZ8NcmOFtpA+W+JZqfpkKfqG940N0=; b=rUZBteAr5v+U8qwKU/EV+D3SGyiJl6wyJgwFbIGO6s/y0+ZsKGtrRB8lfJj9zneg++ 2KD2vqbJIYHJNBTE4UiliMTHc3g8RazzcjU+cbBoy7ORYu5MuwJKCYgYTopIX96UNi23 A9x+N5eKpB+tk+IxbyXFPOTWBl66E19LKLK1TZpeQY8TBY7ulzzVHOwSue3z00FkUIjT xigbA4v3WAKJe541nZqnhyHRAZgwmSxDLostLh/21VcO2iXcfuVvbti/CsL7rFRlg1R2 TIV+A9UiIaXHBGu65zGv4hp4dAdL84USkhC4NvO34Uq3nZzcF4gN+3897jSA15AamLan jGtA== X-Gm-Message-State: APt69E0yuzXBuVnsMIlCbEDUc9/WqAtg8V4RtS8vXdR5G7b/EDB8J54c wUKEGNF9R6SQz9BqLNxsT28m6g== X-Received: by 2002:a1c:1701:: with SMTP id 1-v6mr1327240wmx.38.1528459829981; Fri, 08 Jun 2018 05:10:29 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:29 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 06/11] cpufreq/schedutil: use dl utilization tracking Date: Fri, 8 Jun 2018 14:09:49 +0200 Message-Id: <1528459794-13066-7-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that we have both the dl class bandwidth requirement and the dl class utilization, we can detect when CPU is fully used so we should run at max. Otherwise, we keep using the dl bandwidth requirement to define the utilization of the CPU Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 24 +++++++++++++++--------- kernel/sched/sched.h | 7 ++++++- 2 files changed, 21 insertions(+), 10 deletions(-) -- 2.7.4 diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 32f97fb..25cee59 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -56,6 +56,7 @@ struct sugov_cpu { /* The fields below are only needed when sharing a policy: */ unsigned long util_cfs; unsigned long util_dl; + unsigned long bw_dl; unsigned long util_rt; unsigned long max; @@ -179,6 +180,7 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); sg_cpu->util_cfs = cpu_util_cfs(rq); sg_cpu->util_dl = cpu_util_dl(rq); + sg_cpu->bw_dl = cpu_bw_dl(rq); sg_cpu->util_rt = cpu_util_rt(rq); } @@ -190,20 +192,24 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) if (rq->rt.rt_nr_running) return sg_cpu->max; - util = sg_cpu->util_dl; - util += sg_cpu->util_cfs; + util = sg_cpu->util_cfs; util += sg_cpu->util_rt; + if ((util + sg_cpu->util_dl) >= sg_cpu->max) + return sg_cpu->max; + /* - * Utilization required by DEADLINE must always be granted while, for - * FAIR, we use blocked utilization of IDLE CPUs as a mechanism to - * gracefully reduce the frequency when no tasks show up for longer + * As there is still idle time on the CPU, we need to compute the + * utilization level of the CPU. + * Bandwidth required by DEADLINE must always be granted while, for + * FAIR and RT, we use blocked utilization of IDLE CPUs as a mechanism + * to gracefully reduce the frequency when no tasks show up for longer * periods of time. - * - * Ideally we would like to set util_dl as min/guaranteed freq and - * util_cfs + util_dl as requested freq. However, cpufreq is not yet - * ready for such an interface. So, we only do the latter for now. */ + + /* Add DL bandwidth requirement */ + util += sg_cpu->bw_dl; + return min(sg_cpu->max, util); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4526ba6..bc4305f 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2192,11 +2192,16 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {} #endif #ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL -static inline unsigned long cpu_util_dl(struct rq *rq) +static inline unsigned long cpu_bw_dl(struct rq *rq) { return (rq->dl.running_bw * SCHED_CAPACITY_SCALE) >> BW_SHIFT; } +static inline unsigned long cpu_util_dl(struct rq *rq) +{ + return READ_ONCE(rq->avg_dl.util_avg); +} + static inline unsigned long cpu_util_cfs(struct rq *rq) { unsigned long util = READ_ONCE(rq->cfs.avg.util_avg); From patchwork Fri Jun 8 12:09:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137957 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp792356lji; Fri, 8 Jun 2018 05:11:43 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKwlXBTHQte/DkM0M2g1mQp47tfLxZ7p7kfYIZwoUQfAgHPUn/nlaK7VXehxzSewxBCly2y X-Received: by 2002:a17:902:5a3:: with SMTP id f32-v6mr6418895plf.109.1528459903829; Fri, 08 Jun 2018 05:11:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459903; cv=none; d=google.com; s=arc-20160816; b=uxnUIRT1vuEUyoREDxBEZo4AqZzgTjQJXtRPjU7PDIaSO0xF1vbU/QCG6xgY0P5jmi dq5GVYp79XVlxopnm9iMxmMRLlarjiZlSYHowirHwKOIt2zC9Vb5pp5flg29rOc/3NRV ZQD3/vND6n62QiOw6b1NXDrTNuJfWWa0vpUt7rPcKCF0vUWGWnJO3aPgrkvza61G2kIX AZ6UBU7VsqHEWWwksYFkVPH3X1v2kXxKfXoeeFD0Ew/XGw8vM6IK0YWHSChLhe7tlS+3 XMiP+v/ggF62JfJMh3K60wVJTHzsBfZacsW4SYuGJhJGSvF6eXCxVaPHrjpO+Ipv4RCx R7MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=U+Tog/mH6Ne//NBSS9jdySMokBre49Tx7H+19ZJqohc=; b=rbZQjO4XaWfgH7ZgbQKJfC6GI5VQOMbgZsnPMpTik9o/amTUyPFtgVYIm3E7Mku83x v/y0Ey8ZAWyntZQo+lRE3RagLoKknzkA5gdvNi97vtzAJUW0UO7mGNZIcf5dJ/RbBXve sCfNVt+mvBgZjU9PyIpGytmZddI5D82b/lf+s+WGzKmqy/zQMbpFd3zWn5bkq56EL7LR D4EoP71auZCWZMMXws32RJz5JAEZaGWHmxeUreKbnOL8AM5Lpr84rBhmJpVIYs5K78D/ qcEVUJvNAQPIi712zQjjTUIuSu1W/AR7NQNLvt0OvXehUca+qYwOau7q+FWeqw+tNTlE fI3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Os1IOBxZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x15-v6si10476277pgv.389.2018.06.08.05.11.43; Fri, 08 Jun 2018 05:11:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Os1IOBxZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753093AbeFHMLl (ORCPT + 30 others); Fri, 8 Jun 2018 08:11:41 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:36210 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752886AbeFHMKd (ORCPT ); Fri, 8 Jun 2018 08:10:33 -0400 Received: by mail-wr0-f194.google.com with SMTP id f16-v6so13172794wrm.3 for ; Fri, 08 Jun 2018 05:10:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=U+Tog/mH6Ne//NBSS9jdySMokBre49Tx7H+19ZJqohc=; b=Os1IOBxZprzgMlHk9msuDMtGg4X789KwpEDAxB3GCEU0kJo8oi5g58lB1gmwau5Ikz 2zjm0uUxnvxs7YZWzuXk55gqOaAADJXJa0olq9l1QSrrdLpL14uKMox9+N33kWsTwGML Sge/ISn/Cron0VYRvpmWtAldidzKHmcvtORgs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=U+Tog/mH6Ne//NBSS9jdySMokBre49Tx7H+19ZJqohc=; b=kjVNHmaJGIdsDG/dX1EI3ROhAXj7xY8iUoO0AdlhU3tPePor4OWOv0n7Y/RAMC28b4 52yBDdzGP6EcV4oMldvzXk5G8/jdILcriHZ33uV+uRxEcFZBvXxATvxe5NhYdy1yzA4W 9qiBc+qiJ02DfztMdwZvQowXMShVvcxB7hrgc9A6VIlZlbwnLD9vf2gqGtVu+GTN7Bvo td32T7x4x0sJ7jsJulKrEx6FJ6ONBP+PuKswLoHK+5kjAkOaUOsB7brpQxIolW0vfPFW l1XWin6nJstPXSX2rRk50IHIgLtq2lsam0EyYzDpy+xmUIrph21I+7q94SBjMdUtJ5v8 aj9w== X-Gm-Message-State: APt69E2AkrrQgw37y/urQFG/V6jY6Rw9uiP4yG6BmJsb3j9V8z7QN91j qs2sUlUigjsskVuMrJW8EVjcRQ== X-Received: by 2002:adf:fc05:: with SMTP id i5-v6mr4835704wrr.157.1528459831870; Fri, 08 Jun 2018 05:10:31 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:31 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 07/11] sched/irq: add irq utilization tracking Date: Fri, 8 Jun 2018 14:09:50 +0200 Message-Id: <1528459794-13066-8-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org interrupt and steal time are the only remaining activities tracked by rt_avg. Like for sched classes, we can use PELT to track their average utilization of the CPU. But unlike sched class, we don't track when entering/leaving interrupt; Instead, we take into account the time spent under interrupt context when we update rqs' clock (rq_clock_task). This also means that we have to decay the normal context time and account for interrupt time during the update. That's also important to note that because rq_clock == rq_clock_task + interrupt time and rq_clock_task is used by a sched class to compute its utilization, the util_avg of a sched class only reflects the utilization of the time spent in normal context and not of the whole time of the CPU. Adding the utilization of interrupt gives an more accurate estimate of utilization of CPU. The CPU utilization is : avg_irq + (1 - avg_irq / max capacity) * /Sum avg_rq Most of the time, avg_irq is small and neglictible so the use of the approximation CPU utilization = /Sum avg_rq was enough Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 4 +++- kernel/sched/fair.c | 13 ++++++++++--- kernel/sched/pelt.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/pelt.h | 16 ++++++++++++++++ kernel/sched/sched.h | 3 +++ 5 files changed, 72 insertions(+), 4 deletions(-) -- 2.7.4 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d155518..ab58288 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -16,6 +16,8 @@ #include "../workqueue_internal.h" #include "../smpboot.h" +#include "pelt.h" + #define CREATE_TRACE_POINTS #include @@ -184,7 +186,7 @@ static void update_rq_clock_task(struct rq *rq, s64 delta) #if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) if ((irq_delta + steal) && sched_feat(NONTASK_CAPACITY)) - sched_rt_avg_update(rq, irq_delta + steal); + update_irq_load_avg(rq, irq_delta + steal); #endif } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 71fe74a..cc7a6e2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7290,7 +7290,7 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } -static inline bool others_rqs_have_blocked(struct rq *rq) +static inline bool others_have_blocked(struct rq *rq) { if (READ_ONCE(rq->avg_rt.util_avg)) return true; @@ -7298,6 +7298,11 @@ static inline bool others_rqs_have_blocked(struct rq *rq) if (READ_ONCE(rq->avg_dl.util_avg)) return true; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + if (READ_ONCE(rq->avg_irq.util_avg)) + return true; +#endif + return false; } @@ -7362,8 +7367,9 @@ static void update_blocked_averages(int cpu) } update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + update_irq_load_avg(rq, 0); /* Don't need periodic decay once load/util_avg are null */ - if (others_rqs_have_blocked(rq)) + if (others_have_blocked(rq)) done = false; #ifdef CONFIG_NO_HZ_COMMON @@ -7432,9 +7438,10 @@ static inline void update_blocked_averages(int cpu) update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + update_irq_load_avg(rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq) && !others_rqs_have_blocked(rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !others_have_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index b86405e..b43e2af 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -351,3 +351,43 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } + +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +/* + * irq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_irq_load_avg(struct rq *rq, u64 running) +{ + int ret = 0; + /* + * We know the time that has been used by interrupt since last update + * but we don't when. Let be pessimistic and assume that interrupt has + * happened just before the update. This is not so far from reality + * because interrupt will most probably wake up task and trig an update + * of rq clock during which the metric si updated. + * We start to decay with normal context time and then we add the + * interrupt context time. + * We can safely remove running from rq->clock because + * rq->clock += delta with delta >= running + */ + ret = ___update_load_sum(rq->clock - running, rq->cpu, &rq->avg_irq, + 0, + 0, + 0); + ret += ___update_load_sum(rq->clock, rq->cpu, &rq->avg_irq, + 1, + 1, + 1); + + if (ret) + ___update_load_avg(&rq->avg_irq, 1, 1); + + return ret; +} +#endif diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index 0e4f912..d2894db 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -6,6 +6,16 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); int update_dl_rq_load_avg(u64 now, struct rq *rq, int running); +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +int update_irq_load_avg(struct rq *rq, u64 running); +#else +static inline int +update_irq_load_avg(struct rq *rq, u64 running) +{ + return 0; +} +#endif + /* * When a task is dequeued, its estimated utilization should not be update if * its util_avg has not been updated at least once. @@ -51,6 +61,12 @@ update_dl_rq_load_avg(u64 now, struct rq *rq, int running) { return 0; } + +static inline int +update_irq_load_avg(struct rq *rq, u64 running) +{ + return 0; +} #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index bc4305f..b534a43 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -850,6 +850,9 @@ struct rq { u64 age_stamp; struct sched_avg avg_rt; struct sched_avg avg_dl; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + struct sched_avg avg_irq; +#endif u64 idle_stamp; u64 avg_idle; From patchwork Fri Jun 8 12:09:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137953 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791198lji; Fri, 8 Jun 2018 05:10:42 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKYTGflPlK9HsvxikvOlho4aLGcjkuHmVc6z0FQGT3Orbf3jZuvDYTLNE/2pqKovAj/2wFO X-Received: by 2002:a63:79c9:: with SMTP id u192-v6mr5082861pgc.6.1528459842431; Fri, 08 Jun 2018 05:10:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459842; cv=none; d=google.com; s=arc-20160816; b=VE4p003t233w3y9cNfHQF18DGfLtTrBoZ7WD+COE5yN6jtkFGxR+hzytP/muRxGoP7 m2Q0LKOdvBVVteIB2Z4X+J8MAcK8287Y0Ti4+9TDsgBIoihEQrifMKqZCub1Tcfmkb5Z FMQ1HHLqvM7mLFvJp923YDztfi7ISKVtIcH9Us49nHExF8L6leWp+rYHyGXhYeGRvFE5 dhsVNzDu3WlVe4vNpshIYOiVs4jtepTANxTBs3ArnSkxCrzAqdkA55nuETUDFNHBtRWF Wg7YbJ4rJ+vdfcS4hT6aWUkF6PTPLgyxfv0KP6lsmPae08lFHDcWkMz8Ro4k5BgH97+l hacw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=HwoWBjC6wKOoRZe0zFWtP5iX3jFqmWoF+0Wz0wG8mrA=; b=ZeuzcWcp1qQY+bfo7v9MFngAnZTXycV9qHAVXBty8jIIV+tCuXHqZcc/nEqBAicInQ Hgn2jNjmOcVT1Zjx48hk2++cTsenwLRTLJ+bgbBkZSCsqqidqsAOCfyDv4RJCN6NdgDn lV7nhnmlUP1/EtzKmHJ1elIGQqW9eHBQvDPW9e4M0k7/ummHPcllVzb3c8nXa3tGq4nw NdXO3RcqBo7ijF/nLvBTJhVC5SWVWd+JQdC8LJLpMZgmNiK2zI1mU2PUxU51dYrxUl/3 bJestQ6BW+X2/r0IK7W3j8efonuZ3Vgw66UG+62d5zHyS1LoQIaY2CQr4/Iky/+kT9SA 01YQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BGcXzz8+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e2-v6si35109018pls.575.2018.06.08.05.10.42; Fri, 08 Jun 2018 05:10:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BGcXzz8+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753015AbeFHMKk (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:40 -0400 Received: from mail-wr0-f196.google.com ([209.85.128.196]:42985 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752957AbeFHMKf (ORCPT ); Fri, 8 Jun 2018 08:10:35 -0400 Received: by mail-wr0-f196.google.com with SMTP id w10-v6so13151965wrk.9 for ; Fri, 08 Jun 2018 05:10:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HwoWBjC6wKOoRZe0zFWtP5iX3jFqmWoF+0Wz0wG8mrA=; b=BGcXzz8+TlqcQkjoQdCEtUV1YfujY8fKZo9uS+xfo0uPDncMZfEOcp+ywO0BdkV6+j nb58WAYAtMVv6JwQZ6ZIJv/8c9BL5P/F+9vmPUS33zRndfiwRAB1W2tPkrCcdvC5Ydzp lv9j/yk9ZaCd4phqGjDaftohNuODiYInQIJmE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HwoWBjC6wKOoRZe0zFWtP5iX3jFqmWoF+0Wz0wG8mrA=; b=rIwpWz4CQDKUw6t7HojBUONBxOv485LvsVdFshKOYZ/YYtsJqhV62GiOlnrTXDZqyn xYzjKLaE4l2yN9m5csFQUpPlw/GaGMxhUZVABsZlIR+r+fiIB/yBlP0xDSaHk7vd4sN3 x/Q44SlJ/39wYYqOoGDjZQdgNj63GpoyQDD/vscrW4DVreq3rjCPqPkE+m1kIUnvx011 acAVw/QcDX6tzLMS+V+szrebe/M+VUDcW+hcrsAZd8v/5Do1b+61TZ9nc9U0UckdyfDt /10vzScvfrc1mRnTNS5r6TlQ65XKdN/hTLx07SbMc016rra3p+KSD1dFzJukD9uOa2GQ peVQ== X-Gm-Message-State: APt69E34fzR+GH+T4plpMIaCyj5Cv9L0Do1GiK3O8rByEcKc3EeJ6I7m LH86L47Xwno8ccc9oILIVRp6Yg== X-Received: by 2002:adf:a851:: with SMTP id l75-v6mr4699300wrc.140.1528459834084; Fri, 08 Jun 2018 05:10:34 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:33 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 08/11] cpufreq/schedutil: take into account interrupt Date: Fri, 8 Jun 2018 14:09:51 +0200 Message-Id: <1528459794-13066-9-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The time spent under interrupt can be significant but it is not reflected in the utilization of CPU when deciding to choose an OPP. Now that we have access to this metric, schedutil can take it into account when selecting the OPP for a CPU. rqs utilization don't see the time spend under interrupt context and report their value in the normal context time window. We need to compensate this when adding interrupt utilization The CPU utilization is : irq util_avg + (1 - irq util_avg / max capacity ) * /Sum rq util_avg A test with iperf on hikey (octo arm64) gives: iperf -c server_address -r -t 5 w/o patch w/ patch Tx 276 Mbits/sec 304 Mbits/sec +10% Rx 299 Mbits/sec 328 Mbits/sec +09% 8 iterations stdev is lower than 1% Only WFI idle state is enable (shallowest diel state) Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 25 +++++++++++++++++++++---- kernel/sched/sched.h | 13 +++++++++++++ 2 files changed, 34 insertions(+), 4 deletions(-) -- 2.7.4 diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 25cee59..092c310 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -58,6 +58,7 @@ struct sugov_cpu { unsigned long util_dl; unsigned long bw_dl; unsigned long util_rt; + unsigned long util_irq; unsigned long max; /* The field below is for single-CPU policies only: */ @@ -182,21 +183,30 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->util_dl = cpu_util_dl(rq); sg_cpu->bw_dl = cpu_bw_dl(rq); sg_cpu->util_rt = cpu_util_rt(rq); + sg_cpu->util_irq = cpu_util_irq(rq); } static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) { struct rq *rq = cpu_rq(sg_cpu->cpu); - unsigned long util; + unsigned long util, max = sg_cpu->max; if (rq->rt.rt_nr_running) return sg_cpu->max; + if (unlikely(sg_cpu->util_irq >= max)) + return max; + + /* Sum rq utilization */ util = sg_cpu->util_cfs; util += sg_cpu->util_rt; - if ((util + sg_cpu->util_dl) >= sg_cpu->max) - return sg_cpu->max; :confirm b9 + /* + * Interrupt time is not seen by rqs utilization so we can compare + * them with the CPU capacity + */ + if ((util + sg_cpu->util_dl) >= max) + return max; /* * As there is still idle time on the CPU, we need to compute the @@ -207,10 +217,17 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) * periods of time. */ + /* Weight rqs utilization to normal context window */ + util *= (max - sg_cpu->util_irq); + util /= max; + + /* Add interrupt utilization */ + util += sg_cpu->util_irq; + /* Add DL bandwidth requirement */ util += sg_cpu->bw_dl; - return min(sg_cpu->max, util); + return min(max, util); } static void sugov_set_iowait_boost(struct sugov_cpu *sg_cpu, u64 time, unsigned int flags) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b534a43..873b567 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2221,4 +2221,17 @@ static inline unsigned long cpu_util_rt(struct rq *rq) { return rq->avg_rt.util_avg; } + +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +static inline unsigned long cpu_util_irq(struct rq *rq) +{ + return rq->avg_irq.util_avg; +} +#else +static inline unsigned long cpu_util_irq(struct rq *rq) +{ + return 0; +} + +#endif #endif From patchwork Fri Jun 8 12:09:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137956 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791968lji; Fri, 8 Jun 2018 05:11:22 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKP5joA9YMihOFh0JyqTqH0+cvTB+PbORiAj+zCuBDHnOi5BQ49JYuCQh//w35qb6j6XMUn X-Received: by 2002:a63:5fc4:: with SMTP id t187-v6mr4947186pgb.183.1528459882360; Fri, 08 Jun 2018 05:11:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459882; cv=none; d=google.com; s=arc-20160816; b=C0oQkEVyuZ0WDEXLcjhS6leHHklc9SF9g5MPt0eEBaiiWfoun/O4U8R/qHMmikULTH Sr+FeQ0YMOwRYpTxOJEO1KXthdUI3B5PErxvi4bVzjxw5uAlVlQpV9NSZlYzyZl5R8aP 1meXAsyGuRkZKinF3TpGHEeuc5mmumHKzx5o53MV/giqj0V3WKRBmRWima49LfNuo8Jp q+hR3pA0z7w6d0GCLNElpNN9lPEMhyxvFkMZyqq5T0RDg66tMEvQKU3+Zwl5mAB3ujQ2 ymf3fPfCx9adRRi4GuyU7r18YsemdfgL5tlvfHcawsTo2GRtUctEWBOlOeGknBdcyGDJ kDHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=pE+h1VuN59ZRcVvHCdR88kNf6zFNrgFECds97E/YUjc=; b=crvAv8nP/upluDvXAldQ6lnoye6PpzHHIQTLqRHuYdnccpptg5DoBiDh/9iXoUEn1x wfxfrnU5T0Uic59OKwdC1bpqn4j4LKbrfQyCbeWTq1O4za+ZBvaRELpNbn9mWC5zBMDs /tQ+Y9IiYKaYnOnxM0dFvfTfbz4TknNUOoUj1iEu95eOyfY/KyrK6+/tYYv5/tWdPbeD RwU7ApVLdaVdMF+7nxIANPl2OrFjE0KGZCQqu4pk8HcbEtjlcENPs22D5Wa8sHTH6s0J qUk6h6xpfthXV0DbF9qtr0elg7Db+6K584yXzv3fe43VBkTbC94ZUvs1o67G0vVvW4Wq PVbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eUzTuj/E; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 1-v6si56309434plv.16.2018.06.08.05.11.22; Fri, 08 Jun 2018 05:11:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eUzTuj/E; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932157AbeFHMLT (ORCPT + 30 others); Fri, 8 Jun 2018 08:11:19 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:40162 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752990AbeFHMKi (ORCPT ); Fri, 8 Jun 2018 08:10:38 -0400 Received: by mail-wm0-f67.google.com with SMTP id n5-v6so3265958wmc.5 for ; Fri, 08 Jun 2018 05:10:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pE+h1VuN59ZRcVvHCdR88kNf6zFNrgFECds97E/YUjc=; b=eUzTuj/EJznv+ZBeXTJveSQv9cLz2aA2lx7toZySZanODcUu27xRotOKHEYQCp6JpV ak68H5qqSQUk5oCOv/+2f2M3/C81vBxB2C6BHzxXz//UeyKqLlHhC80qN1u8qi6IjUyD OtPPs+Ci01sJ06ajI4/k/JjQ2JCUuNZlW5B7A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pE+h1VuN59ZRcVvHCdR88kNf6zFNrgFECds97E/YUjc=; b=DKIdivLKxmZtprSfWJHKUQU82VO8Uyh+0thkTHX2eCK39msFzJ+J79QcGeH/51Sp8e CO+rhYeNeZqxmn0Y2/tosHEptHduraOBNp9wwuP6Y5K4YVLilu8hNxu+4a3A8cyO1AsV yEprm/gmZsNi6uDgZ8T9l0vOse0eBWfS1hNTH0jig403HwbD4mooyQ7YLwmofIQxMCOn 2duWwxGx90zD6QVOfkd3P9RD1lU+UpLTpe2qVJNVS6YDp5jNXrcV07tMeI4UBLJWb+vx 70MPCxI8peywJxPQ33HWK861F7m2YAMYdxbiNKfNi9rnQexKj9ovW/GOhpT3GGw+PBgl neGg== X-Gm-Message-State: APt69E0M2i9X8h2AaZoGqUsIXTcyBZOz6uAp9zww9CbKgWMr8hpLUuB1 z2tlymeW6nVxpy7W33FPaaCP3w== X-Received: by 2002:a1c:256:: with SMTP id 83-v6mr1498778wmc.41.1528459836902; Fri, 08 Jun 2018 05:10:36 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:36 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 09/11] sched: use pelt for scale_rt_capacity() Date: Fri, 8 Jun 2018 14:09:52 +0200 Message-Id: <1528459794-13066-10-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The utilization of the CPU by rt, dl and interrupts are now tracked with PELT so we can use these metrics instead of rt_avg to evaluate the remaining capacity available for cfs class. scale_rt_capacity() behavior has been changed and now returns the remaining capacity available for cfs instead of a scaling factor because rt, dl and interrupt provide now absolute utilization value. The same formula as schedutil is used: irq util_avg + (1 - irq util_avg / max capacity ) * /Sum rq util_avg but the implementation is different because it doesn't return the same value and doesn't benefit of the same optimization Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/deadline.c | 2 -- kernel/sched/fair.c | 41 +++++++++++++++++++---------------------- kernel/sched/pelt.c | 2 +- kernel/sched/rt.c | 2 -- 4 files changed, 20 insertions(+), 27 deletions(-) -- 2.7.4 diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 596097f..c882d156 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1180,8 +1180,6 @@ static void update_curr_dl(struct rq *rq) curr->se.exec_start = now; cgroup_account_cputime(curr, delta_exec); - sched_rt_avg_update(rq, delta_exec); - if (dl_entity_is_special(dl_se)) return; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cc7a6e2..fefd71b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7551,39 +7551,36 @@ static inline int get_sd_load_idx(struct sched_domain *sd, static unsigned long scale_rt_capacity(int cpu) { struct rq *rq = cpu_rq(cpu); - u64 total, used, age_stamp, avg; - s64 delta; - - /* - * Since we're reading these variables without serialization make sure - * we read them once before doing sanity checks on them. - */ - age_stamp = READ_ONCE(rq->age_stamp); - avg = READ_ONCE(rq->rt_avg); - delta = __rq_clock_broken(rq) - age_stamp; + unsigned long max = arch_scale_cpu_capacity(NULL, cpu); + unsigned long used, irq, free; - if (unlikely(delta < 0)) - delta = 0; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + irq = READ_ONCE(rq->avg_irq.util_avg); - total = sched_avg_period() + delta; + if (unlikely(irq >= max)) + return 1; +#endif - used = div_u64(avg, total); + used = READ_ONCE(rq->avg_rt.util_avg); + used += READ_ONCE(rq->avg_dl.util_avg); - if (likely(used < SCHED_CAPACITY_SCALE)) - return SCHED_CAPACITY_SCALE - used; + if (unlikely(used >= max)) + return 1; - return 1; + free = max - used; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + free *= (max - irq); + free /= max; +#endif + return free; } static void update_cpu_capacity(struct sched_domain *sd, int cpu) { - unsigned long capacity = arch_scale_cpu_capacity(sd, cpu); + unsigned long capacity = scale_rt_capacity(cpu); struct sched_group *sdg = sd->groups; - cpu_rq(cpu)->cpu_capacity_orig = capacity; - - capacity *= scale_rt_capacity(cpu); - capacity >>= SCHED_CAPACITY_SHIFT; + cpu_rq(cpu)->cpu_capacity_orig = arch_scale_cpu_capacity(sd, cpu); if (!capacity) capacity = 1; diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index b43e2af..e2d53a6 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -237,7 +237,7 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna */ sa->load_avg = div_u64(load * sa->load_sum, divider); sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); - sa->util_avg = sa->util_sum / divider; + WRITE_ONCE(sa->util_avg, sa->util_sum / divider); } /* diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index e8c08a8..4fd2b57 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -970,8 +970,6 @@ static void update_curr_rt(struct rq *rq) curr->se.exec_start = now; cgroup_account_cputime(curr, delta_exec); - sched_rt_avg_update(rq, delta_exec); - if (!rt_bandwidth_enabled()) return; From patchwork Fri Jun 8 12:09:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137955 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791655lji; Fri, 8 Jun 2018 05:11:04 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKvDpYjboGHSLNKbOyUufIY+Wx2TOOKU//SrNuH3BbDYtmFLLNUZW1g7mDh5rj+PiObaLn5 X-Received: by 2002:a17:902:57d8:: with SMTP id g24-v6mr6506800plj.116.1528459864555; Fri, 08 Jun 2018 05:11:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459864; cv=none; d=google.com; s=arc-20160816; b=efiXy4Sv34TcN0HgMFYUH1RK5ALlZqwLRSxXkX7ZaDJPhQvFksNqsX5wlMJkQ8oqG0 zf09Js6PgJgde3pIt6k9FXOxBDOvomdmKewU3xEwbuFrZzVtU8HYF0jO4z7B/bh0QwV4 9cem8rRlQOradWjzFYh/p1LonM9mPKR0YdCb28sVKVqsfx4L2ViOWWpHW2W5qkDVkW3C ichmblJ7CS/AeuaeYkpjMCV1XCxCTCwcUm15Ax2JTu/UcUrtBTcuB4JUQuwZZ0KFkMVH BczWe7Ok2JINkpF3Lp4Q91xphvWTOZInCpobw57t61Mz6F7D/Dr2PT3QL9+5pvMmEC8/ rzQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=58hdi6z7ZlZyTvSIm6ukRmd6PdHbk/qiy6BqniuQYhU=; b=jmujK+rxwO870LQyMO3cZiZMME5P8fUliGiL0T5MrGD3zAQIxzsSWXaNjwMguMuEgx Oo/1pvB4Tiuytx0wQyrY8DqRVEqZLaazcYCWeCaJ++ucQztNL6qY5iLkD3pDIUyNl1oF 9WItJZ4qqKb/HwaQnTHeXqgdZce/c89ceuqzSTy00vf9KaF+9rk8z9rHu2PszTZAQ3lU 0Xj3AMXg0hohE7MgKf5/W8p3ualIoU28ykjBFXHVRrSQRXwcnPnGO6sNP0OpCEro5hOR 9DSW/DVrnffeD0Mj9VRySmMlvgH9ltUp1iZRzcOTgd94QMkUFe25avmLKOrewQhBTnx+ 1n+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GxldxOz+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r63-v6si54729164plb.366.2018.06.08.05.11.04; Fri, 08 Jun 2018 05:11:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GxldxOz+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753048AbeFHMKs (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:48 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:44369 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753016AbeFHMKm (ORCPT ); Fri, 8 Jun 2018 08:10:42 -0400 Received: by mail-wr0-f193.google.com with SMTP id x4-v6so5016902wro.11 for ; Fri, 08 Jun 2018 05:10:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=58hdi6z7ZlZyTvSIm6ukRmd6PdHbk/qiy6BqniuQYhU=; b=GxldxOz+1lbh48KaJ2SSUoO/92Mu8xpihJzxMwSv1MCwYocOsUphm/mCAEo4yIaNzQ hSCd3DjHx4K/c+BTdNiJmwQsnkWDeTfP10nIcpRWCOd11/SWjm/QWk70st1/VmI6/et4 IRaad6tHSgLkyPzKKtGCA7csljFugLYsY3jRA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=58hdi6z7ZlZyTvSIm6ukRmd6PdHbk/qiy6BqniuQYhU=; b=XHhSbRX/0djnTsqwZdahszNbztjU+INEUlVWduq581gwq97z7dKJfI0aSoeubhwiD6 BwA91Nzvqi9bPudXehK57MRV6BiaHaVCPVCL2J0utP59331S6vb5GS7b06JhDbQjhEUj Bs3Szr5jb1VkLfoBP1L25E8Hd4MkGHLvsFGp+tpgWgabiUajhRgS+VS/tn7s7YePln8C yhA6G9WPKRmnDMAsBINjdhUTtR1ONw+lKQThDLIvRg8w4urLWv9HWi1OoG3GWFjyTPrR 8ugzCX001pw3j4NBSvk5e3yyF2WfCwDFYxbgR9CKuXdaWVIwGZ39sLEEAd5+npgvGKox tzpQ== X-Gm-Message-State: APt69E0Q/KT9/kNCCf1HpaH01zs3o1veC0Edt44FktvtRoINR8Rj0DFp aI3EZ3YWqa2/FjKmcR2UxZeYRw== X-Received: by 2002:adf:b60d:: with SMTP id f13-v6mr4673161wre.186.1528459840661; Fri, 08 Jun 2018 05:10:40 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:40 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar Subject: [PATCH v6 10/11] sched: remove rt_avg code Date: Fri, 8 Jun 2018 14:09:53 +0200 Message-Id: <1528459794-13066-11-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org rt_avg is no more used anywhere so we can remove all related code Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 26 -------------------------- kernel/sched/fair.c | 2 -- kernel/sched/sched.h | 17 ----------------- 3 files changed, 45 deletions(-) -- 2.7.4 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ab58288..213d277 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -650,23 +650,6 @@ bool sched_can_stop_tick(struct rq *rq) return true; } #endif /* CONFIG_NO_HZ_FULL */ - -void sched_avg_update(struct rq *rq) -{ - s64 period = sched_avg_period(); - - while ((s64)(rq_clock(rq) - rq->age_stamp) > period) { - /* - * Inline assembly required to prevent the compiler - * optimising this loop into a divmod call. - * See __iter_div_u64_rem() for another example of this. - */ - asm("" : "+rm" (rq->age_stamp)); - rq->age_stamp += period; - rq->rt_avg /= 2; - } -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_RT_GROUP_SCHED) || (defined(CONFIG_FAIR_GROUP_SCHED) && \ @@ -5710,13 +5693,6 @@ void set_rq_offline(struct rq *rq) } } -static void set_cpu_rq_start_time(unsigned int cpu) -{ - struct rq *rq = cpu_rq(cpu); - - rq->age_stamp = sched_clock_cpu(cpu); -} - /* * used to mark begin/end of suspend/resume: */ @@ -5834,7 +5810,6 @@ static void sched_rq_cpu_starting(unsigned int cpu) int sched_cpu_starting(unsigned int cpu) { - set_cpu_rq_start_time(cpu); sched_rq_cpu_starting(cpu); sched_tick_start(cpu); return 0; @@ -6102,7 +6077,6 @@ void __init sched_init(void) #ifdef CONFIG_SMP idle_thread_set_boot_cpu(); - set_cpu_rq_start_time(smp_processor_id()); #endif init_sched_fair_class(); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fefd71b..3594692 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5323,8 +5323,6 @@ static void cpu_load_update(struct rq *this_rq, unsigned long this_load, this_rq->cpu_load[i] = (old_load * (scale - 1) + new_load) >> i; } - - sched_avg_update(this_rq); } /* Used instead of source_load when we know the type == 0 */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 873b567..1faab06 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -846,8 +846,6 @@ struct rq { struct list_head cfs_tasks; - u64 rt_avg; - u64 age_stamp; struct sched_avg avg_rt; struct sched_avg avg_dl; #if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) @@ -1712,11 +1710,6 @@ extern const_debug unsigned int sysctl_sched_time_avg; extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -static inline u64 sched_avg_period(void) -{ - return (u64)sysctl_sched_time_avg * NSEC_PER_MSEC / 2; -} - #ifdef CONFIG_SCHED_HRTICK /* @@ -1753,8 +1746,6 @@ unsigned long arch_scale_freq_capacity(int cpu) #endif #ifdef CONFIG_SMP -extern void sched_avg_update(struct rq *rq); - #ifndef arch_scale_cpu_capacity static __always_inline unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu) @@ -1765,12 +1756,6 @@ unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu) return SCHED_CAPACITY_SCALE; } #endif - -static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) -{ - rq->rt_avg += rt_delta * arch_scale_freq_capacity(cpu_of(rq)); - sched_avg_update(rq); -} #else #ifndef arch_scale_cpu_capacity static __always_inline @@ -1779,8 +1764,6 @@ unsigned long arch_scale_cpu_capacity(void __always_unused *sd, int cpu) return SCHED_CAPACITY_SCALE; } #endif -static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) { } -static inline void sched_avg_update(struct rq *rq) { } #endif struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf) From patchwork Fri Jun 8 12:09:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 137954 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp791419lji; Fri, 8 Jun 2018 05:10:53 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLAXysgrf/owzqjwyekOCCMKgGve2dINeUJtYAJX6CD/gfTayVcTabFi/BwEyQqzNcZNsZ7 X-Received: by 2002:a17:902:650c:: with SMTP id b12-v6mr6543387plk.31.1528459852888; Fri, 08 Jun 2018 05:10:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528459852; cv=none; d=google.com; s=arc-20160816; b=GAuSdKXQpnW2RDpGD0Brc16dID7CIrXR6sfP/Eqnk/76apQQ22suM5ziI08CrrhsQB At7rDKbIxlv4rYrVb5OYp+vsc1Mnrni2bVqdNCoT4YtVoPgCgvR6CMprcEZ+gxuZBtTd VgZf9Y/ezekFs7SdcKDoh4FZS3VN76T7q3sTOnwPSaNadbwjErc2lbExXyyzTzAnDDWV e4gJqFyf+1K92HLnEeTkfoYHtLp2mKh+YfM66zo7ojpMzutwaqVTMkjNZqadhGZnhAl5 brHsMcRYoo3ZnbRnhQk23C64pZiijgiLlqHeefEiAl4HCoWYUhpBNl9qUur2oIrBSYaG kM4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=rkvjbfQsQjKzXzCOUy1KX6h5OggUX2Y49yDI9+C1Ggs=; b=IgBPBYGE9SLe9siDfzMnWyzSgnuivq9BwNwJtgGTrAAjI6nuneV++PTRAl3qzebxnT 3n2sPTgpZWosQJJCtG1a8VaXX2uDQDcF64KiwhwI5Y+OmVAT/WxrN/wKDRbdpQY8SzKq fMLp9Be3+vXnJ+gKubI7yYm2djreJbwr65nc9jTIvMU9n+NT8RpX4buD9lBJfgE2uqo4 EFOsvlg0HQM3+0jPKbM4lY9BRI+8BCI96RzD+IFcbImxNoeICzoJAr2cn3bXEMXAqagN sCWDlReriWWC+d7fSYVGXzskb+rZVZPUSsB4yjtkbtBFeYB4jr9t0osOfBUjR+anCTqe Ndyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nx9V8eP3; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r63-v6si54729164plb.366.2018.06.08.05.10.52; Fri, 08 Jun 2018 05:10:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Nx9V8eP3; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932124AbeFHMKu (ORCPT + 30 others); Fri, 8 Jun 2018 08:10:50 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:38475 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753020AbeFHMKo (ORCPT ); Fri, 8 Jun 2018 08:10:44 -0400 Received: by mail-wr0-f194.google.com with SMTP id e18-v6so4914550wrs.5 for ; Fri, 08 Jun 2018 05:10:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rkvjbfQsQjKzXzCOUy1KX6h5OggUX2Y49yDI9+C1Ggs=; b=Nx9V8eP3HUOiBHLrxP9RBWvGKlLR9+A37fQAb9baIxrodTpunR2bvSZp6csEoICECe X7+dxAsUqe4h3ba1O4vgIgZ2AmgHYCs7YAEfs/FW8U5OsjXkE8iFGM+66w9paiCfTYj0 dizyeGKPn/pfLVylp+4oHcM+UGLXY0S1zeeRA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rkvjbfQsQjKzXzCOUy1KX6h5OggUX2Y49yDI9+C1Ggs=; b=ruQy+5+XpWfsf4q60+0KnWTgP94BXRX5xcmQbvr6HOlEZ8KRH/w0XgfBgTIctAaWwh PFUec1a4YjBdlCNG5lAl5bIWcGK4qurjN8XxySFCyiKRIlzLLHzvhZ0or0218nAt16uZ YUn2GztBf6gttOfCZc9/zjkQ/psQT5PhTytULhksyABurS/yISlswcgDFtLINsgG9RoR oni3oUC3E53zTbhmOPjdy6I8bRrnaB1Y4cLWuhgd+ATbg1x/CjAlOzpRoFehJeMOIAAu jfR82wDm/vY6z64vwgkrjo632ftR5a6WYRzmjpSLOE7pTSJs2mxXTLgEa9td/JX2DyRY uc5Q== X-Gm-Message-State: APt69E0CsDNm8lW68hcVoaWdDbWk6lTfZ3vprwWX5MbYH6uIg76PvLBS iRotuck59AZzViwN+6JSnRlN6g== X-Received: by 2002:adf:e542:: with SMTP id z2-v6mr5072182wrm.111.1528459843201; Fri, 08 Jun 2018 05:10:43 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:6c67:7ea:9f4d:8968]) by smtp.gmail.com with ESMTPSA id b204-v6sm1546003wmh.22.2018.06.08.05.10.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jun 2018 05:10:42 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, Vincent Guittot , Ingo Molnar , Kees Cook , "Luis R. Rodriguez" Subject: [PATCH v6 11/11] proc/sched: remove unused sched_time_avg_ms Date: Fri, 8 Jun 2018 14:09:54 +0200 Message-Id: <1528459794-13066-12-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> References: <1528459794-13066-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org /proc/sys/kernel/sched_time_avg_ms entry is not used anywhere. Remove it Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Kees Cook Cc: "Luis R. Rodriguez" Signed-off-by: Vincent Guittot --- include/linux/sched/sysctl.h | 1 - kernel/sched/core.c | 8 -------- kernel/sched/sched.h | 1 - kernel/sysctl.c | 8 -------- 4 files changed, 18 deletions(-) -- 2.7.4 diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 1c1a151..913488d 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -40,7 +40,6 @@ extern unsigned int sysctl_numa_balancing_scan_size; #ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost; extern __read_mostly unsigned int sysctl_sched_nr_migrate; -extern __read_mostly unsigned int sysctl_sched_time_avg; int sched_proc_update_handler(struct ctl_table *table, int write, void __user *buffer, size_t *length, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 213d277..9894bc7 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -46,14 +46,6 @@ const_debug unsigned int sysctl_sched_features = const_debug unsigned int sysctl_sched_nr_migrate = 32; /* - * period over which we average the RT time consumption, measured - * in ms. - * - * default: 1s - */ -const_debug unsigned int sysctl_sched_time_avg = MSEC_PER_SEC; - -/* * period over which we measure -rt task CPU usage in us. * default: 1s */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1faab06..6766a3c 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1706,7 +1706,6 @@ extern void deactivate_task(struct rq *rq, struct task_struct *p, int flags); extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); -extern const_debug unsigned int sysctl_sched_time_avg; extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 6a78cf7..d77a959 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -368,14 +368,6 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, - { - .procname = "sched_time_avg_ms", - .data = &sysctl_sched_time_avg, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = proc_dointvec_minmax, - .extra1 = &one, - }, #ifdef CONFIG_SCHEDSTATS { .procname = "sched_schedstats",