From patchwork Mon Apr 14 20:41:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881604 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4238A1F3B8A; Mon, 14 Apr 2025 20:52:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663964; cv=none; b=ssVwYIimYSTb9Ijd9fWSpiBELcJJFCX225mQPq4+vpqyl5aeWUJYZFR/yKK0oUknQ0eH0mk+NV1S9UKgCIxOeR3aNFrScqcOFdelEGsS3cEWBI0YLmQtNIvAXGDUjsFRWDdWmu8F2BHDGDlmlECWDkthP+VtP2FmoDaXl++Djvo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663964; c=relaxed/simple; bh=XlkGXxzoA+hEjJubrQTH3D1wqONB8G+bZQDNafTAF90=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Zs2fwNFemp+QVrqPrW2w8Qjtc5ZPo7Ovx07/eoqcF1PIgdtYHznjxUpU+inpgNO9q6DeoAROaqXYSnQUWuTES0D+RkosslVafOfTtyQJXM3uGfPmR0T6LXPTqdDjiTBy6O8+vQ1aJTKCrnmSo0dQT3XE1vUChFDtQWBceRX9uzY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=VF0ZJye9; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="VF0ZJye9" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id 1FC696625E0; Mon, 14 Apr 2025 22:52:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744663955; bh=XlkGXxzoA+hEjJubrQTH3D1wqONB8G+bZQDNafTAF90=; h=From:Subject:Date; b=VF0ZJye9yKwpNPlPvYAf8iPaeV01050/exUmhCSFSiSySwCUr/KW0MlMDd4CtwTFA ffy4nfbQ5qexAHXTzg/PpOOCi9+qd+us6RwD1AWYS9mavbgVigmLefBU4ZDKJ6ZjZv npxp2mtAzdmBZxw1bmUaH11nlKBRdc1bh3t1lSFzUWrmi7V/e4K9klq15ziyqeZ7Sl +C51CCj1hBDeP9R+8/GYj8b79nu0FNGUuf1wp/BOzsnbBN2OM+bI7BcOBtmiYgnAh7 NQZQ5ZPE7GSvE1ehj0SwsH35rL1vy7b+wkn4tyZ6hI58StVUJg28SMVMcSpWC0jS5p 1NWv59hsaO7Hg== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v1 1/5] cpufreq/sched: Check fast_switch_enabled when setting need_freq_update Date: Mon, 14 Apr 2025 22:41:32 +0200 Message-ID: <8533207.T7Z3S40VBb@rjwysocki.net> In-Reply-To: <3364921.aeNJFYEL58@rjwysocki.net> References: <3364921.aeNJFYEL58@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdduheeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepfeduudeutdeugfelffduieegiedtueefledvjeegffdttefhhffhtefhleejgfetnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucfkphepudelhedrudefiedrudelrdelgeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpeduleehrddufeeirdduledrleegpdhhvghlohepkhhrvggrtghhvghrrdhlohgtrghlnhgvthdpmhgrihhlfhhrohhmpehrjhifsehrjhifhihsohgtkhhirdhnvghtpdhnsggprhgtphhtthhopeduuddprhgtphhtthhopehlihhnuhigqdhpmhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlihhnuhigqdhkvghrnhgvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehvihhrvghshhdrkhhumhgrrheslhhinhgrrhhordhorhhgpdhrtghpthhtohepshhrihhnihhvrghsrdhprghnughruhhvrggurgeslhhinhhugidrihhnthgvlhd X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki Commit 8e461a1cb43d ("cpufreq: schedutil: Fix superfluous updates caused by need_freq_update") overlooked the fact that when fast swtching is enabled, it is the only way to pick up new policy limits and so need_freq_update needs to be set in that case when limits_changed is set. This causes policy limits updates to be missed in some cases, so make sugov_should_update_freq() also set need_freq_update when the fast_switch_enabled policy flag is set. Fixes: 8e461a1cb43d ("cpufreq: schedutil: Fix superfluous updates caused by need_freq_update") Closes: https://lore.kernel.org/lkml/Z_Tlc6Qs-tYpxWYb@linaro.org/ Reported-by: Stephan Gerhold Signed-off-by: Rafael J. Wysocki --- kernel/sched/cpufreq_schedutil.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -83,7 +83,9 @@ if (unlikely(sg_policy->limits_changed)) { sg_policy->limits_changed = false; - sg_policy->need_freq_update = cpufreq_driver_test_flags(CPUFREQ_NEED_UPDATE_LIMITS); + sg_policy->need_freq_update = + sg_policy->policy->fast_switch_enabled || + cpufreq_driver_test_flags(CPUFREQ_NEED_UPDATE_LIMITS); return true; } From patchwork Mon Apr 14 20:45:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881171 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16BB21EFFA2; Mon, 14 Apr 2025 20:52:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663962; cv=none; b=JUdDn2UL3PgaXYojnTeI+WxcWiCn1MBwKy8s1hcpkGOn0PMvcRlCuHWXAQE9YL4HUfwmAvhqgr7Mjbli1EybV6GmCKHGnYSGOPBZ3iuyllNnmuestFhO7e8OgwEqj5OyXGDensbt5tc6Dd6EcoFJ+yNKb12P2m7clClkDEOU8z0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663962; c=relaxed/simple; bh=eaMUlr2OVoDVhrTB1FmD26RARmqHIb0rHt9dweslJaw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eJE1avKcneJFleNpr40PthXkeqLFtCrGt/WaFxu/25RScT8fz9uyHFR/I0Hoe0Anj3gTG14q4n/fNzQdLqFBW+1XPpjUBU/25hRQIk8xq14TU3eE3z0aoe5MWXt800cS+LhDYfvkc8vOW4Bv7MeJjcLppedBrhsHP+n7kg88hkQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=Zq2V7ATl; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="Zq2V7ATl" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id 3BD476625D3; Mon, 14 Apr 2025 22:52:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744663954; bh=eaMUlr2OVoDVhrTB1FmD26RARmqHIb0rHt9dweslJaw=; h=From:Subject:Date; b=Zq2V7ATlF23mAFADjiYzJC/PqXstmuzzpv7cTnJ4xYAAhmVFjn6wN1Yd2+Odx8u8k iQxxkMVpyhu2P2bRucZbOWqqhEO4L4dZXTEQek6N82kPBFU4aWsxwJgRnd0pzIyP5y kNjMey1BE4nnm2ALRFqgZi1d5ko1BmT5JvQsKj98d94beMtNFZZJFe1hiF95piAYf4 so/ijEq5EHxRaN6/MdjN7SW2sIbTo7BDuwyiXdjHUmwqYWy4ky7SBXnI008GRfOeuj deX0jl9HJ1orJC5cWjqiDxv9hoAx4lZ6EJd+Xe2vAtc4vHkLMq/JmaWp7rf7Cmy3p6 JCkEB/jPByJIA== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v1 2/5] cpufreq/sched: Explicitly synchronize limits_changed flag handling Date: Mon, 14 Apr 2025 22:45:50 +0200 Message-ID: <7811331.EvYhyI6sBW@rjwysocki.net> In-Reply-To: <3364921.aeNJFYEL58@rjwysocki.net> References: <3364921.aeNJFYEL58@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdduheeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepvdffueeitdfgvddtudegueejtdffteetgeefkeffvdeftddttdeuhfegfedvjefhnecukfhppeduleehrddufeeirdduledrleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepudelhedrudefiedrudelrdelgedphhgvlhhopehkrhgvrggthhgvrhdrlhhotggrlhhnvghtpdhmrghilhhfrhhomheprhhjfiesrhhjfiihshhotghkihdrnhgvthdpnhgspghrtghpthhtohepuddupdhrtghpthhtoheplhhinhhugidqphhmsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvhhirhgvshhhrdhkuhhmrghrsehlihhnrghrohdrohhrghdprhgtphhtthhopehsrhhinhhivhgrshdrphgrnhgurhhuvhgruggrsehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepmhgrrhhiohdrlhh X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki The handling of the limits_changed flag in struct sugov_policy needs to be explicitly synchronized to ensure that cpufreq policy limits updates will not be missed in some cases. Without that synchronization it is theoretically possible that the limits_changed update in sugov_should_update_freq() will be reordered with respect to the reads of the policy limits in cpufreq_driver_resolve_freq() and in that case, if the limits_changed update in sugov_limits() clobbers the one in sugov_should_update_freq(), the new policy limits may not take effect for a long time. Likewise, the limits_changed update in sugov_limits() may theoretically get reordered with respect to the updates of the policy limits in cpufreq_set_policy() and if sugov_should_update_freq() runs between them, the policy limits change may be missed. To ensure that the above situations will not take place, add memory barriers preventing the reordering in question from taking place and add READ_ONCE() and WRITE_ONCE() annotations around all of the limits_changed flag updates to prevent the compiler from messing up with that code. Fixes: 600f5badb78c ("cpufreq: schedutil: Don't skip freq update when limits change") Cc: 5.3+ # 5.3+ Signed-off-by: Rafael J. Wysocki --- kernel/sched/cpufreq_schedutil.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -81,11 +81,22 @@ if (!cpufreq_this_cpu_can_update(sg_policy->policy)) return false; - if (unlikely(sg_policy->limits_changed)) { - sg_policy->limits_changed = false; + if (unlikely(READ_ONCE(sg_policy->limits_changed))) { + WRITE_ONCE(sg_policy->limits_changed, false); sg_policy->need_freq_update = sg_policy->policy->fast_switch_enabled || cpufreq_driver_test_flags(CPUFREQ_NEED_UPDATE_LIMITS); + + /* + * The above limits_changed update must occur before the reads + * of policy limits in cpufreq_driver_resolve_freq() or a policy + * limits update might be missed, so use a memory barrier to + * ensure it. + * + * This pairs with the write memory barrier in sugov_limits(). + */ + smp_mb(); + return true; } @@ -367,7 +378,7 @@ static inline void ignore_dl_rate_limit(struct sugov_cpu *sg_cpu) { if (cpu_bw_dl(cpu_rq(sg_cpu->cpu)) > sg_cpu->bw_min) - sg_cpu->sg_policy->limits_changed = true; + WRITE_ONCE(sg_cpu->sg_policy->limits_changed, true); } static inline bool sugov_update_single_common(struct sugov_cpu *sg_cpu, @@ -873,7 +884,16 @@ mutex_unlock(&sg_policy->work_lock); } - sg_policy->limits_changed = true; + /* + * The limits_changed update below must take place before the updates + * of policy limits in cpufreq_set_policy() or a policy limits update + * might be missed, so use a memory barrier to ensure it. + * + * This pairs with the memory barrier in sugov_should_update_freq(). + */ + smp_wmb(); + + WRITE_ONCE(sg_policy->limits_changed, true); } struct cpufreq_governor schedutil_gov = { From patchwork Mon Apr 14 20:46:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881605 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0BD01EB5D6; Mon, 14 Apr 2025 20:52:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663960; cv=none; b=gUIY07UeptYUYWIuPxSgtd/RoPbe8L7J50RqB1kOauGPo8qMSJuLgPHuClKsjTSZ2kk8zrWFrNptSjTIsGsjg4tBwF8lAHLS1mq0XxUtpKCEAxgXOBas3/sfuT+5Aj4jgzuRfD40D8i4Y+u533kjgwbX8izyvW2Bgi8SOIMCIqo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663960; c=relaxed/simple; bh=nXmokx4zsrPOcbRyjM4FeJRr/pD4smrvip028fQHIoI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n+8DmE5agzlOliwWOQgmOy2xcDailiMz7aAN7sNpzUTDOzpjVIBO9IAUQg+RCs2CXrWc0qYTXcRrAm928Q8HC3pQejCPh0XI2IqcVNVyAzbNJksiWat7J2DcB3LQqJWSLK6kEls56T8PC1jBL3mByylzPQKW7M253bX8RMoJ4kY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=M4e5jaTS; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="M4e5jaTS" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id 559E56625C7; Mon, 14 Apr 2025 22:52:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744663953; bh=nXmokx4zsrPOcbRyjM4FeJRr/pD4smrvip028fQHIoI=; h=From:Subject:Date; b=M4e5jaTSPihhj0hS5+tQx+bGdbRnup1YqrhgUBFhYvc/Xg0sja5JfXRoUgcg1ljhk ryYUCv9cSIOWaaY214z/oz87NYqNdP8X2zwWibJhitzkb+OEfg0kzdYiQv8zmnrx9/ f79cYHL0XwmvgHAtzPLc5UWrLxXIG8wLHbPCfYN0OwzAlWbLdB93t/f+rzrfrtrLc2 tWa93hPGlfiWzQ91ftPBRRoJgSxf+4b4S04VU1+Mqhq/THfDHxyR58Kq5dZuPymwVL 7Nnjpw/sF9lHlnf4p7Zdq+3t4iYssbkyShnydzh7T6oQG04MxJS1FGkZsJTvYmbUPj DXYcyMhWHPrQw== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v1 3/5] cpufreq: Rename __resolve_freq() to clamp_and_resolve_freq() Date: Mon, 14 Apr 2025 22:46:29 +0200 Message-ID: <3642289.iIbC2pHGDl@rjwysocki.net> In-Reply-To: <3364921.aeNJFYEL58@rjwysocki.net> References: <3364921.aeNJFYEL58@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdduheeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepvdffueeitdfgvddtudegueejtdffteetgeefkeffvdeftddttdeuhfegfedvjefhnecukfhppeduleehrddufeeirdduledrleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepudelhedrudefiedrudelrdelgedphhgvlhhopehkrhgvrggthhgvrhdrlhhotggrlhhnvghtpdhmrghilhhfrhhomheprhhjfiesrhhjfiihshhotghkihdrnhgvthdpnhgspghrtghpthhtohepuddupdhrtghpthhtoheplhhinhhugidqphhmsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvhhirhgvshhhrdhkuhhmrghrsehlihhnrghrohdrohhrghdprhgtphhtthhopehsrhhinhhivhgrshdrphgrnhgurhhuvhgruggrsehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepmhgrrhhiohdrlhh X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki In preparation for subsequent changes rename a function in the cpufreq core as per the subject and while at it, clean up some white space around the declaration for that function. No functional impact. Signed-off-by: Rafael J. Wysocki --- drivers/cpufreq/cpufreq.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -490,8 +490,9 @@ } EXPORT_SYMBOL_GPL(cpufreq_disable_fast_switch); -static unsigned int __resolve_freq(struct cpufreq_policy *policy, - unsigned int target_freq, unsigned int relation) +static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, + unsigned int target_freq, + unsigned int relation) { unsigned int idx; @@ -520,7 +521,7 @@ unsigned int cpufreq_driver_resolve_freq(struct cpufreq_policy *policy, unsigned int target_freq) { - return __resolve_freq(policy, target_freq, CPUFREQ_RELATION_LE); + return clamp_and_resolve_freq(policy, target_freq, CPUFREQ_RELATION_LE); } EXPORT_SYMBOL_GPL(cpufreq_driver_resolve_freq); @@ -2338,7 +2339,7 @@ if (cpufreq_disabled()) return -ENODEV; - target_freq = __resolve_freq(policy, target_freq, relation); + target_freq = clamp_and_resolve_freq(policy, target_freq, relation); pr_debug("target for CPU %u: %u kHz, relation %u, requested %u kHz\n", policy->cpu, target_freq, relation, old_target_freq); @@ -2634,8 +2635,8 @@ */ policy->min = new_data.min; policy->max = new_data.max; - policy->min = __resolve_freq(policy, policy->min, CPUFREQ_RELATION_L); - policy->max = __resolve_freq(policy, policy->max, CPUFREQ_RELATION_H); + policy->min = clamp_and_resolve_freq(policy, policy->min, CPUFREQ_RELATION_L); + policy->max = clamp_and_resolve_freq(policy, policy->max, CPUFREQ_RELATION_H); trace_cpu_frequency_limits(policy); cpufreq_update_pressure(policy); From patchwork Mon Apr 14 20:50:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881172 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 717651C2335; Mon, 14 Apr 2025 20:52:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663958; cv=none; b=qBymJLkhJkGAZ6tIucwKc7Y/FhCufkrTkQFPGhVP2EjQbwDWLqfKdadT9nCQYT6UEGdWDvmXLYbIERZIsw22S4EILHoGUbH4kDrH4d64wg8JWsNnSUj+YJZL2GQW1X2hwg/9WV4Fr4bxKpwz/6YM8o1+KBnjkYM5VI8BSQPrlCc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663958; c=relaxed/simple; bh=phD2FDOBIBTw1PXwjxza+ygz79z1ab7wLcOgmHO1XXs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pZeJNVvDVpzBqG92kbcCqz2iBzgDAEEhco1vq2K08kkbXJj9mI5mrzZ4fFOMDaW48NpiPdRY95ji3nDaKAk70m4KBF/EjcPQ84mt2beRMyTshBkA7vH0awGi3lJw40jw81Ohd0XTRvCSPpn2daq4Y96kaOv9p766HE/qnoWJl1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=YlAqLYl5; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="YlAqLYl5" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id 64CDD6625C6; Mon, 14 Apr 2025 22:52:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744663952; bh=phD2FDOBIBTw1PXwjxza+ygz79z1ab7wLcOgmHO1XXs=; h=From:Subject:Date; b=YlAqLYl5FxfI+3ZpA27gnb1ytK9KGI+oTgUHepkxoekEE5F1NuLAkxqPVYXugVIeJ poAl/fI0lVqsyEimzitaCTzAwwgOz0ROLNtbqQsFYrQxS2u0iFvVgr2KmoYbQJfbN3 zGMwQNnrH7kdpqm9oH93kS9MatMba7lb+vldrbOzSUE69iJVjDx8iXtpzwrx+iPO+k HrB4AJA8lJvW4Gs0e1eYLojQG8X0QHelytN+k0pCXvBAV9KD+lrvZ7Ok/LpYMFrznD KPrCryhl1oEFY9f/rX1HTdFmolnaOHQaMAHctbgwcZYgnsEzhdfvqMfS+EQg0s9FIC /4ZIr4/FSSBjw== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v1 4/5] cpufreq: Avoid inconsistent policy->min and policy->max Date: Mon, 14 Apr 2025 22:50:15 +0200 Message-ID: <6059554.MhkbZ0Pkbq@rjwysocki.net> In-Reply-To: <3364921.aeNJFYEL58@rjwysocki.net> References: <3364921.aeNJFYEL58@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdduheeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepvdffueeitdfgvddtudegueejtdffteetgeefkeffvdeftddttdeuhfegfedvjefhnecukfhppeduleehrddufeeirdduledrleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepudelhedrudefiedrudelrdelgedphhgvlhhopehkrhgvrggthhgvrhdrlhhotggrlhhnvghtpdhmrghilhhfrhhomheprhhjfiesrhhjfiihshhotghkihdrnhgvthdpnhgspghrtghpthhtohepuddupdhrtghpthhtoheplhhinhhugidqphhmsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvhhirhgvshhhrdhkuhhmrghrsehlihhnrghrohdrohhrghdprhgtphhtthhopehsrhhinhhivhgrshdrphgrnhgurhhuvhgruggrsehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepmhgrrhhiohdrlhh X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki Since cpufreq_driver_resolve_freq() can run in parallel with cpufreq_set_policy() and there is no synchronization between them, the former may access policy->min and policy->max while the latter is updating them and it may see intermediate values of them due to the way the update is carried out. Also the compiler is free to apply any optimizations it wants both to the stores in cpufreq_set_policy() and to the loads in cpufreq_driver_resolve_freq() which may result in additional inconsistencies. To address this, use WRITE_ONCE() when updating policy->min and policy->max in cpufreq_set_policy() and use READ_ONCE() for reading them in cpufreq_driver_resolve_freq(). Moreover, rearrange the update in cpufreq_set_policy() to avoid storing intermediate values in policy->min and policy->max with the help of the observation that their new values are expected to be properly ordered upfront. Also modify cpufreq_driver_resolve_freq() to take the possible reverse ordering of policy->min and policy->max, which may happen depending on the ordering of operations when this function and cpufreq_set_policy() run concurrently, into account by always honoring the max when it turns out to be less than the min (in case it comes from thermal throttling or similar). Fixes: 151717690694 ("cpufreq: Make policy min/max hard requirements") Cc: 5.16+ # 5.16+ Signed-off-by: Rafael J. Wysocki --- drivers/cpufreq/cpufreq.c | 46 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 10 deletions(-) --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -490,14 +490,12 @@ } EXPORT_SYMBOL_GPL(cpufreq_disable_fast_switch); -static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, - unsigned int target_freq, - unsigned int relation) +static unsigned int __resolve_freq(struct cpufreq_policy *policy, + unsigned int target_freq, + unsigned int relation) { unsigned int idx; - target_freq = clamp_val(target_freq, policy->min, policy->max); - if (!policy->freq_table) return target_freq; @@ -507,6 +505,15 @@ return policy->freq_table[idx].frequency; } +static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, + unsigned int target_freq, + unsigned int relation) +{ + target_freq = clamp_val(target_freq, policy->min, policy->max); + + return __resolve_freq(policy, target_freq, relation); +} + /** * cpufreq_driver_resolve_freq - Map a target frequency to a driver-supported * one. @@ -521,7 +528,22 @@ unsigned int cpufreq_driver_resolve_freq(struct cpufreq_policy *policy, unsigned int target_freq) { - return clamp_and_resolve_freq(policy, target_freq, CPUFREQ_RELATION_LE); + unsigned int min = READ_ONCE(policy->min); + unsigned int max = READ_ONCE(policy->max); + + /* + * If this function runs in parallel with cpufreq_set_policy(), it may + * read policy->min before the update and policy->max after the update + * or the other way around, so there is no ordering guarantee. + * + * Resolve this by always honoring the max (in case it comes from + * thermal throttling or similar). + */ + if (unlikely(min > max)) + min = max; + + return __resolve_freq(policy, clamp_val(target_freq, min, max), + CPUFREQ_RELATION_LE); } EXPORT_SYMBOL_GPL(cpufreq_driver_resolve_freq); @@ -2632,11 +2654,15 @@ * Resolve policy min/max to available frequencies. It ensures * no frequency resolution will neither overshoot the requested maximum * nor undershoot the requested minimum. + * + * Avoid storing intermediate values in policy->max or policy->min and + * compiler optimizations around them because them may be accessed + * concurrently by cpufreq_driver_resolve_freq() during the update. */ - policy->min = new_data.min; - policy->max = new_data.max; - policy->min = clamp_and_resolve_freq(policy, policy->min, CPUFREQ_RELATION_L); - policy->max = clamp_and_resolve_freq(policy, policy->max, CPUFREQ_RELATION_H); + WRITE_ONCE(policy->max, __resolve_freq(policy, new_data.max, CPUFREQ_RELATION_H)); + new_data.min = __resolve_freq(policy, new_data.min, CPUFREQ_RELATION_L); + WRITE_ONCE(policy->min, new_data.min > policy->max ? policy->max : new_data.min); + trace_cpu_frequency_limits(policy); cpufreq_update_pressure(policy); From patchwork Mon Apr 14 20:51:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881606 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64F0F1A3031; Mon, 14 Apr 2025 20:52:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663956; cv=none; b=mqn8sluick0XyvgzPJvHBeh4HbX3I6Q0bS71k2COOb216MxWJKbHW69zHSSqdpsK++rJPUxDOYgbIr5oUoDMPzB4oB2GgRsIcWDvRFfk0c6SPZ9UVLDLZ6ZmmMODFSC00yM7AUcRPLRrzyRZGr/8NHfrmsWUSIhlgUoR23B4VZQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744663956; c=relaxed/simple; bh=pZWSsQarRZslv+h8hP8m2w0AENVl4g70iL/csmnKLyA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oLn6BjGWC5fnIq3R2kT6eFjQd9iR9yRfY0UpRF1nBAb4ioIww2O0m7CsbBZ0TLAYKALwhQJLQv7zbX9lKY9D8tFV4Kq64KXDs0AyQbNoKV2vYoD+ihoOxKrV1uVNv7XW/0ZnwJhTXu7zGm6/098l7ODE8IC8fhMQBfP1rUV5qkw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=P5ZMF4zq; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="P5ZMF4zq" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id 867F46625BF; Mon, 14 Apr 2025 22:52:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744663952; bh=pZWSsQarRZslv+h8hP8m2w0AENVl4g70iL/csmnKLyA=; h=From:Subject:Date; b=P5ZMF4zqu8SrkOAsC9SjkUEX75z3bPxvwvyBWx/9JtkPKYl9cHz8ETPKhmoLfSSmu K0ahuaM9UGFdqwz+0C0k2jsZ04F3XEkLt5vyxi5C1EQOyH0qQDQhzDkmY/UfzIk58R xim/6/cgbXSEW+wDOIAIqEwf0iVr/J5fikGVmRbPQUMWVOZ9jbG3YyqKe5XqYTAQP6 yRzEKIS4HMl31Tdal67K9jilw+awIZqLILj7cg5tevTlGlVzgyU8SRZRMZmtaDZ31b +hgSLNq7dV5mRAuthoZZRKSeyxXkLXcXNSbys9uTeP4hocsnVzgRtfyWMtDyOCU7AO bAZhx3D4XOg+g== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v1 5/5] cpufreq: Eliminate clamp_and_resolve_freq() Date: Mon, 14 Apr 2025 22:51:03 +0200 Message-ID: <8574928.NyiUUSuA9g@rjwysocki.net> In-Reply-To: <3364921.aeNJFYEL58@rjwysocki.net> References: <3364921.aeNJFYEL58@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdduheejucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepvdffueeitdfgvddtudegueejtdffteetgeefkeffvdeftddttdeuhfegfedvjefhnecukfhppeduleehrddufeeirdduledrleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepudelhedrudefiedrudelrdelgedphhgvlhhopehkrhgvrggthhgvrhdrlhhotggrlhhnvghtpdhmrghilhhfrhhomheprhhjfiesrhhjfiihshhotghkihdrnhgvthdpnhgspghrtghpthhtohepuddupdhrtghpthhtoheplhhinhhugidqphhmsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvhhirhgvshhhrdhkuhhmrghrsehlihhnrghrohdrohhrghdprhgtphhtthhopehsrhhinhhivhgrshdrphgrnhgurhhuvhgruggrsehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepmhgrrhhiohdrlhh X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki Fold clamp_and_resolve_freq() into __cpufreq_driver_target() which is its only remaining caller. No intentional functional impact. Signed-off-by: Rafael J. Wysocki --- drivers/cpufreq/cpufreq.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -505,15 +505,6 @@ return policy->freq_table[idx].frequency; } -static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, - unsigned int target_freq, - unsigned int relation) -{ - target_freq = clamp_val(target_freq, policy->min, policy->max); - - return __resolve_freq(policy, target_freq, relation); -} - /** * cpufreq_driver_resolve_freq - Map a target frequency to a driver-supported * one. @@ -2361,7 +2352,8 @@ if (cpufreq_disabled()) return -ENODEV; - target_freq = clamp_and_resolve_freq(policy, target_freq, relation); + target_freq = clamp_val(target_freq, policy->min, policy->max); + target_freq = __resolve_freq(policy, target_freq, relation); pr_debug("target for CPU %u: %u kHz, relation %u, requested %u kHz\n", policy->cpu, target_freq, relation, old_target_freq);