From patchwork Mon Feb 24 02:21:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868765 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67E7C1A08B8; Mon, 24 Feb 2025 02:22:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363743; cv=none; b=sE/aa19PIn2xX4BNbLXxcsQWmuaVVsGt8vluoHiQiTBkmzmwATpQo2NI0O+RfOjJYifIEDaJ2J09tHH2hQJ28h90cfoFZDlmyFRFSDvtmR4oFXc8+OBkeE+v6zSpha73tBwyUX+l2fKrkWyMna+WFnedEBiO9bOR7QTqSGveVMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363743; c=relaxed/simple; bh=Tl0eobmzKFR8IE72hC7Onl6yHZ8FilaP/b+maBCQA8E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Fz3Cw7M3HRDuh/NQY+6sTHE0wUTcQxJlnIRlyZIsO1wru7weTEzPJNIO747q4c20yWmncHzuRabkFCP8Thwe+yMaUEXOFGnRCT7AOq5THXe6CjUNoRPyv5GMEWmRX7pRPy55V4T6vP81MYWxukfxiHF31wBSWvDPaIjx1GbD3BM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MltGf9+X; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MltGf9+X" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7c0892e4b19so484888585a.3; Sun, 23 Feb 2025 18:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363740; x=1740968540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=D1poumkMbwz8WUDhgcTGwAxfP5yNlPlZcTKSuNaDv0I=; b=MltGf9+XcGiqNjM1BPFni22Upfufkq6Vi0EDWNs4hfHR/vYGWf/Bdjb61sYsj8gwaV JPpgaYEn62qrEmfkDWxrGyyXyjdyDVchqZyt49mm+7JMaKqJ8BBm5w48Pb95J7LChrvv iExHsbmZUJqE792IKfdVxrLccq8c9hfqojX55iQYadYfy+vMvQhWMqOaOTTDrL+boZ2x Ao4F4A9Mt4DhVZ5DgfXloIVaw9Vz48smlviMSgwbYvuUh1C7cBe+nIVd0iHUsrvhNyqs pMW9N468LBvESiosj9gibkj/xRmptqjpTohHXRe8mnpo5H/2rrzX/izC+PCGC3CN4n/Z TQxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363740; x=1740968540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=D1poumkMbwz8WUDhgcTGwAxfP5yNlPlZcTKSuNaDv0I=; b=LadNns5EwlcH2KWeG96IfZMAJ2QSdmMtbkDzhxs4n9qE51/xDa/HPElYsAjaZUTLEE Xv5/8DLHfHHHUCFeHKhA6eTlP6mQeBdO8G73/hkrXgV/XKPSSeTIVyN5zovIEJM4fHS4 ShjD0GP+swMEZ0CuTeBuxCziMVLwqU4Tt7fFoEe5EGuwPEGqrb89bE7JCnn6QPZyKm4I AZSJUIVsa2f50xagswtbAmNK3Y8qJ/yn0Zr/DVuOLHDioDCO37MVfBppnzGUr8FtnS5b vLXGOZAa+vBBGeBz0gB97sjVOWP9jNZfwUAW3apvwSS+r0B6dOWzgVVCC57LFk3EKNf8 fRoQ== X-Forwarded-Encrypted: i=1; AJvYcCVWhhEPPjF98DI4O5iN/q4s6T8Tm2Cb2QZj6dR8lqS+Vk/c3xkYbZ2Ty+K1/HU8Ixm+O78=@vger.kernel.org, AJvYcCVaq+sMkKIV15Q0rCrFFyo4kWz/StQ9jsmwNe1I7uiNjc40eEakjO+Bl9abJK2q2bQ0sQrmK8yglHEEAnRo@vger.kernel.org, AJvYcCWVkl8HGdMyo0iVP9YGHPjT5lazk10VGBAygMWze8L7SuAvTb4SwIY0MTT+pOkj3Dtyu6Zz0iIleQfr2v+L9NjH@vger.kernel.org X-Gm-Message-State: AOJu0YyyaNXKFbowWH+NJxEeoBctGe2aWb6RvPuB43wasS+vIZxyX1yL sm+ePgCeeiJ13sq1Xysg3p3fjMBqL/TMtDYDdi9lOrf9mAo6FHqs X-Gm-Gg: ASbGncvooaTYHaEj5f8OYWsBQcy2R94dtG9/8Kx9sh4lL6Ng7x43XGLQo4LK++2lqiX xdiLk9xknXda/6Dcu36hmPiUCAYt3+Goy0D2armR4NyZR+8yijc2ixLfEpbaBzZZrGfw3f7gB9p /nEo7W4jeq3gXMc9xoklVNpfD74kBBHfuleAVHyIvLYOIb4iH76NjgSmtEFNUjoiRqHwQf+oUSa XsTVLLJ6A3SHLZZWj7N0wdlWOX3Ua/QbruO4IDA4zzHErUJ8onauExqO419kaqqloj7UgWRKyqG ZckyIBV+TEc/p6oR5H8wqdIQFLBMeqH7mecpvEzGXJsIJGUbiSxYVfbpye8N6KbnE0LQUInwGSN rdIfjcGJdzn3xTdGM X-Google-Smtp-Source: AGHT+IEbG51k5BQfXyudC0joXeQ0yrs56uBl53PyH/gx5UidLvBI1pZnngtdFjGKW1kJtA5lDdnEOQ== X-Received: by 2002:a05:620a:4113:b0:7c0:a389:f23c with SMTP id af79cd13be357-7c0cef7a394mr1888342285a.49.1740363740269; Sun, 23 Feb 2025 18:22:20 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0a609cb1bsm856081785a.111.2025.02.23.18.22.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:19 -0800 (PST) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id 58A4F1200043; Sun, 23 Feb 2025 21:22:19 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Sun, 23 Feb 2025 21:22:19 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:18 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Neeraj Upadhyay , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 02/20] srcu: Define SRCU_READ_FLAVOR_ALL in terms of symbols Date: Sun, 23 Feb 2025 18:21:56 -0800 Message-Id: <20250224022214.12037-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit defines SRCU_READ_FLAVOR_ALL in terms of the SRCU_READ_FLAVOR_* definitions instead of a hexadecimal constant. Suggested-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index d7ba46e74f58..f6f779b9d9ff 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -47,7 +47,8 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). #define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). #define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite(). -#define SRCU_READ_FLAVOR_ALL 0x7 // All of the above. +#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \ + SRCU_READ_FLAVOR_LITE) // All of the above. #ifdef CONFIG_TINY_SRCU #include From patchwork Mon Feb 24 02:21:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868764 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57F811A7253; Mon, 24 Feb 2025 02:22:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363746; cv=none; b=K/t5bf10PlHWwWSKPyykL3f9zGTe/JQeJ1YG+KewgotueeOpWYgKOJFhOvp25dwJJ5ICwB+Xuh4EFbA7ce32snpByY41hIszpe5Wg1tKGlnn7BLDYpsTIr7h3frzM2NYlLV8/Qqdd2TRTnOgbh6t+sM2aagzaRq+n4ksnT9PeqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363746; c=relaxed/simple; bh=wANPTy97bY/zTdpU+z3ZCDuJvLVgmYnNsjBC6Zp8iT4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=an1i3JkNjHodotAUaMID8IFFgcpv3wwRV5yS9VX6la00eQ89P6ru8dTYMcuCfwtG1DHyVJI/yxblZ0c/8gFMtLra8ZjGNIuk7E4Zf4eNILaD8BCNhK+EUf95EXWDh6cCnLe6oWf9KnLy3UAIl9ADDcAADcBwTlhiSsIiTyZgeNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V8ZQi1vX; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V8ZQi1vX" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-471fabc5bf5so22118501cf.3; Sun, 23 Feb 2025 18:22:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363743; x=1740968543; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=qOLMEk+pK7yaifZ1uynRQqEOYQw0oxUKxlJ/oKqOFnU=; b=V8ZQi1vXaLPxirPuDrtavSTG2HOBhbfCAW00ECngBvTufAhCBT6O+L58jy2yrVQHld Gc79KMaOSmAsBzWJ6B8jjyJyFnNyPctW9IiNjNV6Hk3X/PM9G0SARpLUxPSglK7W4pCz bWe8wQ6ZyVKNbq7Dp1YKNACQypHiXwP/Qlm7kg+z+yf0T7eyp3nWeosqDx/JUlfFnP+1 7s+Iy27xrvaazxVQ1WIptU3tA4in+7T5pytxdt9WAMqJ9/cFuMqAEJlH1pMgz0wfw5vc tFz5ndHwVDwiizdRxLP1ykSGKGNSoaPNXHY6ynOtQpaRnBysC3e6wk8cmml1s2B+lNSB 21MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363743; x=1740968543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qOLMEk+pK7yaifZ1uynRQqEOYQw0oxUKxlJ/oKqOFnU=; b=LeU4av4pworWOsjijcpFWNhjFe1pLiK7Qv0ivj3Lvc+xKXsEzY4vQslG4iXpIh4JHp lXQj0cYF/38V4AvnKC5eiSqlAHmyW3k1bYbNsxzVyGZ5Pt3eKHvlk9X7eNxm5j4JpHLe tuRb9hMUY1EKeBRor36q/o40wHOG/XTTMYlSXg4UYZ83yhnuhLn+3wZ2ucKhIcOiXK0G /+7qrqqhQkUicxS+9eAteqF6taemPShTs1NeonyxLM52KAG2zOuJ3IhCFWQH2E0y83Vs jzB7tJ+F4a/z1awzJpR8LgyPBJKoFssuFHhPgLmNtSxgTEJgERZ31XhbDJBgAB1zpcqJ kBhA== X-Forwarded-Encrypted: i=1; AJvYcCUkD1depFC10FcWeGdkQVZ2zXJ6vg1vzCsGJQFkfySNMMVBMFYnJdDYIx/utwVJyKocpj98yKgvm3h/9WH9CpGU@vger.kernel.org, AJvYcCVTG0XxA+RJwbJ1SekjDI9BA0yFZJXoLiLN6huaJwmkaxTENWiEzE0GRikgNEMe+VmWZdU=@vger.kernel.org, AJvYcCVUZLeVZu5riZRluogMryASgRa5BwZMPu6GFnRvbZoBt6RpelT+hsgLJUKDsu+WJWZDpH5HBuqkQttKr4iI@vger.kernel.org X-Gm-Message-State: AOJu0YxxIck14CiLslo8iMknwKUJ3EMZtwCRuv9JLtsQGleEizGWSnTO YrDtw43kY+IxSx0/81OZhX0BLrpQpTz26+nstFLc/rwNPCZLedGo X-Gm-Gg: ASbGnctDeUEF3OobbIquwTqrPkLejXtL0KtLKrsiuhcyCfslg3S9ekZsm0KOA+Ac14C uxIijixTK0wTuoL5PwBNYTR/uOoR91iVCl+g/pSjwprw/MY0qrL1FUUo2RFX+zVYBVsgPzAF28n vgaIkOGbS54IozVSGivEdwbrHOT0kEpalzyFb0UX0lYEkr9e/+k0oRQh7kYiF/nueyMrP+WYoL9 8SBiN8L2R28Uxg5MQtmzjhDUo6/8CvaWtOTwW7vxHoWqrNTiOj34ziWFrQC4maZqEEq1QzpP7Os gsOT4xlYwnXqoe4M98R9YMziZ357tJt3CYRHNn8gZItslq7YDMhXwmB2rGQXd2cbqB+IViLFlzr H6ZDUWn0pIHGtmpOR X-Google-Smtp-Source: AGHT+IFx4DEUzY8iXU9N91XMG8aD4j9QaEqr6DyvWhkoV6w4KF1Me5GNMAhFv0AUVhGfqJH7aUxvhQ== X-Received: by 2002:a05:622a:2d2:b0:471:a523:6ac1 with SMTP id d75a77b69052e-472228a7e9bmr150011431cf.6.1740363743099; Sun, 23 Feb 2025 18:22:23 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-471ed70947fsm89446881cf.14.2025.02.23.18.22.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:22 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 166D91200043; Sun, 23 Feb 2025 21:22:22 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:22 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:21 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 04/20] srcu: Pull ->srcu_{un,}lock_count into a new srcu_ctr structure Date: Sun, 23 Feb 2025 18:21:58 -0800 Message-Id: <20250224022214.12037-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit prepares for array-index-free srcu_read_lock*() by moving the ->srcu_{un,}lock_count fields into a new srcu_ctr structure. This will permit ->srcu_index to be replaced by a per-CPU pointer to this structure. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 13 +++-- kernel/rcu/srcutree.c | 115 +++++++++++++++++++-------------------- 2 files changed, 66 insertions(+), 62 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index b17814c9d1c7..c794d599db5c 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -17,14 +17,19 @@ struct srcu_node; struct srcu_struct; +/* One element of the srcu_data srcu_ctrs array. */ +struct srcu_ctr { + atomic_long_t srcu_locks; /* Locks per CPU. */ + atomic_long_t srcu_unlocks; /* Unlocks per CPU. */ +}; + /* * Per-CPU structure feeding into leaf srcu_node, similar in function * to rcu_node. */ struct srcu_data { /* Read-side state. */ - atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */ - atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */ + struct srcu_ctr srcu_ctrs[2]; /* Locks and unlocks per CPU. */ int srcu_reader_flavor; /* Reader flavor for srcu_struct structure? */ /* Values: SRCU_READ_FLAVOR_.* */ @@ -221,7 +226,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */ + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); /* Y */ barrier(); /* Avoid leaking the critical section. */ return idx; } @@ -240,7 +245,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) { barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */ + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); /* Z */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); } diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e69ce9d59abf..d7ee2f345e19 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -116,8 +116,9 @@ do { \ /* * Initialize SRCU per-CPU data. Note that statically allocated * srcu_struct structures might already have srcu_read_lock() and - * srcu_read_unlock() running against them. So if the is_static parameter - * is set, don't initialize ->srcu_lock_count[] and ->srcu_unlock_count[]. + * srcu_read_unlock() running against them. So if the is_static + * parameter is set, don't initialize ->srcu_ctrs[].srcu_locks and + * ->srcu_ctrs[].srcu_unlocks. */ static void init_srcu_struct_data(struct srcu_struct *ssp) { @@ -128,8 +129,6 @@ static void init_srcu_struct_data(struct srcu_struct *ssp) * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ - BUILD_BUG_ON(ARRAY_SIZE(sdp->srcu_lock_count) != - ARRAY_SIZE(sdp->srcu_unlock_count)); for_each_possible_cpu(cpu) { sdp = per_cpu_ptr(ssp->sda, cpu); spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); @@ -429,10 +428,10 @@ static bool srcu_gp_is_expedited(struct srcu_struct *ssp) } /* - * Computes approximate total of the readers' ->srcu_lock_count[] values - * for the rank of per-CPU counters specified by idx, and returns true if - * the caller did the proper barrier (gp), and if the count of the locks - * matches that of the unlocks passed in. + * Computes approximate total of the readers' ->srcu_ctrs[].srcu_locks + * values for the rank of per-CPU counters specified by idx, and returns + * true if the caller did the proper barrier (gp), and if the count of + * the locks matches that of the unlocks passed in. */ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, unsigned long unlocks) { @@ -443,7 +442,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_lock_count[idx]); + sum += atomic_long_read(&sdp->srcu_ctrs[idx].srcu_locks); if (IS_ENABLED(CONFIG_PROVE_RCU)) mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } @@ -455,8 +454,8 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns } /* - * Returns approximate total of the readers' ->srcu_unlock_count[] values - * for the rank of per-CPU counters specified by idx. + * Returns approximate total of the readers' ->srcu_ctrs[].srcu_unlocks + * values for the rank of per-CPU counters specified by idx. */ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, unsigned long *rdm) { @@ -467,7 +466,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, u for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_unlock_count[idx]); + sum += atomic_long_read(&sdp->srcu_ctrs[idx].srcu_unlocks); mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), @@ -510,9 +509,9 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * been no readers on this index at some point in this function. * But there might be more readers, as a task might have read * the current ->srcu_idx but not yet have incremented its CPU's - * ->srcu_lock_count[idx] counter. In fact, it is possible + * ->srcu_ctrs[idx].srcu_locks counter. In fact, it is possible * that most of the tasks have been preempted between fetching - * ->srcu_idx and incrementing ->srcu_lock_count[idx]. And there + * ->srcu_idx and incrementing ->srcu_ctrs[idx].srcu_locks. And there * could be almost (ULONG_MAX / sizeof(struct task_struct)) tasks * in a system whose address space was fully populated with memory. * Call this quantity Nt. @@ -521,36 +520,36 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * code for a long time. That now-preempted updater has already * flipped ->srcu_idx (possibly during the preceding grace period), * done an smp_mb() (again, possibly during the preceding grace - * period), and summed up the ->srcu_unlock_count[idx] counters. + * period), and summed up the ->srcu_ctrs[idx].srcu_unlocks counters. * How many times can a given one of the aforementioned Nt tasks - * increment the old ->srcu_idx value's ->srcu_lock_count[idx] + * increment the old ->srcu_idx value's ->srcu_ctrs[idx].srcu_locks * counter, in the absence of nesting? * * It can clearly do so once, given that it has already fetched - * the old value of ->srcu_idx and is just about to use that value - * to index its increment of ->srcu_lock_count[idx]. But as soon as - * it leaves that SRCU read-side critical section, it will increment - * ->srcu_unlock_count[idx], which must follow the updater's above - * read from that same value. Thus, as soon the reading task does - * an smp_mb() and a later fetch from ->srcu_idx, that task will be - * guaranteed to get the new index. Except that the increment of - * ->srcu_unlock_count[idx] in __srcu_read_unlock() is after the - * smp_mb(), and the fetch from ->srcu_idx in __srcu_read_lock() - * is before the smp_mb(). Thus, that task might not see the new - * value of ->srcu_idx until the -second- __srcu_read_lock(), - * which in turn means that this task might well increment - * ->srcu_lock_count[idx] for the old value of ->srcu_idx twice, - * not just once. + * the old value of ->srcu_idx and is just about to use that + * value to index its increment of ->srcu_ctrs[idx].srcu_locks. + * But as soon as it leaves that SRCU read-side critical section, + * it will increment ->srcu_ctrs[idx].srcu_unlocks, which must + * follow the updater's above read from that same value. Thus, + * as soon the reading task does an smp_mb() and a later fetch from + * ->srcu_idx, that task will be guaranteed to get the new index. + * Except that the increment of ->srcu_ctrs[idx].srcu_unlocks + * in __srcu_read_unlock() is after the smp_mb(), and the fetch + * from ->srcu_idx in __srcu_read_lock() is before the smp_mb(). + * Thus, that task might not see the new value of ->srcu_idx until + * the -second- __srcu_read_lock(), which in turn means that this + * task might well increment ->srcu_ctrs[idx].srcu_locks for the + * old value of ->srcu_idx twice, not just once. * * However, it is important to note that a given smp_mb() takes * effect not just for the task executing it, but also for any * later task running on that same CPU. * - * That is, there can be almost Nt + Nc further increments of - * ->srcu_lock_count[idx] for the old index, where Nc is the number - * of CPUs. But this is OK because the size of the task_struct - * structure limits the value of Nt and current systems limit Nc - * to a few thousand. + * That is, there can be almost Nt + Nc further increments + * of ->srcu_ctrs[idx].srcu_locks for the old index, where Nc + * is the number of CPUs. But this is OK because the size of + * the task_struct structure limits the value of Nt and current + * systems limit Nc to a few thousand. * * OK, but what about nesting? This does impose a limit on * nesting of half of the size of the task_struct structure @@ -581,10 +580,10 @@ static bool srcu_readers_active(struct srcu_struct *ssp) for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_lock_count[0]); - sum += atomic_long_read(&sdp->srcu_lock_count[1]); - sum -= atomic_long_read(&sdp->srcu_unlock_count[0]); - sum -= atomic_long_read(&sdp->srcu_unlock_count[1]); + sum += atomic_long_read(&sdp->srcu_ctrs[0].srcu_locks); + sum += atomic_long_read(&sdp->srcu_ctrs[1].srcu_locks); + sum -= atomic_long_read(&sdp->srcu_ctrs[0].srcu_unlocks); + sum -= atomic_long_read(&sdp->srcu_ctrs[1].srcu_unlocks); } return sum; } @@ -746,7 +745,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) int idx; idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ return idx; } @@ -760,7 +759,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -777,7 +776,7 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); idx = READ_ONCE(ssp->srcu_idx) & 0x1; - atomic_long_inc(&sdp->srcu_lock_count[idx]); + atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_locks); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ return idx; } @@ -793,7 +792,7 @@ void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ - atomic_long_inc(&sdp->srcu_unlock_count[idx]); + atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_unlocks); } EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); @@ -1123,17 +1122,17 @@ static void srcu_flip(struct srcu_struct *ssp) /* * Because the flip of ->srcu_idx is executed only if the * preceding call to srcu_readers_active_idx_check() found that - * the ->srcu_unlock_count[] and ->srcu_lock_count[] sums matched - * and because that summing uses atomic_long_read(), there is - * ordering due to a control dependency between that summing and - * the WRITE_ONCE() in this call to srcu_flip(). This ordering - * ensures that if this updater saw a given reader's increment from - * __srcu_read_lock(), that reader was using a value of ->srcu_idx - * from before the previous call to srcu_flip(), which should be - * quite rare. This ordering thus helps forward progress because - * the grace period could otherwise be delayed by additional - * calls to __srcu_read_lock() using that old (soon to be new) - * value of ->srcu_idx. + * the ->srcu_ctrs[].srcu_unlocks and ->srcu_ctrs[].srcu_locks sums + * matched and because that summing uses atomic_long_read(), + * there is ordering due to a control dependency between that + * summing and the WRITE_ONCE() in this call to srcu_flip(). + * This ordering ensures that if this updater saw a given reader's + * increment from __srcu_read_lock(), that reader was using a value + * of ->srcu_idx from before the previous call to srcu_flip(), + * which should be quite rare. This ordering thus helps forward + * progress because the grace period could otherwise be delayed + * by additional calls to __srcu_read_lock() using that old (soon + * to be new) value of ->srcu_idx. * * This sum-equality check and ordering also ensures that if * a given call to __srcu_read_lock() uses the new value of @@ -1914,8 +1913,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) struct srcu_data *sdp; sdp = per_cpu_ptr(ssp->sda, cpu); - u0 = data_race(atomic_long_read(&sdp->srcu_unlock_count[!idx])); - u1 = data_race(atomic_long_read(&sdp->srcu_unlock_count[idx])); + u0 = data_race(atomic_long_read(&sdp->srcu_ctrs[!idx].srcu_unlocks)); + u1 = data_race(atomic_long_read(&sdp->srcu_ctrs[idx].srcu_unlocks)); /* * Make sure that a lock is always counted if the corresponding @@ -1923,8 +1922,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) */ smp_rmb(); - l0 = data_race(atomic_long_read(&sdp->srcu_lock_count[!idx])); - l1 = data_race(atomic_long_read(&sdp->srcu_lock_count[idx])); + l0 = data_race(atomic_long_read(&sdp->srcu_ctrs[!idx].srcu_locks)); + l1 = data_race(atomic_long_read(&sdp->srcu_ctrs[idx].srcu_locks)); c0 = l0 - u0; c1 = l1 - u1; From patchwork Mon Feb 24 02:22:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868763 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28DF51C6FEC; Mon, 24 Feb 2025 02:22:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363749; cv=none; b=T8VY8Bulv0z5VlHDARi8y4XFblvfVXN4MGlUEt9JNiFXUvXpF/yXtn90KFWZHBzi23kfKNwMFr7CCbC3sOyuXm9Ja3V9XsaKWO6b5iShKngna4VCzcvcD4RUVWBGrIfEYn2DLswGTgwMumsQ6xVih5P39zdUv8Sq8rjbA+S+NOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363749; c=relaxed/simple; bh=n8KYK+ApLOt1uHZCuF43++Q8gIRMXbNEKX0ciRrabXg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GhjGqXDTnWgcltalZb535Qc6pLtOG2CILGCZUkeE0yA7gGl6SI/hn97IEbp2vX0RfCitBcToIlUWuNDsgUjQ4/1ytsMdC5xkT6ueu4NyGOFGKFOIGsCTgwi1wsb7lZjYcgKDQCoAPuFCzgLMQNoM6lq061HGdWGmb/t8EZIyznw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mVVtcLTl; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mVVtcLTl" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c09a30e14dso688316085a.0; Sun, 23 Feb 2025 18:22:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363746; x=1740968546; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=AcTo3KBDVWrf1oe45FpKbyw6ahcrt2yvV77yr4g8K3g=; b=mVVtcLTlHQXj/KKTCK2fvFuZOEce7Px7xt25KIwN188XTTvCCg3Nh7XvMEdxAMeA3l vANHClV2dRNlBLZpmls87ufEHGxqBKbVluz6uVRmjIzy4Bc7mKYVDw3Bf7oXgByWrnDK i2FnoSpVRxhb0bLfRXLy5/jqa9svJJ8oCh5ulKBb6XbMAzaXEvchNJlc9dWhuzwLKSQD 6Xc8HDLv0qPZNhfUy4VYtQIgd1I2GxAjLBT95KgZhBFKRQoy6QqPLZo0JKmqdvErXnwe ecItlmryEyDnpeYwHETbmlgI/XjDtcndrlMcDpD1ETTdDpa/b67UcDG7dsv1+NAOoFV8 HjrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363746; x=1740968546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=AcTo3KBDVWrf1oe45FpKbyw6ahcrt2yvV77yr4g8K3g=; b=I0t9xiSj8qHOf3kBnqD6F2BxImfj5fZSCiaPz+gf/mo55L07/SSOw+4zjC7d1gsPAo ZFmRTu9+vhZLuCNF5DRkmB+xQ9qsZUSW4DCBBnawg53LzVwD8eKcaBas7axmhxu6P4k1 gYg7POYQDXhD9gto+7JbprgycYVwbqqTjLbwDQXDL7+eGG152LOynkFe+nayeFEBvvK+ osMtrBnchCUb4cS9oKk7u4bzCVqpC4f3FlRzRMt9cpjiNtiyBvu9pq9j8lLwR0M/ZOj5 Cgo0zPwiPY8T2bLk8WjEgin4fC85mKGq75bMTW365Dq13KTYcn3GvZsPvYBNeHmDAjxX VRAg== X-Forwarded-Encrypted: i=1; AJvYcCWM78JkMnXhH1pWrNyhiUqcVin3V72pimR1BGlrprdqAeh1nDUmwenygI7YVnSoxOFXruV5YnJBx0QoC2bjLOg2@vger.kernel.org, AJvYcCXO+2NTXleUgNrYDpQ368legic6Et5/O6gRKr624yZL2hsmgmsvjUKysHji2QdECE6Zs+PqG6KI8Nl0HneS@vger.kernel.org, AJvYcCXdb9GeWNFz0DB5/Ny3f/5Gt87gqAB3U3ip6t+w17fO0jgpC8vKbjlmQ5gO9CkfIr219/E=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4uBmsIMixYnqloJbs64c9xdAai9r2aLO6RM9F/kBRCog6fWdz wWVcIrzp5wIw1DsJzMVPXsqvb7HR409vzSuzZeqgeaFpNbemmU5O X-Gm-Gg: ASbGncvB5rbADtlTojnk3m4JvJumGBoRC8XxKkAc7Z4ZNEhIi4aKYKJJxL40BajAwPP a/5Scr397XNqXH7VdazElfpb1jxMyJJ4CE7T74uenfDXzHtOcCNSSKmbzAoUgg7Iu4xd9ncsoMp Yj/Y+SsOnuvEz1VRhHUTyVbCSAyUWHK4Ve9Y1DfsvIaDAa85ia6kds0lcTsTLQNuRyyy6T5Oz4/ cPXB7qqZm1ubzBQNiYkY5TBYsu0DhoVc3KYE8Cf3yBcla97ylN2/oW5SExPYPpUDxcZwRz7KbNy s+HII2w8CtP6i/aJooY08Olm/FxoQH0xkrmbVDY2QYnMCzgqjLwTDkexdS7X2qrrxtdbZXYkSvI UEZOBq6jxOshncrEl X-Google-Smtp-Source: AGHT+IGEM+/HZwVspmar+q03xk2siTeyFzKw3jnpholWrfwhHL/2SXZ2X7t+5uARsJftW+6Gd12JeA== X-Received: by 2002:a05:620a:4045:b0:7c0:c282:702d with SMTP id af79cd13be357-7c0cef561aemr1595736785a.39.1740363745924; Sun, 23 Feb 2025 18:22:25 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0c2d9f679sm552767685a.5.2025.02.23.18.22.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:25 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id D791F1200066; Sun, 23 Feb 2025 21:22:24 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:24 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:24 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 06/20] srcu: Make Tree SRCU updates independent of ->srcu_idx Date: Sun, 23 Feb 2025 18:22:00 -0800 Message-Id: <20250224022214.12037-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit makes Tree SRCU updates independent of ->srcu_idx, then drop ->srcu_idx. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 1 - kernel/rcu/srcutree.c | 68 ++++++++++++++++++++-------------------- 2 files changed, 34 insertions(+), 35 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 1b01ced61a45..6b7eba59f384 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -100,7 +100,6 @@ struct srcu_usage { * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - unsigned int srcu_idx; /* Current rdr array element. */ struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ struct lockdep_map dep_map; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7efde1a2344e..247bdf42fb54 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -246,7 +246,6 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_sup->srcu_cb_mutex); mutex_init(&ssp->srcu_sup->srcu_gp_mutex); - ssp->srcu_idx = 0; ssp->srcu_sup->srcu_gp_seq = SRCU_GP_SEQ_INITIAL_VAL; ssp->srcu_sup->srcu_barrier_seq = 0; mutex_init(&ssp->srcu_sup->srcu_barrier_mutex); @@ -510,38 +509,39 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * If the locks are the same as the unlocks, then there must have * been no readers on this index at some point in this function. * But there might be more readers, as a task might have read - * the current ->srcu_idx but not yet have incremented its CPU's + * the current ->srcu_ctrp but not yet have incremented its CPU's * ->srcu_ctrs[idx].srcu_locks counter. In fact, it is possible * that most of the tasks have been preempted between fetching - * ->srcu_idx and incrementing ->srcu_ctrs[idx].srcu_locks. And there - * could be almost (ULONG_MAX / sizeof(struct task_struct)) tasks - * in a system whose address space was fully populated with memory. - * Call this quantity Nt. + * ->srcu_ctrp and incrementing ->srcu_ctrs[idx].srcu_locks. And + * there could be almost (ULONG_MAX / sizeof(struct task_struct)) + * tasks in a system whose address space was fully populated + * with memory. Call this quantity Nt. * - * So suppose that the updater is preempted at this point in the - * code for a long time. That now-preempted updater has already - * flipped ->srcu_idx (possibly during the preceding grace period), - * done an smp_mb() (again, possibly during the preceding grace - * period), and summed up the ->srcu_ctrs[idx].srcu_unlocks counters. - * How many times can a given one of the aforementioned Nt tasks - * increment the old ->srcu_idx value's ->srcu_ctrs[idx].srcu_locks - * counter, in the absence of nesting? + * So suppose that the updater is preempted at this + * point in the code for a long time. That now-preempted + * updater has already flipped ->srcu_ctrp (possibly during + * the preceding grace period), done an smp_mb() (again, + * possibly during the preceding grace period), and summed up + * the ->srcu_ctrs[idx].srcu_unlocks counters. How many times + * can a given one of the aforementioned Nt tasks increment the + * old ->srcu_ctrp value's ->srcu_ctrs[idx].srcu_locks counter, + * in the absence of nesting? * * It can clearly do so once, given that it has already fetched - * the old value of ->srcu_idx and is just about to use that + * the old value of ->srcu_ctrp and is just about to use that * value to index its increment of ->srcu_ctrs[idx].srcu_locks. * But as soon as it leaves that SRCU read-side critical section, * it will increment ->srcu_ctrs[idx].srcu_unlocks, which must - * follow the updater's above read from that same value. Thus, - * as soon the reading task does an smp_mb() and a later fetch from - * ->srcu_idx, that task will be guaranteed to get the new index. + * follow the updater's above read from that same value. Thus, + as soon the reading task does an smp_mb() and a later fetch from + * ->srcu_ctrp, that task will be guaranteed to get the new index. * Except that the increment of ->srcu_ctrs[idx].srcu_unlocks * in __srcu_read_unlock() is after the smp_mb(), and the fetch - * from ->srcu_idx in __srcu_read_lock() is before the smp_mb(). - * Thus, that task might not see the new value of ->srcu_idx until + * from ->srcu_ctrp in __srcu_read_lock() is before the smp_mb(). + * Thus, that task might not see the new value of ->srcu_ctrp until * the -second- __srcu_read_lock(), which in turn means that this * task might well increment ->srcu_ctrs[idx].srcu_locks for the - * old value of ->srcu_idx twice, not just once. + * old value of ->srcu_ctrp twice, not just once. * * However, it is important to note that a given smp_mb() takes * effect not just for the task executing it, but also for any @@ -1095,7 +1095,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, /* * Wait until all readers counted by array index idx complete, but * loop an additional time if there is an expedited grace period pending. - * The caller must ensure that ->srcu_idx is not changed while checking. + * The caller must ensure that ->srcu_ctrp is not changed while checking. */ static bool try_check_zero(struct srcu_struct *ssp, int idx, int trycount) { @@ -1113,14 +1113,14 @@ static bool try_check_zero(struct srcu_struct *ssp, int idx, int trycount) } /* - * Increment the ->srcu_idx counter so that future SRCU readers will + * Increment the ->srcu_ctrp counter so that future SRCU readers will * use the other rank of the ->srcu_(un)lock_count[] arrays. This allows * us to wait for pre-existing readers in a starvation-free manner. */ static void srcu_flip(struct srcu_struct *ssp) { /* - * Because the flip of ->srcu_idx is executed only if the + * Because the flip of ->srcu_ctrp is executed only if the * preceding call to srcu_readers_active_idx_check() found that * the ->srcu_ctrs[].srcu_unlocks and ->srcu_ctrs[].srcu_locks sums * matched and because that summing uses atomic_long_read(), @@ -1128,15 +1128,15 @@ static void srcu_flip(struct srcu_struct *ssp) * summing and the WRITE_ONCE() in this call to srcu_flip(). * This ordering ensures that if this updater saw a given reader's * increment from __srcu_read_lock(), that reader was using a value - * of ->srcu_idx from before the previous call to srcu_flip(), + * of ->srcu_ctrp from before the previous call to srcu_flip(), * which should be quite rare. This ordering thus helps forward * progress because the grace period could otherwise be delayed * by additional calls to __srcu_read_lock() using that old (soon - * to be new) value of ->srcu_idx. + * to be new) value of ->srcu_ctrp. * * This sum-equality check and ordering also ensures that if * a given call to __srcu_read_lock() uses the new value of - * ->srcu_idx, this updater's earlier scans cannot have seen + * ->srcu_ctrp, this updater's earlier scans cannot have seen * that reader's increments, which is all to the good, because * this grace period need not wait on that reader. After all, * if those earlier scans had seen that reader, there would have @@ -1151,7 +1151,6 @@ static void srcu_flip(struct srcu_struct *ssp) */ smp_mb(); /* E */ /* Pairs with B and C. */ - WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); // Flip the counter. WRITE_ONCE(ssp->srcu_ctrp, &ssp->sda->srcu_ctrs[!(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0])]); @@ -1466,8 +1465,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_expedited); * * Wait for the count to drain to zero of both indexes. To avoid the * possible starvation of synchronize_srcu(), it waits for the count of - * the index=((->srcu_idx & 1) ^ 1) to drain to zero at first, - * and then flip the srcu_idx and wait for the count of the other index. + * the index=!(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]) to drain to zero + * at first, and then flip the ->srcu_ctrp and wait for the count of the + * other index. * * Can block; must be called from process context. * @@ -1693,7 +1693,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) /* * Because readers might be delayed for an extended period after - * fetching ->srcu_idx for their index, at any point in time there + * fetching ->srcu_ctrp for their index, at any point in time there * might well be readers using both idx=0 and idx=1. We therefore * need to wait for readers to clear from both index values before * invoking a callback. @@ -1721,7 +1721,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) } if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) == SRCU_STATE_SCAN1) { - idx = 1 ^ (ssp->srcu_idx & 1); + idx = !(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]); if (!try_check_zero(ssp, idx, 1)) { mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ @@ -1739,7 +1739,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) * SRCU read-side critical sections are normally short, * so check at least twice in quick succession after a flip. */ - idx = 1 ^ (ssp->srcu_idx & 1); + idx = !(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]); if (!try_check_zero(ssp, idx, 2)) { mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ @@ -1897,7 +1897,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) int ss_state = READ_ONCE(ssp->srcu_sup->srcu_size_state); int ss_state_idx = ss_state; - idx = ssp->srcu_idx & 0x1; + idx = ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]; if (ss_state < 0 || ss_state >= ARRAY_SIZE(srcu_size_state_name)) ss_state_idx = ARRAY_SIZE(srcu_size_state_name) - 1; pr_alert("%s%s Tree SRCU g%ld state %d (%s)", From patchwork Mon Feb 24 02:22:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868762 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83FC61D86C6; Mon, 24 Feb 2025 02:22:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363752; cv=none; b=aEvUdNqlpqFbpA1t0/9s7fyYC1ZN23bBYsTi/jDq3K3OyIGymzbDxQUlPsI18qGSI+WpF2YU2s+yLQyfXHLauZ2ouCcQOgH1jPEArpLInCVj7XuxpFM4RCjDABpMVj13my5kp1jvAviLTife5o5xSU+Zy6rlCV3z/m4tGwVgR0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363752; c=relaxed/simple; bh=/9l1ioIqmq/Hpm/G/Z1p7cG+v9hvH+kj+ihFoubN/j8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bEvj1osPqBVYJRfgQy8R23sRkbBosmucYm8sK42h1jAxA4WUGvn7Bch913/nVM6H7lykxgngSUP3SDp2q8cbaHJ0cxH//7so3NbWWrGNQlbREib4VCAu2oobtWdNj4e6RUBWlE/uqYND57PTlG/wvmilZt95em4SxRynn2MwkGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QK2NElW1; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QK2NElW1" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-6e67f377236so32018176d6.1; Sun, 23 Feb 2025 18:22:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363749; x=1740968549; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=fYMRQ6Hd0Y9BVpEL39V5BvX9VXJ2sUq5YH9vLc7Ts4Y=; b=QK2NElW1rhlj58o8BUG188Zp91jbJ3A73bSHYwoUdrooJqq4T2eOGscnCwFgUveoi2 QMPed5ccUIRJQnfLpxqhVxloD7j4Z+9K8z/z6tUQdKN6rmpuQr5DgXn0f3iB1GLLYueQ oYueNfKRAleLi5lBx8xHiyTeupTe+QUYZQ8DFuDgWQDiAaW+drjqOsDKdG8qKNCFXzXK TfXDCngwdz0X3bF+x2+vLTqNS2GY9OxTMIndpGRrszK2lw7VE3P+VkFb/VyCQTTpNfck QrKlgAj6v9+sJ+LdtNt+qV3j2w90y9FWFri6tJrfW4vUFIK+GPH+XVaBVAs9it7PJr1j PlZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363749; x=1740968549; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=fYMRQ6Hd0Y9BVpEL39V5BvX9VXJ2sUq5YH9vLc7Ts4Y=; b=W1qmnV1jml8wTbVvT4/UKHTWxdQFzLnYm3mK2NZ52ZgljTH/3YvivNPmUJVrGkr1p7 emwV0nostRh2TyrO3LiMOaQuTeuAIJXJFkFDSb5DLZiIc//6yJrmrOSQYleZIS4OgKXO /H5ATtUlDii2sTwPs+y0IsfyN4F0DGwolfspp8XBd8f1xxhEmJy0cKXcH1hwmAsYxOaJ +tsAPy0+N5OQ7InZtXd4iVirLpdcvTex6XO2tGETddW/L1/CvMyfgJ3fMBovqVjneMXB EXbT3T+JzOas3AVFLxvDDWfaif+cAf3vllqXKntdNFRWSPnPliXliog0s2zSZV4MhSib VUuA== X-Forwarded-Encrypted: i=1; AJvYcCU7P/XtN6Yu7Wa6jj4Hd6KjJiqB/4VMnp9AkOP8wJMEfYpI6pCdT7u8HNIsKAu4wkl6e3HrUjbEuPHVvRMbbesk@vger.kernel.org, AJvYcCUmfpGGIWG5jtr1TSv6b5sgFUCHbFGKtukRuy9wxruczzupQzhux+YT5IhD3/F622jY9/M=@vger.kernel.org, AJvYcCVBECuNADc+6TIHXEzBefQlxtTTjSed3j3YqMpm4cpustNJlSNedjbm1+BNgS+XL5oxzKQfhx9jWe1Td6MR@vger.kernel.org X-Gm-Message-State: AOJu0YyJNvOiBka9wC5JTSx3/CSQluiIsSZbrwCxa4hcAfou8onbREK9 oyjH3Hrc3ZpaOSFNKo07WswwVuusU7gj8WEBta0W7xTfIuTk4yXa X-Gm-Gg: ASbGnctiQp/UhMPszkXHiOql9PqoOZC1GjtrJidCvTWQC8fVTxW2H83j4S6IdxjLbfD TjZu0zIdQ39u6sa8fTspaDM4pYYILfUm5ljlBXIBqNUIMv9SbY3zzzcXWCvgRek5CMrJDbrJ6x/ 2xniM6pcK2IVJHUfYgdMNGGLJbdHOxolecLO1aybS7fmshUUi2JX/olhS3Bw6SR5Y+RAzMY9k2x qlhgvKtWv8+5sWyAZroeLJB3ixTkYrdAJH5Vt5ramkCUm7cJZMRP6kqHcIu5Ija7h+2OgtYLriC qi1KdDSQvpfOUxO+IC3gH6fdMPnw9+qeD1NrjiipU7dzmy92WTuWorb+C+KFcl/rC1d28+xzVZ5 /x17nj8fsO3IaM4He X-Google-Smtp-Source: AGHT+IHClRbnQ4ctEnkp5istwWDgkUN2nCCWyWoj8RsUc0KxJ45c4T22hIVU190OvqEKtZUeDqyb/A== X-Received: by 2002:a05:6214:194c:b0:6e6:5d9a:9171 with SMTP id 6a1803df08f44-6e6b00fc43bmr180572626d6.23.1740363749320; Sun, 23 Feb 2025 18:22:29 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d779201sm128381556d6.2.2025.02.23.18.22.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:29 -0800 (PST) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 79A0D1200043; Sun, 23 Feb 2025 21:22:28 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sun, 23 Feb 2025 21:22:28 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:27 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 08/20] srcu: Rename srcu_check_read_flavor_lite() to srcu_check_read_flavor_force() Date: Sun, 23 Feb 2025 18:22:02 -0800 Message-Id: <20250224022214.12037-9-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit renames the srcu_check_read_flavor_lite() function to srcu_check_read_flavor_force() and adds a read_flavor argument in order to support an srcu_read_lock_fast() variant that is to avoid array indexing in both the lock and unlock primitives. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 2 +- include/linux/srcutiny.h | 2 +- include/linux/srcutree.h | 10 ++++++---- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index f6f779b9d9ff..ca00b9af7c23 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -279,7 +279,7 @@ static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_read_flavor_lite(ssp); + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_LITE); retval = __srcu_read_lock_lite(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 1321da803274..b347bde1aac2 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -82,7 +82,7 @@ static inline void srcu_barrier(struct srcu_struct *ssp) } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#define srcu_check_read_flavor_lite(ssp) do { } while (0) +#define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6b7eba59f384..e29cc57eac81 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -251,16 +251,18 @@ static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); -// Record _lite() usage even for CONFIG_PROVE_RCU=n kernels. -static inline void srcu_check_read_flavor_lite(struct srcu_struct *ssp) +// Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is +// needed only for flavors that require grace-period smp_mb() calls to be +// promoted to synchronize_rcu(). +static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, int read_flavor) { struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE)) + if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) return; // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. - __srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); + __srcu_check_read_flavor(ssp, read_flavor); } // Record non-_lite() usage only for CONFIG_PROVE_RCU=y kernels. From patchwork Mon Feb 24 02:22:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868761 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 999751A2390; Mon, 24 Feb 2025 02:22:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363756; cv=none; b=K9/OVLAJo/9B2W1AzhCxS0CSJ9q+AboECmbBSqk0MW6Xvj8SBcEdnZfsuEFVWQ7WyAxNpheTNPwztuZ0DhV504QT97LYAN2VCI3tGPaJSgZpj1kLU4rkKWpcMkTEDSmAsHSjV2fB7Z+dMU8kHvj+6s4vOsrr03EWHoUOeRTK/ds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363756; c=relaxed/simple; bh=TgHuhjSSnyReNfhJRKsOVKxVBGTwz2BqA8pdzW/huhg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VzZaWczv32ozCbk7ieSP66y2T0ao60QJxeLs+d8oCVWvFNW0Wver755zP4FPbL6g0e9a+UU/t0WQ+xGGDdzoUY+vCc3LT3lb2wcreimUOYNUD/jlo6j7SRYWrynW9YTTjDyDtkZGLM+ag+HIntQrpOqt4N+NL2najYLf2mteXCk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M4/Go+t2; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M4/Go+t2" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-471fe5e0a80so34221141cf.1; Sun, 23 Feb 2025 18:22:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363753; x=1740968553; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=YGLAsuKK/f3ek23KfjWTWNPSOmHZjb22aDmgZC2NlGs=; b=M4/Go+t2K82cr8/gqI6/u/FWnhuuxGwEaAKiFEeiX+cJHiHUdeKZbK2S2HLg2IT2q1 plsfqURXB9wAX9hzohIc1PzYWEST579uWwHa6Et14zVV/BKnet/MHtPU20YndoTUBN22 Ecapjr636Z9LIAddpV14FfJCktlI8VdIWHJW/bJMNhlCpYgCoX+RIHkSfwA/82aFOudB x1Z47+7vPbQBIl5Y1JdU9wis0kEfDNRaBzbJVIS+6e2RBBUbiIPic/73L/eKvLQf5t0o UPkBLARvgNPTycuQMHon27puRKcAzIIjL1iRlTvaWVhdwNp6VaUQCsWX4JacDBfNt9Vz +piw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363753; x=1740968553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YGLAsuKK/f3ek23KfjWTWNPSOmHZjb22aDmgZC2NlGs=; b=J4o1tRCM51P0eLpzT+Cw6ko0BvWGq/gpDEQK6+dJ8LnpBE4oJjTNZhiEcMDrtOHhnR vAqAp+M8UC8o7p10V3tpBxwqTeF+7BEzNxAGq6Vo6G7Y1zG1EWQDxdmwAcIq91LCtCSL 0Fyb+L2q7q7DORas2AZ3S5nIFlCyBPKkFRgt36uqsplcDNS7LMIu6NASW4TNxMIOp096 Hh4iHHU/PZsPMSqxjVMMWwU1jDVKh3HfjjrKK5jkimBCaE+zJyEs3x7uO/6+ffpbqTSV XrAsydcyl6/e+WtVaVDWzd0lVBj+DdHnGs1yTd97aBO5eRpMsaTh/FnhlWBrlU4QPadd BJ/g== X-Forwarded-Encrypted: i=1; AJvYcCWHnPXicsQ2xC5Fq0wAeEgikdXPb9hyBd+a3AgiUTApG5zmav4QrG+eDJhejwM71NCdeEk=@vger.kernel.org, AJvYcCXHyUkJrUeffTncDx2lY1SQY1dztyS90FwpPRxKFi1+ZqfX0UEqEsAduRmOQnk1CGoMz9x0AqAG2cXcL6re@vger.kernel.org, AJvYcCXgMsN5RxWUDWsrZSJfEkmzqdF9Te9W8MO8tWn2tmkvog6kIONEiFXcUeKhB38n/rj0ZnKn5endN4nDuWCnPzra@vger.kernel.org X-Gm-Message-State: AOJu0Yz1X5RiN/u8eFobXJDYME39RO6ULqjawtqfB1kVL6Gxzl+LTKFg Uy3o0N2O9CX1qPAOjJi5ORyXGa/i3TR8vH7Dj/9w4p2v9Xo4YjOe X-Gm-Gg: ASbGncuYQx4qUn980X2tapHbjBlFaVJVHfVEwfviHJpw+1k2w+AwdNF2jUET5OBMHAk 2UDyQM5ktT/NmTKH9wyDNzSHPYbGf3NRPdHOJAMkPdLC3e5qQmKF7DGFc5J8j9XQFs73WwH/kUg yz8sTR9chhvPuI/u9ep4CBq+sqX7VfaVViZEGjBu40V3iLS219MimjrskjnKExYOf2fRpwCuea2 lr6M3fkhhntRK7dERfBta6icGQ/mhdGDvKUPfQlhdrW8qU7zwbpOzqrdVOZoJf3/YNnSCNIiv4l 97vHnFoFVpF0RIW+WruGw1lII806VeQn2Cyn200sZR1bOKDlB1FdfBcNxDcAL03V1VVrUAHeT+J 6FdAK7fXYjAOE/qDK X-Google-Smtp-Source: AGHT+IGUSog6nwxcNA18fQoC5dGEQkV9c1YwPYJYckbB/HQopiZwY63sHGbjZDO/LBtKd+Y4ciduqg== X-Received: by 2002:a05:622a:10c:b0:472:3e4:53fe with SMTP id d75a77b69052e-47224611fb7mr150683901cf.0.1740363753370; Sun, 23 Feb 2025 18:22:33 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-47209a1b133sm62076051cf.70.2025.02.23.18.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:33 -0800 (PST) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 78D701200043; Sun, 23 Feb 2025 21:22:32 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sun, 23 Feb 2025 21:22:32 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:31 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 10/20] srcu: Pull pointer-to-integer conversion into __srcu_ptr_to_ctr() Date: Sun, 23 Feb 2025 18:22:04 -0800 Message-Id: <20250224022214.12037-11-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit abstracts the srcu_read_lock*() pointer-to-integer conversion into a new __srcu_ptr_to_ctr(). This will be used in rcutorture for testing an srcu_read_lock_fast() that returns a pointer rather than an integer. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 9 ++++++++- kernel/rcu/srcutree.c | 4 ++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index e29cc57eac81..f41bb3a55a04 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -211,6 +211,13 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); +// Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that +// element's index. +static inline bool __srcu_ptr_to_ctr(struct srcu_struct *ssp, struct srcu_ctr __percpu *scpp) +{ + return scpp - &ssp->sda->srcu_ctrs[0]; +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns an index that must be passed to the matching @@ -228,7 +235,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); this_cpu_inc(scp->srcu_locks.counter); /* Y */ barrier(); /* Avoid leaking the critical section. */ - return scp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scp); } /* diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 8b5c50bc98e5..a91651866485 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -753,7 +753,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) this_cpu_inc(scp->srcu_locks.counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ - return scp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scp); } EXPORT_SYMBOL_GPL(__srcu_read_lock); @@ -783,7 +783,7 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) atomic_long_inc(&scp->srcu_locks); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ - return scpp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scpp); } EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); From patchwork Mon Feb 24 02:22:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868760 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A7FF1FC0F3; Mon, 24 Feb 2025 02:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363758; cv=none; b=tBCyP+H/NqXXG6U83/jWA2CJDpzQryDjOTMoDBQU9vZ2+sd5a7FboALQidTlSJjttPON+HHFrJRoZ0Bl1KoJB5Fjn6qAIv8esgovWuvw+3BaDpEjiOLNZbL37sPjxyw4eSiC3hkDfTf/mI5uXdkvSbIgnlcVbAnwSOlAsYwzvVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363758; c=relaxed/simple; bh=vYGr4oase6oAhv/S3mUkEU7TFl9z4yEqQk+LuMvaIG0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ajf4yj5QAE6bknpFi5wt4gufo5+XxFkzoI8pVY35wB67GEq6J7jopGY/Nx0JW03J/LU8iMJwIIoLqYa0g0IKWlXvV1vLyrt9HvHyxCvf/U0MwiLnG7RwTjDT+B/wixKM73lFyeQQO3UQHbQtSe9AfULXK3vJuMSyg/N77aNnwpc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Wyrlz9Ge; arc=none smtp.client-ip=209.85.160.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wyrlz9Ge" Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-47210ab1283so57653511cf.1; Sun, 23 Feb 2025 18:22:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363756; x=1740968556; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=9xc+k+HYzIVy9Ic6oMLdV0EroWNk+iKQ9YparoGG0Ik=; b=Wyrlz9GeIxY2YjJXe5tWrd95p9p7msqGEXxXi7/xqTBOWk15M67/8i6xjkyjPEnxfY 7whIWckBQfsOppeoMzX8gqiDmThtLSLLx7CNewck+VgH5IJkC1iwKJf5iN/bZOiTc4OL NKhF3NEHxA2Q5IFTOetiKYqIavYrWiQRYF0g+mZLItX2t6zviYxXoLtlkiHRFPDPwtlK H3b0/jHXKExi8KFpXcC8TlMT0UlYsqiK6jGTqfpBEiEOABU45LTPaMGA9WCLHqYRSBxL IbBcZ27krs1gKqlvRQmqJ57HZGCaMQMN/iM1T1zxVBtiuk8Bd5q3fPp9UCo2BYzHwiVI cbFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363756; x=1740968556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=9xc+k+HYzIVy9Ic6oMLdV0EroWNk+iKQ9YparoGG0Ik=; b=XAE0xVenYtwI2iEnypHaVncaOZqYupa3hnDtyZ7ZBsAFyw3ELng6Z1MJJkebQVcP62 Ah2cTVjzHbxz1C1zEUyn59iMrfDmuYoHT7WX/qLup+GYqFXH/eS+8CJjRHRLayxdPXSH Xec7Mp983Y1iOgZXz+QYFq/sbup4Xc1aRoT5qZavA9p7iKtDOGreM2Ezq4ItGeJDSOSZ +eVZFflxVnvIZFS8Qdd4fWDdzHOtR/YQQ9BJt1uu2NH0e/Br9BO08DD/eT+bX78d+UyU ggKAuwZQ+LAiJrH1kbJ5rYG3CrE/UVZvyD3m541b3+nldk+qSPgo0f6+Nk/mZyc/1P7W 72OQ== X-Forwarded-Encrypted: i=1; AJvYcCUYVXgGI3Zhbm/451ylKto2Ngp9o9DlE/s1UC69F1nXqBsu6qsxAdaSEooXf3P2gv4SYVU=@vger.kernel.org, AJvYcCV7hMTbtS+HFsQDNoNrGKiBA4375C3eYAOqWfBQHqkv++sCUMSrIMfgK+hg2njCuPStzrRH4RE8GR+EqgIxyeib@vger.kernel.org, AJvYcCWECpV444VOB/ZhaCFlRFBaz6KZBnvGKirat0hzS+OdIT+CsaVSIbbFybjEdiax1rjAxrOU3ktMyvb/Bfr4@vger.kernel.org X-Gm-Message-State: AOJu0YzacwxkFy1U4fXaZEG9uhAgBn6kIpF5pzn3ch39XObqenW8uHL0 MiK6DKfYGCkMdfRNfKAZ7a6M9A7n/yzD/iTG8lrLtsO81fLCne/2 X-Gm-Gg: ASbGncuk+uIJ5IBm0xjHXwHwGLSBkzx9TEGNCKQEXUkzF2ov4zrWTARI0HozmIV+QMt +/+A6W9szofq0x1Q/vMWVAZgVsY5qRLCHHK7aTgCnX+piE6BiNTykKy18DfCV0YNNpcWUFgd7HE JxtPmMNEcqCWfdz5BkCKrX9jsqfzpQsSBzEJXluB73NAqyoS257qkzIPsDmnzCIddzGE4LEq+ti MS45pMURDV0rDEpgxQbY5z/U6qO9DQVNvbTvH4QOzmWw7U7ORpuwhTFWOZPmfrEkmpo6yLf2Zml t32jIFk83dfPJ3T58Fv4T6G+QvkcoTHBhJA+Bz1ZwfBbylKjM8IeRaXjz0MKYnt4oGBx2m/Du4z rhGNJVYEgLd2Klof+ X-Google-Smtp-Source: AGHT+IFYTMVp6JYLVO/whvadX5BcZfW43b4gbDKbaMnb7W7u3y3MHECALfKNOxhOC+OdWqH3s0bwIw== X-Received: by 2002:a05:622a:388:b0:471:fde4:f09e with SMTP id d75a77b69052e-4722295d3c7mr159638211cf.36.1740363755990; Sun, 23 Feb 2025 18:22:35 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c09d5980e1sm952821385a.13.2025.02.23.18.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:35 -0800 (PST) Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 2987F1200043; Sun, 23 Feb 2025 21:22:35 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Sun, 23 Feb 2025 21:22:35 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:34 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 12/20] srcu: Move SRCU Tree/Tiny definitions from srcu.h Date: Sun, 23 Feb 2025 18:22:06 -0800 Message-Id: <20250224022214.12037-13-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" There are a couple of definitions under "#ifdef CONFIG_TINY_SRCU" in include/linux/srcu.h. There is no point in them being there, so this commit moves them to include/linux/srcutiny.h and include/linux/srcutree.c, thus eliminating that #ifdef. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 10 +--------- include/linux/srcutiny.h | 3 +++ include/linux/srcutree.h | 1 + 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 505f5bdce444..2bd0e24e9b55 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -52,6 +52,7 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_LITE // Flavors requiring synchronize_rcu() // instead of smp_mb(). +void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); #ifdef CONFIG_TINY_SRCU #include @@ -64,15 +65,6 @@ int init_srcu_struct(struct srcu_struct *ssp); void call_srcu(struct srcu_struct *ssp, struct rcu_head *head, void (*func)(struct rcu_head *head)); void cleanup_srcu_struct(struct srcu_struct *ssp); -int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); -void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); -#ifdef CONFIG_TINY_SRCU -#define __srcu_read_lock_lite __srcu_read_lock -#define __srcu_read_unlock_lite __srcu_read_unlock -#else // #ifdef CONFIG_TINY_SRCU -int __srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp); -void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) __releases(ssp); -#endif // #else // #ifdef CONFIG_TINY_SRCU void synchronize_srcu(struct srcu_struct *ssp); #define SRCU_GET_STATE_COMPLETED 0x1 diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index b347bde1aac2..a6194f7a7e34 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -71,6 +71,9 @@ static inline int __srcu_read_lock(struct srcu_struct *ssp) return idx; } +#define __srcu_read_lock_lite __srcu_read_lock +#define __srcu_read_unlock_lite __srcu_read_unlock + static inline void synchronize_srcu_expedited(struct srcu_struct *ssp) { synchronize_srcu(ssp); diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 55fa400624bb..ef3065c0cadc 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -207,6 +207,7 @@ struct srcu_struct { #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); From patchwork Mon Feb 24 02:22:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868759 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB6F0202C44; Mon, 24 Feb 2025 02:22:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363761; cv=none; b=QzRYVMCIvYU4fambvBZFZq/HH6zIAaCzz0DkWENbCUg/jp8wciA3+Lbd3qC1RrjIr6Vb+X5d5CyUNIzxi9iBBVYuEPxMcfn+ENg78/T3IhWOmWlf/X+F5eELGaJMkedTHELH7B/gzJ281FYE5Mw50lpge6Jr00bB/gzHQjpQXVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363761; c=relaxed/simple; bh=x1RagGnEsGngNw3EY8nNKEZy+pH7qTCaixBwp6WaeUM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jvdPrjBFDbn4V/XaKIRGAlM485biFgPD3CinoAJyFBYvl03qIe9luwyO/svYXZlld6bRYnk9g9QrddBdxsQbBZaaozXW10RKtR02gUDoG0PsQhFVXtOScqyoJVILcleAEpmm5HL+I+b2JNWJRQoO/PKeeoW0uNamzP6G+J5Pdxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X84TZ4gq; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X84TZ4gq" Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-6e6621960eeso48879686d6.0; Sun, 23 Feb 2025 18:22:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363759; x=1740968559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=LYVXngARAHEiVLhrUvNNltGXmcHFU3g/uWrZ+f3iCoY=; b=X84TZ4gq6tQ3qHFKwFwJosUXNsyME9HArhW4ireHougv+e7EZR410YOyreB5r5MqVM WGWd4L4lqaRPXJGbInj8XxzlXlxtWGOU6tHZxHl6V1t3nvrboHD/U5M+K2Xw4zVuQjug KNc8UcuY58rSClsN8NL88XTQH3YuePTZ9zTKaabOz/b+p5pBiIyqwRyWKBz9H/fvjx8+ o3DyPk4c9ovVXU3Y27jTCZ4E6XANvJdEgZcWnRZHzFFtFL0ozvBHIXZIc4n0qXl5sFSD 3ENHD8muRKo29rMfzQUvNsyCyPQOv1no5BL2ODDINwP6bd3MjDPSa6LZHrnwLZz/IBRH r38A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363759; x=1740968559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LYVXngARAHEiVLhrUvNNltGXmcHFU3g/uWrZ+f3iCoY=; b=vojje5ZMZoEZNuZL5UxlcLe59C97ENJyTQ6PRem2dGl4Mi7dAR0SXBK5x7XlKxvMft kX+VUrE+4/2n1hNY2XkWrdtlgKabqheC+PdVmEKy+anIQzh7k8b5pOy9IfWq6TQCCp4n o7YqulNTXZaV+elBEARtEx1m/0lmU6qwxt0zeBpdr2HbUTsEH/gRk9dZYp2civjQhyOJ VQ3Prd2pRM8GRwvT5qMQJEs7pj35fEFKb9wudWqyjGGQx70wvAI7FrTCuK7t9yN2zist HoB7NVWqutLYzaziq3v/26TUqTGGnL47+/94uNHXW0gmasgz76e0GK5sjJPFgm9c7HTc dzBg== X-Forwarded-Encrypted: i=1; AJvYcCUqkc6Owwky4vy2RGsuifaLbKw6FMOp3VknVwRRLL5NCy/CNCwjqy9QcT9RPGY8zWJW7XjAfPJLBANKjmbcuEvW@vger.kernel.org, AJvYcCV7fZieXfRWUnnPM5a9wpzBatFbZbmM4i19FKGXwuwYcgvq0+kRmvAILYX/Ly29A6MmHus=@vger.kernel.org, AJvYcCWCS+u4g/FBbCW1HAIkEps0IZsSSLEwVlgm+Le4pv70/q79kePxsHzAv5+COMrRoS7t7SIuvgyu5o4cEuC+@vger.kernel.org X-Gm-Message-State: AOJu0Yyktw6NfxiIbpsG0uVmMIml3fZZ/3gGGSSrcaA5to7NRsEcRyFn glQSCxC7Mkhzf1t/ZbTWdmWxgJyi3rbDCI0GuQ8rhwa9ODpwkmOT X-Gm-Gg: ASbGnct5epEnVQyZDF878nneLW3Xflj/GFgHNvfU636AiktYwRisbvc/9noTg8GWCA7 wIpChg/VgUlUdKMBdE0Oka2xt/RhCTgOCW8eIOrleok4UL9S+d4EZPxtb6Rt4d522ram2HWPk9n DB9cVUYoGUmCoKfg3rh0X41YO3dbjK+MdLxqU6xSmKbkkWMByJTGE1pccDyfSKlVTDOE87P06wu tUSjEydr31wojt99zbZna25tKiqG/YVvfDjZZvcBXk2x/abamcB48e+b1xlHwfwt2lyU/7fvCCi Oo7FyOhGYJNMSi5yPToj4FE9tcFv78P650p2sErdyNU9LcD+6OVZZr4aJ7ScMRRJW1t3fcJhomT ZXKM7jr/UpEizxhVL X-Google-Smtp-Source: AGHT+IHZEXr3FBaKPE3j25LAFJ0ZFPpAKJCewtDs9aZxscFGONzRa6lLREjHVfy2UJiJ5PQf0mtAdQ== X-Received: by 2002:a05:6214:1250:b0:6e2:3761:71b0 with SMTP id 6a1803df08f44-6e6ae72d99bmr175752656d6.5.1740363758761; Sun, 23 Feb 2025 18:22:38 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d71721csm128183396d6.0.2025.02.23.18.22.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:38 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id DD8B81200043; Sun, 23 Feb 2025 21:22:37 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:37 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepvdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:37 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 14/20] rcutorture: Add ability to test srcu_read_{,un}lock_fast() Date: Sun, 23 Feb 2025 18:22:08 -0800 Message-Id: <20250224022214.12037-15-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit permits rcutorture to test srcu_read_{,un}lock_fast(), which is specified by the rcutorture.reader_flavor=0x8 kernel boot parameter. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/rcutorture.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 1d2de50fb5d6..1bd3eaa0b8e7 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -677,6 +677,7 @@ static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) static int srcu_torture_read_lock(void) { int idx; + struct srcu_ctr __percpu *scp; int ret = 0; if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL)) { @@ -694,6 +695,12 @@ static int srcu_torture_read_lock(void) WARN_ON_ONCE(idx & ~0x1); ret += idx << 2; } + if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + scp = srcu_read_lock_fast(srcu_ctlp); + idx = __srcu_ptr_to_ctr(srcu_ctlp, scp); + WARN_ON_ONCE(idx & ~0x1); + ret += idx << 3; + } return ret; } @@ -719,6 +726,8 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1))); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3)); if (reader_flavor & SRCU_READ_FLAVOR_LITE) srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2); if (reader_flavor & SRCU_READ_FLAVOR_NMI) From patchwork Mon Feb 24 02:22:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868758 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 566A32066F2; Mon, 24 Feb 2025 02:22:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; cv=none; b=GAw6Wu93R14ZZuL2b4iV0hKWHlbpT1vCCSfg08+Kfc9rdt1WnCTI5mUX1dxTH+IJs8ympySLXC1N5j0puOhs4LxxjwOgA3399IX2TlejHSylmMmDKA748QF6nYWjqKBdGF70wEmA/LTIvXKziLBKDv0yZ3EOjzP3dTcI1wbJ6cU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; c=relaxed/simple; bh=kAUdkuuHIe0hHKVg4sC/0Qh5Fb6pN442dDVYb1BISTo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G1gDYd4f7mNNFdU/KH0zWXN99Z66Ac5Zv3oLqY8ZGXaoBRqZiC6tYawpzbSHwLM0Kovunq4DL+XWSr/pocuglPpsKFUuULraPiiq0JyKgW2uYrkPiXt8dA8MyIGniRuPuRS9rPs9omh/cQzmBWJqloHXDgHL2/o3VuTRA/6VpVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UheFGIen; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UheFGIen" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7c0ade6036aso492449185a.0; Sun, 23 Feb 2025 18:22:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363761; x=1740968561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=HoXXo7te2qdCoqRU2DAsTQhDKL9OLvKsbtcGFLuSrXc=; b=UheFGIenFPdOK3I3vZbk7qW684gRr4c72LvY3IBr+Ob+BwejHRuFSgACYSvYru7o8G AhAsbmkd0uZHfFLyOpJNRO1ZwIgBEq9tF1WctxAJgZuHmDQe3hh4Fj6gmp8ih22K0MJx FaRFRRRCbh+01+Gth/+DzB1GL15L93UdmtXu2anPQdP4mS2xj5A00eWppFuWlKRuuqum Fz6xh/3UtE7/Ii07EXftNW0qEvBVN/Yyv2Wkyzw8IxswpQAwGe5lK+aoAqwgvTeJbps9 2CtCDt7PuHFCHPeWnVkcMAY26YCfGSBPZuEMccjgMQQykF6ubdjh9a00DhCvoQkTlJPR yY6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363761; x=1740968561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HoXXo7te2qdCoqRU2DAsTQhDKL9OLvKsbtcGFLuSrXc=; b=OOiRb9VxLk/Sl45uyYK1vGrYZX2hmkRuKW2REauAD6mSn7QfKPlHkLTHvkhfHev2vE jerxJ3lWHAVdCTB47tKU/mL27cJPibRRcxhDaMJDCrmnD7xDTK6SOnF99A8ACaRkdxDp D16Ech6aXysrKjHI/ynOFht1xgIgJ+Vvpf0sV5CXeN1By+afYr8IbaTNkI8/bDkUaJmb PioIt8qA15YoB7lymsXUmXI4bZYYb0ZgppNtmALYcvisdqV+awqVUVH77cwIOhancSMM qDnenmac+kw8yb2TGu1mw4ZkrxIMzowOVpYIsxfzNtwnIovz4P7cTWn1uD5EUreN1iJe RYgQ== X-Forwarded-Encrypted: i=1; AJvYcCV9j8HQCrArVAJmxwHAhIRsc34Dhxl3YT1EhQpzHznDTPYRh3e6BXYHqURsx9u2a1I8MNI=@vger.kernel.org, AJvYcCX2iCRegL8FV/NH5OpxK+wuvwnh7bjgTjoPWc4ipyTBm1EjIhtxB7vji0uJHslQwxWrnA0a5RAnOFMxy8de2Vlp@vger.kernel.org, AJvYcCXZjdV0aKsYfvFIQVdn0VwacPZvpjvhammA6ocpETxPA3qtKwvQijQNZP4pOYUfoOr9P3VToJe94CIP1LO/@vger.kernel.org X-Gm-Message-State: AOJu0YzSVItVtFvl+pv12tHp4U5xL/I4rr9eEkJGqMq0rKOq4Baei1m7 Gnbjt4sNZEVbLkIAec2Rekbgrqn2P7rMbbi9bmteALH4dZ/IBFdq X-Gm-Gg: ASbGncvtcprgay+9DY4IwvYc2xsrQhReVwCYSnqJ58reZlVSzQt8ob1EDgwhGXd+GcO JvqLIRte4iHPozauSABDvEO1Maz4XigS5WObvd3eDmxrORMC3ckCLSXHlJ9pvr1fmTcMJXNISSv UTTE6G4ObP3REPQNVgCr9h8kY1s/peYOVoQu7LlhFh34wDW5RrZ13IqBuymcPTNYP37E9xXTykd 8upkcfkrYKxklWKXqZRcwTGaDrWG2HExGM9pLVhSolOfFUXOi1wdmn/kSJGlJ1r6XWeEY08fbJj 4z8gVljCtCLIOz3Z8uPiw/x+xtBXR28c9Qj95uawOCznILp3vH74ge3H+to593V9fXOfjGwMY5N 62nEpTHZILGw27AJR X-Google-Smtp-Source: AGHT+IEZNGEvL+X6WQsjqqBoVy8Xl/SkDXAOELoRPMIY4g9ibmVObQC07YvQ0QLcC5b11t6JdM6nkw== X-Received: by 2002:a05:6214:d07:b0:6e6:69e4:825e with SMTP id 6a1803df08f44-6e6ae8f3b6cmr123259076d6.21.1740363761352; Sun, 23 Feb 2025 18:22:41 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e69c93e8d0sm56723336d6.108.2025.02.23.18.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:41 -0800 (PST) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id 880FF1200043; Sun, 23 Feb 2025 21:22:40 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Sun, 23 Feb 2025 21:22:40 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedujedpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:40 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH rcu 16/20] rcutorture: Make scenario SRCU-P use srcu_read_lock_fast() Date: Sun, 23 Feb 2025 18:22:10 -0800 Message-Id: <20250224022214.12037-17-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit causes the rcutorture SRCU-P scenario use the srcu_read_lock_fast() and srcu_read_unlock_fast() functions. This will cause these two functions to be regularly tested by several developers (myself included), for example, those who use torture.sh as an RCU acceptance test. Signed-off-by: Paul E. McKenney Signed-off-by: Boqun Feng --- tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot index 2db39f298d18..fb61703690cb 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot @@ -2,3 +2,4 @@ rcutorture.torture_type=srcud rcupdate.rcu_self_test=1 rcutorture.fwd_progress=3 srcutree.big_cpu_lim=5 +rcutorture.reader_flavor=0x8 From patchwork Mon Feb 24 02:22:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868757 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24EA724060C; Mon, 24 Feb 2025 02:22:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363766; cv=none; b=phjiAzXHsgPj000u6mr8q3s1CjfiDH1Xr0j7lMSeWxv6DXL8wgt/80u+YFYFbqfNmnpWIROR53HUBYKu4C5kWyefMXTMkForlPxJ626V1IGdD4gby8CtumKPKmkLyZf/CJ8lqbgRY0PEuWu0BeSBtM8P90GeswE7POFQpelKL+8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363766; c=relaxed/simple; bh=mRU9WCM67lJH5/lbA8uOg4PhTTj8coyG+QePvNGba3k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Iz/E8RKGavcLcA9O+Tx5r4lsEOFirbxFZmOgUYLXRKMR1k06Mcp6sm6eYHQGFXcR2qCgwKMY6W2Rowd3Dv7MJmOsNn9rxqsfNfLnqftBd4adeZz0gRSxbmsbyLZp6Vm+D4C0d5qKozQZQbyZKrhlATiqtIQJujnsxFefckjcxEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=STa3TsHG; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="STa3TsHG" Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6e67fad4671so35417956d6.1; Sun, 23 Feb 2025 18:22:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363764; x=1740968564; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=eg6dUqpGoKhiV+cpu434M/fIW4z0zJ0cT+ywlwyprkA=; b=STa3TsHGJtVb+MCCGHC2sB6gvb5cnWXjN892Yue9Jq0rRDfO5B6LsCbQ2O79PP9llf XeTe8iY5qTueKaqUtCQE5XGkjFFfKhPDs/4PunaCEH1+Bp/LErEY4bHQQOliqEPSqQP8 X8TZFKEdARr1Qi4gKXTgwRlYmAYmBZ/S6dmH6BN3Cw078SJlG+FUpqgq9hWfpduqrttg EP5W0RQTZxykJYB4YcLuZHspxiDwMhdzrO0QHvEYdwS6Dj1yQ0ZQBvArBzsCtgoxJods W1l0n6yClTMmUnfa4pkqWmff8WbCH97dqilf/0WkgpHsmsiZlP3U4pZ6BshT92sGpkCb 2mnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363764; x=1740968564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eg6dUqpGoKhiV+cpu434M/fIW4z0zJ0cT+ywlwyprkA=; b=P6JgmEPlTWPEEpPo61yT7mEKnV8sPqV7feH29rRluMgJSFnaCr/FW8fsS2hVKwIw4w T3+IpupwRxUIIexn1V+NxK6/lkCDbLk58qtQnhfZVSshjOYnrxUoAup8iAVo14ZhI7Bu ZR6gKEf5cAY4rsUlpJb7rIzu+SttnaHOkQ0dvA1dI0g65+3FJ2tH+aCIHHEt2pWFmRFi XYQEv8bcBMCfeKWmTpn9MZ3bsDOawsVdMTpb86c2LmN8MspWbMRWRx7qHH28dvWuvnjZ 9HPnGZLRum5HXWftmpvMheIBWsGdpwyoHfS6XtIA3JX4TbCCEEmFFJ5r8vcDJwfxsmAt iD0Q== X-Forwarded-Encrypted: i=1; AJvYcCWBf4aFLi3TAcMMCcglyC2x1blEYzEHWVe0SArJYvy62s1jPsTqQcnoxDYb1ZMqATOOm5gA9iaWKvwi0E8U@vger.kernel.org, AJvYcCWK0UD0UnHItb49ZFez36bm5Z5Uhrpk55u8X9LBwHHilUXa/tpvGjJtDxzFskJFC3NN2cGDlxI6Bkohswqbv7WD@vger.kernel.org, AJvYcCWiq2n5Sx/nQ/TRi4fa/Ml6MdUxFhy2S+KZj83ONKN22VAEgR28RdNGAV24/2sO7dl/cUY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/E8f16js9tvlpk7F1RfRwiUgDUhaZi5TPhXHVxvs6DXFppBqx lpygdtc5iLfbU8r9zxUQk5ScEOUec1ibWfbQ3K7j/KTI8e4RQPNW X-Gm-Gg: ASbGncvD+ML09oDBiJ19x7uzYqt8Cpkw+Bq68wifSP2hZX7UZVvxdd9cGIpubPITmBX 6h9JVTIKXy4GYy16cXKinaspR4Xd2Uv2PzkHHz5blubKCO/J3ITfKy2AcIiIa34isgzon/SJVjC i3mDowyISWjku8OChMGVpCtlaBXUs83+S8/6ZpJu+2iaN5uxaTH8ODoX39sx1BKRGCfSpImXIY3 yB0VhDYL3riOQ/NMfkysRR/8Jb95IPBsgYc2hK161wxzr8/9SXiAWxPBT4K7COA/Eavd/FQECZJ JaGyPjggmPMnIRvCq8bAEXXjHnNnjjorKj/AUhhMzc4htyU2qnTK3lhWWcDcOewhHMRPnbljK+O aFCptKovjORYMkGv+ X-Google-Smtp-Source: AGHT+IGzIhpbr9iU21ineA2QNZuQdz9ARj2vtfEGBqgEatKHIidqLPnjQqCQPKzUU874OlLqnK72cg== X-Received: by 2002:ad4:5fcd:0:b0:6e4:4011:9df7 with SMTP id 6a1803df08f44-6e6ae7f876emr161379806d6.16.1740363764052; Sun, 23 Feb 2025 18:22:44 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e6b1759af6sm36879046d6.93.2025.02.23.18.22.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:43 -0800 (PST) Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfauth.phl.internal (Postfix) with ESMTP id 404341200066; Sun, 23 Feb 2025 21:22:43 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Sun, 23 Feb 2025 21:22:43 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:42 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 18/20] srcu: Document that srcu_{read_lock,down_read}() can share srcu_struct Date: Sun, 23 Feb 2025 18:22:12 -0800 Message-Id: <20250224022214.12037-19-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds a sentence to the srcu_down_read() function's kernel-doc header noting that it is permissible to use srcu_down_read() and srcu_read_lock() on the same srcu_struct, even concurrently. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index a0df80baaccf..317eab82a5f0 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -359,7 +359,8 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) * srcu_down_read() nor srcu_up_read() may be invoked from an NMI handler. * * Calls to srcu_down_read() may be nested, similar to the manner in - * which calls to down_read() may be nested. + * which calls to down_read() may be nested. The same srcu_struct may be + * used concurrently by srcu_down_read() and srcu_read_lock(). */ static inline int srcu_down_read(struct srcu_struct *ssp) __acquires(ssp) { From patchwork Mon Feb 24 02:22:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 868756 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C06F241C87; Mon, 24 Feb 2025 02:22:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363769; cv=none; b=i9D3NoZk1FrM7o/PtsdPN9RuN+k6PYGYiM2X3gcxe5uqs8KWu6m0D/6RZ68PmcDZIITdz9gFl0TwkIgtb7DwCuV4ccFNaj0vnxbtbXeGLYiTt5bm5Wtpi78EYIlUOoBvbhamUCecT/E5rb8aeOaBcPL9YHO+6CRJa+qFoNg9FnA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363769; c=relaxed/simple; bh=Z7wcVTQ2PMbca+VamTK60Bfots1ruiWDLXTJSx+zoDc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DRTRyZMb1krESw/GuSAbqsndk2fl0cAlweIjuj9p6XKtHVPoKgWHb1XGMzUR3Kf8p+jAWpnNTsG64IbEDSIUtEMbTwOSGAe9YKES9jXdBbM6bPy/tW7LMwRJaWDKliYmcvYJief5dBti/G/Wz/cBiYAZmOGfr+Jpc+wci64me1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UTIdZnLC; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UTIdZnLC" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-47210ab1283so57654981cf.1; Sun, 23 Feb 2025 18:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363767; x=1740968567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=Meuv+MydLFL2FaXVwlNSjl8ifFrRUaf5oa8eloYWNAY=; b=UTIdZnLC/We+kcPwiP+quF3DmhU7+nqO6mZukeiFLr/z0iFRtVbn8yd1MHijbCSxcj EDxrxCfWN8rNocFsx4xt06RT+34A1t0EeoA+PDp4tPiN1hBK1P86ZNUK9Tclu3W9pAQZ 1nIyzFNG2Y3V6fBmMedi8pvBcPCqNAnRBQl2Eobnt6nMul63IaydXeI4gV2r8ViCE1RO 1AIjLR8MUBLDS2AXE+jvS2lWqGxA8/i68rF5QdZiTAbtYjs0ZEF70yBGAuawzuxQbYgA jiY8mOfxJhujYHpoa2fNetkSeirlMuY/1bgHg7CmZ6XWmFw5i0CdusBg3m1l/CiLh15X iudA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363767; x=1740968567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Meuv+MydLFL2FaXVwlNSjl8ifFrRUaf5oa8eloYWNAY=; b=BoemCvro8iGuVbkte5zGSW4mLaTpJJvqZBXQUaWDOQEd3uTe5NYbfkVLYcBekVdofz ovtbhSSbP+VQH3FPMHwRi56onZIsNWajGqMRhp+8LR+37/LeTduDo0ZopZsbNQ5NGC8c PgV0PefAd1zegVR8DWFVg/QrX2F1dxqz5eH10JAFm08c2e1b7Ia0hr00Eg9e2leuSUCe noP5DZxkIN+MvO8rVniHHo6tVtBFA00jRyWsOOPPs2HoUDEwppD5DxyY2grL4pmJN2XH tKrfOY4AaI5CPbtkFh+cDWkoBh4rxQjM2cV65otETw3V9LIu467mWJSlVBl3XcBaUISW SEPg== X-Forwarded-Encrypted: i=1; AJvYcCU9nohnHBZH8xsO5AyH6umHDefn6FRH0Eo17QAs6M1kq7QaieJBfKXMYWLYBbQEj2K9fUg=@vger.kernel.org, AJvYcCUGywl/xdpWZmvl/G32A7kNc6iqpAjGNHc5lrK1KA2yHVulytKP3QcYqV+gkeZ1No8A9IbCYq5IdTSDG45yng2p@vger.kernel.org, AJvYcCXsnEDSpVA6W0NBvNxITCtIeMuiIY1HRsy0fKvtXpdJDCY06GvsHpfFERuyAbNEkgofRDMpDOAjPCI+Vfmm@vger.kernel.org X-Gm-Message-State: AOJu0YyNrfqKPNF95weA39UXJlbmZ3qYLkZfAJ5CoprvPhkcgTIzSDdm QuNn05tdGf5w+70kXSVGe37GDT5LTqfL1SvNwco4VlzupcuYJmCU X-Gm-Gg: ASbGnctVTHvrE2GE706rYrp+LlRPNE0GFG3fN7g45tytpv0cUvpFLHQy4a+hY6/AOsp cYejlhe8//T31lNDC0Qb3BMWDOn8eFigXJSN69fHOUCYLzyVJ8rxD7Y0REPAdZdd3hIFfrd0E7I hS+VSLWnfvG0gZ5LouRbWSOmZf0PHU1nzACgBe66IBlBe8GYLiKOeFFc4BZp6Mj2diRtBgqN66n tmSElMaZrFceAOghW1afa1YHTucr9o7yq1+Ti9QbGVLlpzGOSMD4RpGIobhqEBBk8QBkU71mon1 eH6QjAB4SnrXT/JAUZXJRiyVPaabJGP9YR+wWTFZGZfBfPo5TtSed7fBxdbgBA14JfkiomSrKPY rLPkGSSDOScL5beOm X-Google-Smtp-Source: AGHT+IFiS5JCKSf9kpXYqQvd4C20SO9WeFyiqD0ODJRSs4FwlxUKUEaIQb+jwmPk1FCG3Lz3izvoTA== X-Received: by 2002:ac8:5804:0:b0:472:9ea:887b with SMTP id d75a77b69052e-472228b3738mr106238181cf.10.1740363766868; Sun, 23 Feb 2025 18:22:46 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4720b1fe010sm59942551cf.60.2025.02.23.18.22.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:46 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id F0E6C1200043; Sun, 23 Feb 2025 21:22:45 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:45 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepieenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedukedpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:45 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov Subject: [PATCH rcu 20/20] srcu: Make SRCU-fast also be NMI-safe Date: Sun, 23 Feb 2025 18:22:14 -0800 Message-Id: <20250224022214.12037-21-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" BPF uses rcu_read_lock_trace() in NMI context, so srcu_read_lock_fast() must be NMI-safe if it is to have any chance of addressing RCU Tasks Trace use cases. This commit therefore causes srcu_read_lock_fast() and srcu_read_unlock_fast() to use atomic_long_inc() instead of this_cpu_inc() on architectures that support NMIs but do not have NMI-safe implementations of this_cpu_inc(). Note that both x86 and arm64 have NMI-safe implementations of this_cpu_inc(), and thus do not pay the performance penalty inherent in atomic_inc_long(). It is tempting to use this trick to fold srcu_read_lock_nmisafe() into srcu_read_lock(), but this would need careful thought, review, and performance analysis. Though those smp_mb() calls might well make performance a non-issue. Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index bdc467efce3a..8bed7e6cc4c1 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -231,17 +231,24 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ss * srcu_struct. Returns a pointer that must be passed to the matching * srcu_read_unlock_fast(). * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. + * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side + * critical sections either because they disables interrupts, because they + * are a single instruction, or because they are a read-modify-write atomic + * operation, depending on the whims of the architecture. + * + * This means that __srcu_read_lock_fast() is not all that fast + * on architectures that support NMIs but do not supply NMI-safe + * implementations of this_cpu_inc(). */ static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct *ssp) { struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast()."); - this_cpu_inc(scp->srcu_locks.counter); /* Y */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_locks.counter); /* Y */ + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */ barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -252,15 +259,22 @@ static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct * different CPU than that which was incremented by the corresponding * srcu_read_lock_fast(), but it must be within the same task. * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. + * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side + * critical sections either because they disables interrupts, because they + * are a single instruction, or because they are a read-modify-write atomic + * operation, depending on the whims of the architecture. + * + * This means that __srcu_read_unlock_fast() is not all that fast + * on architectures that support NMIs but do not supply NMI-safe + * implementations of this_cpu_inc(). */ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) { barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast()."); }