From patchwork Wed Jul 20 18:26:19 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 2878 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 43C2C2405D for ; Wed, 20 Jul 2011 18:26:40 +0000 (UTC) Received: from mail-qw0-f52.google.com (mail-qw0-f52.google.com [209.85.216.52]) by fiordland.canonical.com (Postfix) with ESMTP id 12780A186C9 for ; Wed, 20 Jul 2011 18:26:39 +0000 (UTC) Received: by qwb8 with SMTP id 8so365918qwb.11 for ; Wed, 20 Jul 2011 11:26:39 -0700 (PDT) Received: by 10.229.25.212 with SMTP id a20mr7419911qcc.148.1311186399397; Wed, 20 Jul 2011 11:26:39 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.229.217.78 with SMTP id hl14cs121057qcb; Wed, 20 Jul 2011 11:26:39 -0700 (PDT) Received: by 10.52.71.211 with SMTP id x19mr3006227vdu.69.1311186398247; Wed, 20 Jul 2011 11:26:38 -0700 (PDT) Received: from e6.ny.us.ibm.com (e6.ny.us.ibm.com [32.97.182.146]) by mx.google.com with ESMTPS id a17si357777vcf.28.2011.07.20.11.26.38 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 20 Jul 2011 11:26:38 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.146 as permitted sender) client-ip=32.97.182.146; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.146 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from d01relay03.pok.ibm.com (d01relay03.pok.ibm.com [9.56.227.235]) by e6.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p6KI2UHn010772 for ; Wed, 20 Jul 2011 14:02:30 -0400 Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p6KIQbGg132302 for ; Wed, 20 Jul 2011 14:26:37 -0400 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p6KIQRlc024249 for ; Wed, 20 Jul 2011 14:26:37 -0400 Received: from paulmck-ThinkPad-W500 (paulmck-ThinkPad-W500.beaverton.ibm.com [9.47.24.65]) by d01av01.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p6KIQPin024160; Wed, 20 Jul 2011 14:26:25 -0400 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 2989213F802; Wed, 20 Jul 2011 11:26:25 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, patches@linaro.org, greearb@candelatech.com, edt@aei.ca, "Paul E. McKenney" Subject: [PATCH tip/core/urgent 3/7] rcu: Streamline code produced by __rcu_read_unlock() Date: Wed, 20 Jul 2011 11:26:19 -0700 Message-Id: <1311186383-24819-3-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.3.2 In-Reply-To: <20110720182512.GA22946@linux.vnet.ibm.com> References: <20110720182512.GA22946@linux.vnet.ibm.com> Given some common flag combinations, particularly -Os, gcc will inline rcu_read_unlock_special() despite its being in an unlikely() clause. Use noinline to prohibit this misoptimization. In addition, move the second barrier() in __rcu_read_unlock() so that it is not on the common-case code path. This will allow the compiler to generate better code for the common-case path through __rcu_read_unlock(). Suggested-by: Linus Torvalds Signed-off-by: Paul E. McKenney Acked-by: Mathieu Desnoyers --- kernel/rcutree_plugin.h | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h index 3a0ae03..4d2c068 100644 --- a/kernel/rcutree_plugin.h +++ b/kernel/rcutree_plugin.h @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t, * notify RCU core processing or task having blocked during the RCU * read-side critical section. */ -static void rcu_read_unlock_special(struct task_struct *t) +static noinline void rcu_read_unlock_special(struct task_struct *t) { int empty; int empty_exp; @@ -391,11 +391,11 @@ void __rcu_read_unlock(void) struct task_struct *t = current; barrier(); /* needed if we ever invoke rcu_read_unlock in rcutree.c */ - --t->rcu_read_lock_nesting; - barrier(); /* decrement before load of ->rcu_read_unlock_special */ - if (t->rcu_read_lock_nesting == 0 && - unlikely(ACCESS_ONCE(t->rcu_read_unlock_special))) - rcu_read_unlock_special(t); + if (--t->rcu_read_lock_nesting == 0) { + barrier(); /* decr before ->rcu_read_unlock_special load */ + if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special))) + rcu_read_unlock_special(t); + } #ifdef CONFIG_PROVE_LOCKING WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0); #endif /* #ifdef CONFIG_PROVE_LOCKING */