From patchwork Thu Mar 18 18:42:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 405333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7597CC4332D for ; Thu, 18 Mar 2021 18:43:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 231A964F53 for ; Thu, 18 Mar 2021 18:43:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232725AbhCRSmo (ORCPT ); Thu, 18 Mar 2021 14:42:44 -0400 Received: from mail-40133.protonmail.ch ([185.70.40.133]:22084 "EHLO mail-40133.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232675AbhCRSmg (ORCPT ); Thu, 18 Mar 2021 14:42:36 -0400 Date: Thu, 18 Mar 2021 18:42:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1616092954; bh=uga/a2V2PAZhoIbIvMFpYDGIy1+3tARpQIkiT76KT3w=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=g+3uQ5rbOLP8AEv2fyaFf/ESl7FCCuqGZ4kYFQ5fg7g6BHeobyHW4uYMyLMPrFwT8 93lfmMvjWSm4Iij/A+3wYYovZDDdEkurVnsdWd6yCqK3gSX5iNfRAmQxXt4E4v2LtT 5901nfIkdM6UmnxfI8pDeGtOfnGl84JMVQLh6ka2uqvo+SzvCQYa6gwpQB9v2Lr3ah zLa3mb5+NrEkW3IqGvJQbqSP05UnuSnJyGbZbTkAjGJA5xiyjaFY5vs9pxGwggyUJR r6FSgbcTPY7HZ/GDWnLoChC6TRgtofpGmt8t1eMrrprRTXVF0vOgshekcKGhpjPxz6 vBDiHSI7DWZmg== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Alexander Lobakin , Mauro Carvalho Chehab , Kirill Tkhai , Bartosz Golaszewski , Leon Romanovsky , Eric Dumazet , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH net-next 1/4] gro: make net/gro.h self-contained Message-ID: <20210318184157.700604-2-alobakin@pm.me> In-Reply-To: <20210318184157.700604-1-alobakin@pm.me> References: <20210318184157.700604-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org If some source file includes , but doesn't include : In file included from net/8021q/vlan_core.c:7: ./include/net/gro.h:6:1: warning: data definition has no type or storage class 6 | INDIRECT_CALLABLE_DECLARE(struct sk_buff *ipv6_gro_receive(struct list_head *, | ^~~~~~~~~~~~~~~~~~~~~~~~~ ./include/net/gro.h:6:1: error: type defaults to ‘int’ in declaration of ‘INDIRECT_CALLABLE_DECLARE’ [-Werror=implicit-int] [...] Include directly. It's small and won't pull lots of dependencies. Also add some incomplete struct declarations to be fully stacked. Fixes: 04f00ab2275f ("net/core: move gro function declarations to separate header ") Signed-off-by: Alexander Lobakin --- include/net/gro.h | 5 +++++ 1 file changed, 5 insertions(+) -- 2.31.0 diff --git a/include/net/gro.h b/include/net/gro.h index 8a6eb5303cc4..27c38b36df16 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -3,6 +3,11 @@ #ifndef _NET_IPV6_GRO_H #define _NET_IPV6_GRO_H +#include + +struct list_head; +struct sk_buff; + INDIRECT_CALLABLE_DECLARE(struct sk_buff *ipv6_gro_receive(struct list_head *, struct sk_buff *)); INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int)); From patchwork Thu Mar 18 18:42:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 404573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CF6EC433E6 for ; Thu, 18 Mar 2021 18:43:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E76A564F4D for ; Thu, 18 Mar 2021 18:43:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232704AbhCRSmn (ORCPT ); Thu, 18 Mar 2021 14:42:43 -0400 Received: from mail-40133.protonmail.ch ([185.70.40.133]:31689 "EHLO mail-40133.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232622AbhCRSmf (ORCPT ); Thu, 18 Mar 2021 14:42:35 -0400 Date: Thu, 18 Mar 2021 18:42:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1616092953; bh=CZPzXPr6kxKmjKYiFth7lEmlditzWiKAt31a6bYz0BI=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=FIaVLXSW92dxr8oytIZCizii/i9XUVemC0T7TIRFjAl8uwZpExxjQ7C3+J5qZMr4W wwTyCbbGbZndoyKBBvrGYBmryILMnlvz6RobHBpFBkBBdlQ1i1TT2nqw1WeCuAmYTB yDraprzPaWRcHKCv4BKVytT64iZCjXFeB0+XMqZ0rcFlVbUjB57tUVJR5QOtqCAwLe 8aayIKPCRWVZuQYFOvvVtuYaiA2kx+5bjFKjKFbrFMj9tR4Om6jX8PT0e8zpnuE4tC 5gpLrw10tKVeR7UODv2P8EOCVlvB5NLFkaXNCHs5srn/NlZ1MGgsv1ROlw4lXYt79u EP2wMyhm3hm4Q== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Alexander Lobakin , Mauro Carvalho Chehab , Kirill Tkhai , Bartosz Golaszewski , Leon Romanovsky , Eric Dumazet , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH net-next 2/4] gro: add combined call_gro_receive() + INDIRECT_CALL_INET() helper Message-ID: <20210318184157.700604-3-alobakin@pm.me> In-Reply-To: <20210318184157.700604-1-alobakin@pm.me> References: <20210318184157.700604-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org call_gro_receive() is used to limit GRO recursion, but it works only with callback pointers. There's a combined version of call_gro_receive() + INDIRECT_CALL_2() in , but it doesn't check for IPv6 modularity. Add a similar new helper to cover both of these. It can and will be used to avoid retpoline overhead when IP header lies behind another offloaded proto. Signed-off-by: Alexander Lobakin --- include/net/gro.h | 8 ++++++++ 1 file changed, 8 insertions(+) -- 2.31.0 diff --git a/include/net/gro.h b/include/net/gro.h index 27c38b36df16..01edaf3fdda0 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -14,4 +14,12 @@ INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *inet_gro_receive(struct list_head *, struct sk_buff *)); INDIRECT_CALLABLE_DECLARE(int inet_gro_complete(struct sk_buff *, int)); + +#define indirect_call_gro_receive_inet(cb, f2, f1, head, skb) \ +({ \ + unlikely(gro_recursion_inc_test(skb)) ? \ + NAPI_GRO_CB(skb)->flush |= 1, NULL : \ + INDIRECT_CALL_INET(cb, f2, f1, head, skb); \ +}) + #endif /* _NET_IPV6_GRO_H */ From patchwork Thu Mar 18 18:42:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 405332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A29C1C433E0 for ; Thu, 18 Mar 2021 18:43:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 614CD64F2B for ; Thu, 18 Mar 2021 18:43:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232771AbhCRSnP (ORCPT ); Thu, 18 Mar 2021 14:43:15 -0400 Received: from mail-40134.protonmail.ch ([185.70.40.134]:23574 "EHLO mail-40134.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232710AbhCRSmp (ORCPT ); Thu, 18 Mar 2021 14:42:45 -0400 Date: Thu, 18 Mar 2021 18:42:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1616092962; bh=X+A01bVgCW9C0LcweX802pafQYX/MaNpf8hWlmvRiLI=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=ihdVBEmjgnxfyRPkNRf6p8/oxzuoUGhnrp6NznWS9bSE5WC/C6TJe4eq3wNehnTwo XSxncp9rTywDwaLZ1VkljA4hoViA1ArxFmrMPpObCe9IXTCR8pJ6iXi5qtE3ukYhGJ c9vnxFBFBMryQwg3N4vEJvCrXTojaMWwR5YX8YYC5U56dZFTcp7bP8/g50FB41MAWE Oaam0BjnCsS/z9LFNcGyfN2TSSVDHn1RKB3nxpXu/Rx6rjR/kwhZsVLlTvIVVH7127 /kAoc0UH3ivKn2fmFvTK7TpJugrpOKwtL16iEbx62f50ZKAJ1imdN8d8dCaXmfJXaE j1OdCDOS/OFtQ== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Alexander Lobakin , Mauro Carvalho Chehab , Kirill Tkhai , Bartosz Golaszewski , Leon Romanovsky , Eric Dumazet , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH net-next 3/4] vlan/8021q: avoid retpoline overhead on GRO Message-ID: <20210318184157.700604-4-alobakin@pm.me> In-Reply-To: <20210318184157.700604-1-alobakin@pm.me> References: <20210318184157.700604-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The two most popular headers going after VLAN are IPv4 and IPv6. Retpoline overhead for them is addressed only in dev_gro_receive(), when they lie right after the outermost Ethernet header. Use the indirect call wrappers in VLAN GRO receive code to reduce the penalty on receiving tagged frames (when hardware stripping is off or not available). Signed-off-by: Alexander Lobakin --- net/8021q/vlan_core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) -- 2.31.0 diff --git a/net/8021q/vlan_core.c b/net/8021q/vlan_core.c index 78ec2e1b14d1..59bc13b5f14f 100644 --- a/net/8021q/vlan_core.c +++ b/net/8021q/vlan_core.c @@ -4,6 +4,7 @@ #include #include #include +#include #include "vlan.h" bool vlan_do_receive(struct sk_buff **skbp) @@ -495,7 +496,10 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head, skb_gro_pull(skb, sizeof(*vhdr)); skb_gro_postpull_rcsum(skb, vhdr, sizeof(*vhdr)); - pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb); + + pp = indirect_call_gro_receive_inet(ptype->callbacks.gro_receive, + ipv6_gro_receive, inet_gro_receive, + head, skb); out_unlock: rcu_read_unlock(); @@ -515,7 +519,9 @@ static int vlan_gro_complete(struct sk_buff *skb, int nhoff) rcu_read_lock(); ptype = gro_find_complete_by_type(type); if (ptype) - err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(*vhdr)); + err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete, + ipv6_gro_complete, inet_gro_complete, + skb, nhoff + sizeof(*vhdr)); rcu_read_unlock(); return err; From patchwork Thu Mar 18 18:42:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 404572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8315FC433E9 for ; Thu, 18 Mar 2021 18:43:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3F29164F6B for ; Thu, 18 Mar 2021 18:43:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232721AbhCRSnN (ORCPT ); Thu, 18 Mar 2021 14:43:13 -0400 Received: from mail-40133.protonmail.ch ([185.70.40.133]:35379 "EHLO mail-40133.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232670AbhCRSmm (ORCPT ); Thu, 18 Mar 2021 14:42:42 -0400 Date: Thu, 18 Mar 2021 18:42:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1616092960; bh=lRsfcB6MOCDLr07XVaSyak4dBxm5+WyXdc1BMSXra2g=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=pKCNFQL/xr/CXt9HvoPgr9+uy4yPhiw//L6nBrmR539JfzGk91q/hhycZHWlBhKe0 pEqihAX8BbLHii5g2NL91Rq+QVzslf02BZD6x6w8p31NMh+R7VxdhY/WT8qByzmqFa vGXL+j+v9MZmhur5h1LOMiKOB4isYNyyQedMOyiHHxaq+l3Oi66aiDrV2xb2tsHjgU OyJuHv8AMcALbcG6cJ+uMNaRJ0OqH4y9iz1mBtXCV8DF4r1LX2zHxi9TE9I6XMkk6i rrQomPXgnCtRRWaYr182NwLOSEjaMDHDR8lYDnn+6rUTw7PweuQS2ik0rCDEUF41V7 exGiYWEwFqVJQ== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Alexander Lobakin , Mauro Carvalho Chehab , Kirill Tkhai , Bartosz Golaszewski , Leon Romanovsky , Eric Dumazet , netdev@vger.kernel.org, linux-kernel@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH net-next 4/4] ethernet: avoid retpoline overhead on TEB (GENEVE, NvGRE, VxLAN) GRO Message-ID: <20210318184157.700604-5-alobakin@pm.me> In-Reply-To: <20210318184157.700604-1-alobakin@pm.me> References: <20210318184157.700604-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The two most popular headers going after Ethernet are IPv4 and IPv6. Retpoline overhead for them is addressed only in dev_gro_receive(), when they lie right after the outermost Ethernet header. Use the indirect call wrappers in TEB (Transparent Ethernet Bridging, such as GENEVE, NvGRE, VxLAN etc.) GRO receive code to reduce the penalty when processing the inner headers. Signed-off-by: Alexander Lobakin --- net/ethernet/eth.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) -- 2.31.0 diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c index e01cf766d2c5..933b427122be 100644 --- a/net/ethernet/eth.c +++ b/net/ethernet/eth.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include @@ -449,7 +450,10 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) skb_gro_pull(skb, sizeof(*eh)); skb_gro_postpull_rcsum(skb, eh, sizeof(*eh)); - pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb); + + pp = indirect_call_gro_receive_inet(ptype->callbacks.gro_receive, + ipv6_gro_receive, inet_gro_receive, + head, skb); out_unlock: rcu_read_unlock(); @@ -473,8 +477,9 @@ int eth_gro_complete(struct sk_buff *skb, int nhoff) rcu_read_lock(); ptype = gro_find_complete_by_type(type); if (ptype != NULL) - err = ptype->callbacks.gro_complete(skb, nhoff + - sizeof(struct ethhdr)); + err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete, + ipv6_gro_complete, inet_gro_complete, + skb, nhoff + sizeof(*eh)); rcu_read_unlock(); return err;