From patchwork Tue Jun 8 18:27:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "gregkh@linuxfoundation.org" X-Patchwork-Id: 455834 Delivered-To: patch@linaro.org Received: by 2002:a02:735a:0:0:0:0:0 with SMTP id a26csp4032224jae; Tue, 8 Jun 2021 12:14:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyz/g0iZcxec6akanny0EifewzXH3sIdLzwijrEqczFGz4UtP0Aoc3/9leZOonsB1W2AA0j X-Received: by 2002:a17:906:70d4:: with SMTP id g20mr12073119ejk.327.1623179645612; Tue, 08 Jun 2021 12:14:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1623179645; cv=none; d=google.com; s=arc-20160816; b=IBxAZTZvXD8j22VlSOvJ6ns1y7VKUL/Il3DCW5woMPLvAyoYynlhKuf1HXqFGkXGyq wyCaKbzHboMkQ/mhbD9jquCtpssOdvJyRW7h1U0+d7nQDGaoCjd8/hK2dpZP8+t7PNF0 GoPBtx3ZQ/gKNfk9TBPShqZHtlJmPd1JA6Muz0xBVqQ19yYEDl+5TyWavRbKbylXty3T lJigLtPre24bM/34R8ABsAWqamS52PxM/n6IiFCK4UWvq/gxhwMdN7zOgdg9WMJqbCIV G6xrqjwXGrtyH+o2OvFBC5D54dFRpprEWZQIU/wT7i97BrTUgDv4TOF2aDUp5EDkO4is cybw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Yf2bw83u6WPOHV9YCwCIY0Y/sFMOsHM+dppo4P0iqSc=; b=OO/u1beoKF8qxxtCi/PyyYfkOaP+nn+WtRakjQcvNubzfRpBYgFNpX5tuE/nJ6BRXN tQ/5Usrm8tf7Js4v5TKF1/W++MSMJxwpUsgbWhg6zicjEvA8RPSN0iMP5RYbSUqOLLqb 5bHX+HIDuuscenvVpVpC4bv74IbREEKWGeT/Y2rLKujehR23WNhAMNHeewjfvuVo+ipu f2BG3WDuJWfhsze+UsES0mSO/7B9VNc4B+hQcz0NDwW+SlinqiOcgiBuvmLr5gDFrh8D rqlfXC3Ep1ElBJPp3zvhNUh+UZNjDdaWPDOU5pVtq0jA0SIn2lS/YcdQqOsW25y4F8HG uJHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b="Z/AWCqXi"; spf=pass (google.com: domain of stable-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id l23si509825ejb.573.2021.06.08.12.14.05; Tue, 08 Jun 2021 12:14:05 -0700 (PDT) Received-SPF: pass (google.com: domain of stable-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b="Z/AWCqXi"; spf=pass (google.com: domain of stable-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235765AbhFHTOB (ORCPT + 12 others); Tue, 8 Jun 2021 15:14:01 -0400 Received: from mail.kernel.org ([198.145.29.99]:59066 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234695AbhFHTL7 (ORCPT ); Tue, 8 Jun 2021 15:11:59 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2364561951; Tue, 8 Jun 2021 18:49:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1623178162; bh=0hXEeibGul9nJre4TcFiqpF9IJB0HprDogCDkcaYm0I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z/AWCqXi7WpQKuCo8huKwgf+ozurK2Q4GjEzCMlG8iEtc3UcExaEESwqfINoCX+rH p06y7iL9+sSgEbltws5qIg5GzQL0MLuOElBOmzwaRyrhkqpgQnnxX1Fj5jW42kNWBO U5pm1Msfx8zs21jQgm0TPMJARD+86ZjQmbvdhFZw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Arnd Bergmann , Matthew Wilcox , "Jason A. Donenfeld" , "David S. Miller" Subject: [PATCH 5.12 101/161] wireguard: allowedips: allocate nodes in kmem_cache Date: Tue, 8 Jun 2021 20:27:11 +0200 Message-Id: <20210608175948.875081842@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210608175945.476074951@linuxfoundation.org> References: <20210608175945.476074951@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Jason A. Donenfeld commit dc680de28ca849dfe589dc15ac56d22505f0ef11 upstream. The previous commit moved from O(n) to O(1) for removal, but in the process introduced an additional pointer member to a struct that increased the size from 60 to 68 bytes, putting nodes in the 128-byte slab. With deployed systems having as many as 2 million nodes, this represents a significant doubling in memory usage (128 MiB -> 256 MiB). Fix this by using our own kmem_cache, that's sized exactly right. This also makes wireguard's memory usage more transparent in tools like slabtop and /proc/slabinfo. Fixes: e7096c131e51 ("net: WireGuard secure network tunnel") Suggested-by: Arnd Bergmann Suggested-by: Matthew Wilcox Cc: stable@vger.kernel.org Signed-off-by: Jason A. Donenfeld Signed-off-by: David S. Miller Signed-off-by: Greg Kroah-Hartman --- drivers/net/wireguard/allowedips.c | 31 +++++++++++++++++++++++++------ drivers/net/wireguard/allowedips.h | 5 ++++- drivers/net/wireguard/main.c | 10 +++++++++- 3 files changed, 38 insertions(+), 8 deletions(-) --- a/drivers/net/wireguard/allowedips.c +++ b/drivers/net/wireguard/allowedips.c @@ -6,6 +6,8 @@ #include "allowedips.h" #include "peer.h" +static struct kmem_cache *node_cache; + static void swap_endian(u8 *dst, const u8 *src, u8 bits) { if (bits == 32) { @@ -40,6 +42,11 @@ static void push_rcu(struct allowedips_n } } +static void node_free_rcu(struct rcu_head *rcu) +{ + kmem_cache_free(node_cache, container_of(rcu, struct allowedips_node, rcu)); +} + static void root_free_rcu(struct rcu_head *rcu) { struct allowedips_node *node, *stack[128] = { @@ -49,7 +56,7 @@ static void root_free_rcu(struct rcu_hea while (len > 0 && (node = stack[--len])) { push_rcu(stack, node->bit[0], &len); push_rcu(stack, node->bit[1], &len); - kfree(node); + kmem_cache_free(node_cache, node); } } @@ -164,7 +171,7 @@ static int add(struct allowedips_node __ return -EINVAL; if (!rcu_access_pointer(*trie)) { - node = kzalloc(sizeof(*node), GFP_KERNEL); + node = kmem_cache_zalloc(node_cache, GFP_KERNEL); if (unlikely(!node)) return -ENOMEM; RCU_INIT_POINTER(node->peer, peer); @@ -180,7 +187,7 @@ static int add(struct allowedips_node __ return 0; } - newnode = kzalloc(sizeof(*newnode), GFP_KERNEL); + newnode = kmem_cache_zalloc(node_cache, GFP_KERNEL); if (unlikely(!newnode)) return -ENOMEM; RCU_INIT_POINTER(newnode->peer, peer); @@ -213,10 +220,10 @@ static int add(struct allowedips_node __ return 0; } - node = kzalloc(sizeof(*node), GFP_KERNEL); + node = kmem_cache_zalloc(node_cache, GFP_KERNEL); if (unlikely(!node)) { list_del(&newnode->peer_list); - kfree(newnode); + kmem_cache_free(node_cache, newnode); return -ENOMEM; } INIT_LIST_HEAD(&node->peer_list); @@ -306,7 +313,7 @@ void wg_allowedips_remove_by_peer(struct if (child) child->parent_bit = node->parent_bit; *rcu_dereference_protected(node->parent_bit, lockdep_is_held(lock)) = child; - kfree_rcu(node, rcu); + call_rcu(&node->rcu, node_free_rcu); /* TODO: Note that we currently don't walk up and down in order to * free any potential filler nodes. This means that this function @@ -350,4 +357,16 @@ struct wg_peer *wg_allowedips_lookup_src return NULL; } +int __init wg_allowedips_slab_init(void) +{ + node_cache = KMEM_CACHE(allowedips_node, 0); + return node_cache ? 0 : -ENOMEM; +} + +void wg_allowedips_slab_uninit(void) +{ + rcu_barrier(); + kmem_cache_destroy(node_cache); +} + #include "selftest/allowedips.c" --- a/drivers/net/wireguard/allowedips.h +++ b/drivers/net/wireguard/allowedips.h @@ -19,7 +19,7 @@ struct allowedips_node { u8 bits[16] __aligned(__alignof(u64)); /* Keep rarely used members at bottom to be beyond cache line. */ - struct allowedips_node *__rcu *parent_bit; /* XXX: this puts us at 68->128 bytes instead of 60->64 bytes!! */ + struct allowedips_node *__rcu *parent_bit; union { struct list_head peer_list; struct rcu_head rcu; @@ -53,4 +53,7 @@ struct wg_peer *wg_allowedips_lookup_src bool wg_allowedips_selftest(void); #endif +int wg_allowedips_slab_init(void); +void wg_allowedips_slab_uninit(void); + #endif /* _WG_ALLOWEDIPS_H */ --- a/drivers/net/wireguard/main.c +++ b/drivers/net/wireguard/main.c @@ -21,10 +21,15 @@ static int __init mod_init(void) { int ret; + ret = wg_allowedips_slab_init(); + if (ret < 0) + goto err_allowedips; + #ifdef DEBUG + ret = -ENOTRECOVERABLE; if (!wg_allowedips_selftest() || !wg_packet_counter_selftest() || !wg_ratelimiter_selftest()) - return -ENOTRECOVERABLE; + goto err_peer; #endif wg_noise_init(); @@ -50,6 +55,8 @@ err_netlink: err_device: wg_peer_uninit(); err_peer: + wg_allowedips_slab_uninit(); +err_allowedips: return ret; } @@ -58,6 +65,7 @@ static void __exit mod_exit(void) wg_genetlink_uninit(); wg_device_uninit(); wg_peer_uninit(); + wg_allowedips_slab_uninit(); } module_init(mod_init);