From patchwork Sat Mar 10 15:21:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131299 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244688lja; Sat, 10 Mar 2018 07:23:01 -0800 (PST) X-Google-Smtp-Source: AG47ELueHdVLVftnGFfDrqLw0P5XNtgG0lWKVlqMNLqAhOPHv2YlkrJrlqCT8Op4CxN/9PU5tYZL X-Received: by 10.98.137.147 with SMTP id n19mr2277227pfk.193.1520695381319; Sat, 10 Mar 2018 07:23:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695381; cv=none; d=google.com; s=arc-20160816; b=IVSu3ZnS9P7UWyiK3kkF2yMA21XO/0/h4CddsdHaljYBWtHD0hBfcySPm/gPDyeiuL NgIVhvgKycRfRZnkyIhcZQlKGNRbrv8Q0OhmFJjhuKhhEQHnu0slOsqJmv7yRaySFXOA sm9LicxNM44Aw/r1REx3lcoB2OghpqoFTgoF2VSWynkHy7fONSCGCFRcdKi47t5GPaNl nUTVf/jVva77dfJKDCiqvU5vitkTcjQW6UvyCetMUYlCIW4XlBYipRnTeAJAuDcixlvz 83VWVkSQR2hgqvTLZ5mn84sgipqvVpvTjLRan8ipyOoQ/wCJuTMQIXUD+sVwAf9Makgj 6u5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=0pSO1I7/1d3SIyd/CIYxopugahniWYaNXOHMgXM3sEc=; b=CkTN8bq097IfZ1JRG8L5hv8yUs6cdPxxzmyBQzrzC+dkEbds9190WSw0URoBgVkQtI RDBA842hy60kRNOktsuZsSpBwHjstO0Gcku/tSc6KJZLQ6p1o2J8xtvdk4fnATk9A4Og k6x8UGt+Bmwm4JzADbebip6EGUs5w1fB7OBtrTXTi1eeufgS7cpnX/gLwQOeTn1/p54S WE+4TmjNbCixYCdmt/MeACZiVvEWDABMQl9MhKL/vI+7Hl60vxlcIpy07TKwJ83jG93V VxR8ZnY3lRczf9Ye40YQTHrVKcnwL5IEsZPr6VEUFnRWF6QseZTcKX9s8v++ZQuh0Efg O0lA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=RknFKhhG; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.23.01; Sat, 10 Mar 2018 07:23:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=RknFKhhG; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932315AbeCJPW7 (ORCPT + 4 others); Sat, 10 Mar 2018 10:22:59 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:43528 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932308AbeCJPW5 (ORCPT ); Sat, 10 Mar 2018 10:22:57 -0500 Received: by mail-wr0-f193.google.com with SMTP id a63so8024182wrc.10 for ; Sat, 10 Mar 2018 07:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RYUXyT4yhMIRanK7ihkLKdOKlgN14RDXrNInu0AEK8g=; b=RknFKhhG0D7vyNolv/HJayLrHRepnfuk8E4K5bw7LPuka/mELechLkjTQttpHSUJwJ VkHkVreuHUqi9lVl8N+2jMmTT7NWy2MNDe1TpSQxouT5uHI28Y6WSEkuToVG9+lVXqHO jDxGKUAZUJupEAtqbchw96mD4gOLS8vyUaNH0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RYUXyT4yhMIRanK7ihkLKdOKlgN14RDXrNInu0AEK8g=; b=gDeeOem3nVlnX+NTNiChfjlkvT89Xs4Tpr621lwhx0X2YRT+2SKOPLaBPwy20AXQkw HRSGkiikx47C1CJkP11zLA3y8GvcIQvEmPttWNpk1+Fx1lbyNvnjnq+8Kc2mCVd6Lc4g R2iGWa/h3cBQ0MdKarsvfbxaBAbP+hfprXxKetphO5k0DQ0m7ddO7bAnGPD2drRICKFu NI/VxlJIzsr7jBYUpEdNjRVcDrWf+LoiQPdGGPwe1aVR7nfQeYi0x7BrQQWKD6PeDW5Q I5AmMwGw7ocEPcUSfMKEbswTeTIkr0Q5m42B0H7JXNOvRThS8Rk2gKWHzcp2Og1vGYqt qWeg== X-Gm-Message-State: AElRT7FGo7pjzKwYWrIqK2zU1uTQPiysR1QxWT7QIRS1Xwpn8+ozZcxc xrVQjZhnrHXBH8l7z2PlOcy2tHuhM9Q= X-Received: by 10.223.136.15 with SMTP id d15mr1890013wrd.127.1520695375885; Sat, 10 Mar 2018 07:22:55 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:55 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 09/23] crypto: arm64/sha256-neon - play nice with CONFIG_PREEMPT kernels Date: Sat, 10 Mar 2018 15:21:54 +0000 Message-Id: <20180310152208.10369-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Tweak the SHA256 update routines to invoke the SHA256 block transform block by block, to avoid excessive scheduling delays caused by the NEON algorithm running with preemption disabled. Also, remove a stale comment which no longer applies now that kernel mode NEON is actually disallowed in some contexts. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha256-glue.c | 36 +++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) -- 2.15.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c index b064d925fe2a..e8880ccdc71f 100644 --- a/arch/arm64/crypto/sha256-glue.c +++ b/arch/arm64/crypto/sha256-glue.c @@ -89,21 +89,32 @@ static struct shash_alg algs[] = { { static int sha256_update_neon(struct shash_desc *desc, const u8 *data, unsigned int len) { - /* - * Stacking and unstacking a substantial slice of the NEON register - * file may significantly affect performance for small updates when - * executing in interrupt context, so fall back to the scalar code - * in that case. - */ + struct sha256_state *sctx = shash_desc_ctx(desc); + if (!may_use_simd()) return sha256_base_do_update(desc, data, len, (sha256_block_fn *)sha256_block_data_order); - kernel_neon_begin(); - sha256_base_do_update(desc, data, len, - (sha256_block_fn *)sha256_block_neon); - kernel_neon_end(); + while (len > 0) { + unsigned int chunk = len; + + /* + * Don't hog the CPU for the entire time it takes to process all + * input when running on a preemptible kernel, but process the + * data block by block instead. + */ + if (IS_ENABLED(CONFIG_PREEMPT) && + chunk + sctx->count % SHA256_BLOCK_SIZE > SHA256_BLOCK_SIZE) + chunk = SHA256_BLOCK_SIZE - + sctx->count % SHA256_BLOCK_SIZE; + kernel_neon_begin(); + sha256_base_do_update(desc, data, chunk, + (sha256_block_fn *)sha256_block_neon); + kernel_neon_end(); + data += chunk; + len -= chunk; + } return 0; } @@ -117,10 +128,9 @@ static int sha256_finup_neon(struct shash_desc *desc, const u8 *data, sha256_base_do_finalize(desc, (sha256_block_fn *)sha256_block_data_order); } else { - kernel_neon_begin(); if (len) - sha256_base_do_update(desc, data, len, - (sha256_block_fn *)sha256_block_neon); + sha256_update_neon(desc, data, len); + kernel_neon_begin(); sha256_base_do_finalize(desc, (sha256_block_fn *)sha256_block_neon); kernel_neon_end();