From patchwork Thu Apr 24 10:46:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884076 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D2E81BC41 for ; Thu, 24 Apr 2025 10:47:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491625; cv=none; b=tizfC7iEJu1OnTqKB45O9WnvSbxVTFS2D4A5sEWfa6GfoWedLhldmDEEA/Er7Pyt2x/QyXiNdDiwSKkUEWD8mKsGccauUAktgdmEKyQUuNnNufT2GVEr1HTrCZKDY/2Eh5YhGCopoInEElvAWkYME1LqqVXYFEtQnDmswyundg8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491625; c=relaxed/simple; bh=Ke57qn4hRCxqEoAnvXopwdr1hZy//WRZ9YyiHok06ns=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=S8dgy7Chj4YLff8QHe4fFmcJDfMtbc6ueeVgs5a3ExuYqr9DTdYttSewTOWv8GIm+8itGuRuMbXxJSI6OysfITtNp0KL+YtkFhQuayGthsTgaOgrhrg4eh8LqBjS4+/Tjbw71KwnWRK5sJltqHp5j0CY3d8k73jRy+Buasz2+Nk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=dIWudVt4; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="dIWudVt4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=HmauAosGkBrZxlNmethMiiN/nycK6gwyAD1FkNssueY=; b=dIWudVt4K4OMJmP90dVIym521Q KH0/HtgtTpd/ng+DQDEP2eNyl12glv/O7/M8HHibv+hSAYF1cTwZRk/hSxbceyTIiDGagnrAx1sEn 2kWWUTvbY5AL3fCGoacxVFjc45xYAM/CKzDktonuaUzl5NoWDEyQ9dNWJzmAFSMqfWWCw2gGBths6 IvDPYK95m4lAGv6n6NGqbHCb4rLO8K0VoHIEggNO3zxaTZWIAdK041dCZLQPXlcwYeAG2av70ILbw SE0CFHkghuLpr9RhtirJA7QqxeObRev/CSKJfnnclbkH9VPYHd3GBwFkDxnWunV4yDB/i5cJgQd5v r9ilCZFw==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6c-000fLQ-1f; Thu, 24 Apr 2025 18:46:59 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:46:58 +0800 Date: Thu, 24 Apr 2025 18:46:58 +0800 Message-Id: In-Reply-To: References: From: Herbert Xu Subject: [PATCH 01/15] crypto: lib/sha256 - Move partial block handling out To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Extract the common partial block handling into a helper macro that can be reused by other library code. Also delete the unused sha256_base_do_finalize function. Signed-off-by: Herbert Xu --- include/crypto/internal/blockhash.h | 52 +++++++++++++++++++++++++++++ include/crypto/sha2.h | 9 +++-- include/crypto/sha256_base.h | 38 ++------------------- 3 files changed, 62 insertions(+), 37 deletions(-) create mode 100644 include/crypto/internal/blockhash.h diff --git a/include/crypto/internal/blockhash.h b/include/crypto/internal/blockhash.h new file mode 100644 index 000000000000..4184e2337d68 --- /dev/null +++ b/include/crypto/internal/blockhash.h @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Handle partial blocks for block hash. + * + * Copyright (c) 2015 Linaro Ltd + * Copyright (c) 2025 Herbert Xu + */ + +#ifndef _CRYPTO_INTERNAL_BLOCKHASH_H +#define _CRYPTO_INTERNAL_BLOCKHASH_H + +#include +#include + +#define BLOCK_HASH_UPDATE_BASE(block, state, src, nbytes, bs, dv, buf, \ + buflen) \ + ({ \ + unsigned int _nbytes = (nbytes); \ + unsigned int _buflen = (buflen); \ + typeof(block) _block = (block); \ + typeof(state) _state = (state); \ + unsigned int _bs = (bs); \ + unsigned int _dv = (dv); \ + const u8 *_src = (src); \ + u8 *_buf = (buf); \ + while ((_buflen + _nbytes) >= _bs) { \ + unsigned int len = _nbytes; \ + const u8 *data = _src; \ + int blocks, remain; \ + if (_buflen) { \ + remain = _bs - _buflen; \ + memcpy(_buf + _buflen, _src, remain); \ + data = _buf; \ + len = _bs; \ + } \ + remain = len % bs; \ + blocks = (len - remain) / _dv; \ + _block(_state, data, blocks); \ + _src += len - remain - _buflen; \ + _nbytes -= len - remain - _buflen; \ + _buflen = 0; \ + } \ + memcpy(_buf + _buflen, _src, _nbytes); \ + _buflen += _nbytes; \ + }) + +#define BLOCK_HASH_UPDATE(block, state, src, nbytes, bs, buf, buflen) \ + BLOCK_HASH_UPDATE_BASE(block, state, src, nbytes, bs, 1, buf, buflen) +#define BLOCK_HASH_UPDATE_BLOCKS(block, state, src, nbytes, bs, buf, buflen) \ + BLOCK_HASH_UPDATE_BASE(block, state, src, nbytes, bs, bs, buf, buflen) + +#endif /* _CRYPTO_INTERNAL_BLOCKHASH_H */ diff --git a/include/crypto/sha2.h b/include/crypto/sha2.h index abbd882f7849..f873c2207b1e 100644 --- a/include/crypto/sha2.h +++ b/include/crypto/sha2.h @@ -71,8 +71,13 @@ struct crypto_sha256_state { }; struct sha256_state { - u32 state[SHA256_DIGEST_SIZE / 4]; - u64 count; + union { + struct crypto_sha256_state ctx; + struct { + u32 state[SHA256_DIGEST_SIZE / 4]; + u64 count; + }; + }; u8 buf[SHA256_BLOCK_SIZE]; }; diff --git a/include/crypto/sha256_base.h b/include/crypto/sha256_base.h index 08cd5e41d4fd..9f284bed5a51 100644 --- a/include/crypto/sha256_base.h +++ b/include/crypto/sha256_base.h @@ -8,6 +8,7 @@ #ifndef _CRYPTO_SHA256_BASE_H #define _CRYPTO_SHA256_BASE_H +#include #include #include #include @@ -40,35 +41,10 @@ static inline int lib_sha256_base_do_update(struct sha256_state *sctx, sha256_block_fn *block_fn) { unsigned int partial = sctx->count % SHA256_BLOCK_SIZE; - struct crypto_sha256_state *state = (void *)sctx; sctx->count += len; - - if (unlikely((partial + len) >= SHA256_BLOCK_SIZE)) { - int blocks; - - if (partial) { - int p = SHA256_BLOCK_SIZE - partial; - - memcpy(sctx->buf + partial, data, p); - data += p; - len -= p; - - block_fn(state, sctx->buf, 1); - } - - blocks = len / SHA256_BLOCK_SIZE; - len %= SHA256_BLOCK_SIZE; - - if (blocks) { - block_fn(state, data, blocks); - data += blocks * SHA256_BLOCK_SIZE; - } - partial = 0; - } - if (len) - memcpy(sctx->buf + partial, data, len); - + BLOCK_HASH_UPDATE_BLOCKS(block_fn, &sctx->ctx, data, len, + SHA256_BLOCK_SIZE, sctx->buf, partial); return 0; } @@ -140,14 +116,6 @@ static inline int lib_sha256_base_do_finalize(struct sha256_state *sctx, return lib_sha256_base_do_finup(state, sctx->buf, partial, block_fn); } -static inline int sha256_base_do_finalize(struct shash_desc *desc, - sha256_block_fn *block_fn) -{ - struct sha256_state *sctx = shash_desc_ctx(desc); - - return lib_sha256_base_do_finalize(sctx, block_fn); -} - static inline int __sha256_base_finish(u32 state[SHA256_DIGEST_SIZE / 4], u8 *out, unsigned int digest_size) { From patchwork Thu Apr 24 10:47:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884075 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 145161F0E47 for ; Thu, 24 Apr 2025 10:47:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491629; cv=none; b=hxGWooKP58A3UxBXrAjygxbPpAiHLX4aa4g9vALuf5GKBlBqrW36YYAsdBe3rQugBhyoTBUR9vwG9l91edJXZNA0ZGh2pg1Lb+hAXjwbGO/yRwlhPf2zlWTcBAGgZjnFkNxBlZPlhpMZGY1NBXS4YbwMZovdTf7N/aLfF3Tz6ZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491629; c=relaxed/simple; bh=UsM0XbEusUr0lC4MllKynTLIvoGmPTFMixDGT8FDOv8=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=iuBTFe+8kwTb9buJw7tCLfAShm6zsgUO+4UTppwMwIPm0B8oPzW/SU1HE688DHQHP6oyxNvzlSYdgFNCWr1iaCAe51egDctgInXIWoVymayMU+El6UgzVUINCCEvJkOQsd3YXdIYmBwv0Bp8ZsZjBMUaYh7EUq9ywucGzguIw+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=D1edC4hQ; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="D1edC4hQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=0HlilcKT3IskbtVTsgPUPvbWYKBws35GdN7HbjtHKYc=; b=D1edC4hQwhFBhCJzlp4cFfLOZQ q26uglV3MewubVrvTwwE9OzPg4QfjgFPB2r91viEOraIQ5/Rh6WhmvGWhlyUbYB7Bbi9KOU/HONP9 FLM5NmBM+uKzQI9X+IUK4bA/d7irl88YXPjRat6ylfE7TaltAKQ6C7uVoCkwUFPsYSPB1Uxzu2sgm YGOxk3WqoNb0WTuw8hv3TPcAoUdvRqSR5jUzExMmxRCbk050ve7AYLXbdlJXsKMKqM+SfL33SXzdH GhGD89xEJjI5ZekQEou9eoBJVlzhV3N3iXea7NwWjnC7eUPHHt2cydBvDTaPfMlRth+KuxQ1KfgLI X2oJ4TcQ==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6h-000fLl-0V; Thu, 24 Apr 2025 18:47:04 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:03 +0800 Date: Thu, 24 Apr 2025 18:47:03 +0800 Message-Id: <4d5023d077557cb2e3bc1f620bc888e9c0e855e8.1745490652.git.herbert@gondor.apana.org.au> In-Reply-To: References: From: Herbert Xu Subject: [PATCH 03/15] crypto: arm/poly1305 - Add block-only interface To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Add block-only interface. Also remove the unnecessary SIMD fallback path. Signed-off-by: Herbert Xu --- arch/arm/lib/crypto/poly1305-armv4.pl | 4 +- arch/arm/lib/crypto/poly1305-glue.c | 77 +++++++++++++++------------ 2 files changed, 46 insertions(+), 35 deletions(-) diff --git a/arch/arm/lib/crypto/poly1305-armv4.pl b/arch/arm/lib/crypto/poly1305-armv4.pl index 6d79498d3115..d57c6e2fc84a 100644 --- a/arch/arm/lib/crypto/poly1305-armv4.pl +++ b/arch/arm/lib/crypto/poly1305-armv4.pl @@ -43,9 +43,9 @@ $code.=<<___; #else # define __ARM_ARCH__ __LINUX_ARM_ARCH__ # define __ARM_MAX_ARCH__ __LINUX_ARM_ARCH__ -# define poly1305_init poly1305_init_arm +# define poly1305_init poly1305_block_init_arch # define poly1305_blocks poly1305_blocks_arm -# define poly1305_emit poly1305_emit_arm +# define poly1305_emit poly1305_emit_arch .globl poly1305_blocks_neon #endif diff --git a/arch/arm/lib/crypto/poly1305-glue.c b/arch/arm/lib/crypto/poly1305-glue.c index 42d0ebde1ae1..e9082d0d9e99 100644 --- a/arch/arm/lib/crypto/poly1305-glue.c +++ b/arch/arm/lib/crypto/poly1305-glue.c @@ -7,20 +7,28 @@ #include #include -#include -#include -#include +#include #include #include +#include #include +#include #include -void poly1305_init_arm(void *state, const u8 *key); -void poly1305_blocks_arm(void *state, const u8 *src, u32 len, u32 hibit); -void poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit); -void poly1305_emit_arm(void *state, u8 *digest, const u32 *nonce); +asmlinkage void poly1305_block_init_arch(struct poly1305_block_state *state, + const u8 key[POLY1305_BLOCK_SIZE]); +EXPORT_SYMBOL_GPL(poly1305_block_init_arch); +asmlinkage void poly1305_blocks_arm(struct poly1305_block_state *state, + const u8 *src, u32 len, u32 hibit); +asmlinkage void poly1305_blocks_neon(struct poly1305_block_state *state, + const u8 *src, u32 len, u32 hibit); +asmlinkage void poly1305_emit_arch(const struct poly1305_state *state, + u8 digest[POLY1305_DIGEST_SIZE], + const u32 nonce[4]); +EXPORT_SYMBOL_GPL(poly1305_emit_arch); -void __weak poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit) +void __weak poly1305_blocks_neon(struct poly1305_block_state *state, + const u8 *src, u32 len, u32 hibit) { } @@ -28,21 +36,39 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) { - poly1305_init_arm(&dctx->h, key); dctx->s[0] = get_unaligned_le32(key + 16); dctx->s[1] = get_unaligned_le32(key + 20); dctx->s[2] = get_unaligned_le32(key + 24); dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; + poly1305_block_init_arch(&dctx->state, key); } EXPORT_SYMBOL(poly1305_init_arch); +void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, + unsigned int len, u32 padbit) +{ + len = round_down(len, POLY1305_BLOCK_SIZE); + if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && + static_branch_likely(&have_neon)) { + do { + unsigned int todo = min_t(unsigned int, len, SZ_4K); + + kernel_neon_begin(); + poly1305_blocks_neon(state, src, todo, padbit); + kernel_neon_end(); + + len -= todo; + src += todo; + } while (len); + } else + poly1305_blocks_arm(state, src, len, padbit); +} +EXPORT_SYMBOL_GPL(poly1305_blocks_arch); + void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, unsigned int nbytes) { - bool do_neon = IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && - crypto_simd_usable(); - if (unlikely(dctx->buflen)) { u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); @@ -52,30 +78,15 @@ void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, dctx->buflen += bytes; if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_arm(&dctx->h, dctx->buf, - POLY1305_BLOCK_SIZE, 1); + poly1305_blocks_arch(&dctx->state, dctx->buf, + POLY1305_BLOCK_SIZE, 1); dctx->buflen = 0; } } if (likely(nbytes >= POLY1305_BLOCK_SIZE)) { - unsigned int len = round_down(nbytes, POLY1305_BLOCK_SIZE); - - if (static_branch_likely(&have_neon) && do_neon) { - do { - unsigned int todo = min_t(unsigned int, len, SZ_4K); - - kernel_neon_begin(); - poly1305_blocks_neon(&dctx->h, src, todo, 1); - kernel_neon_end(); - - len -= todo; - src += todo; - } while (len); - } else { - poly1305_blocks_arm(&dctx->h, src, len, 1); - src += len; - } + poly1305_blocks_arch(&dctx->state, src, nbytes, 1); + src += round_down(nbytes, POLY1305_BLOCK_SIZE); nbytes %= POLY1305_BLOCK_SIZE; } @@ -92,10 +103,10 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) dctx->buf[dctx->buflen++] = 1; memset(dctx->buf + dctx->buflen, 0, POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arm(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0); + poly1305_blocks_arch(&dctx->state, dctx->buf, POLY1305_BLOCK_SIZE, 0); } - poly1305_emit_arm(&dctx->h, dst, dctx->s); + poly1305_emit_arch(&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx){}; } EXPORT_SYMBOL(poly1305_final_arch); From patchwork Thu Apr 24 10:47:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884074 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812411F0E47 for ; Thu, 24 Apr 2025 10:47:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491633; cv=none; b=bKEvcjC3vRt/a0ItIC2HuraKcQ52hgXy/Z0TGF5Yq3G9i4j38yng+LqMHmsjYVpMjAXeguvz7HH8Yfnj83TvkSDdTSqOINffZuK1FseAPl88yheiKNib5xBGibMkSkHgNrILaPeOc0zEr7vzpad762am9yEz4nPjPOJsBi9EMbI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491633; c=relaxed/simple; bh=MppoNuwbKwzAWErSnM7IVUKSs/CGSDIGBmRVSuOaJ2M=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=dLu/WBBUSJzlBY00lznfvEd5XsrAZZSsuDTMLkhQTuGegAelAb5Iwb190s/nFn+mrp1htaYT8DIlehaRbxWJf/LrLjclIuBLVMmbVU++yFoyshlj/X5hA6xqNyhsNXUyTsRu+/PtjL5sM0qDhw9ykpfqBl3qY0SF10/29eRp6LQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=YMgstW5q; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="YMgstW5q" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=iqtms7F7kkvqGAx6C7Vu9N/6a0sOPJJAEFz5Uc1UeJ0=; b=YMgstW5q3uvbKcIGQ1bc45EGbj bMK3HdSLY0X0wam4bUKca87vllmkEaAEk69olSnyfhXuViOPJF+NTJJ/YVmP40m5MeVzKxkgqzpff PX+rTXZejqOCAJuw4uij90NjQnjbXP6yxpNXjbgonxKgn6m7gMtlpWEdjgpHyRiC9v6Tx/uDCgJ3y Ts0k2ap04VFJMGrf3TN13EI9mNaVFHNy6PNQwq7N1qz2L6c7449smG06pCHpO0BhTT+AhrxJmEqEx cDgttmSwOdEpgTDLMgjcNpfTF7thaeqX+rQkVr+b2NcXPzEMqVSivDpKkm/ADdgs2XSWOrcomulIE kwqWU92w==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6l-000fM7-2L; Thu, 24 Apr 2025 18:47:08 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:07 +0800 Date: Thu, 24 Apr 2025 18:47:07 +0800 Message-Id: In-Reply-To: References: From: Herbert Xu Subject: [PATCH 05/15] crypto: mips/poly1305 - Add block-only interface To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Add block-only interface. Signed-off-by: Herbert Xu --- arch/mips/lib/crypto/poly1305-glue.c | 28 ++++++++++++++++++--------- arch/mips/lib/crypto/poly1305-mips.pl | 12 ++++++------ 2 files changed, 25 insertions(+), 15 deletions(-) diff --git a/arch/mips/lib/crypto/poly1305-glue.c b/arch/mips/lib/crypto/poly1305-glue.c index 576e7a58e0b1..fc00e96b2a5b 100644 --- a/arch/mips/lib/crypto/poly1305-glue.c +++ b/arch/mips/lib/crypto/poly1305-glue.c @@ -5,23 +5,32 @@ * Copyright (C) 2019 Linaro Ltd. */ -#include +#include #include +#include #include +#include #include -asmlinkage void poly1305_init_mips(void *state, const u8 *key); -asmlinkage void poly1305_blocks_mips(void *state, const u8 *src, u32 len, u32 hibit); -asmlinkage void poly1305_emit_mips(void *state, u8 *digest, const u32 *nonce); +asmlinkage void poly1305_block_init_arch(struct poly1305_block_state *state, + const u8 key[POLY1305_BLOCK_SIZE]); +EXPORT_SYMBOL_GPL(poly1305_block_init_arch); +asmlinkage void poly1305_blocks_arch(struct poly1305_block_state *state, + const u8 *src, u32 len, u32 hibit); +EXPORT_SYMBOL_GPL(poly1305_blocks_arch); +asmlinkage void poly1305_emit_arch(const struct poly1305_state *state, + u8 digest[POLY1305_DIGEST_SIZE], + const u32 nonce[4]); +EXPORT_SYMBOL_GPL(poly1305_emit_arch); void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) { - poly1305_init_mips(&dctx->h, key); dctx->s[0] = get_unaligned_le32(key + 16); dctx->s[1] = get_unaligned_le32(key + 20); dctx->s[2] = get_unaligned_le32(key + 24); dctx->s[3] = get_unaligned_le32(key + 28); dctx->buflen = 0; + poly1305_block_init_arch(&dctx->state, key); } EXPORT_SYMBOL(poly1305_init_arch); @@ -37,7 +46,7 @@ void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, dctx->buflen += bytes; if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_mips(&dctx->h, dctx->buf, + poly1305_blocks_arch(&dctx->state, dctx->buf, POLY1305_BLOCK_SIZE, 1); dctx->buflen = 0; } @@ -46,7 +55,7 @@ void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, if (likely(nbytes >= POLY1305_BLOCK_SIZE)) { unsigned int len = round_down(nbytes, POLY1305_BLOCK_SIZE); - poly1305_blocks_mips(&dctx->h, src, len, 1); + poly1305_blocks_arch(&dctx->state, src, len, 1); src += len; nbytes %= POLY1305_BLOCK_SIZE; } @@ -64,10 +73,11 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) dctx->buf[dctx->buflen++] = 1; memset(dctx->buf + dctx->buflen, 0, POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_mips(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0); + poly1305_blocks_arch(&dctx->state, dctx->buf, + POLY1305_BLOCK_SIZE, 0); } - poly1305_emit_mips(&dctx->h, dst, dctx->s); + poly1305_emit_arch(&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx){}; } EXPORT_SYMBOL(poly1305_final_arch); diff --git a/arch/mips/lib/crypto/poly1305-mips.pl b/arch/mips/lib/crypto/poly1305-mips.pl index b05bab884ed2..399f10c3e385 100644 --- a/arch/mips/lib/crypto/poly1305-mips.pl +++ b/arch/mips/lib/crypto/poly1305-mips.pl @@ -93,9 +93,9 @@ $code.=<<___; #endif #ifdef __KERNEL__ -# define poly1305_init poly1305_init_mips -# define poly1305_blocks poly1305_blocks_mips -# define poly1305_emit poly1305_emit_mips +# define poly1305_init poly1305_block_init_arch +# define poly1305_blocks poly1305_blocks_arch +# define poly1305_emit poly1305_emit_arch #endif #if defined(__MIPSEB__) && !defined(MIPSEB) @@ -565,9 +565,9 @@ $code.=<<___; #endif #ifdef __KERNEL__ -# define poly1305_init poly1305_init_mips -# define poly1305_blocks poly1305_blocks_mips -# define poly1305_emit poly1305_emit_mips +# define poly1305_init poly1305_block_init_arch +# define poly1305_blocks poly1305_blocks_arch +# define poly1305_emit poly1305_emit_arch #endif #if defined(__MIPSEB__) && !defined(MIPSEB) From patchwork Thu Apr 24 10:47:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884073 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52C23225A47 for ; Thu, 24 Apr 2025 10:47:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491638; cv=none; b=WeKeFCpx9fiK02BMdTWfn8ZZaKoA9swuFbomc7CB9YRrBFjusRWIAokzMb/8xmsoyXENwDjVJGapMASDZIbd7SOaeEBN41Vy6dVwvhQSSIyEh2c9/kPKgFSdQV9J+GH6MbOr2Gj7flBvHdE2mWfpWt+BBeBhbSFy8KYJFefPZaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491638; c=relaxed/simple; bh=HjFR15Ky8DLW9iaO+TCUn+GXnu9fhXrRxH53DG+tap0=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=WwWmcm0WAKarlU+Lmh8amMfpxUhIHtpMigiLA1zwu1CvyITX9DBd6tC3H3ah/NUOyYxVPfVJtYTzLbbf4Xc3AsfHH0IWUgwRorQF9JtecPUBPtDpM3MZgfxy3ZeaYJIRExuKJ4pryFgR5gt8IAN50h60M3J/9NATBo6Gnv5rXcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=ZCkpGX4D; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="ZCkpGX4D" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=15SOCZxTa5FLBXuBXRIEacLsZm8i8T81fycrn6Jneao=; b=ZCkpGX4DngKXAyzMlZvWO7KZSz UnaSs+31iQ+XLvLBV6tEanPIS/EDzQOWpeMGqyTtBTs7OlkE3xhYBQ19MBCL+f//ue6uA66eH2WGv 44uu5Ps/VobxBbPj7w6n2xJyxmENO/dVkVb+98/5zXuCyISOxlimTBXjhXZaBJrlsH4aLnpt36lfV 86c0UBA/dv9SF4iRMwp9c4geak9zR1cBkCHUjuKEmlTE2HdwQRBuoCI+f4UBEmB9Bgc0953D17A9w kBU+9odv+uqMbty7e9ixEMYLDL/p8Tj5G72ArjRvzqQ4heAZMPh2dXXS5Oq3NcP+oQ2iDvcfMMtW0 ieoa5+rw==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6q-000fMT-19; Thu, 24 Apr 2025 18:47:13 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:12 +0800 Date: Thu, 24 Apr 2025 18:47:12 +0800 Message-Id: In-Reply-To: References: From: Herbert Xu Subject: [PATCH 07/15] crypto: x86/poly1305 - Add block-only interface To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Add block-only interface. Also remove the unnecessary SIMD fallback path. Signed-off-by: Herbert Xu --- arch/x86/lib/crypto/poly1305_glue.c | 124 +++++++++++----------------- 1 file changed, 50 insertions(+), 74 deletions(-) diff --git a/arch/x86/lib/crypto/poly1305_glue.c b/arch/x86/lib/crypto/poly1305_glue.c index cff35ca5822a..b45818c51223 100644 --- a/arch/x86/lib/crypto/poly1305_glue.c +++ b/arch/x86/lib/crypto/poly1305_glue.c @@ -3,34 +3,15 @@ * Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. */ -#include -#include +#include +#include +#include #include #include #include #include +#include #include -#include -#include - -asmlinkage void poly1305_init_x86_64(void *ctx, - const u8 key[POLY1305_BLOCK_SIZE]); -asmlinkage void poly1305_blocks_x86_64(void *ctx, const u8 *inp, - const size_t len, const u32 padbit); -asmlinkage void poly1305_emit_x86_64(void *ctx, u8 mac[POLY1305_DIGEST_SIZE], - const u32 nonce[4]); -asmlinkage void poly1305_emit_avx(void *ctx, u8 mac[POLY1305_DIGEST_SIZE], - const u32 nonce[4]); -asmlinkage void poly1305_blocks_avx(void *ctx, const u8 *inp, const size_t len, - const u32 padbit); -asmlinkage void poly1305_blocks_avx2(void *ctx, const u8 *inp, const size_t len, - const u32 padbit); -asmlinkage void poly1305_blocks_avx512(void *ctx, const u8 *inp, - const size_t len, const u32 padbit); - -static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx); -static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx2); -static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx512); struct poly1305_arch_internal { union { @@ -45,64 +26,55 @@ struct poly1305_arch_internal { struct { u32 r2, r1, r4, r3; } rn[9]; }; -/* The AVX code uses base 2^26, while the scalar code uses base 2^64. If we hit - * the unfortunate situation of using AVX and then having to go back to scalar - * -- because the user is silly and has called the update function from two - * separate contexts -- then we need to convert back to the original base before - * proceeding. It is possible to reason that the initial reduction below is - * sufficient given the implementation invariants. However, for an avoidance of - * doubt and because this is not performance critical, we do the full reduction - * anyway. Z3 proof of below function: https://xn--4db.cc/ltPtHCKN/py - */ -static void convert_to_base2_64(void *ctx) +asmlinkage void poly1305_init_x86_64(struct poly1305_block_state *state, + const u8 key[POLY1305_BLOCK_SIZE]); +asmlinkage void poly1305_blocks_x86_64(struct poly1305_arch_internal *ctx, + const u8 *inp, + const size_t len, const u32 padbit); +asmlinkage void poly1305_emit_x86_64(const struct poly1305_state *ctx, + u8 mac[POLY1305_DIGEST_SIZE], + const u32 nonce[4]); +asmlinkage void poly1305_emit_avx(const struct poly1305_state *ctx, + u8 mac[POLY1305_DIGEST_SIZE], + const u32 nonce[4]); +asmlinkage void poly1305_blocks_avx(struct poly1305_arch_internal *ctx, + const u8 *inp, const size_t len, + const u32 padbit); +asmlinkage void poly1305_blocks_avx2(struct poly1305_arch_internal *ctx, + const u8 *inp, const size_t len, + const u32 padbit); +asmlinkage void poly1305_blocks_avx512(struct poly1305_arch_internal *ctx, + const u8 *inp, + const size_t len, const u32 padbit); + +static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx2); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(poly1305_use_avx512); + +void poly1305_block_init_arch(struct poly1305_block_state *state, + const u8 key[POLY1305_BLOCK_SIZE]) { - struct poly1305_arch_internal *state = ctx; - u32 cy; - - if (!state->is_base2_26) - return; - - cy = state->h[0] >> 26; state->h[0] &= 0x3ffffff; state->h[1] += cy; - cy = state->h[1] >> 26; state->h[1] &= 0x3ffffff; state->h[2] += cy; - cy = state->h[2] >> 26; state->h[2] &= 0x3ffffff; state->h[3] += cy; - cy = state->h[3] >> 26; state->h[3] &= 0x3ffffff; state->h[4] += cy; - state->hs[0] = ((u64)state->h[2] << 52) | ((u64)state->h[1] << 26) | state->h[0]; - state->hs[1] = ((u64)state->h[4] << 40) | ((u64)state->h[3] << 14) | (state->h[2] >> 12); - state->hs[2] = state->h[4] >> 24; -#define ULT(a, b) ((a ^ ((a ^ b) | ((a - b) ^ b))) >> (sizeof(a) * 8 - 1)) - cy = (state->hs[2] >> 2) + (state->hs[2] & ~3ULL); - state->hs[2] &= 3; - state->hs[0] += cy; - state->hs[1] += (cy = ULT(state->hs[0], cy)); - state->hs[2] += ULT(state->hs[1], cy); -#undef ULT - state->is_base2_26 = 0; + poly1305_init_x86_64(state, key); } +EXPORT_SYMBOL_GPL(poly1305_block_init_arch); -static void poly1305_simd_init(void *ctx, const u8 key[POLY1305_BLOCK_SIZE]) +void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *inp, + unsigned int len, u32 padbit) { - poly1305_init_x86_64(ctx, key); -} - -static void poly1305_simd_blocks(void *ctx, const u8 *inp, size_t len, - const u32 padbit) -{ - struct poly1305_arch_internal *state = ctx; + struct poly1305_arch_internal *ctx = + container_of(&state->h.h, struct poly1305_arch_internal, h); /* SIMD disables preemption, so relax after processing each page. */ BUILD_BUG_ON(SZ_4K < POLY1305_BLOCK_SIZE || SZ_4K % POLY1305_BLOCK_SIZE); - if (!static_branch_likely(&poly1305_use_avx) || - (len < (POLY1305_BLOCK_SIZE * 18) && !state->is_base2_26) || - !crypto_simd_usable()) { - convert_to_base2_64(ctx); + if (!static_branch_likely(&poly1305_use_avx)) { poly1305_blocks_x86_64(ctx, inp, len, padbit); return; } do { - const size_t bytes = min_t(size_t, len, SZ_4K); + const unsigned int bytes = min(len, SZ_4K); kernel_fpu_begin(); if (static_branch_likely(&poly1305_use_avx512)) @@ -117,24 +89,26 @@ static void poly1305_simd_blocks(void *ctx, const u8 *inp, size_t len, inp += bytes; } while (len); } +EXPORT_SYMBOL_GPL(poly1305_blocks_arch); -static void poly1305_simd_emit(void *ctx, u8 mac[POLY1305_DIGEST_SIZE], - const u32 nonce[4]) +void poly1305_emit_arch(const struct poly1305_state *ctx, + u8 mac[POLY1305_DIGEST_SIZE], const u32 nonce[4]) { if (!static_branch_likely(&poly1305_use_avx)) poly1305_emit_x86_64(ctx, mac, nonce); else poly1305_emit_avx(ctx, mac, nonce); } +EXPORT_SYMBOL_GPL(poly1305_emit_arch); void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) { - poly1305_simd_init(&dctx->h, key); dctx->s[0] = get_unaligned_le32(&key[16]); dctx->s[1] = get_unaligned_le32(&key[20]); dctx->s[2] = get_unaligned_le32(&key[24]); dctx->s[3] = get_unaligned_le32(&key[28]); dctx->buflen = 0; + poly1305_block_init_arch(&dctx->state, key); } EXPORT_SYMBOL(poly1305_init_arch); @@ -151,14 +125,15 @@ void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, dctx->buflen += bytes; if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_simd_blocks(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 1); + poly1305_blocks_arch(&dctx->state, dctx->buf, + POLY1305_BLOCK_SIZE, 1); dctx->buflen = 0; } } if (likely(srclen >= POLY1305_BLOCK_SIZE)) { bytes = round_down(srclen, POLY1305_BLOCK_SIZE); - poly1305_simd_blocks(&dctx->h, src, bytes, 1); + poly1305_blocks_arch(&dctx->state, src, bytes, 1); src += bytes; srclen -= bytes; } @@ -176,10 +151,11 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) dctx->buf[dctx->buflen++] = 1; memset(dctx->buf + dctx->buflen, 0, POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_simd_blocks(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0); + poly1305_blocks_arch(&dctx->state, dctx->buf, + POLY1305_BLOCK_SIZE, 0); } - poly1305_simd_emit(&dctx->h, dst, dctx->s); + poly1305_emit_arch(&dctx->h, dst, dctx->s); memzero_explicit(dctx, sizeof(*dctx)); } EXPORT_SYMBOL(poly1305_final_arch); From patchwork Thu Apr 24 10:47:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884072 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76A56225A47 for ; Thu, 24 Apr 2025 10:47:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491643; cv=none; b=alFqdSDPhlcsTiWfN9t4+KrloI+POf+8dr7d7kgquUreUcjwvOTqjmAEfY393pvT0SRNlvPnA1AVRfPg5rGAK+m02L27zMl07Rmk1IwXeMweBX5WMy8ObpQ9zC+2WBUJcpB5J7EJgLgXa7chIF9d375Uj+B1fhto+L5Ui/6uRYc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491643; c=relaxed/simple; bh=UsMZBCD5SaCF48KNqiXQ031n4pNhyPVGFmDcLfoshwI=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=EPg91Xnnvq61yDJrINjYTPv4vKZivUjADfWMEH+ypH4do+J7csuVkTieNYT1ZAUDCdNiDrEjf+bxLPCqp2mYHg46rVKhfCHDumdVK3WozWxmLqHatZ8YmkOBAUF4wRdt3lz1annNV4FMlR3BwrsSveH+lFmAMK8INvKCTFT5O0c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=hXZPtSm1; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="hXZPtSm1" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=jU4PS2dP4ny+hkgHM9GdrByIvEyYgtPJ80/WwT6ZJ8Y=; b=hXZPtSm1tEPXcJV97ay8d3FP0f ptn7QkAnDYINHZRAF5ER5YdGPafvCd7sUSekz4Gxxo/tgPWs2I/NtHQSlJYyKB0uBLazh5CgauG5V 57LqB2sd7x7Y6wE6Yp1eCWqWqXqdQpl0zr4szALsnrMkkc7NykC9knry7VmIaH7hlq1jdmzXv5TNJ tbhcwB6r4yTJkjnlMg/abhQGlazkD+FOUi/79sMPkAXDvrSRZ7kFEzxDyJSxKC0iHjHcm8ur4ylXV j8yfd9DVgZaQ/bmo0m6aScMGjp5XvLGOz6VZUd/aI2qIPiW3U/rvXvJKxfMi3sTKBe7eQmC6eOGG0 feVtQcag==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6v-000fMp-00; Thu, 24 Apr 2025 18:47:18 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:16 +0800 Date: Thu, 24 Apr 2025 18:47:16 +0800 Message-Id: <6c08ba96e4cb6a6219e06bb77006cba91e6e84a2.1745490652.git.herbert@gondor.apana.org.au> In-Reply-To: References: From: Herbert Xu Subject: [PATCH 09/15] crypto: lib/poly1305 - Use block-only interface To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Now that every architecture provides a block function, use that to implement the lib/poly1305 and remove the old per-arch code. Signed-off-by: Herbert Xu --- arch/arm/lib/crypto/poly1305-glue.c | 57 ------------------- arch/arm64/lib/crypto/poly1305-glue.c | 58 ------------------- arch/mips/lib/crypto/poly1305-glue.c | 60 -------------------- arch/powerpc/lib/crypto/poly1305-p10-glue.c | 63 --------------------- arch/x86/lib/crypto/poly1305_glue.c | 60 -------------------- include/crypto/poly1305.h | 53 ++--------------- lib/crypto/poly1305.c | 42 +++++++++----- 7 files changed, 33 insertions(+), 360 deletions(-) diff --git a/arch/arm/lib/crypto/poly1305-glue.c b/arch/arm/lib/crypto/poly1305-glue.c index e9082d0d9e99..aa7402523f41 100644 --- a/arch/arm/lib/crypto/poly1305-glue.c +++ b/arch/arm/lib/crypto/poly1305-glue.c @@ -12,7 +12,6 @@ #include #include #include -#include #include asmlinkage void poly1305_block_init_arch(struct poly1305_block_state *state, @@ -34,17 +33,6 @@ void __weak poly1305_blocks_neon(struct poly1305_block_state *state, static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); -void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) -{ - dctx->s[0] = get_unaligned_le32(key + 16); - dctx->s[1] = get_unaligned_le32(key + 20); - dctx->s[2] = get_unaligned_le32(key + 24); - dctx->s[3] = get_unaligned_le32(key + 28); - dctx->buflen = 0; - poly1305_block_init_arch(&dctx->state, key); -} -EXPORT_SYMBOL(poly1305_init_arch); - void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, unsigned int len, u32 padbit) { @@ -66,51 +54,6 @@ void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, } EXPORT_SYMBOL_GPL(poly1305_blocks_arch); -void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, - unsigned int nbytes) -{ - if (unlikely(dctx->buflen)) { - u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - nbytes -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - } - - if (likely(nbytes >= POLY1305_BLOCK_SIZE)) { - poly1305_blocks_arch(&dctx->state, src, nbytes, 1); - src += round_down(nbytes, POLY1305_BLOCK_SIZE); - nbytes %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(nbytes)) { - dctx->buflen = nbytes; - memcpy(dctx->buf, src, nbytes); - } -} -EXPORT_SYMBOL(poly1305_update_arch); - -void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) -{ - if (unlikely(dctx->buflen)) { - dctx->buf[dctx->buflen++] = 1; - memset(dctx->buf + dctx->buflen, 0, - POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arch(&dctx->state, dctx->buf, POLY1305_BLOCK_SIZE, 0); - } - - poly1305_emit_arch(&dctx->h, dst, dctx->s); - *dctx = (struct poly1305_desc_ctx){}; -} -EXPORT_SYMBOL(poly1305_final_arch); - bool poly1305_is_arch_optimized(void) { /* We always can use at least the ARM scalar implementation. */ diff --git a/arch/arm64/lib/crypto/poly1305-glue.c b/arch/arm64/lib/crypto/poly1305-glue.c index 9fdb5bd3dbb0..ab7b120bcc81 100644 --- a/arch/arm64/lib/crypto/poly1305-glue.c +++ b/arch/arm64/lib/crypto/poly1305-glue.c @@ -12,7 +12,6 @@ #include #include #include -#include #include asmlinkage void poly1305_block_init_arch(struct poly1305_block_state *state, @@ -29,17 +28,6 @@ EXPORT_SYMBOL_GPL(poly1305_emit_arch); static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); -void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) -{ - dctx->s[0] = get_unaligned_le32(key + 16); - dctx->s[1] = get_unaligned_le32(key + 20); - dctx->s[2] = get_unaligned_le32(key + 24); - dctx->s[3] = get_unaligned_le32(key + 28); - dctx->buflen = 0; - poly1305_block_init_arch(&dctx->state, key); -} -EXPORT_SYMBOL(poly1305_init_arch); - void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, unsigned int len, u32 padbit) { @@ -60,52 +48,6 @@ void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, } EXPORT_SYMBOL_GPL(poly1305_blocks_arch); -void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, - unsigned int nbytes) -{ - if (unlikely(dctx->buflen)) { - u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - nbytes -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - } - - if (likely(nbytes >= POLY1305_BLOCK_SIZE)) { - poly1305_blocks_arch(&dctx->state, src, nbytes, 1); - src += round_down(nbytes, POLY1305_BLOCK_SIZE); - nbytes %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(nbytes)) { - dctx->buflen = nbytes; - memcpy(dctx->buf, src, nbytes); - } -} -EXPORT_SYMBOL(poly1305_update_arch); - -void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) -{ - if (unlikely(dctx->buflen)) { - dctx->buf[dctx->buflen++] = 1; - memset(dctx->buf + dctx->buflen, 0, - POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 0); - } - - poly1305_emit_arch(&dctx->h, dst, dctx->s); - memzero_explicit(dctx, sizeof(*dctx)); -} -EXPORT_SYMBOL(poly1305_final_arch); - bool poly1305_is_arch_optimized(void) { /* We always can use at least the ARM64 scalar implementation. */ diff --git a/arch/mips/lib/crypto/poly1305-glue.c b/arch/mips/lib/crypto/poly1305-glue.c index fc00e96b2a5b..4550abe587b3 100644 --- a/arch/mips/lib/crypto/poly1305-glue.c +++ b/arch/mips/lib/crypto/poly1305-glue.c @@ -9,7 +9,6 @@ #include #include #include -#include #include asmlinkage void poly1305_block_init_arch(struct poly1305_block_state *state, @@ -23,65 +22,6 @@ asmlinkage void poly1305_emit_arch(const struct poly1305_state *state, const u32 nonce[4]); EXPORT_SYMBOL_GPL(poly1305_emit_arch); -void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) -{ - dctx->s[0] = get_unaligned_le32(key + 16); - dctx->s[1] = get_unaligned_le32(key + 20); - dctx->s[2] = get_unaligned_le32(key + 24); - dctx->s[3] = get_unaligned_le32(key + 28); - dctx->buflen = 0; - poly1305_block_init_arch(&dctx->state, key); -} -EXPORT_SYMBOL(poly1305_init_arch); - -void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, - unsigned int nbytes) -{ - if (unlikely(dctx->buflen)) { - u32 bytes = min(nbytes, POLY1305_BLOCK_SIZE - dctx->buflen); - - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - nbytes -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - } - - if (likely(nbytes >= POLY1305_BLOCK_SIZE)) { - unsigned int len = round_down(nbytes, POLY1305_BLOCK_SIZE); - - poly1305_blocks_arch(&dctx->state, src, len, 1); - src += len; - nbytes %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(nbytes)) { - dctx->buflen = nbytes; - memcpy(dctx->buf, src, nbytes); - } -} -EXPORT_SYMBOL(poly1305_update_arch); - -void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) -{ - if (unlikely(dctx->buflen)) { - dctx->buf[dctx->buflen++] = 1; - memset(dctx->buf + dctx->buflen, 0, - POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 0); - } - - poly1305_emit_arch(&dctx->h, dst, dctx->s); - *dctx = (struct poly1305_desc_ctx){}; -} -EXPORT_SYMBOL(poly1305_final_arch); - bool poly1305_is_arch_optimized(void) { return true; diff --git a/arch/powerpc/lib/crypto/poly1305-p10-glue.c b/arch/powerpc/lib/crypto/poly1305-p10-glue.c index a33c61efd360..cef56a0e4c12 100644 --- a/arch/powerpc/lib/crypto/poly1305-p10-glue.c +++ b/arch/powerpc/lib/crypto/poly1305-p10-glue.c @@ -10,7 +10,6 @@ #include #include #include -#include #include asmlinkage void poly1305_p10le_4blocks(struct poly1305_block_state *state, const u8 *m, u32 mlen); @@ -45,17 +44,6 @@ void poly1305_block_init_arch(struct poly1305_block_state *dctx, } EXPORT_SYMBOL_GPL(poly1305_block_init_arch); -void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) -{ - dctx->s[0] = get_unaligned_le32(key + 16); - dctx->s[1] = get_unaligned_le32(key + 20); - dctx->s[2] = get_unaligned_le32(key + 24); - dctx->s[3] = get_unaligned_le32(key + 28); - dctx->buflen = 0; - poly1305_block_init_arch(&dctx->state, key); -} -EXPORT_SYMBOL(poly1305_init_arch); - void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, unsigned int len, u32 padbit) { @@ -76,57 +64,6 @@ void poly1305_blocks_arch(struct poly1305_block_state *state, const u8 *src, } EXPORT_SYMBOL_GPL(poly1305_blocks_arch); -void poly1305_update_arch(struct poly1305_desc_ctx *dctx, - const u8 *src, unsigned int srclen) -{ - unsigned int bytes; - - if (!static_key_enabled(&have_p10)) - return poly1305_update_generic(dctx, src, srclen); - - if (unlikely(dctx->buflen)) { - bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - srclen -= bytes; - dctx->buflen += bytes; - if (dctx->buflen < POLY1305_BLOCK_SIZE) - return; - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - - if (likely(srclen >= POLY1305_BLOCK_SIZE)) { - poly1305_blocks_arch(&dctx->state, src, srclen, 1); - src += srclen - (srclen % POLY1305_BLOCK_SIZE); - srclen %= POLY1305_BLOCK_SIZE; - } - - if (unlikely(srclen)) { - dctx->buflen = srclen; - memcpy(dctx->buf, src, srclen); - } -} -EXPORT_SYMBOL(poly1305_update_arch); - -void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) -{ - if (!static_key_enabled(&have_p10)) - return poly1305_final_generic(dctx, dst); - - if (dctx->buflen) { - dctx->buf[dctx->buflen++] = 1; - memset(dctx->buf + dctx->buflen, 0, - POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 0); - } - - poly1305_emit_arch(&dctx->h, dst, dctx->s); -} -EXPORT_SYMBOL(poly1305_final_arch); - bool poly1305_is_arch_optimized(void) { return static_key_enabled(&have_p10); diff --git a/arch/x86/lib/crypto/poly1305_glue.c b/arch/x86/lib/crypto/poly1305_glue.c index b45818c51223..0881b3e73388 100644 --- a/arch/x86/lib/crypto/poly1305_glue.c +++ b/arch/x86/lib/crypto/poly1305_glue.c @@ -10,7 +10,6 @@ #include #include #include -#include #include struct poly1305_arch_internal { @@ -101,65 +100,6 @@ void poly1305_emit_arch(const struct poly1305_state *ctx, } EXPORT_SYMBOL_GPL(poly1305_emit_arch); -void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 key[POLY1305_KEY_SIZE]) -{ - dctx->s[0] = get_unaligned_le32(&key[16]); - dctx->s[1] = get_unaligned_le32(&key[20]); - dctx->s[2] = get_unaligned_le32(&key[24]); - dctx->s[3] = get_unaligned_le32(&key[28]); - dctx->buflen = 0; - poly1305_block_init_arch(&dctx->state, key); -} -EXPORT_SYMBOL(poly1305_init_arch); - -void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, - unsigned int srclen) -{ - unsigned int bytes; - - if (unlikely(dctx->buflen)) { - bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); - memcpy(dctx->buf + dctx->buflen, src, bytes); - src += bytes; - srclen -= bytes; - dctx->buflen += bytes; - - if (dctx->buflen == POLY1305_BLOCK_SIZE) { - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 1); - dctx->buflen = 0; - } - } - - if (likely(srclen >= POLY1305_BLOCK_SIZE)) { - bytes = round_down(srclen, POLY1305_BLOCK_SIZE); - poly1305_blocks_arch(&dctx->state, src, bytes, 1); - src += bytes; - srclen -= bytes; - } - - if (unlikely(srclen)) { - dctx->buflen = srclen; - memcpy(dctx->buf, src, srclen); - } -} -EXPORT_SYMBOL(poly1305_update_arch); - -void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst) -{ - if (unlikely(dctx->buflen)) { - dctx->buf[dctx->buflen++] = 1; - memset(dctx->buf + dctx->buflen, 0, - POLY1305_BLOCK_SIZE - dctx->buflen); - poly1305_blocks_arch(&dctx->state, dctx->buf, - POLY1305_BLOCK_SIZE, 0); - } - - poly1305_emit_arch(&dctx->h, dst, dctx->s); - memzero_explicit(dctx, sizeof(*dctx)); -} -EXPORT_SYMBOL(poly1305_final_arch); - bool poly1305_is_arch_optimized(void) { return static_key_enabled(&poly1305_use_avx); diff --git a/include/crypto/poly1305.h b/include/crypto/poly1305.h index 027d74842cd5..e54abda8cfe9 100644 --- a/include/crypto/poly1305.h +++ b/include/crypto/poly1305.h @@ -55,55 +55,14 @@ struct poly1305_desc_ctx { unsigned int buflen; /* finalize key */ u32 s[4]; - union { - struct { - struct poly1305_state h; - union { - struct poly1305_key opaque_r[CONFIG_CRYPTO_LIB_POLY1305_RSIZE]; - struct poly1305_core_key core_r; - }; - }; - struct poly1305_block_state state; - }; + struct poly1305_block_state state; }; -void poly1305_init_arch(struct poly1305_desc_ctx *desc, - const u8 key[POLY1305_KEY_SIZE]); -void poly1305_init_generic(struct poly1305_desc_ctx *desc, - const u8 key[POLY1305_KEY_SIZE]); - -static inline void poly1305_init(struct poly1305_desc_ctx *desc, const u8 *key) -{ - if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) - poly1305_init_arch(desc, key); - else - poly1305_init_generic(desc, key); -} - -void poly1305_update_arch(struct poly1305_desc_ctx *desc, const u8 *src, - unsigned int nbytes); -void poly1305_update_generic(struct poly1305_desc_ctx *desc, const u8 *src, - unsigned int nbytes); - -static inline void poly1305_update(struct poly1305_desc_ctx *desc, - const u8 *src, unsigned int nbytes) -{ - if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) - poly1305_update_arch(desc, src, nbytes); - else - poly1305_update_generic(desc, src, nbytes); -} - -void poly1305_final_arch(struct poly1305_desc_ctx *desc, u8 *digest); -void poly1305_final_generic(struct poly1305_desc_ctx *desc, u8 *digest); - -static inline void poly1305_final(struct poly1305_desc_ctx *desc, u8 *digest) -{ - if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) - poly1305_final_arch(desc, digest); - else - poly1305_final_generic(desc, digest); -} +void poly1305_init(struct poly1305_desc_ctx *desc, + const u8 key[POLY1305_KEY_SIZE]); +void poly1305_update(struct poly1305_desc_ctx *desc, + const u8 *src, unsigned int nbytes); +void poly1305_final(struct poly1305_desc_ctx *desc, u8 *digest); #if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305) bool poly1305_is_arch_optimized(void); diff --git a/lib/crypto/poly1305.c b/lib/crypto/poly1305.c index ebdfccf378ee..a37b424ee84b 100644 --- a/lib/crypto/poly1305.c +++ b/lib/crypto/poly1305.c @@ -22,47 +22,59 @@ void poly1305_block_init_generic(struct poly1305_block_state *desc, } EXPORT_SYMBOL_GPL(poly1305_block_init_generic); -void poly1305_init_generic(struct poly1305_desc_ctx *desc, - const u8 key[POLY1305_KEY_SIZE]) +void poly1305_init(struct poly1305_desc_ctx *desc, + const u8 key[POLY1305_KEY_SIZE]) { desc->s[0] = get_unaligned_le32(key + 16); desc->s[1] = get_unaligned_le32(key + 20); desc->s[2] = get_unaligned_le32(key + 24); desc->s[3] = get_unaligned_le32(key + 28); desc->buflen = 0; - poly1305_block_init_generic(&desc->state, key); + if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) + poly1305_block_init_arch(&desc->state, key); + else + poly1305_block_init_generic(&desc->state, key); } -EXPORT_SYMBOL_GPL(poly1305_init_generic); +EXPORT_SYMBOL(poly1305_init); -static inline void poly1305_block(struct poly1305_block_state *state, const u8 *src, - unsigned int len) +static inline void poly1305_block(struct poly1305_block_state *state, + const u8 *src, unsigned int len) { - poly1305_blocks_generic(state, src, len, 1); + if (!IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) + poly1305_blocks_generic(state, src, len, 1); + poly1305_blocks_arch(state, src, len, 1); } -void poly1305_update_generic(struct poly1305_desc_ctx *desc, const u8 *src, - unsigned int nbytes) +void poly1305_update(struct poly1305_desc_ctx *desc, + const u8 *src, unsigned int nbytes) { desc->buflen = BLOCK_HASH_UPDATE(&poly1305_block, &desc->state, src, nbytes, POLY1305_BLOCK_SIZE, desc->buf, desc->buflen); } -EXPORT_SYMBOL_GPL(poly1305_update_generic); +EXPORT_SYMBOL(poly1305_update); -void poly1305_final_generic(struct poly1305_desc_ctx *desc, u8 *dst) +void poly1305_final(struct poly1305_desc_ctx *desc, u8 *dst) { if (unlikely(desc->buflen)) { desc->buf[desc->buflen++] = 1; memset(desc->buf + desc->buflen, 0, POLY1305_BLOCK_SIZE - desc->buflen); - poly1305_blocks_generic(&desc->state, desc->buf, - POLY1305_BLOCK_SIZE, 0); + if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) + poly1305_blocks_arch(&desc->state, desc->buf, + POLY1305_BLOCK_SIZE, 0); + else + poly1305_blocks_generic(&desc->state, desc->buf, + POLY1305_BLOCK_SIZE, 0); } - poly1305_emit_generic(&desc->h, dst, desc->s); + if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305)) + poly1305_emit_arch(&desc->state.h, dst, desc->s); + else + poly1305_emit_generic(&desc->state.h, dst, desc->s); *desc = (struct poly1305_desc_ctx){}; } -EXPORT_SYMBOL_GPL(poly1305_final_generic); +EXPORT_SYMBOL(poly1305_final); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Martin Willi "); From patchwork Thu Apr 24 10:47:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884071 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C763223DC6 for ; Thu, 24 Apr 2025 10:47:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491647; cv=none; b=ViLSJkI4lr0iu78+CkzDY1cng+oRCjwCK3vfN1W1F2sQPnXnTmKFBeMEU+0bn0ZWAM8aGvzuG/L5t80wRY1fh/82TAqyw4WeO/9X5+StmBW12MCjS8LqIDAq4mA/xAavzSoW0kEe+xqWxXx5vzKFnrIDE9wN/Q9zZNKvMtwQz8o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491647; c=relaxed/simple; bh=CBRU/eGA6+8aaYxf0VrvEFjY5A+UA/Fgt8K/oBY8J2s=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=U8JJJ8DqZqZUXYdFf6K3LtW1YL2Rq8vmQWzl4tBu7Ue5NxhaWSG9dPi0WrknB6kcUPerK/Ba9Ye5XOFO3Xauq78fHNa85Xm2TajlTUVUq8K4q4MCqQHXWC+VxCWvSK1WfmvjOTGi7mDNRqHh9P5A1imtWNC4CY9XBvrmH0ZiBeI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=WsXXVa4o; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="WsXXVa4o" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=c6GpcTkVft1TAZ051ptj2wdqaG8QtNQc6h/yaPSV8Dw=; b=WsXXVa4oTgdyU1qTwujYJWwm7Q E9ox/uMaLYS0qxq+ud9lDr3LlsFEh7EIZ92d+i4MUjqSwWIsaPTLXE1Wg877Z47o/iOl8DSD9RLgY uE6vVW6dfAcI/L7jRPqoapYdIQuQH+c5PZAoMXlHlJNG14cx2//LFqmkCmPNwmtE3K7uoSSemhoaD cg570t/DfwKSlnLesAND6759y/hdHHo/p5Uhs7lB5mVm6xZj1BLp3rr1tPYe2ZFUKx912UDA4M6yM S0HIjbK3WIDeEMSXyql5juFfuENERpE5ZL3F+t/VG8p2zmqgMfplcy5qEpiXWOZMRBdJ/Oj6v4PMx lQ/vi6lw==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u6z-000fNZ-1v; Thu, 24 Apr 2025 18:47:22 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:21 +0800 Date: Thu, 24 Apr 2025 18:47:21 +0800 Message-Id: <82fa7de99e3cff9935673912f0f1b774e210c7de.1745490652.git.herbert@gondor.apana.org.au> In-Reply-To: References: From: Herbert Xu Subject: [PATCH 11/15] crypto: testmgr/poly1305 - Use setkey on poly1305 To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Use the new setkey interface for poly1305 instead of supplying the key via the first two blocks. Signed-off-by: Herbert Xu --- crypto/testmgr.h | 112 ++++++++++++++++++++++++++--------------------- 1 file changed, 62 insertions(+), 50 deletions(-) diff --git a/crypto/testmgr.h b/crypto/testmgr.h index afc10af59b0a..09db05b90b5c 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -8842,6 +8842,11 @@ static const struct hash_testvec hmac_sha3_512_tv_template[] = { static const struct hash_testvec poly1305_tv_template[] = { { /* Test Vector #1 */ + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, .plaintext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -8849,20 +8854,17 @@ static const struct hash_testvec poly1305_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .psize = 96, + .psize = 64, .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #2 */ - .plaintext = "\x00\x00\x00\x00\x00\x00\x00\x00" + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x36\xe5\xf6\xb5\xc5\xe0\x60\x70" - "\xf0\xef\xca\x96\x22\x7a\x86\x3e" - "\x41\x6e\x79\x20\x73\x75\x62\x6d" + "\xf0\xef\xca\x96\x22\x7a\x86\x3e", + .ksize = 32, + .plaintext = "\x41\x6e\x79\x20\x73\x75\x62\x6d" "\x69\x73\x73\x69\x6f\x6e\x20\x74" "\x6f\x20\x74\x68\x65\x20\x49\x45" "\x54\x46\x20\x69\x6e\x74\x65\x6e" @@ -8909,15 +8911,16 @@ static const struct hash_testvec poly1305_tv_template[] = { "\x20\x77\x68\x69\x63\x68\x20\x61" "\x72\x65\x20\x61\x64\x64\x72\x65" "\x73\x73\x65\x64\x20\x74\x6f", - .psize = 407, + .psize = 375, .digest = "\x36\xe5\xf6\xb5\xc5\xe0\x60\x70" "\xf0\xef\xca\x96\x22\x7a\x86\x3e", }, { /* Test Vector #3 */ - .plaintext = "\x36\xe5\xf6\xb5\xc5\xe0\x60\x70" + .key = "\x36\xe5\xf6\xb5\xc5\xe0\x60\x70" "\xf0\xef\xca\x96\x22\x7a\x86\x3e" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x41\x6e\x79\x20\x73\x75\x62\x6d" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\x41\x6e\x79\x20\x73\x75\x62\x6d" "\x69\x73\x73\x69\x6f\x6e\x20\x74" "\x6f\x20\x74\x68\x65\x20\x49\x45" "\x54\x46\x20\x69\x6e\x74\x65\x6e" @@ -8964,15 +8967,16 @@ static const struct hash_testvec poly1305_tv_template[] = { "\x20\x77\x68\x69\x63\x68\x20\x61" "\x72\x65\x20\x61\x64\x64\x72\x65" "\x73\x73\x65\x64\x20\x74\x6f", - .psize = 407, + .psize = 375, .digest = "\xf3\x47\x7e\x7c\xd9\x54\x17\xaf" "\x89\xa6\xb8\x79\x4c\x31\x0c\xf0", }, { /* Test Vector #4 */ - .plaintext = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" + .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" "\x47\x39\x17\xc1\x40\x2b\x80\x09" - "\x9d\xca\x5c\xbc\x20\x70\x75\xc0" - "\x27\x54\x77\x61\x73\x20\x62\x72" + "\x9d\xca\x5c\xbc\x20\x70\x75\xc0", + .ksize = 32, + .plaintext = "\x27\x54\x77\x61\x73\x20\x62\x72" "\x69\x6c\x6c\x69\x67\x2c\x20\x61" "\x6e\x64\x20\x74\x68\x65\x20\x73" "\x6c\x69\x74\x68\x79\x20\x74\x6f" @@ -8988,73 +8992,79 @@ static const struct hash_testvec poly1305_tv_template[] = { "\x68\x65\x20\x6d\x6f\x6d\x65\x20" "\x72\x61\x74\x68\x73\x20\x6f\x75" "\x74\x67\x72\x61\x62\x65\x2e", - .psize = 159, + .psize = 127, .digest = "\x45\x41\x66\x9a\x7e\xaa\xee\x61" "\xe7\x08\xdc\x7c\xbc\xc5\xeb\x62", }, { /* Test Vector #5 */ - .plaintext = "\x02\x00\x00\x00\x00\x00\x00\x00" + .key = "\x02\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xff\xff\xff\xff\xff\xff\xff\xff" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff", - .psize = 48, + .psize = 16, .digest = "\x03\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #6 */ - .plaintext = "\x02\x00\x00\x00\x00\x00\x00\x00" + .key = "\x02\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\x02\x00\x00\x00\x00\x00\x00\x00" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .ksize = 32, + .plaintext = "\x02\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .psize = 48, + .psize = 16, .digest = "\x03\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #7 */ - .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + .key = "\x01\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xff\xff\xff\xff\xff\xff\xff\xff" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" "\xf0\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" "\x11\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .psize = 80, + .psize = 48, .digest = "\x05\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #8 */ - .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + .key = "\x01\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xff\xff\xff\xff\xff\xff\xff\xff" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" "\xfb\xfe\xfe\xfe\xfe\xfe\xfe\xfe" "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" "\x01\x01\x01\x01\x01\x01\x01\x01" "\x01\x01\x01\x01\x01\x01\x01\x01", - .psize = 80, + .psize = 48, .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #9 */ - .plaintext = "\x02\x00\x00\x00\x00\x00\x00\x00" + .key = "\x02\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xfd\xff\xff\xff\xff\xff\xff\xff" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xfd\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff", - .psize = 48, + .psize = 16, .digest = "\xfa\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff", }, { /* Test Vector #10 */ - .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + .key = "\x01\x00\x00\x00\x00\x00\x00\x00" "\x04\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xe3\x35\x94\xd7\x50\x5e\x43\xb9" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xe3\x35\x94\xd7\x50\x5e\x43\xb9" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x33\x94\xd7\x50\x5e\x43\x79\xcd" "\x01\x00\x00\x00\x00\x00\x00\x00" @@ -9062,24 +9072,30 @@ static const struct hash_testvec poly1305_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x01\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .psize = 96, + .psize = 64, .digest = "\x14\x00\x00\x00\x00\x00\x00\x00" "\x55\x00\x00\x00\x00\x00\x00\x00", }, { /* Test Vector #11 */ - .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + .key = "\x01\x00\x00\x00\x00\x00\x00\x00" "\x04\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\xe3\x35\x94\xd7\x50\x5e\x43\xb9" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ksize = 32, + .plaintext = "\xe3\x35\x94\xd7\x50\x5e\x43\xb9" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x33\x94\xd7\x50\x5e\x43\x79\xcd" "\x01\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .psize = 80, + .psize = 48, .digest = "\x13\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", }, { /* Regression test for overflow in AVX2 implementation */ + .key = "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .ksize = 32, .plaintext = "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" @@ -9113,12 +9129,8 @@ static const struct hash_testvec poly1305_tv_template[] = { "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" "\xff\xff\xff\xff", - .psize = 300, + .psize = 268, .digest = "\xfb\x5e\x96\xd8\x61\xd5\xc7\xc8" "\x78\xe5\x87\xcc\x2d\x5a\x22\xe1", } From patchwork Thu Apr 24 10:47:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884070 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B48E22D7B0 for ; Thu, 24 Apr 2025 10:47:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491651; cv=none; b=kug5WfMhRaORe1UFzqTVUZpStA2zXAuK3NnnkYVjDlUr+JG6b4Rt+eykD7UHfQ/xK41gon5pSURJNtqBhlQaLFxCRUmKwht6ErJ464FnGMl9o9nIVDH/g7YnXwAlpjfnPuX3vjF8tNMD1h72OBVU0FfqaPorsL+BWRPE0KNo7iQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491651; c=relaxed/simple; bh=TdN1ASQ+5tJT/4s35Zlto2UupUyBX21IFUiow77AhWI=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=bXxWonxmnYx5zRwV3FBnsjbHAydlP1OXXShnxON9CPxLKyajoB4bD8zWU+MTo2kaobZMWW0wBqGeWpxT2j81xJKjLDC32OPY2TRhd8CSer4b+9AyNpNLoTr8Pb2U99QL8b/3axX907cTRTG4twcoG8Qt00ScZE2lFfe1w/22dw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=Mu93zsbV; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="Mu93zsbV" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=YpLTxnnw/+/57TnZpiX/XQjlZFNLzpA5fAtBF0t/0I8=; b=Mu93zsbVxS9r+HcS/emoX4gHSj 5csP+GqchYIk6yXrEOk31IQhcnDl0F0LvdDVj6JfDuC+6WGUBj/c/pQAnUGEk3UDJ04iouy+F8Kg/ NSI7iLDzn3b33kIBvrucNbnIBYI9TmTkIVkiAAfah49q0vPNMG8NJVHpZvSR2w/p1DvHUablsBXCe uoUEF17n3V2ziSvMeuAKfzCFk9BZkAyZOsxHnO7rurBOCs+uGGtLYbrBIbaMyE1dy96HEZf0opwc+ idCR9tJ4fKsie8gJjNrXU1Mv/ZQGNpeCF3f3HH+Uy3/sn/VHkopcqB5/XTAxgjIocdlidqinasoGD wVQexQjQ==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u74-000fOJ-0f; Thu, 24 Apr 2025 18:47:27 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:26 +0800 Date: Thu, 24 Apr 2025 18:47:26 +0800 Message-Id: <81e1f3a513fda4e49f0b10527d3540e205ad375f.1745490652.git.herbert@gondor.apana.org.au> In-Reply-To: References: From: Herbert Xu Subject: [PATCH 13/15] crypto: arm64/polyval - Use API partial block handling To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Use the Crypto API partial block handling. Also remove the unnecessary SIMD fallback path. Signed-off-by: Herbert Xu --- arch/arm64/crypto/polyval-ce-glue.c | 73 ++++++++--------------------- 1 file changed, 20 insertions(+), 53 deletions(-) diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyval-ce-glue.c index 0a3b5718df85..c4e653688ea0 100644 --- a/arch/arm64/crypto/polyval-ce-glue.c +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -15,17 +15,15 @@ * ARMv8 Crypto Extensions instructions to implement the finite field operations. */ -#include +#include #include -#include #include -#include -#include +#include +#include +#include #include #include -#include -#include -#include +#include #define NUM_KEY_POWERS 8 @@ -38,7 +36,6 @@ struct polyval_tfm_ctx { struct polyval_desc_ctx { u8 buffer[POLYVAL_BLOCK_SIZE]; - u32 bytes; }; asmlinkage void pmull_polyval_update(const struct polyval_tfm_ctx *keys, @@ -48,25 +45,16 @@ asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op2); static void internal_polyval_update(const struct polyval_tfm_ctx *keys, const u8 *in, size_t nblocks, u8 *accumulator) { - if (likely(crypto_simd_usable())) { - kernel_neon_begin(); - pmull_polyval_update(keys, in, nblocks, accumulator); - kernel_neon_end(); - } else { - polyval_update_non4k(keys->key_powers[NUM_KEY_POWERS-1], in, - nblocks, accumulator); - } + kernel_neon_begin(); + pmull_polyval_update(keys, in, nblocks, accumulator); + kernel_neon_end(); } static void internal_polyval_mul(u8 *op1, const u8 *op2) { - if (likely(crypto_simd_usable())) { - kernel_neon_begin(); - pmull_polyval_mul(op1, op2); - kernel_neon_end(); - } else { - polyval_mul_non4k(op1, op2); - } + kernel_neon_begin(); + pmull_polyval_mul(op1, op2); + kernel_neon_end(); } static int polyval_arm64_setkey(struct crypto_shash *tfm, @@ -103,49 +91,27 @@ static int polyval_arm64_update(struct shash_desc *desc, { struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); const struct polyval_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm); - u8 *pos; unsigned int nblocks; - unsigned int n; - if (dctx->bytes) { - n = min(srclen, dctx->bytes); - pos = dctx->buffer + POLYVAL_BLOCK_SIZE - dctx->bytes; - - dctx->bytes -= n; - srclen -= n; - - while (n--) - *pos++ ^= *src++; - - if (!dctx->bytes) - internal_polyval_mul(dctx->buffer, - tctx->key_powers[NUM_KEY_POWERS-1]); - } - - while (srclen >= POLYVAL_BLOCK_SIZE) { + do { /* allow rescheduling every 4K bytes */ nblocks = min(srclen, 4096U) / POLYVAL_BLOCK_SIZE; internal_polyval_update(tctx, src, nblocks, dctx->buffer); srclen -= nblocks * POLYVAL_BLOCK_SIZE; src += nblocks * POLYVAL_BLOCK_SIZE; - } + } while (srclen >= POLYVAL_BLOCK_SIZE); - if (srclen) { - dctx->bytes = POLYVAL_BLOCK_SIZE - srclen; - pos = dctx->buffer; - while (srclen--) - *pos++ ^= *src++; - } - - return 0; + return srclen; } -static int polyval_arm64_final(struct shash_desc *desc, u8 *dst) +static int polyval_arm64_finup(struct shash_desc *desc, const u8 *src, + unsigned int len, u8 *dst) { struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); const struct polyval_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm); - if (dctx->bytes) { + if (len) { + crypto_xor(dctx->buffer, src, len); internal_polyval_mul(dctx->buffer, tctx->key_powers[NUM_KEY_POWERS-1]); } @@ -159,13 +125,14 @@ static struct shash_alg polyval_alg = { .digestsize = POLYVAL_DIGEST_SIZE, .init = polyval_arm64_init, .update = polyval_arm64_update, - .final = polyval_arm64_final, + .finup = polyval_arm64_finup, .setkey = polyval_arm64_setkey, .descsize = sizeof(struct polyval_desc_ctx), .base = { .cra_name = "polyval", .cra_driver_name = "polyval-ce", .cra_priority = 200, + .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY, .cra_blocksize = POLYVAL_BLOCK_SIZE, .cra_ctxsize = sizeof(struct polyval_tfm_ctx), .cra_module = THIS_MODULE, From patchwork Thu Apr 24 10:47:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herbert Xu X-Patchwork-Id: 884069 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4685C221FCC for ; Thu, 24 Apr 2025 10:47:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=144.6.53.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491656; cv=none; b=r2GV/j8OLGQ88w3EmZrT3mplZLYvBGwh4Kw9qbpCjHmpnuc7hwTAPZZ0GtOqzZ01uuNN6vrYwy+pcrTxruMYNoOxTI2fsYDV95HIwn7akpz1kDY1ELiBUVZ7ci53OJhv/VsHP9HlK+mfENXXKNlYpcNcWdNUnu1IDpFwbLim9PA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491656; c=relaxed/simple; bh=1pfPyVZVRvHXTMzpTjLn7qatF+RMJ1aGTlpkru9PJk4=; h=Date:Message-Id:In-Reply-To:References:From:Subject:To; b=k8u1Fr7yzgYpqYU0VeZ0rQOjRwlYF7g3YsuQE5VwJevlzznzAydBAAqffFEgxzvjHYuQfJmsBPAYw2xXukeA/t5dnHXlyx67ju05gp9fVvWTkwfyWe2yg2RsGaXL8RXVONXIfD4YlaMYTO3+3O21pHGU3yE+LUaTgux9mMfFJ+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au; spf=pass smtp.mailfrom=gondor.apana.org.au; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b=NPbnS3/y; arc=none smtp.client-ip=144.6.53.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gondor.apana.org.au Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hmeau.com header.i=@hmeau.com header.b="NPbnS3/y" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=To:Subject:From:References:In-Reply-To:Message-Id:Date:Sender: Reply-To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=QcTf8gBn5TASfxlGhLjJaS9cqJZGrJ005HS2C+MIj9g=; b=NPbnS3/ymNfSiE46tx6hAHs3XQ fMqqklWxMo5DwsaHkf3Bywv7dhMtbN+K5WkWhIz4hUusqWBP3p3Q/IzQX6vw8ei6Ad0hT9GiGl7TP toDhYLwkq9KQCPlY9AhEZRshVCMDCc54n9MgEwmweC5ZvadoH88oXFv8GLw5B3Rg2MdlkYxTYq5qu iMcDK7Ga54BXprINQL+2onxFaKFORA6ERYEHuWTF+br7CuYZTowzjGtmQKb8PrXx96Ur6RL0IL1oV GiPj6tFQWyyjGnpEWxum1ueSOvfHBtNhKX1QVeRIEL0ML/Urba2DkiojwuqOBvTG8k57UmRNKFJVj fvuKV6iw==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1u7u78-000fOg-2g; Thu, 24 Apr 2025 18:47:31 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Thu, 24 Apr 2025 18:47:30 +0800 Date: Thu, 24 Apr 2025 18:47:30 +0800 Message-Id: <18dd6ca13b6924cd28be515dc49d18edf8c85664.1745490652.git.herbert@gondor.apana.org.au> In-Reply-To: References: From: Herbert Xu Subject: [PATCH 15/15] crypto: polyval-generic - Use API partial block handling To: Linux Crypto Mailing List Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Use the Crypto API partial block handling. The accelerated export format on x86/arm64 is easier to use so switch the generic polyval algorithm to use that format instead. Signed-off-by: Herbert Xu --- crypto/polyval-generic.c | 120 +++++++++++++-------------------------- include/crypto/polyval.h | 8 --- 2 files changed, 40 insertions(+), 88 deletions(-) diff --git a/crypto/polyval-generic.c b/crypto/polyval-generic.c index 4f98910bcdb5..ffd174e75420 100644 --- a/crypto/polyval-generic.c +++ b/crypto/polyval-generic.c @@ -44,15 +44,15 @@ * */ -#include -#include #include -#include #include -#include -#include +#include +#include +#include #include #include +#include +#include struct polyval_tfm_ctx { struct gf128mul_4k *gf128; @@ -63,7 +63,6 @@ struct polyval_desc_ctx { u8 buffer[POLYVAL_BLOCK_SIZE]; be128 buffer128; }; - u32 bytes; }; static void copy_and_reverse(u8 dst[POLYVAL_BLOCK_SIZE], @@ -76,46 +75,6 @@ static void copy_and_reverse(u8 dst[POLYVAL_BLOCK_SIZE], put_unaligned(swab64(b), (u64 *)&dst[0]); } -/* - * Performs multiplication in the POLYVAL field using the GHASH field as a - * subroutine. This function is used as a fallback for hardware accelerated - * implementations when simd registers are unavailable. - * - * Note: This function is not used for polyval-generic, instead we use the 4k - * lookup table implementation for finite field multiplication. - */ -void polyval_mul_non4k(u8 *op1, const u8 *op2) -{ - be128 a, b; - - // Assume one argument is in Montgomery form and one is not. - copy_and_reverse((u8 *)&a, op1); - copy_and_reverse((u8 *)&b, op2); - gf128mul_x_lle(&a, &a); - gf128mul_lle(&a, &b); - copy_and_reverse(op1, (u8 *)&a); -} -EXPORT_SYMBOL_GPL(polyval_mul_non4k); - -/* - * Perform a POLYVAL update using non4k multiplication. This function is used - * as a fallback for hardware accelerated implementations when simd registers - * are unavailable. - * - * Note: This function is not used for polyval-generic, instead we use the 4k - * lookup table implementation of finite field multiplication. - */ -void polyval_update_non4k(const u8 *key, const u8 *in, - size_t nblocks, u8 *accumulator) -{ - while (nblocks--) { - crypto_xor(accumulator, in, POLYVAL_BLOCK_SIZE); - polyval_mul_non4k(accumulator, key); - in += POLYVAL_BLOCK_SIZE; - } -} -EXPORT_SYMBOL_GPL(polyval_update_non4k); - static int polyval_setkey(struct crypto_shash *tfm, const u8 *key, unsigned int keylen) { @@ -154,56 +113,53 @@ static int polyval_update(struct shash_desc *desc, { struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm); - u8 *pos; u8 tmp[POLYVAL_BLOCK_SIZE]; - int n; - if (dctx->bytes) { - n = min(srclen, dctx->bytes); - pos = dctx->buffer + dctx->bytes - 1; - - dctx->bytes -= n; - srclen -= n; - - while (n--) - *pos-- ^= *src++; - - if (!dctx->bytes) - gf128mul_4k_lle(&dctx->buffer128, ctx->gf128); - } - - while (srclen >= POLYVAL_BLOCK_SIZE) { + do { copy_and_reverse(tmp, src); crypto_xor(dctx->buffer, tmp, POLYVAL_BLOCK_SIZE); gf128mul_4k_lle(&dctx->buffer128, ctx->gf128); src += POLYVAL_BLOCK_SIZE; srclen -= POLYVAL_BLOCK_SIZE; - } + } while (srclen >= POLYVAL_BLOCK_SIZE); - if (srclen) { - dctx->bytes = POLYVAL_BLOCK_SIZE - srclen; - pos = dctx->buffer + POLYVAL_BLOCK_SIZE - 1; - while (srclen--) - *pos-- ^= *src++; - } - - return 0; + return srclen; } -static int polyval_final(struct shash_desc *desc, u8 *dst) +static int polyval_finup(struct shash_desc *desc, const u8 *src, + unsigned int len, u8 *dst) { struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); - const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm); - if (dctx->bytes) - gf128mul_4k_lle(&dctx->buffer128, ctx->gf128); + if (len) { + u8 tmp[POLYVAL_BLOCK_SIZE] = {}; + + memcpy(tmp, src, len); + polyval_update(desc, tmp, POLYVAL_BLOCK_SIZE); + } copy_and_reverse(dst, dctx->buffer); return 0; } -static void polyval_exit_tfm(struct crypto_tfm *tfm) +static int polyval_export(struct shash_desc *desc, void *out) { - struct polyval_tfm_ctx *ctx = crypto_tfm_ctx(tfm); + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + copy_and_reverse(out, dctx->buffer); + return 0; +} + +static int polyval_import(struct shash_desc *desc, const void *in) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + copy_and_reverse(dctx->buffer, in); + return 0; +} + +static void polyval_exit_tfm(struct crypto_shash *tfm) +{ + struct polyval_tfm_ctx *ctx = crypto_shash_ctx(tfm); gf128mul_free_4k(ctx->gf128); } @@ -212,17 +168,21 @@ static struct shash_alg polyval_alg = { .digestsize = POLYVAL_DIGEST_SIZE, .init = polyval_init, .update = polyval_update, - .final = polyval_final, + .finup = polyval_finup, .setkey = polyval_setkey, + .export = polyval_export, + .import = polyval_import, + .exit_tfm = polyval_exit_tfm, + .statesize = sizeof(struct polyval_desc_ctx), .descsize = sizeof(struct polyval_desc_ctx), .base = { .cra_name = "polyval", .cra_driver_name = "polyval-generic", .cra_priority = 100, + .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY, .cra_blocksize = POLYVAL_BLOCK_SIZE, .cra_ctxsize = sizeof(struct polyval_tfm_ctx), .cra_module = THIS_MODULE, - .cra_exit = polyval_exit_tfm, }, }; diff --git a/include/crypto/polyval.h b/include/crypto/polyval.h index 1d630f371f77..d2e63743e592 100644 --- a/include/crypto/polyval.h +++ b/include/crypto/polyval.h @@ -8,15 +8,7 @@ #ifndef _CRYPTO_POLYVAL_H #define _CRYPTO_POLYVAL_H -#include -#include - #define POLYVAL_BLOCK_SIZE 16 #define POLYVAL_DIGEST_SIZE 16 -void polyval_mul_non4k(u8 *op1, const u8 *op2); - -void polyval_update_non4k(const u8 *key, const u8 *in, - size_t nblocks, u8 *accumulator); - #endif