From patchwork Sun Jun 1 22:44:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893898 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B660239E69; Sun, 1 Jun 2025 22:45:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; cv=none; b=IrCR4HpjvinWTlFxrQ0csLUNwGWAD7G2pfUxr+bIibmy5Hc/oO4nj9WnOmRHKO3QzPWLwtHZDsfTX5C36r5ojq3286uPMNBZtCjqIXZ/3SjKKx3bE65IInggCz3v1QrP8dBYC4Sa6ZcRkO1876Yvx3hEmC5+HDDB9H1fj+bEwtE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; c=relaxed/simple; bh=HDU9nSTLbE55YbS5EUuBOLBm8L+KFIjICgPyy22jxdc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fYXZH1X0OWLKAxCzRYne4xmFhxC4momFHWiyuxAl9720beefHPTpyxJHJT5Sp3hc0BVbwwTpizxMCbGo4f+YPMHkx3E2r513oJDQv2vtbWmkN5GZzIoxH8js+PHALAutUoGshMq/DbyPA96uy1UTuGZgjOJFFNp0clohvzPD2MY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DNPBjqUP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DNPBjqUP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C7E3C4CEF3; Sun, 1 Jun 2025 22:45:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817953; bh=HDU9nSTLbE55YbS5EUuBOLBm8L+KFIjICgPyy22jxdc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DNPBjqUP++VnaPZeQEjCA9PyqCJOY1jpIJYbHsl58QNltzexLH+k8ZpJP826lKJc6 4uiNjnztbjOQdF4v00x2dEGtWcRRGMZXT4P2/7aRYie0N3dfqpFTBMtr4tB/gTk+ro 2C8fZYfDf+KKdNV9QH1Fvbe22PkJswPRcWoBHgi7Ae73mX/+cFwVJzfwqCEaqPyQYL dYvjAgD1stt+XlvcSxaIVj8HfZoVRj9GfkwMdVGe18TiSaMerfLHUZXzoq3RcwpSA+ kDVCYaQs7AB87EEgn9MXaT1pxlISTeshxy5D2VGrJt0fqsYGg703b67vwpKi5cdlPV 9yCrZlVd3VVRQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 01/13] crypto/crc32: register only one shash_alg Date: Sun, 1 Jun 2025 15:44:29 -0700 Message-ID: <20250601224441.778374-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Stop unnecessarily registering a "crc32-generic" shash_alg when a "crc32-$(ARCH)" shash_alg is registered too. While every algorithm does need to have a generic implementation to ensure uniformity of support across platforms, that doesn't mean that we need to make the generic implementation available through crypto_shash when an optimized implementation is also available. Registering the generic shash_alg did allow users of the crypto_shash or crypto_ahash APIs to request the generic implementation specifically, instead of an optimized one. However, the only known use case for that was the differential fuzz tests in crypto/testmgr.c. Equivalent test coverage is now provided by crc_kunit. Besides simplifying crypto/crc32.c, this change eliminates the need for the library to provide crc32_le_base() as part of its interface. Later patches will make crc32_le_base() be internal to the library. Signed-off-by: Eric Biggers --- crypto/crc32.c | 69 ++++++++------------------------------------------ 1 file changed, 11 insertions(+), 58 deletions(-) diff --git a/crypto/crc32.c b/crypto/crc32.c index cc371d42601fd..b61d5663d0bac 100644 --- a/crypto/crc32.c +++ b/crypto/crc32.c @@ -57,33 +57,16 @@ static int crc32_init(struct shash_desc *desc) static int crc32_update(struct shash_desc *desc, const u8 *data, unsigned int len) { u32 *crcp = shash_desc_ctx(desc); - *crcp = crc32_le_base(*crcp, data, len); - return 0; -} - -static int crc32_update_arch(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - u32 *crcp = shash_desc_ctx(desc); - *crcp = crc32_le(*crcp, data, len); return 0; } /* No final XOR 0xFFFFFFFF, like crc32_le */ -static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len, - u8 *out) -{ - put_unaligned_le32(crc32_le_base(*crcp, data, len), out); - return 0; -} - -static int __crc32_finup_arch(u32 *crcp, const u8 *data, unsigned int len, - u8 *out) +static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out) { put_unaligned_le32(crc32_le(*crcp, data, len), out); return 0; } @@ -91,16 +74,10 @@ static int crc32_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { return __crc32_finup(shash_desc_ctx(desc), data, len, out); } -static int crc32_finup_arch(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - return __crc32_finup_arch(shash_desc_ctx(desc), data, len, out); -} - static int crc32_final(struct shash_desc *desc, u8 *out) { u32 *crcp = shash_desc_ctx(desc); put_unaligned_le32(*crcp, out); @@ -111,72 +88,48 @@ static int crc32_digest(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { return __crc32_finup(crypto_shash_ctx(desc->tfm), data, len, out); } -static int crc32_digest_arch(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - return __crc32_finup_arch(crypto_shash_ctx(desc->tfm), data, len, out); -} - -static struct shash_alg algs[] = {{ +static struct shash_alg alg = { .setkey = crc32_setkey, .init = crc32_init, .update = crc32_update, .final = crc32_final, .finup = crc32_finup, .digest = crc32_digest, .descsize = sizeof(u32), .digestsize = CHKSUM_DIGEST_SIZE, .base.cra_name = "crc32", - .base.cra_driver_name = "crc32-generic", .base.cra_priority = 100, .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY, .base.cra_blocksize = CHKSUM_BLOCK_SIZE, .base.cra_ctxsize = sizeof(u32), .base.cra_module = THIS_MODULE, .base.cra_init = crc32_cra_init, -}, { - .setkey = crc32_setkey, - .init = crc32_init, - .update = crc32_update_arch, - .final = crc32_final, - .finup = crc32_finup_arch, - .digest = crc32_digest_arch, - .descsize = sizeof(u32), - .digestsize = CHKSUM_DIGEST_SIZE, - - .base.cra_name = "crc32", - .base.cra_driver_name = "crc32-" __stringify(ARCH), - .base.cra_priority = 150, - .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY, - .base.cra_blocksize = CHKSUM_BLOCK_SIZE, - .base.cra_ctxsize = sizeof(u32), - .base.cra_module = THIS_MODULE, - .base.cra_init = crc32_cra_init, -}}; - -static int num_algs; +}; static int __init crc32_mod_init(void) { - /* register the arch flavor only if it differs from the generic one */ - num_algs = 1 + ((crc32_optimizations() & CRC32_LE_OPTIMIZATION) != 0); + const char *driver_name = + (crc32_optimizations() & CRC32_LE_OPTIMIZATION) ? + "crc32-" __stringify(ARCH) : + "crc32-generic"; + + strscpy(alg.base.cra_driver_name, driver_name, CRYPTO_MAX_ALG_NAME); - return crypto_register_shashes(algs, num_algs); + return crypto_register_shash(&alg); } static void __exit crc32_mod_fini(void) { - crypto_unregister_shashes(algs, num_algs); + crypto_unregister_shash(&alg); } module_init(crc32_mod_init); module_exit(crc32_mod_fini); MODULE_AUTHOR("Alexander Boyko "); MODULE_DESCRIPTION("CRC32 calculations wrapper for lib/crc32"); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("crc32"); -MODULE_ALIAS_CRYPTO("crc32-generic"); From patchwork Sun Jun 1 22:44:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893695 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9609823BCE7; Sun, 1 Jun 2025 22:45:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; cv=none; b=J8hFCl1X7QEqofPcEUR9b4kWWPediqgCNc0d4NCc03OZRqbPmyAbGZpwaix12maE9ZNWywP0dmsY5eWZKPMlD+gDfGfdQCrnE7bI8YKzZo8gyMsNXMBg8BRsyGHX2vJ96jcpBNkv+wW5gsdRG055HVSVdt1LQIb4hzPrSQKrQL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; c=relaxed/simple; bh=DuymbRdfQNASqdv95GmKI7xo6X3aqFKpkNJXYJg46+g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=szrDzpYQhtWMZQLzBHc1b1B/hbP00nbu0hb8KdCAhtPCic5xgkyr/11VO+D6bs/hojbT9aqfI1ROfrl95E5hHhouyZZWuHYuKoRdVAPoDSUxNFCWVcOVnT8b4CBDCtMD2e7Ucl93r/WKLQSIWnBXvrYhXjBUad3IfAcxSKAbVLE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=s5KZx5zF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="s5KZx5zF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97E5FC4AF09; Sun, 1 Jun 2025 22:45:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817954; bh=DuymbRdfQNASqdv95GmKI7xo6X3aqFKpkNJXYJg46+g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s5KZx5zFP0pmjW+yPAWdh9HSDJ2eN9w2ECvi4PqHcvu7LAXDpxsbC+W620rIBoLrE DznkxM4gfP4EgC5XNbW/1IZSwXBfLwYb/IAzXBTSTWzJC4xcVQe3Kc8S5SaI7DgHDo zB1yc7zunGBGTw5P/TzAbj5R/07z6Jz4uDjDkknV7a/3ATwioeZfpMpSDAJIGlwvmZ lDcK7YxikgkmIs33oVtz5QhNny84wp9XVplAzVmdhy00Q1kCZDwflOiPDRUlhAlYk2 Eg0xXw+k7lVB/hD59SGVjKJq3uODrL4J9EKIvDT8fqH39+CwyxjA3VwspzvlJX9AJO fhH3nLtfKMqaw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 02/13] crypto/crc32c: register only one shash_alg Date: Sun, 1 Jun 2025 15:44:30 -0700 Message-ID: <20250601224441.778374-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Stop unnecessarily registering a "crc32c-generic" shash_alg when a "crc32c-$(ARCH)" shash_alg is registered too. While every algorithm does need to have a generic implementation to ensure uniformity of support across platforms, that doesn't mean that we need to make the generic implementation available through crypto_shash when an optimized implementation is also available. Registering the generic shash_alg did allow users of the crypto_shash or crypto_ahash APIs to request the generic implementation specifically, instead of an optimized one. However, the only known use case for that was the differential fuzz tests in crypto/testmgr.c. Equivalent test coverage is now provided by crc_kunit. Besides simplifying crypto/crc32c.c, this change eliminates the need for the library to provide crc32c_base() as part of its interface. Later patches will make crc32c_base() be internal to the library. Signed-off-by: Eric Biggers --- crypto/crc32c.c | 70 +++++++------------------------------------------ 1 file changed, 10 insertions(+), 60 deletions(-) diff --git a/crypto/crc32c.c b/crypto/crc32c.c index e5377898414a2..e160695682f16 100644 --- a/crypto/crc32c.c +++ b/crypto/crc32c.c @@ -83,19 +83,10 @@ static int chksum_setkey(struct crypto_shash *tfm, const u8 *key, static int chksum_update(struct shash_desc *desc, const u8 *data, unsigned int length) { struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - ctx->crc = crc32c_base(ctx->crc, data, length); - return 0; -} - -static int chksum_update_arch(struct shash_desc *desc, const u8 *data, - unsigned int length) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - ctx->crc = crc32c(ctx->crc, data, length); return 0; } static int chksum_final(struct shash_desc *desc, u8 *out) @@ -105,17 +96,10 @@ static int chksum_final(struct shash_desc *desc, u8 *out) put_unaligned_le32(~ctx->crc, out); return 0; } static int __chksum_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out) -{ - put_unaligned_le32(~crc32c_base(*crcp, data, len), out); - return 0; -} - -static int __chksum_finup_arch(u32 *crcp, const u8 *data, unsigned int len, - u8 *out) { put_unaligned_le32(~crc32c(*crcp, data, len), out); return 0; } @@ -125,98 +109,64 @@ static int chksum_finup(struct shash_desc *desc, const u8 *data, struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); return __chksum_finup(&ctx->crc, data, len, out); } -static int chksum_finup_arch(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - - return __chksum_finup_arch(&ctx->crc, data, len, out); -} - static int chksum_digest(struct shash_desc *desc, const u8 *data, unsigned int length, u8 *out) { struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm); return __chksum_finup(&mctx->key, data, length, out); } -static int chksum_digest_arch(struct shash_desc *desc, const u8 *data, - unsigned int length, u8 *out) -{ - struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm); - - return __chksum_finup_arch(&mctx->key, data, length, out); -} - static int crc32c_cra_init(struct crypto_tfm *tfm) { struct chksum_ctx *mctx = crypto_tfm_ctx(tfm); mctx->key = ~0; return 0; } -static struct shash_alg algs[] = {{ +static struct shash_alg alg = { .digestsize = CHKSUM_DIGEST_SIZE, .setkey = chksum_setkey, .init = chksum_init, .update = chksum_update, .final = chksum_final, .finup = chksum_finup, .digest = chksum_digest, .descsize = sizeof(struct chksum_desc_ctx), .base.cra_name = "crc32c", - .base.cra_driver_name = "crc32c-generic", .base.cra_priority = 100, .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY, .base.cra_blocksize = CHKSUM_BLOCK_SIZE, .base.cra_ctxsize = sizeof(struct chksum_ctx), .base.cra_module = THIS_MODULE, .base.cra_init = crc32c_cra_init, -}, { - .digestsize = CHKSUM_DIGEST_SIZE, - .setkey = chksum_setkey, - .init = chksum_init, - .update = chksum_update_arch, - .final = chksum_final, - .finup = chksum_finup_arch, - .digest = chksum_digest_arch, - .descsize = sizeof(struct chksum_desc_ctx), - - .base.cra_name = "crc32c", - .base.cra_driver_name = "crc32c-" __stringify(ARCH), - .base.cra_priority = 150, - .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY, - .base.cra_blocksize = CHKSUM_BLOCK_SIZE, - .base.cra_ctxsize = sizeof(struct chksum_ctx), - .base.cra_module = THIS_MODULE, - .base.cra_init = crc32c_cra_init, -}}; - -static int num_algs; +}; static int __init crc32c_mod_init(void) { - /* register the arch flavor only if it differs from the generic one */ - num_algs = 1 + ((crc32_optimizations() & CRC32C_OPTIMIZATION) != 0); + const char *driver_name = + (crc32_optimizations() & CRC32C_OPTIMIZATION) ? + "crc32c-" __stringify(ARCH) : + "crc32c-generic"; + + strscpy(alg.base.cra_driver_name, driver_name, CRYPTO_MAX_ALG_NAME); - return crypto_register_shashes(algs, num_algs); + return crypto_register_shash(&alg); } static void __exit crc32c_mod_fini(void) { - crypto_unregister_shashes(algs, num_algs); + crypto_unregister_shash(&alg); } module_init(crc32c_mod_init); module_exit(crc32c_mod_fini); MODULE_AUTHOR("Clay Haapala "); MODULE_DESCRIPTION("CRC32c (Castagnoli) calculations wrapper for lib/crc32c"); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("crc32c"); -MODULE_ALIAS_CRYPTO("crc32c-generic"); From patchwork Sun Jun 1 22:44:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893897 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9A8523BCF3; Sun, 1 Jun 2025 22:45:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; cv=none; b=UMqKJ8K+O6yApmLxXHuFD7SiF2ea6A7vFmw11Wg2HR+PNVWnOP9sO5NOJAZqFV+rIZGTuKQW2SVBdyI5m6GBNT2YO2tSw+w+XMokN4adwPkz5TaZSkNWVNGJNBvePFgWVqmgIJ6pQb+8zpUbOJuS735RorjaiR3mjjavn/RY8QM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817954; c=relaxed/simple; bh=n02zpmiKFoZCFseh0go3tL+B28l1c/RKe0UcUZvIUPU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KfPJ3aHB4JP05ekE5siPYpagdL1/2VpGnE5DJ6C7SCmpnBLyEM13hlbQuFVIs8u8ETPgrUxEWgW60ktmg3RQ3jiin7Fk7DUKH7bAfLTFzAGcLg6dbHZEeznxBdtS5JYpnnddqk7d9s6dAWhm5Jbb/2KKnpXMlBhwuLXB6FpN4C8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pz+DFv84; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pz+DFv84" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F07EC4AF0C; Sun, 1 Jun 2025 22:45:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817954; bh=n02zpmiKFoZCFseh0go3tL+B28l1c/RKe0UcUZvIUPU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pz+DFv84GlPqBaQzPmtnVnqB9ixuVmKdzCU/KW+UhCtBiaEdlTTHMj7hCpJ50o0DV 3wqzFLjNboGqMH8jDK420YbG+Sz07wknsXCpkVnOiREJMMdKcIvzH/ZeYFoHdGi3GA LmOKWrQOjWgP/q2YDdK/o1WXAzanaX1r0uiHy5zttbKr60OCF2EihyVsprf5xeTn0B Y8/p/zCxXES4PW4Y4AA+QG00iOX7cKylWQSD8a1yZQQBRNAk7qGDwCa2leJEXnB6Es wTVc5XY7f4DtGgiVoOdjGT9Hlx4t8fZ8s9vRcDdA8s43aeR5rVnggY1pmkfWEgPbw/ u8mpkAxEBA6+w== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 03/13] lib/crc: move files into lib/crc/ Date: Sun, 1 Jun 2025 15:44:31 -0700 Message-ID: <20250601224441.778374-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move all CRC files in lib/ into a subdirectory lib/crc/ to keep them from cluttering up the main lib/ directory. Signed-off-by: Eric Biggers --- Documentation/core-api/kernel-api.rst | 14 ++-- MAINTAINERS | 3 +- include/linux/crc64.h | 3 - lib/Kconfig | 87 +------------------- lib/Kconfig.debug | 21 ----- lib/Makefile | 32 +------- lib/crc/.gitignore | 5 ++ lib/crc/Kconfig | 111 ++++++++++++++++++++++++++ lib/crc/Makefile | 32 ++++++++ lib/{ => crc}/crc-ccitt.c | 3 - lib/{ => crc}/crc-itu-t.c | 0 lib/{ => crc}/crc-t10dif.c | 0 lib/{ => crc}/crc16.c | 0 lib/{ => crc}/crc32.c | 0 lib/{ => crc}/crc4.c | 0 lib/{ => crc}/crc64.c | 0 lib/{ => crc}/crc7.c | 0 lib/{ => crc}/crc8.c | 0 lib/{ => crc}/gen_crc32table.c | 4 +- lib/{ => crc}/gen_crc64table.c | 11 +-- lib/crc/tests/Makefile | 2 + lib/{ => crc}/tests/crc_kunit.c | 0 lib/tests/Makefile | 1 - 23 files changed, 164 insertions(+), 165 deletions(-) create mode 100644 lib/crc/.gitignore create mode 100644 lib/crc/Kconfig create mode 100644 lib/crc/Makefile rename lib/{ => crc}/crc-ccitt.c (98%) rename lib/{ => crc}/crc-itu-t.c (100%) rename lib/{ => crc}/crc-t10dif.c (100%) rename lib/{ => crc}/crc16.c (100%) rename lib/{ => crc}/crc32.c (100%) rename lib/{ => crc}/crc4.c (100%) rename lib/{ => crc}/crc64.c (100%) rename lib/{ => crc}/crc7.c (100%) rename lib/{ => crc}/crc8.c (100%) rename lib/{ => crc}/gen_crc32table.c (95%) rename lib/{ => crc}/gen_crc64table.c (81%) create mode 100644 lib/crc/tests/Makefile rename lib/{ => crc}/tests/crc_kunit.c (100%) diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index ae92a2571388a..c4642d9f13a9c 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -134,28 +134,28 @@ Arithmetic Overflow Checking :internal: CRC Functions ------------- -.. kernel-doc:: lib/crc4.c +.. kernel-doc:: lib/crc/crc4.c :export: -.. kernel-doc:: lib/crc7.c +.. kernel-doc:: lib/crc/crc7.c :export: -.. kernel-doc:: lib/crc8.c +.. kernel-doc:: lib/crc/crc8.c :export: -.. kernel-doc:: lib/crc16.c +.. kernel-doc:: lib/crc/crc16.c :export: -.. kernel-doc:: lib/crc32.c +.. kernel-doc:: lib/crc/crc32.c -.. kernel-doc:: lib/crc-ccitt.c +.. kernel-doc:: lib/crc/crc-ccitt.c :export: -.. kernel-doc:: lib/crc-itu-t.c +.. kernel-doc:: lib/crc/crc-itu-t.c :export: Base 2 log and power Functions ------------------------------ diff --git a/MAINTAINERS b/MAINTAINERS index 75da23034e18d..724f46458a480 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6335,12 +6335,11 @@ L: linux-crypto@vger.kernel.org S: Maintained T: git https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crc-next F: Documentation/staging/crc* F: arch/*/lib/crc* F: include/linux/crc* -F: lib/crc* -F: lib/tests/crc_kunit.c +F: lib/crc/ F: scripts/gen-crc-consts.py CREATIVE SB0540 M: Bastien Nocera L: linux-input@vger.kernel.org diff --git a/include/linux/crc64.h b/include/linux/crc64.h index 41de30b907dff..b6aa290a79312 100644 --- a/include/linux/crc64.h +++ b/include/linux/crc64.h @@ -1,9 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * See lib/crc64.c for the related specification and polynomial arithmetic. - */ #ifndef _LINUX_CRC64_H #define _LINUX_CRC64_H #include diff --git a/lib/Kconfig b/lib/Kconfig index 6c1b8f1842678..3db88ab0d2b4c 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -134,98 +134,13 @@ config TRACE_MMIO_ACCESS depends on TRACING && ARCH_HAVE_TRACE_MMIO_ACCESS help Create tracepoints for MMIO read/write operations. These trace events can be used for logging all MMIO read/write operations. +source "lib/crc/Kconfig" source "lib/crypto/Kconfig" -config CRC_CCITT - tristate - help - The CRC-CCITT library functions. Select this if your module uses any - of the functions from . - -config CRC16 - tristate - help - The CRC16 library functions. Select this if your module uses any of - the functions from . - -config CRC_T10DIF - tristate - help - The CRC-T10DIF library functions. Select this if your module uses - any of the functions from . - -config ARCH_HAS_CRC_T10DIF - bool - -config CRC_T10DIF_ARCH - tristate - default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS - -config CRC_ITU_T - tristate - help - The CRC-ITU-T library functions. Select this if your module uses - any of the functions from . - -config CRC32 - tristate - select BITREVERSE - help - The CRC32 library functions. Select this if your module uses any of - the functions from or . - -config ARCH_HAS_CRC32 - bool - -config CRC32_ARCH - tristate - default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS - -config CRC64 - tristate - help - The CRC64 library functions. Select this if your module uses any of - the functions from . - -config ARCH_HAS_CRC64 - bool - -config CRC64_ARCH - tristate - default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS - -config CRC4 - tristate - help - The CRC4 library functions. Select this if your module uses any of - the functions from . - -config CRC7 - tristate - help - The CRC7 library functions. Select this if your module uses any of - the functions from . - -config CRC8 - tristate - help - The CRC8 library functions. Select this if your module uses any of - the functions from . - -config CRC_OPTIMIZATIONS - bool "Enable optimized CRC implementations" if EXPERT - default y - help - Disabling this option reduces code size slightly by disabling the - architecture-optimized implementations of any CRC variants that are - enabled. CRC checksumming performance may get much slower. - - Keep this enabled unless you're really trying to minimize the size of - the kernel. config XXHASH tristate config AUDIT_GENERIC diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ebe33181b6e6e..3fda96761adbc 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2899,31 +2899,10 @@ config HW_BREAKPOINT_KUNIT_TEST help Tests for hw_breakpoint constraints accounting. If unsure, say N. -config CRC_KUNIT_TEST - tristate "KUnit tests for CRC functions" if !KUNIT_ALL_TESTS - depends on KUNIT - default KUNIT_ALL_TESTS - select CRC7 - select CRC16 - select CRC_T10DIF - select CRC32 - select CRC64 - help - Unit tests for the CRC library functions. - - This is intended to help people writing architecture-specific - optimized versions. If unsure, say N. - -config CRC_BENCHMARK - bool "Benchmark for the CRC functions" - depends on CRC_KUNIT_TEST - help - Include benchmarks in the KUnit test suite for the CRC functions. - config SIPHASH_KUNIT_TEST tristate "Perform selftest on siphash functions" if !KUNIT_ALL_TESTS depends on KUNIT default KUNIT_ALL_TESTS help diff --git a/lib/Makefile b/lib/Makefile index c38582f187dd8..14a5928bb57f5 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -120,11 +120,11 @@ CFLAGS_kobject_uevent.o += -DDEBUG endif obj-$(CONFIG_DEBUG_INFO_REDUCED) += debug_info.o CFLAGS_debug_info.o += $(call cc-option, -femit-struct-debug-detailed=any) -obj-y += math/ crypto/ tests/ vdso/ +obj-y += math/ crc/ crypto/ tests/ vdso/ obj-$(CONFIG_GENERIC_IOMAP) += iomap.o obj-$(CONFIG_HAS_IOMEM) += iomap_copy.o devres.o obj-$(CONFIG_CHECK_SIGNATURE) += check_signature.o obj-$(CONFIG_DEBUG_LOCKING_API_SELFTESTS) += locking-selftest.o @@ -146,19 +146,10 @@ obj-$(CONFIG_DEBUG_OBJECTS) += debugobjects.o obj-$(CONFIG_BITREVERSE) += bitrev.o obj-$(CONFIG_LINEAR_RANGES) += linear_ranges.o obj-$(CONFIG_PACKING) += packing.o obj-$(CONFIG_PACKING_KUNIT_TEST) += packing_test.o -obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o -obj-$(CONFIG_CRC16) += crc16.o -obj-$(CONFIG_CRC_T10DIF)+= crc-t10dif.o -obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o -obj-$(CONFIG_CRC32) += crc32.o -obj-$(CONFIG_CRC64) += crc64.o -obj-$(CONFIG_CRC4) += crc4.o -obj-$(CONFIG_CRC7) += crc7.o -obj-$(CONFIG_CRC8) += crc8.o obj-$(CONFIG_XXHASH) += xxhash.o obj-$(CONFIG_GENERIC_ALLOCATOR) += genalloc.o obj-$(CONFIG_842_COMPRESS) += 842/ obj-$(CONFIG_842_DECOMPRESS) += 842/ @@ -292,31 +283,10 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o obj-$(CONFIG_ASN1) += asn1_decoder.o obj-$(CONFIG_ASN1_ENCODER) += asn1_encoder.o obj-$(CONFIG_FONT_SUPPORT) += fonts/ -hostprogs := gen_crc32table -hostprogs += gen_crc64table -clean-files := crc32table.h -clean-files += crc64table.h - -$(obj)/crc32.o: $(obj)/crc32table.h - -quiet_cmd_crc32 = GEN $@ - cmd_crc32 = $< > $@ - -$(obj)/crc32table.h: $(obj)/gen_crc32table - $(call cmd,crc32) - -$(obj)/crc64.o: $(obj)/crc64table.h - -quiet_cmd_crc64 = GEN $@ - cmd_crc64 = $< > $@ - -$(obj)/crc64table.h: $(obj)/gen_crc64table - $(call cmd,crc64) - # # Build a fast OID lookip registry from include/linux/oid_registry.h # obj-$(CONFIG_OID_REGISTRY) += oid_registry.o diff --git a/lib/crc/.gitignore b/lib/crc/.gitignore new file mode 100644 index 0000000000000..a9e48103c9fb1 --- /dev/null +++ b/lib/crc/.gitignore @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only +/crc32table.h +/crc64table.h +/gen_crc32table +/gen_crc64table diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig new file mode 100644 index 0000000000000..e0e7168b74c75 --- /dev/null +++ b/lib/crc/Kconfig @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# Kconfig for the kernel's cyclic redundancy check (CRC) library code + +config CRC4 + tristate + help + The CRC4 library functions. Select this if your module uses any of + the functions from . + +config CRC7 + tristate + help + The CRC7 library functions. Select this if your module uses any of + the functions from . + +config CRC8 + tristate + help + The CRC8 library functions. Select this if your module uses any of + the functions from . + +config CRC16 + tristate + help + The CRC16 library functions. Select this if your module uses any of + the functions from . + +config CRC_CCITT + tristate + help + The CRC-CCITT library functions. Select this if your module uses any + of the functions from . + +config CRC_ITU_T + tristate + help + The CRC-ITU-T library functions. Select this if your module uses + any of the functions from . + +config CRC_T10DIF + tristate + help + The CRC-T10DIF library functions. Select this if your module uses + any of the functions from . + +config ARCH_HAS_CRC_T10DIF + bool + +config CRC_T10DIF_ARCH + tristate + default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS + +config CRC32 + tristate + select BITREVERSE + help + The CRC32 library functions. Select this if your module uses any of + the functions from or . + +config ARCH_HAS_CRC32 + bool + +config CRC32_ARCH + tristate + default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS + +config CRC64 + tristate + help + The CRC64 library functions. Select this if your module uses any of + the functions from . + +config ARCH_HAS_CRC64 + bool + +config CRC64_ARCH + tristate + default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS + +config CRC_OPTIMIZATIONS + bool "Enable optimized CRC implementations" if EXPERT + default y + help + Disabling this option reduces code size slightly by disabling the + architecture-optimized implementations of any CRC variants that are + enabled. CRC checksumming performance may get much slower. + + Keep this enabled unless you're really trying to minimize the size of + the kernel. + +config CRC_KUNIT_TEST + tristate "KUnit tests for CRC functions" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + select CRC7 + select CRC16 + select CRC_T10DIF + select CRC32 + select CRC64 + help + Unit tests for the CRC library functions. + + This is intended to help people writing architecture-specific + optimized versions. If unsure, say N. + +config CRC_BENCHMARK + bool "Benchmark for the CRC functions" + depends on CRC_KUNIT_TEST + help + Include benchmarks in the KUnit test suite for the CRC functions. diff --git a/lib/crc/Makefile b/lib/crc/Makefile new file mode 100644 index 0000000000000..ff4c30dda4528 --- /dev/null +++ b/lib/crc/Makefile @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# Makefile for the kernel's cyclic redundancy check (CRC) library code + +obj-$(CONFIG_CRC4) += crc4.o +obj-$(CONFIG_CRC7) += crc7.o +obj-$(CONFIG_CRC8) += crc8.o +obj-$(CONFIG_CRC16) += crc16.o +obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o +obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o +obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o +obj-$(CONFIG_CRC32) += crc32.o +obj-$(CONFIG_CRC64) += crc64.o +obj-y += tests/ + +hostprogs := gen_crc32table gen_crc64table +clean-files := crc32table.h crc64table.h + +$(obj)/crc32.o: $(obj)/crc32table.h +$(obj)/crc64.o: $(obj)/crc64table.h + +quiet_cmd_crc32 = GEN $@ + cmd_crc32 = $< > $@ + +quiet_cmd_crc64 = GEN $@ + cmd_crc64 = $< > $@ + +$(obj)/crc32table.h: $(obj)/gen_crc32table + $(call cmd,crc32) + +$(obj)/crc64table.h: $(obj)/gen_crc64table + $(call cmd,crc64) diff --git a/lib/crc-ccitt.c b/lib/crc/crc-ccitt.c similarity index 98% rename from lib/crc-ccitt.c rename to lib/crc/crc-ccitt.c index 9cddf35d3b66e..8d2bc419230b3 100644 --- a/lib/crc-ccitt.c +++ b/lib/crc/crc-ccitt.c @@ -1,9 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only -/* - * linux/lib/crc-ccitt.c - */ #include #include #include diff --git a/lib/crc-itu-t.c b/lib/crc/crc-itu-t.c similarity index 100% rename from lib/crc-itu-t.c rename to lib/crc/crc-itu-t.c diff --git a/lib/crc-t10dif.c b/lib/crc/crc-t10dif.c similarity index 100% rename from lib/crc-t10dif.c rename to lib/crc/crc-t10dif.c diff --git a/lib/crc16.c b/lib/crc/crc16.c similarity index 100% rename from lib/crc16.c rename to lib/crc/crc16.c diff --git a/lib/crc32.c b/lib/crc/crc32.c similarity index 100% rename from lib/crc32.c rename to lib/crc/crc32.c diff --git a/lib/crc4.c b/lib/crc/crc4.c similarity index 100% rename from lib/crc4.c rename to lib/crc/crc4.c diff --git a/lib/crc64.c b/lib/crc/crc64.c similarity index 100% rename from lib/crc64.c rename to lib/crc/crc64.c diff --git a/lib/crc7.c b/lib/crc/crc7.c similarity index 100% rename from lib/crc7.c rename to lib/crc/crc7.c diff --git a/lib/crc8.c b/lib/crc/crc8.c similarity index 100% rename from lib/crc8.c rename to lib/crc/crc8.c diff --git a/lib/gen_crc32table.c b/lib/crc/gen_crc32table.c similarity index 95% rename from lib/gen_crc32table.c rename to lib/crc/gen_crc32table.c index 6d03425b849e4..9a7f31658e355 100644 --- a/lib/gen_crc32table.c +++ b/lib/crc/gen_crc32table.c @@ -1,9 +1,9 @@ // SPDX-License-Identifier: GPL-2.0 #include -#include "../include/linux/crc32poly.h" -#include "../include/generated/autoconf.h" +#include "../../include/linux/crc32poly.h" +#include "../../include/generated/autoconf.h" #include static uint32_t crc32table_le[256]; static uint32_t crc32table_be[256]; static uint32_t crc32ctable_le[256]; diff --git a/lib/gen_crc64table.c b/lib/crc/gen_crc64table.c similarity index 81% rename from lib/gen_crc64table.c rename to lib/crc/gen_crc64table.c index e05a4230a0a08..f2be9f62bab71 100644 --- a/lib/gen_crc64table.c +++ b/lib/crc/gen_crc64table.c @@ -1,16 +1,9 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Generate lookup table for the table-driven CRC64 calculation. - * - * gen_crc64table is executed in kernel build time and generates - * lib/crc64table.h. This header is included by lib/crc64.c for - * the table-driven CRC64 calculation. - * - * See lib/crc64.c for more information about which specification - * and polynomial arithmetic that gen_crc64table.c follows to - * generate the lookup table. + * This host program runs at kernel build time and generates the lookup tables + * used by the generic CRC64 code. * * Copyright 2018 SUSE Linux. * Author: Coly Li */ #include diff --git a/lib/crc/tests/Makefile b/lib/crc/tests/Makefile new file mode 100644 index 0000000000000..65f63c318ef5e --- /dev/null +++ b/lib/crc/tests/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_CRC_KUNIT_TEST) += crc_kunit.o diff --git a/lib/tests/crc_kunit.c b/lib/crc/tests/crc_kunit.c similarity index 100% rename from lib/tests/crc_kunit.c rename to lib/crc/tests/crc_kunit.c diff --git a/lib/tests/Makefile b/lib/tests/Makefile index 56d6450144828..741d3ac2cba25 100644 --- a/lib/tests/Makefile +++ b/lib/tests/Makefile @@ -8,11 +8,10 @@ obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o obj-$(CONFIG_BITS_TEST) += test_bits.o obj-$(CONFIG_BLACKHOLE_DEV_KUNIT_TEST) += blackhole_dev_kunit.o obj-$(CONFIG_CHECKSUM_KUNIT) += checksum_kunit.o obj-$(CONFIG_CMDLINE_KUNIT_TEST) += cmdline_kunit.o obj-$(CONFIG_CPUMASK_KUNIT_TEST) += cpumask_kunit.o -obj-$(CONFIG_CRC_KUNIT_TEST) += crc_kunit.o CFLAGS_fortify_kunit.o += $(call cc-disable-warning, unsequenced) CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-overread) CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-truncation) CFLAGS_fortify_kunit.o += $(DISABLE_STRUCTLEAK_PLUGIN) obj-$(CONFIG_FORTIFY_KUNIT_TEST) += fortify_kunit.o From patchwork Sun Jun 1 22:44:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893694 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAD5323C50D; Sun, 1 Jun 2025 22:45:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817955; cv=none; b=rzJgsnkWctDKThPZPD3+SQ2ZKU/mWoccw0z2yF3qXVu1CRgWdU3JzLL54xuXvCI/DyxANpl6IdofnrxOqxPKK25r8kgVcXWeL/+hBA9Fq07PQOoPHyFpm9yaeZwqmLrfiqbUNySnVfFSNd6ISxvIPlhci14ptn6e5u2KolRdhkM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817955; c=relaxed/simple; bh=4RudxKWuudSCiaU23la1n8R/MDLiWSDgg4YvblvnOqY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RF0OuKshjgO7oIMjmMUjGP6FMkku9W22q04ophTWmcxnHkBa42ojN7ZYo/k9RwxyDwuSbo6J/JVU+yp8QRmXToQV7VJwS7fMLN2FYeFcjU4WAINqT6NPlja6hBuJj/+6YqXiCpMUayXitmCUghf12QFY0IMQW9gg+vhax56C94U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uBTZo2bT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uBTZo2bT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB6B1C4CEE7; Sun, 1 Jun 2025 22:45:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817955; bh=4RudxKWuudSCiaU23la1n8R/MDLiWSDgg4YvblvnOqY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uBTZo2bT4AIvvSiFIRgueNzXSc/t9rRYA5t0ck6kcLqQzEhuA1NQu8UVFZUBAkhVO dNZ1tgq1jJIWHUBBmAc74lnmgZsXOCrEoA7KF9aGyq6EfRQJOUYGn0qXb77HdmwBI/ Wex3E1Ogn4MDe3hPSZm4ZsmuyZvXvfoxiMBRhAlHvyGIwx1DAK/QOLnaVHtt6FFrTb Mh8+depLSbcvz4df/Sq+XGacKfyOCy0bzGD5vwCMgjdIczevRWBxnR83llZclZwzVt 3zp7sk/+reymyNExFgLGgczvT1iK7g8v7865SqSboaO8GcK7syxrfOiy5ARkDcbWv4 m1ZolheymV7Uw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 04/13] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/ Date: Sun, 1 Jun 2025 15:44:32 -0700 Message-ID: <20250601224441.778374-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Rework how lib/crc/ supports arch-optimized code. First, instead of the arch-optimized code being in arch/$(SRCARCH)/lib/, it will now be in lib/crc/$(SRCARCH)/. Second, the API functions (e.g. crc32c()), arch-optimized functions (e.g. crc32c_arch()), and generic functions (e.g. crc32c_base()) will now be part of a single module for each CRC type, allowing better inlining and dead code elimination. The second change is made possible by the first. As an example, consider CONFIG_CRC32=m on x86. We'll now have just crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules were already coupled together and always both got loaded together via direct symbol dependency, so the separation provided no benefit. Note: later I'd like to apply the same design to lib/crypto/ too, where often the API functions are out-of-line so this will work even better. In those cases, for each algorithm we currently have 3 modules all coupled together, e.g. libsha256.ko, libsha256-generic.ko, and sha256-x86.ko. We should have just one, inline things properly, and rely on the compiler's dead code elimination to decide the inclusion of the generic code instead of manually setting it via kconfig. Having arch-specific code outside arch/ was somewhat controversial when Zinc proposed it back in 2018. But I don't think the concerns are warranted. It's better from a technical perspective, as it enables the improvements mentioned above. This model is already successfully used in other places in the kernel such as lib/raid6/. The community of each architecture still remains free to work on the code, even if it's not in arch/. At the time there was also a desire to put the library code in the same files as the old-school crypto API, but that was a mistake; now that the library is separate, that's no longer a constraint either. Signed-off-by: Eric Biggers --- Documentation/core-api/kernel-api.rst | 2 +- MAINTAINERS | 1 - include/linux/crc-t10dif.h | 10 +-- include/linux/crc32.h | 30 +------- include/linux/crc64.h | 19 +---- lib/crc/Kconfig | 18 ++--- lib/crc/Makefile | 19 ++++- lib/crc/{crc-t10dif.c => crc-t10dif-main.c} | 37 ++++++++-- lib/crc/{crc32.c => crc32-main.c} | 77 +++++++++++++++++---- lib/crc/{crc64.c => crc64-main.c} | 47 +++++++++++-- 10 files changed, 164 insertions(+), 96 deletions(-) rename lib/crc/{crc-t10dif.c => crc-t10dif-main.c} (78%) rename lib/crc/{crc32.c => crc32-main.c} (73%) rename lib/crc/{crc64.c => crc64-main.c} (66%) diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index c4642d9f13a9c..9c8370891a39b 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -146,11 +146,11 @@ CRC Functions :export: .. kernel-doc:: lib/crc/crc16.c :export: -.. kernel-doc:: lib/crc/crc32.c +.. kernel-doc:: lib/crc/crc32-main.c .. kernel-doc:: lib/crc/crc-ccitt.c :export: .. kernel-doc:: lib/crc/crc-itu-t.c diff --git a/MAINTAINERS b/MAINTAINERS index 724f46458a480..2aefd2bdbded2 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6333,11 +6333,10 @@ M: Eric Biggers R: Ard Biesheuvel L: linux-crypto@vger.kernel.org S: Maintained T: git https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crc-next F: Documentation/staging/crc* -F: arch/*/lib/crc* F: include/linux/crc* F: lib/crc/ F: scripts/gen-crc-consts.py CREATIVE SB0540 diff --git a/include/linux/crc-t10dif.h b/include/linux/crc-t10dif.h index a559fdff3f7e2..ecc8bc2dd7f4c 100644 --- a/include/linux/crc-t10dif.h +++ b/include/linux/crc-t10dif.h @@ -2,19 +2,11 @@ #ifndef _LINUX_CRC_T10DIF_H #define _LINUX_CRC_T10DIF_H #include -u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len); -u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len); - -static inline u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC_T10DIF_ARCH)) - return crc_t10dif_arch(crc, p, len); - return crc_t10dif_generic(crc, p, len); -} +u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len); static inline u16 crc_t10dif(const u8 *p, size_t len) { return crc_t10dif_update(0, p, len); } diff --git a/include/linux/crc32.h b/include/linux/crc32.h index 8c1883b81b420..87b71bff5951b 100644 --- a/include/linux/crc32.h +++ b/include/linux/crc32.h @@ -3,37 +3,13 @@ #define _LINUX_CRC32_H #include #include -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len); -u32 crc32_le_base(u32 crc, const u8 *p, size_t len); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len); -u32 crc32_be_base(u32 crc, const u8 *p, size_t len); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len); -u32 crc32c_base(u32 crc, const u8 *p, size_t len); - -static inline u32 crc32_le(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32_le_arch(crc, p, len); - return crc32_le_base(crc, p, len); -} - -static inline u32 crc32_be(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32_be_arch(crc, p, len); - return crc32_be_base(crc, p, len); -} - -static inline u32 crc32c(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32c_arch(crc, p, len); - return crc32c_base(crc, p, len); -} +u32 crc32_le(u32 crc, const void *p, size_t len); +u32 crc32_be(u32 crc, const void *p, size_t len); +u32 crc32c(u32 crc, const void *p, size_t len); /* * crc32_optimizations() returns flags that indicate which CRC32 library * functions are using architecture-specific optimizations. Unlike * IS_ENABLED(CONFIG_CRC32_ARCH) it takes into account the different CRC32 diff --git a/include/linux/crc64.h b/include/linux/crc64.h index b6aa290a79312..fc0c06ab1993c 100644 --- a/include/linux/crc64.h +++ b/include/linux/crc64.h @@ -2,28 +2,18 @@ #ifndef _LINUX_CRC64_H #define _LINUX_CRC64_H #include -u64 crc64_be_arch(u64 crc, const u8 *p, size_t len); -u64 crc64_be_generic(u64 crc, const u8 *p, size_t len); -u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len); -u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len); - /** * crc64_be - Calculate bitwise big-endian ECMA-182 CRC64 * @crc: seed value for computation. 0 or (u64)~0 for a new CRC calculation, * or the previous crc64 value if computing incrementally. * @p: pointer to buffer over which CRC64 is run * @len: length of buffer @p */ -static inline u64 crc64_be(u64 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC64_ARCH)) - return crc64_be_arch(crc, p, len); - return crc64_be_generic(crc, p, len); -} +u64 crc64_be(u64 crc, const void *p, size_t len); /** * crc64_nvme - Calculate CRC64-NVME * @crc: seed value for computation. 0 for a new CRC calculation, or the * previous crc64 value if computing incrementally. @@ -31,13 +21,8 @@ static inline u64 crc64_be(u64 crc, const void *p, size_t len) * @len: length of buffer @p * * This computes the CRC64 defined in the NVME NVM Command Set Specification, * *including the bitwise inversion at the beginning and end*. */ -static inline u64 crc64_nvme(u64 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC64_ARCH)) - return ~crc64_nvme_arch(~crc, p, len); - return ~crc64_nvme_generic(~crc, p, len); -} +u64 crc64_nvme(u64 crc, const void *p, size_t len); #endif /* _LINUX_CRC64_H */ diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index e0e7168b74c75..85b0dc774d722 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -46,39 +46,33 @@ config CRC_T10DIF config ARCH_HAS_CRC_T10DIF bool config CRC_T10DIF_ARCH - tristate - default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS + bool + depends on CRC_T10DIF && CRC_OPTIMIZATIONS config CRC32 tristate select BITREVERSE help The CRC32 library functions. Select this if your module uses any of the functions from or . -config ARCH_HAS_CRC32 - bool - config CRC32_ARCH - tristate - default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS + bool + depends on CRC32 && CRC_OPTIMIZATIONS config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of the functions from . -config ARCH_HAS_CRC64 - bool - config CRC64_ARCH - tristate - default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS + bool + depends on CRC64 && CRC_OPTIMIZATIONS config CRC_OPTIMIZATIONS bool "Enable optimized CRC implementations" if EXPERT default y help diff --git a/lib/crc/Makefile b/lib/crc/Makefile index ff4c30dda4528..2f39385f97a5c 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -1,25 +1,40 @@ # SPDX-License-Identifier: GPL-2.0-only # Makefile for the kernel's cyclic redundancy check (CRC) library code +ccflags-y := -I$(src)/$(SRCARCH) + obj-$(CONFIG_CRC4) += crc4.o obj-$(CONFIG_CRC7) += crc7.o obj-$(CONFIG_CRC8) += crc8.o obj-$(CONFIG_CRC16) += crc16.o obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o + obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o +crc-t10dif-y := crc-t10dif-main.o +ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) +endif + obj-$(CONFIG_CRC32) += crc32.o +crc32-y := crc32-main.o +ifeq ($(CONFIG_CRC32_ARCH),y) +endif + obj-$(CONFIG_CRC64) += crc64.o +crc64-y := crc64-main.o +ifeq ($(CONFIG_CRC64_ARCH),y) +endif + obj-y += tests/ hostprogs := gen_crc32table gen_crc64table clean-files := crc32table.h crc64table.h -$(obj)/crc32.o: $(obj)/crc32table.h -$(obj)/crc64.o: $(obj)/crc64table.h +$(obj)/crc32-main.o: $(obj)/crc32table.h +$(obj)/crc64-main.o: $(obj)/crc64table.h quiet_cmd_crc32 = GEN $@ cmd_crc32 = $< > $@ quiet_cmd_crc64 = GEN $@ diff --git a/lib/crc/crc-t10dif.c b/lib/crc/crc-t10dif-main.c similarity index 78% rename from lib/crc/crc-t10dif.c rename to lib/crc/crc-t10dif-main.c index 311c2ab829f15..5d7f298c267ed 100644 --- a/lib/crc/crc-t10dif.c +++ b/lib/crc/crc-t10dif-main.c @@ -48,18 +48,41 @@ static const u16 t10_dif_crc_table[256] = { 0xA415, 0x2FA2, 0x38CC, 0xB37B, 0x1610, 0x9DA7, 0x8AC9, 0x017E, 0x1F65, 0x94D2, 0x83BC, 0x080B, 0xAD60, 0x26D7, 0x31B9, 0xBA0E, 0xF0D8, 0x7B6F, 0x6C01, 0xE7B6, 0x42DD, 0xC96A, 0xDE04, 0x55B3 }; -u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len) +static inline __maybe_unused u16 +crc_t10dif_generic(u16 crc, const u8 *p, size_t len) { - size_t i; + while (len--) + crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ *p++]; + return crc; +} - for (i = 0; i < len; i++) - crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ p[i]]; +#ifdef CONFIG_CRC_T10DIF_ARCH +#include "crc-t10dif.h" /* $(SRCARCH)/crc-t10dif.h */ +#else +#define crc_t10dif_arch crc_t10dif_generic +#endif - return crc; +u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len) +{ + return crc_t10dif_arch(crc, p, len); +} +EXPORT_SYMBOL(crc_t10dif_update); + +#ifdef crc_t10dif_mod_init_arch +static int __init crc_t10dif_mod_init(void) +{ + crc_t10dif_mod_init_arch(); + return 0; +} +subsys_initcall(crc_t10dif_mod_init); + +static void __exit crc_t10dif_mod_exit(void) +{ } -EXPORT_SYMBOL(crc_t10dif_generic); +module_exit(crc_t10dif_mod_exit); +#endif -MODULE_DESCRIPTION("T10 DIF CRC calculation"); +MODULE_DESCRIPTION("CRC-T10DIF library functions"); MODULE_LICENSE("GPL"); diff --git a/lib/crc/crc32.c b/lib/crc/crc32-main.c similarity index 73% rename from lib/crc/crc32.c rename to lib/crc/crc32-main.c index 95429861d3aca..a86b4bed0def7 100644 --- a/lib/crc/crc32.c +++ b/lib/crc/crc32-main.c @@ -29,29 +29,71 @@ #include #include #include "crc32table.h" -MODULE_AUTHOR("Matt Domsch "); -MODULE_DESCRIPTION("Various CRC32 calculations"); -MODULE_LICENSE("GPL"); - -u32 crc32_le_base(u32 crc, const u8 *p, size_t len) +static inline __maybe_unused u32 +crc32_le_base(u32 crc, const u8 *p, size_t len) { while (len--) crc = (crc >> 8) ^ crc32table_le[(crc & 255) ^ *p++]; return crc; } -EXPORT_SYMBOL(crc32_le_base); -u32 crc32c_base(u32 crc, const u8 *p, size_t len) +static inline __maybe_unused u32 +crc32_be_base(u32 crc, const u8 *p, size_t len) +{ + while (len--) + crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++]; + return crc; +} + +static inline __maybe_unused u32 +crc32c_base(u32 crc, const u8 *p, size_t len) { while (len--) crc = (crc >> 8) ^ crc32ctable_le[(crc & 255) ^ *p++]; return crc; } -EXPORT_SYMBOL(crc32c_base); + +#ifdef CONFIG_CRC32_ARCH +#include "crc32.h" /* $(SRCARCH)/crc32.h */ + +u32 crc32_optimizations(void) +{ + return crc32_optimizations_arch(); +} +EXPORT_SYMBOL(crc32_optimizations); +#endif + +#ifndef crc32_le_arch +#define crc32_le_arch crc32_le_base +#endif +#ifndef crc32_be_arch +#define crc32_be_arch crc32_be_base +#endif +#ifndef crc32c_arch +#define crc32c_arch crc32c_base +#endif + +u32 crc32_le(u32 crc, const void *p, size_t len) +{ + return crc32_le_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32_le); + +u32 crc32_be(u32 crc, const void *p, size_t len) +{ + return crc32_be_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32_be); + +u32 crc32c(u32 crc, const void *p, size_t len) +{ + return crc32c_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32c); /* * This multiplies the polynomials x and y modulo the given modulus. * This follows the "little-endian" CRC convention that the lsbit * represents the highest power of x, and the msbit represents x^0. @@ -115,12 +157,21 @@ u32 crc32_le_shift(u32 crc, size_t len) { return crc32_generic_shift(crc, len, CRC32_POLY_LE); } EXPORT_SYMBOL(crc32_le_shift); -u32 crc32_be_base(u32 crc, const u8 *p, size_t len) +#ifdef crc32_mod_init_arch +static int __init crc32_mod_init(void) { - while (len--) - crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++]; - return crc; + crc32_mod_init_arch(); + return 0; } -EXPORT_SYMBOL(crc32_be_base); +subsys_initcall(crc32_mod_init); + +static void __exit crc32_mod_exit(void) +{ +} +module_exit(crc32_mod_exit); +#endif + +MODULE_DESCRIPTION("CRC32 library functions"); +MODULE_LICENSE("GPL"); diff --git a/lib/crc/crc64.c b/lib/crc/crc64-main.c similarity index 66% rename from lib/crc/crc64.c rename to lib/crc/crc64-main.c index 5b1b17057f0ae..e4a6d879e84c3 100644 --- a/lib/crc/crc64.c +++ b/lib/crc/crc64-main.c @@ -36,23 +36,56 @@ #include #include #include #include "crc64table.h" -MODULE_DESCRIPTION("CRC64 calculations"); -MODULE_LICENSE("GPL v2"); - -u64 crc64_be_generic(u64 crc, const u8 *p, size_t len) +static inline __maybe_unused u64 +crc64_be_generic(u64 crc, const u8 *p, size_t len) { while (len--) crc = (crc << 8) ^ crc64table[(crc >> 56) ^ *p++]; return crc; } -EXPORT_SYMBOL_GPL(crc64_be_generic); -u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len) +static inline __maybe_unused u64 +crc64_nvme_generic(u64 crc, const u8 *p, size_t len) { while (len--) crc = (crc >> 8) ^ crc64nvmetable[(crc & 0xff) ^ *p++]; return crc; } -EXPORT_SYMBOL_GPL(crc64_nvme_generic); + +#ifdef CONFIG_CRC64_ARCH +#include "crc64.h" /* $(SRCARCH)/crc64.h */ +#else +#define crc64_be_arch crc64_be_generic +#define crc64_nvme_arch crc64_nvme_generic +#endif + +u64 crc64_be(u64 crc, const void *p, size_t len) +{ + return crc64_be_arch(crc, p, len); +} +EXPORT_SYMBOL_GPL(crc64_be); + +u64 crc64_nvme(u64 crc, const void *p, size_t len) +{ + return ~crc64_nvme_arch(~crc, p, len); +} +EXPORT_SYMBOL_GPL(crc64_nvme); + +#ifdef crc64_mod_init_arch +static int __init crc64_mod_init(void) +{ + crc64_mod_init_arch(); + return 0; +} +subsys_initcall(crc64_mod_init); + +static void __exit crc64_mod_exit(void) +{ +} +module_exit(crc64_mod_exit); +#endif + +MODULE_DESCRIPTION("CRC64 library functions"); +MODULE_LICENSE("GPL"); From patchwork Sun Jun 1 22:44:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893896 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DE1C23C8C9; Sun, 1 Jun 2025 22:45:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817956; cv=none; b=Bcx9TfLCuHoBnpz0LQ2KT87oXto0BjBvc73W4pcxVErKm39n09mee43+Gik38KiPze5O9MqmvKzvwLKugK+unBfoUM5/h+zaHfbV9G6YxD6RRkKk1eUK0ARLsKtGqZEzqfi+zo7JFnzj51XNm+uFxBoYRhg58BE/iBGJhP5M1wQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817956; c=relaxed/simple; bh=Ha9zZQj7Jf+ffkE3L33bjFk9KtVZ+B+rHBHCMXqGDMQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a1BaSPPBjfgQ3k4moICNRrPA3Z4WvCTwquruqSq7xfyh0jLJv+7a8upNmxGqu1rFKbDdspwgkrsOBYvmEIjvNcqft9xw9DWDFMfUBZq81nlP171sEbBbIu+ckKS1IXFLdee4r2/CKb6p3tp2aJ6CigyQVeC6Ro5IX1Kn+0vc2o8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rHB3tBZf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rHB3tBZf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 528E6C4CEF2; Sun, 1 Jun 2025 22:45:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817955; bh=Ha9zZQj7Jf+ffkE3L33bjFk9KtVZ+B+rHBHCMXqGDMQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rHB3tBZfPR0PjTAc5bEh8QvmRpTFmwcLsr5xvk1n2uvTx326V6vHeCrpZ/MFdMfEy /zcFJLTxddvTMQeRtAX9UoabfX1RmUPPO65iIVJ3kdVROHcmp+EEKa5FIHQbl4Ijk2 EqeLs/1TUVUsTxZWWtda0IJjyaN2a8crCnQPzG0CKd7cbv3rk7tCWtRTR6mQEBJlyY Jq/ZBC38H2wbeNQsfSXOVopzeOz2AdR7YRJm5lf0fL2fc2CinDRR8OgtyovH4zNaa1 YGj1BBNx8IDtrfIl0DKH/xjb7B/gY46bWJ8iCRJKwj+VZY05/kRiR81Gt+eXKqMJG+ pYYt37HfRvjiA== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 05/13] lib/crc/arm: migrate arm-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:33 -0700 Message-ID: <20250601224441.778374-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the arm-optimized CRC code from arch/arm/lib/crc* into its new location in lib/crc/arm/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/arm/Kconfig | 2 - arch/arm/lib/Makefile | 6 --- lib/crc/Kconfig | 2 + lib/crc/Makefile | 2 + .../arm/lib => lib/crc/arm}/crc-t10dif-core.S | 0 .../crc-t10dif.c => lib/crc/arm/crc-t10dif.h | 23 ++--------- {arch/arm/lib => lib/crc/arm}/crc32-core.S | 0 arch/arm/lib/crc32.c => lib/crc/arm/crc32.h | 40 +++++-------------- 8 files changed, 16 insertions(+), 59 deletions(-) rename {arch/arm/lib => lib/crc/arm}/crc-t10dif-core.S (100%) rename arch/arm/lib/crc-t10dif.c => lib/crc/arm/crc-t10dif.h (70%) rename {arch/arm/lib => lib/crc/arm}/crc32-core.S (100%) rename arch/arm/lib/crc32.c => lib/crc/arm/crc32.h (69%) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 3072731fe09c5..c531b49aa98ea 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -6,12 +6,10 @@ config ARM select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE if HAVE_KRETPROBES && FRAME_POINTER && !ARM_UNWIND select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_CACHE_LINE_SIZE if OF select ARCH_HAS_CPU_CACHE_ALIASING select ARCH_HAS_CPU_FINALIZE_INIT if MMU - select ARCH_HAS_CRC32 if KERNEL_MODE_NEON - select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DMA_ALLOC if MMU select ARCH_HAS_DMA_OPS select ARCH_HAS_DMA_WRITE_COMBINE if !ARM_DMA_MEM_BUFFERABLE diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile index 91ea0e29107af..d1b9ea2023648 100644 --- a/arch/arm/lib/Makefile +++ b/arch/arm/lib/Makefile @@ -45,11 +45,5 @@ ifeq ($(CONFIG_KERNEL_MODE_NEON),y) CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o endif obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o - -obj-$(CONFIG_CRC32_ARCH) += crc32-arm.o -crc32-arm-y := crc32.o crc32-core.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm.o -crc-t10dif-arm-y := crc-t10dif.o crc-t10dif-core.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 85b0dc774d722..b34a52cdba5d7 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -48,10 +48,11 @@ config ARCH_HAS_CRC_T10DIF bool config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS + default y if ARM && KERNEL_MODE_NEON config CRC32 tristate select BITREVERSE help @@ -59,10 +60,11 @@ config CRC32 the functions from or . config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS + default y if ARM && KERNEL_MODE_NEON config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 2f39385f97a5c..1069edb81a481 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -12,15 +12,17 @@ obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) +crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) +crc32-$(CONFIG_ARM) += arm/crc32-core.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/arm/lib/crc-t10dif-core.S b/lib/crc/arm/crc-t10dif-core.S similarity index 100% rename from arch/arm/lib/crc-t10dif-core.S rename to lib/crc/arm/crc-t10dif-core.S diff --git a/arch/arm/lib/crc-t10dif.c b/lib/crc/arm/crc-t10dif.h similarity index 70% rename from arch/arm/lib/crc-t10dif.c rename to lib/crc/arm/crc-t10dif.h index 1093f8ec13b0b..2edf7e9681d05 100644 --- a/arch/arm/lib/crc-t10dif.c +++ b/lib/crc/arm/crc-t10dif.h @@ -3,16 +3,10 @@ * Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions * * Copyright (C) 2016 Linaro Ltd */ -#include -#include -#include -#include -#include - #include #include #include @@ -23,11 +17,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull); asmlinkage u16 crc_t10dif_pmull64(u16 init_crc, const u8 *buf, size_t len); asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u8 *buf, size_t len, u8 out[16]); -u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) { if (static_branch_likely(&have_pmull)) { if (crypto_simd_usable()) { kernel_neon_begin(); @@ -47,26 +41,15 @@ u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) return crc_t10dif_generic(0, buf, sizeof(buf)); } } return crc_t10dif_generic(crc, data, length); } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_arm_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (elf_hwcap & HWCAP_NEON) { static_branch_enable(&have_neon); if (elf_hwcap2 & HWCAP2_PMULL) static_branch_enable(&have_pmull); } - return 0; } -subsys_initcall(crc_t10dif_arm_init); - -static void __exit crc_t10dif_arm_exit(void) -{ -} -module_exit(crc_t10dif_arm_exit); - -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_DESCRIPTION("Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions"); -MODULE_LICENSE("GPL v2"); diff --git a/arch/arm/lib/crc32-core.S b/lib/crc/arm/crc32-core.S similarity index 100% rename from arch/arm/lib/crc32-core.S rename to lib/crc/arm/crc32-core.S diff --git a/arch/arm/lib/crc32.c b/lib/crc/arm/crc32.h similarity index 69% rename from arch/arm/lib/crc32.c rename to lib/crc/arm/crc32.h index f2bef8849c7c3..19e6e35f6418f 100644 --- a/arch/arm/lib/crc32.c +++ b/lib/crc/arm/crc32.h @@ -4,15 +4,10 @@ * * Copyright (C) 2016 Linaro Ltd */ #include -#include -#include -#include -#include -#include #include #include #include @@ -27,18 +22,19 @@ asmlinkage u32 crc32_pmull_le(const u8 buf[], u32 len, u32 init_crc); asmlinkage u32 crc32_armv8_le(u32 init_crc, const u8 buf[], u32 len); asmlinkage u32 crc32c_pmull_le(const u8 buf[], u32 len, u32 init_crc); asmlinkage u32 crc32c_armv8_le(u32 init_crc, const u8 buf[], u32 len); -static u32 crc32_le_scalar(u32 crc, const u8 *p, size_t len) +static inline u32 crc32_le_scalar(u32 crc, const u8 *p, size_t len) { if (static_branch_likely(&have_crc32)) return crc32_armv8_le(crc, p, len); return crc32_le_base(crc, p, len); } -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (len >= PMULL_MIN_LEN + 15 && static_branch_likely(&have_pmull) && crypto_simd_usable()) { size_t n = -(uintptr_t)p & 15; @@ -55,20 +51,20 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) p += n; len -= n; } return crc32_le_scalar(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); -static u32 crc32c_scalar(u32 crc, const u8 *p, size_t len) +static inline u32 crc32c_scalar(u32 crc, const u8 *p, size_t len) { if (static_branch_likely(&have_crc32)) return crc32c_armv8_le(crc, p, len); return crc32c_base(crc, p, len); } -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (len >= PMULL_MIN_LEN + 15 && static_branch_likely(&have_pmull) && crypto_simd_usable()) { size_t n = -(uintptr_t)p & 15; @@ -85,39 +81,21 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) p += n; len -= n; } return crc32c_scalar(crc, p, len); } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); - -static int __init crc32_arm_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (elf_hwcap2 & HWCAP2_CRC32) static_branch_enable(&have_crc32); if (elf_hwcap2 & HWCAP2_PMULL) static_branch_enable(&have_pmull); - return 0; } -subsys_initcall(crc32_arm_init); -static void __exit crc32_arm_exit(void) -{ -} -module_exit(crc32_arm_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (elf_hwcap2 & (HWCAP2_CRC32 | HWCAP2_PMULL)) return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_DESCRIPTION("Accelerated CRC32(C) using ARM CRC, NEON and Crypto Extensions"); -MODULE_LICENSE("GPL v2"); From patchwork Sun Jun 1 22:44:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893693 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BECBB23D293; Sun, 1 Jun 2025 22:45:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817956; cv=none; b=rlOqnjDJ9JMMTFcWQnt7p2aAjgqgEDWhZChEOrzWJ9tlZ7d99XuKUCITMyXH4kgCgWVRcITFzNsiY9gBxJmWt0AdArF+5vVL1+907UTOZic0VUi9hHu4Dr4blVJI6y9NnEijAeb9RoPVqwvNBpz/tpROR79q55+aYFUZcU9ifYI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817956; c=relaxed/simple; bh=nj3FLjV0XRklapAo1b/4rm9s4AcmV3Wa+beXZi58AgQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YKmfRF9EE3phHhtXMfyNcI5AZksAbZEeRMGEXR7kWTl1ez3LhTXT611u8+0KVDw8UOKDsTqt2jBrMwuD0Pms5l3NlP48qG39uGUfCInbWf0MYrlGZ6+3OHhX4r/9HTGtsy6O55w0AY/LsOOQ/qiyujf0Jglj5wLBIniX0Q6W9Z8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gQuMmCvJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gQuMmCvJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DD52BC4AF0B; Sun, 1 Jun 2025 22:45:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817956; bh=nj3FLjV0XRklapAo1b/4rm9s4AcmV3Wa+beXZi58AgQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gQuMmCvJpvdry21LC124O06TKecBmc+QGeLcYmJlcVawgJ1lYXH480V5TxG/AuNWw KXwFOUsyTnl3M8hgCtaze/Byag/idw16bQUAoeK2P83pS+sM6KLa59k7IO1fcx28op zdzolOGKz/mDyEQJIyXgjZsUJfdZkZMMCQPVulnUZD75QEiDEc6QnBiQCXoaW15N2h 39aHzXNxJK4KZPhhLx4mTUeMCcoPn+ENVuWD0N9APL/V7J5HvxviRomiEVT8CxTkSZ zeU98/4bdMCOBn0D+TpKm1RHJAAJWXrnUpwVKfJo727nURTvQsQMB/Uvf2RuWr0W1K GAVO4dE2gXCjA== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 06/13] lib/crc/arm64: migrate arm64-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:34 -0700 Message-ID: <20250601224441.778374-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the arm64-optimized CRC code from arch/arm64/lib/crc* into its new location in lib/crc/arm64/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/arm64/Kconfig | 2 -- arch/arm64/lib/Makefile | 6 ----- lib/crc/Kconfig | 2 ++ lib/crc/Makefile | 2 ++ .../lib => lib/crc/arm64}/crc-t10dif-core.S | 0 .../crc/arm64/crc-t10dif.h | 22 +++---------------- .../arm64/lib => lib/crc/arm64}/crc32-core.S | 0 .../lib/crc32.c => lib/crc/arm64/crc32.h | 22 ++++++------------- 8 files changed, 14 insertions(+), 42 deletions(-) rename {arch/arm64/lib => lib/crc/arm64}/crc-t10dif-core.S (100%) rename arch/arm64/lib/crc-t10dif.c => lib/crc/arm64/crc-t10dif.h (70%) rename {arch/arm64/lib => lib/crc/arm64}/crc32-core.S (100%) rename arch/arm64/lib/crc32.c => lib/crc/arm64/crc32.h (81%) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 55fc331af3371..cbeac225903f7 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -19,12 +19,10 @@ config ARM64 select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CC_PLATFORM - select ARCH_HAS_CRC32 - select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DMA_OPS if XEN select ARCH_HAS_DMA_PREP_COHERENT diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 027bfa9689c6a..9b255d9332479 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -14,16 +14,10 @@ CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU) endif lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o -obj-$(CONFIG_CRC32_ARCH) += crc32-arm64.o -crc32-arm64-y := crc32.o crc32-core.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm64.o -crc-t10dif-arm64-y := crc-t10dif.o crc-t10dif-core.o - obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index b34a52cdba5d7..663f4dca2bda1 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -49,10 +49,11 @@ config ARCH_HAS_CRC_T10DIF config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON + default y if ARM64 && KERNEL_MODE_NEON config CRC32 tristate select BITREVERSE help @@ -61,10 +62,11 @@ config CRC32 config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON + default y if ARM64 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 1069edb81a481..822d18b57b432 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -13,16 +13,18 @@ obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o +crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) crc32-$(CONFIG_ARM) += arm/crc32-core.o +crc32-$(CONFIG_ARM64) += arm64/crc32-core.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/arm64/lib/crc-t10dif-core.S b/lib/crc/arm64/crc-t10dif-core.S similarity index 100% rename from arch/arm64/lib/crc-t10dif-core.S rename to lib/crc/arm64/crc-t10dif-core.S diff --git a/arch/arm64/lib/crc-t10dif.c b/lib/crc/arm64/crc-t10dif.h similarity index 70% rename from arch/arm64/lib/crc-t10dif.c rename to lib/crc/arm64/crc-t10dif.h index c2ffe4fdb59d1..c4521a7f1ee9b 100644 --- a/arch/arm64/lib/crc-t10dif.c +++ b/lib/crc/arm64/crc-t10dif.h @@ -4,15 +4,10 @@ * * Copyright (C) 2016 - 2017 Linaro Ltd */ #include -#include -#include -#include -#include -#include #include #include #include @@ -24,11 +19,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull); asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len, u8 out[16]); asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); -u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) { if (static_branch_likely(&have_pmull)) { if (crypto_simd_usable()) { kernel_neon_begin(); @@ -48,26 +43,15 @@ u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) return crc_t10dif_generic(0, buf, sizeof(buf)); } } return crc_t10dif_generic(crc, data, length); } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_arm64_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (cpu_have_named_feature(ASIMD)) { static_branch_enable(&have_asimd); if (cpu_have_named_feature(PMULL)) static_branch_enable(&have_pmull); } - return 0; } -subsys_initcall(crc_t10dif_arm64_init); - -static void __exit crc_t10dif_arm64_exit(void) -{ -} -module_exit(crc_t10dif_arm64_exit); - -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_DESCRIPTION("CRC-T10DIF using arm64 NEON and Crypto Extensions"); -MODULE_LICENSE("GPL v2"); diff --git a/arch/arm64/lib/crc32-core.S b/lib/crc/arm64/crc32-core.S similarity index 100% rename from arch/arm64/lib/crc32-core.S rename to lib/crc/arm64/crc32-core.S diff --git a/arch/arm64/lib/crc32.c b/lib/crc/arm64/crc32.h similarity index 81% rename from arch/arm64/lib/crc32.c rename to lib/crc/arm64/crc32.h index ed3acd71178f8..98f3b3fcc729a 100644 --- a/arch/arm64/lib/crc32.c +++ b/lib/crc/arm64/crc32.h @@ -1,11 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only -#include -#include -#include - #include #include #include #include @@ -20,11 +16,12 @@ asmlinkage u32 crc32_be_arm64(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32_le_arm64_4way(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32c_le_arm64_4way(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32_be_arm64_4way(u32 crc, unsigned char const *p, size_t len); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_le_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -39,13 +36,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32_le_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32c_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -60,13 +57,13 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32c_le_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) +#define crc32_be_arch crc32_be_arch +static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_be_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -81,19 +78,14 @@ u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32_be_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32_be_arch); -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (alternative_has_cap_likely(ARM64_HAS_CRC32)) return CRC32_LE_OPTIMIZATION | CRC32_BE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("arm64-optimized CRC32 functions"); From patchwork Sun Jun 1 22:44:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893895 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06EC523D2A9; Sun, 1 Jun 2025 22:45:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817957; cv=none; b=qhTsD9dfFzev4wOA3+UNFyfMCxeH8SpKQ/f3fmhD78Sa9JsSHdF3kMz1ZIIOX8NEANUlvqz2qvR4lWlazbkarcPFcoovrgAGvhMt3uzLdm9+as8qaZHFsf2Yw1pTL/aaU0nBklPkh0P1iFK5rgtZMssIwAw/3+m6GYoInqW/otc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817957; c=relaxed/simple; bh=yoI90iVlquVCitVGa7RjvDmSsdHmtg6em7U4Tn0qZw0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Zy8W5MN5D6C5vqUVw9TQq8MnvaQ5AO0ATxb5N36EJ0a3lYPauK2TiZOfOqqlIr7GXoE2MFuzQje/qIXCX/OxX2nGw6m/a+GYSPU8vUxMXNCmITxjzZ9VNwqrSRRmlm8At3aBEF/rpa6kiFmT0nexeSajx3A9inodW1O1gUufxLc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=X/+oa40f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="X/+oa40f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74559C4CEE7; Sun, 1 Jun 2025 22:45:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817956; bh=yoI90iVlquVCitVGa7RjvDmSsdHmtg6em7U4Tn0qZw0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=X/+oa40fhs3WXlKJD4saBc72E23kv2NhgbcXplwEgCHTSx9q7IEqr1ccshOJ7tgnA 1wvSNnBXanAW0tzJBW0i77u5/lfiDGRSmDSaAEtDLDXaFSfcXR7uEipZlWltGULHKG vmLfT92usKuxNR1te1C5Gj9xrmv8ttsPpoa7oOuSrgYSTO/zWghx+tMvMSyzzhyquP 8bQkBn5G93f6oPP7ZVPETOrTsXb9Nx6g01+InRcz1r/Zt6Z2giIRsqn2+I5unoCbmU Vydkoc1ew8KwpGCw7V2LLbC2a6+CesUEmCU4a0ixOpuS42FGYNReQ3EqOsXAG3DoDU Qol9Wooapbftw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 07/13] lib/crc/loongarch: migrate loongarch-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:35 -0700 Message-ID: <20250601224441.778374-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the loongarch-optimized CRC code from arch/loongarch/lib/crc* into its new location in lib/crc/loongarch/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/loongarch/Kconfig | 1 - arch/loongarch/lib/Makefile | 2 -- lib/crc/Kconfig | 1 + .../crc/loongarch/crc32.h | 34 ++++--------------- 4 files changed, 8 insertions(+), 30 deletions(-) rename arch/loongarch/lib/crc32-loongarch.c => lib/crc/loongarch/crc32.h (71%) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 1a2cf012b8f2f..1b19893a6bdf0 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -13,11 +13,10 @@ config LOONGARCH select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_CPU_FINALIZE_INIT - select ARCH_HAS_CRC32 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile index fae77809048b8..ccea3bbd43531 100644 --- a/arch/loongarch/lib/Makefile +++ b/arch/loongarch/lib/Makefile @@ -9,7 +9,5 @@ lib-y += delay.o memset.o memcpy.o memmove.o \ obj-$(CONFIG_ARCH_SUPPORTS_INT128) += tishift.o obj-$(CONFIG_CPU_HAS_LSX) += xor_simd.o xor_simd_glue.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o - -obj-$(CONFIG_CRC32_ARCH) += crc32-loongarch.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 663f4dca2bda1..ee320ae19ca85 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -63,10 +63,11 @@ config CRC32 config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 + default y if LOONGARCH config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/arch/loongarch/lib/crc32-loongarch.c b/lib/crc/loongarch/crc32.h similarity index 71% rename from arch/loongarch/lib/crc32-loongarch.c rename to lib/crc/loongarch/crc32.h index b37cd8537b459..d5aff7ca96ae9 100644 --- a/arch/loongarch/lib/crc32-loongarch.c +++ b/lib/crc/loongarch/crc32.h @@ -8,12 +8,10 @@ * Copyright (C) 2018 MIPS Tech, LLC * Copyright (C) 2020-2023 Loongson Technology Corporation Limited */ #include -#include -#include #include #define _CRC32(crc, value, size, type) \ do { \ __asm__ __volatile__( \ @@ -26,11 +24,12 @@ do { \ #define CRC32(crc, value, size) _CRC32(crc, value, size, crc) #define CRC32C(crc, value, size) _CRC32(crc, value, size, crcc) static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (!static_branch_likely(&have_crc32)) return crc32_le_base(crc, p, len); while (len >= sizeof(u64)) { @@ -61,13 +60,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) CRC32(crc, value, b); } return crc; } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (!static_branch_likely(&have_crc32)) return crc32c_base(crc, p, len); while (len >= sizeof(u64)) { @@ -98,38 +97,19 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) CRC32C(crc, value, b); } return crc; } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); - -static int __init crc32_loongarch_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (cpu_has_crc32) static_branch_enable(&have_crc32); - return 0; } -subsys_initcall(crc32_loongarch_init); -static void __exit crc32_loongarch_exit(void) -{ -} -module_exit(crc32_loongarch_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_crc32)) return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Min Zhou "); -MODULE_AUTHOR("Huacai Chen "); -MODULE_DESCRIPTION("CRC32 and CRC32C using LoongArch crc* instructions"); -MODULE_LICENSE("GPL v2"); From patchwork Sun Jun 1 22:44:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893692 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 920B423E340; Sun, 1 Jun 2025 22:45:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817957; cv=none; b=Xt0LUP/3KxYOQQ72Q06ku0+ihwdcGVf/xvTfzt5MpCFbCo+8Rkd4FDIIjb+vc04/ri5aNO3gQ/2oI4zKrnsEXbcNJautwUdm8w9sjw32BYpzeQcrT02U09Mgku/P66x9Rtu9TrCAL65CF5nkShHT91qhU1BzZhB3US5iORoYvVw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817957; c=relaxed/simple; bh=apKGde3VrZfQelJPg198iFKP+frhjwkkcGBF4f9uuBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i7maCPgLJVRTOgcSI32J+9hPHsv1kFK6UjoiNvgHXxevJmzyzoHVWj7a2Y/k2zALcUBdcgoGGZU8ljVKl/bx+EuBbrH9r3TEXm7HEkpWrORpA7hdLN79Gwn63YRsBJod6yQNkImku21CcCkemwkXHPr3913sSAtEUVkBN8BXBcg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Zn1ngrvi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Zn1ngrvi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C14AC4CEF3; Sun, 1 Jun 2025 22:45:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817957; bh=apKGde3VrZfQelJPg198iFKP+frhjwkkcGBF4f9uuBU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zn1ngrvibz2nkinD2PEEbiJAzT/lFfENVLFa/KQveCqdfEdnd0LNqEHp1YsV2Tpab l1tIAoLh4EkWGv2pnHwWdbpFO0M+dewFCrtd7LjIhfBn5dI74xg3dyGUvNq4vCJxGC FpbEjQus9K/L1HVXHA1zAjwWWafUYsmUWv+CihYUpDXp1JmqxKIjMoKUaWdNknUGQP +3QhNRmQOb3F6RBhkO4xhbkae7c5y7ACq3ir0zqlg2qipWGqyD2kfcoDvd1J9L3ul5 DsZnfo+4xcVTRfOZkyD0T0W1xMq+5AjdKylLJoTZk+pjAHcGDsSJYL6FIOChhzBtDW 2hb95ofQgtrgQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 08/13] lib/crc/mips: migrate mips-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:36 -0700 Message-ID: <20250601224441.778374-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the mips-optimized CRC code from arch/mips/lib/crc* into its new location in lib/crc/mips/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/mips/Kconfig | 1 - arch/mips/lib/Makefile | 2 -- lib/crc/Kconfig | 1 + .../lib/crc32-mips.c => lib/crc/mips/crc32.h | 35 ++++--------------- 4 files changed, 8 insertions(+), 31 deletions(-) rename arch/mips/lib/crc32-mips.c => lib/crc/mips/crc32.h (82%) diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index fc0772c1bad4a..31d6975674fd6 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -1995,11 +1995,10 @@ config CPU_MIPSR5 select MIPS_SPRAM config CPU_MIPSR6 bool default y if CPU_MIPS32_R6 || CPU_MIPS64_R6 - select ARCH_HAS_CRC32 select CPU_HAS_RIXI select CPU_HAS_DIEI if !CPU_DIEI_BROKEN select HAVE_ARCH_BITREVERSE select MIPS_ASID_BITS_VARIABLE select MIPS_SPRAM diff --git a/arch/mips/lib/Makefile b/arch/mips/lib/Makefile index 9d75845ef78e1..8c40ffb09c420 100644 --- a/arch/mips/lib/Makefile +++ b/arch/mips/lib/Makefile @@ -14,9 +14,7 @@ obj-$(CONFIG_PCI) += iomap-pci.o lib-$(CONFIG_GENERIC_CSUM) := $(filter-out csum_partial.o, $(lib-y)) obj-$(CONFIG_CPU_GENERIC_DUMP_TLB) += dump_tlb.o obj-$(CONFIG_CPU_R3000) += r3k_dump_tlb.o -obj-$(CONFIG_CRC32_ARCH) += crc32-mips.o - # libgcc-style stuff needed in the kernel obj-y += bswapsi.o bswapdi.o multi3.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index ee320ae19ca85..2d99aab4f838d 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -64,10 +64,11 @@ config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 default y if LOONGARCH + default y if MIPS && CPU_MIPSR6 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/arch/mips/lib/crc32-mips.c b/lib/crc/mips/crc32.h similarity index 82% rename from arch/mips/lib/crc32-mips.c rename to lib/crc/mips/crc32.h index 45e4d2c9fbf54..aa7f3228b7db3 100644 --- a/arch/mips/lib/crc32-mips.c +++ b/lib/crc/mips/crc32.h @@ -7,14 +7,10 @@ * Copyright (C) 2014 Linaro Ltd * Copyright (C) 2018 MIPS Tech, LLC */ #include -#include -#include -#include -#include #include #include #ifndef TOOLCHAIN_SUPPORTS_CRC #define _ASM_SET_CRC(OP, SZ, TYPE) \ @@ -62,11 +58,12 @@ do { \ #define CRC32C(crc, value, size) \ _CRC32(crc, value, size, crc32c) static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (!static_branch_likely(&have_crc32)) return crc32_le_base(crc, p, len); if (IS_ENABLED(CONFIG_64BIT)) { @@ -104,13 +101,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) CRC32(crc, value, b); } return crc; } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (!static_branch_likely(&have_crc32)) return crc32c_base(crc, p, len); if (IS_ENABLED(CONFIG_64BIT)) { @@ -147,37 +144,19 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) CRC32C(crc, value, b); } return crc; } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); - -static int __init crc32_mips_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (cpu_have_feature(cpu_feature(MIPS_CRC32))) static_branch_enable(&have_crc32); - return 0; } -subsys_initcall(crc32_mips_init); -static void __exit crc32_mips_exit(void) -{ -} -module_exit(crc32_mips_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_crc32)) return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Marcin Nowakowski X-Patchwork-Id: 893894 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8151423FC42; Sun, 1 Jun 2025 22:45:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817958; cv=none; b=NdZqc+Eq40I9zyS0JV5R7/7HqLvc1Aiiy4oNYMmW7/0DFXRgKOhVxcQJgdZBrzMwfol1PnWFOW8fuWbFT36xtxsTHGUuJ0p/s6V5NpoyYKR5SXWeC1JfQdpX7v/yTZaDjnA3GPxarvWeJ6NGoxLL4dL9NxlQYXtXlG9vPvLxZ7w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817958; c=relaxed/simple; bh=SXwip4eijDXULIBpid24Mjyf9ZlXG+23/x0n84S5DAk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b6zr4F7kFrVCJ9caZkojoeotdiel8kUt9BW3u+V3dOsGak3+MpuQuBc5KmOSXFb9fMMZIBKx5hUas6/HQ5IxMu6eZw7v+a66JuSZGOVE8gz1E9Acn+t2bB1nLf/D75DyEn9ldy33l3bOUKbP7Wf0BrhosgTjOmthGfdmE39g0hk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SkcxPLic; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SkcxPLic" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9758AC4CEF2; Sun, 1 Jun 2025 22:45:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817958; bh=SXwip4eijDXULIBpid24Mjyf9ZlXG+23/x0n84S5DAk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SkcxPLiceM33E7E96La+Jj3NmG09ByoGJwAyDmGTHuuoMFKvExCxns2BraB9NC4px edHT+i+zdVvE0NC6Kr+pxdYF/3AKCj71SQb0LFyU2SLf9Fmgn3MsDYx2+IfPzo9lJN q+lXZVn5BZKTFwywGqXTm1Q5iWMP+MNAViCdl+DD0DngiCHV6W85iA7D9XT4B0bIwf OatYgBTRIzW4ynuA09ap11PYHKXQ5ANged2/+MJzxs6ywXaAsmBdOivRxVBHtERHI/ aif6vRW8+rXKwFkn70nVDS76Q+7gxsvTiNEPMFoFuRydXmrbL2BTE16BcwhR7QAhiq +aquta5JE/QNA== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 09/13] lib/crc/powerpc: migrate powerpc-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:37 -0700 Message-ID: <20250601224441.778374-10-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the powerpc-optimized CRC code from arch/powerpc/lib/crc* into its new location in lib/crc/powerpc/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/powerpc/Kconfig | 2 -- arch/powerpc/lib/Makefile | 6 ---- lib/crc/Kconfig | 2 ++ lib/crc/Makefile | 2 ++ .../crc/powerpc/crc-t10dif.h | 20 ++--------- .../crc/powerpc}/crc-vpmsum-template.S | 0 .../lib/crc32.c => lib/crc/powerpc/crc32.h | 36 +++---------------- .../crc/powerpc}/crc32c-vpmsum_asm.S | 0 .../crc/powerpc}/crct10dif-vpmsum_asm.S | 0 9 files changed, 12 insertions(+), 56 deletions(-) rename arch/powerpc/lib/crc-t10dif.c => lib/crc/powerpc/crc-t10dif.h (75%) rename {arch/powerpc/lib => lib/crc/powerpc}/crc-vpmsum-template.S (100%) rename arch/powerpc/lib/crc32.c => lib/crc/powerpc/crc32.h (64%) rename {arch/powerpc/lib => lib/crc/powerpc}/crc32c-vpmsum_asm.S (100%) rename {arch/powerpc/lib => lib/crc/powerpc}/crct10dif-vpmsum_asm.S (100%) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c3e0cc83f1205..45b4fa7b9b02f 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -125,12 +125,10 @@ config PPC select ARCH_DISABLE_KASAN_INLINE if PPC_RADIX_MMU select ARCH_DMA_DEFAULT_COHERENT if !NOT_COHERENT_CACHE select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_HAS_COPY_MC if PPC64 - select ARCH_HAS_CRC32 if PPC64 && ALTIVEC - select ARCH_HAS_CRC_T10DIF if PPC64 && ALTIVEC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX if STRICT_KERNEL_RWX select ARCH_HAS_DEVMEM_IS_ALLOWED diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index 481f968e42c7b..59de2e2232df6 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -78,12 +78,6 @@ obj-$(CONFIG_FTR_FIXUP_SELFTEST) += feature-fixups-test.o obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o CFLAGS_xor_vmx.o += -mhard-float -maltivec $(call cc-option,-mabi=altivec) # Enable CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include) -obj-$(CONFIG_CRC32_ARCH) += crc32-powerpc.o -crc32-powerpc-y := crc32.o crc32c-vpmsum_asm.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-powerpc.o -crc-t10dif-powerpc-y := crc-t10dif.o crct10dif-vpmsum_asm.o - obj-$(CONFIG_PPC64) += $(obj64-y) diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 2d99aab4f838d..06611610e478f 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -50,10 +50,11 @@ config ARCH_HAS_CRC_T10DIF config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON + default y if PPC64 && ALTIVEC config CRC32 tristate select BITREVERSE help @@ -65,10 +66,11 @@ config CRC32_ARCH depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 default y if LOONGARCH default y if MIPS && CPU_MIPSR6 + default y if PPC64 && ALTIVEC config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 822d18b57b432..f9109b71c1264 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -14,17 +14,19 @@ obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o +crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o +crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/powerpc/lib/crc-t10dif.c b/lib/crc/powerpc/crc-t10dif.h similarity index 75% rename from arch/powerpc/lib/crc-t10dif.c rename to lib/crc/powerpc/crc-t10dif.h index be23ded3a9df6..59e16804a6eae 100644 --- a/arch/powerpc/lib/crc-t10dif.c +++ b/lib/crc/powerpc/crc-t10dif.h @@ -7,14 +7,11 @@ */ #include #include #include -#include #include -#include -#include #include #include #define VMX_ALIGN 16 #define VMX_ALIGN_MASK (VMX_ALIGN-1) @@ -23,11 +20,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto); u32 __crct10dif_vpmsum(u32 crc, unsigned char const *p, size_t len); -u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) +static inline u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) { unsigned int prealign; unsigned int tail; u32 crc = crci; @@ -60,24 +57,13 @@ u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) crc = crc_t10dif_generic(crc, p, tail); } return crc & 0xffff; } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_powerpc_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (cpu_has_feature(CPU_FTR_ARCH_207S) && (cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO)) static_branch_enable(&have_vec_crypto); - return 0; } -subsys_initcall(crc_t10dif_powerpc_init); - -static void __exit crc_t10dif_powerpc_exit(void) -{ -} -module_exit(crc_t10dif_powerpc_exit); - -MODULE_AUTHOR("Daniel Axtens "); -MODULE_DESCRIPTION("CRCT10DIF using vector polynomial multiply-sum instructions"); -MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/lib/crc-vpmsum-template.S b/lib/crc/powerpc/crc-vpmsum-template.S similarity index 100% rename from arch/powerpc/lib/crc-vpmsum-template.S rename to lib/crc/powerpc/crc-vpmsum-template.S diff --git a/arch/powerpc/lib/crc32.c b/lib/crc/powerpc/crc32.h similarity index 64% rename from arch/powerpc/lib/crc32.c rename to lib/crc/powerpc/crc32.h index 0d9befb6e7b83..0dceb04d0ee94 100644 --- a/arch/powerpc/lib/crc32.c +++ b/lib/crc/powerpc/crc32.h @@ -1,13 +1,10 @@ // SPDX-License-Identifier: GPL-2.0-only #include #include #include -#include #include -#include -#include #include #include #define VMX_ALIGN 16 #define VMX_ALIGN_MASK (VMX_ALIGN-1) @@ -16,17 +13,12 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto); u32 __crc32c_vpmsum(u32 crc, const u8 *p, size_t len); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_le_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_le_arch); - -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { unsigned int prealign; unsigned int tail; if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || @@ -56,38 +48,20 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) crc = crc32c_base(crc, p, tail); } return crc; } -EXPORT_SYMBOL(crc32c_arch); - -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); -static int __init crc32_powerpc_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (cpu_has_feature(CPU_FTR_ARCH_207S) && (cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO)) static_branch_enable(&have_vec_crypto); - return 0; } -subsys_initcall(crc32_powerpc_init); -static void __exit crc32_powerpc_exit(void) -{ -} -module_exit(crc32_powerpc_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_vec_crypto)) return CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Anton Blanchard "); -MODULE_DESCRIPTION("CRC32C using vector polynomial multiply-sum instructions"); -MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/lib/crc32c-vpmsum_asm.S b/lib/crc/powerpc/crc32c-vpmsum_asm.S similarity index 100% rename from arch/powerpc/lib/crc32c-vpmsum_asm.S rename to lib/crc/powerpc/crc32c-vpmsum_asm.S diff --git a/arch/powerpc/lib/crct10dif-vpmsum_asm.S b/lib/crc/powerpc/crct10dif-vpmsum_asm.S similarity index 100% rename from arch/powerpc/lib/crct10dif-vpmsum_asm.S rename to lib/crc/powerpc/crct10dif-vpmsum_asm.S From patchwork Sun Jun 1 22:44:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893691 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FF2124113D; Sun, 1 Jun 2025 22:45:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817959; cv=none; b=k1mT33I3AUdjQdiaK48rtA4v3qb8Nauo2sODhMFcJLwS7fYdTvuNPFC03UcQTGwU0XQH9QYi/yLkaySO27hhpGW0DKhfpSJS/lospsPODA9pl/Lk3y/EWEFbHLfk/oOIhWXMupgyRET5/6Zey5UOULBZVUx0F0H//lYbYtPW08E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817959; c=relaxed/simple; bh=bR2IghF04qIfhGMxr7Z+GJfdUQx5LDT1V3vKPSH6Ff4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W2XMeuhKjk9BZ3Wkg7vPU6Y2MI7NIcDGgqumHxQqxqHBDtzdVFtv9cZOcc7NyOMobP/1PFIf1TJ1OU52dm0IvZ0TgEZ+y2hMmxrEJr2ZCxpIHFIySpiaxYYPKl5Bon105OtLjTAMca36mSNZYIUHbLGDGRdME5ekvGrUuSB8gyg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IBTRtWHj; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IBTRtWHj" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E2A3C4CEE7; Sun, 1 Jun 2025 22:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817958; bh=bR2IghF04qIfhGMxr7Z+GJfdUQx5LDT1V3vKPSH6Ff4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IBTRtWHjJ+jupFNO5g3cD6rMnOkFHFGZEMgzNOQ70Q8P8oHYwGGfZUtLjvyOYSiUH i7F2k/afwYgujvklX7H3sBOkqdbEigqlSrolUXN9v5cZ20fMYzKz+4BPXLJh05KOlv Lwl3NZBBcJArPhpitLTXLQcgY0kYIVL5vTxMKoHYr8gpH6ddjOA86+p2iaLEcVMNLu u4zEv21rLTdRnWQM9lijRqZ8BUfa+4S4kyFoIzQ5DqUWMOoqYptStpb+fJAzPqBJqU +yzl6RyX+gtETuNHxzn4wBaoVnvmKElnDhwDkePobv4PTgLbyjf+0m9A39mPF5QDMP zurRH1JGaUr4g== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 10/13] lib/crc/riscv: migrate riscv-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:38 -0700 Message-ID: <20250601224441.778374-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the riscv-optimized CRC code from arch/riscv/lib/crc* into its new location in lib/crc/riscv/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/riscv/Kconfig | 3 --- arch/riscv/lib/Makefile | 6 ------ lib/crc/Kconfig | 3 +++ lib/crc/Makefile | 3 +++ .../lib => lib/crc/riscv}/crc-clmul-consts.h | 0 .../crc/riscv}/crc-clmul-template.h | 0 {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h | 0 .../crc/riscv/crc-t10dif.h | 8 +------- {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c | 0 .../lib/crc32.c => lib/crc/riscv/crc32.h | 20 +++++++------------ {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c | 0 {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c | 0 .../lib/crc64.c => lib/crc/riscv/crc64.h | 11 ++-------- {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c | 0 {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c | 0 15 files changed, 16 insertions(+), 38 deletions(-) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-consts.h (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-template.h (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h (100%) rename arch/riscv/lib/crc-t10dif.c => lib/crc/riscv/crc-t10dif.h (62%) rename {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c (100%) rename arch/riscv/lib/crc32.c => lib/crc/riscv/crc32.h (66%) rename {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c (100%) rename arch/riscv/lib/crc64.c => lib/crc/riscv/crc64.h (65%) rename {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c (100%) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bbec87b793099..b533d9e5086a9 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -22,13 +22,10 @@ config RISCV select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT - select ARCH_HAS_CRC32 if RISCV_ISA_ZBC - select ARCH_HAS_CRC64 if 64BIT && RISCV_ISA_ZBC - select ARCH_HAS_CRC_T10DIF if RISCV_ISA_ZBC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX select ARCH_HAS_FAST_MULTIPLIER diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 0baec92d2f55b..a4f4b48ed3a47 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -14,14 +14,8 @@ ifeq ($(CONFIG_MMU), y) lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o endif lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o -obj-$(CONFIG_CRC32_ARCH) += crc32-riscv.o -crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o -obj-$(CONFIG_CRC64_ARCH) += crc64-riscv.o -crc64-riscv-y := crc64.o crc64_msb.o crc64_lsb.o -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-riscv.o -crc-t10dif-riscv-y := crc-t10dif.o crc16_msb.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 06611610e478f..4044e08cb9739 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -51,10 +51,11 @@ config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON default y if PPC64 && ALTIVEC + default y if RISCV && RISCV_ISA_ZBC config CRC32 tristate select BITREVERSE help @@ -67,20 +68,22 @@ config CRC32_ARCH default y if ARM && KERNEL_MODE_NEON default y if ARM64 default y if LOONGARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC + default y if RISCV && RISCV_ISA_ZBC config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of the functions from . config CRC64_ARCH bool depends on CRC64 && CRC_OPTIMIZATIONS + default y if RISCV && RISCV_ISA_ZBC && 64BIT config CRC_OPTIMIZATIONS bool "Enable optimized CRC implementations" if EXPERT default y help diff --git a/lib/crc/Makefile b/lib/crc/Makefile index f9109b71c1264..e46b35a2ffc04 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -15,23 +15,26 @@ obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o +crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o +crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) +crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o endif obj-y += tests/ hostprogs := gen_crc32table gen_crc64table diff --git a/arch/riscv/lib/crc-clmul-consts.h b/lib/crc/riscv/crc-clmul-consts.h similarity index 100% rename from arch/riscv/lib/crc-clmul-consts.h rename to lib/crc/riscv/crc-clmul-consts.h diff --git a/arch/riscv/lib/crc-clmul-template.h b/lib/crc/riscv/crc-clmul-template.h similarity index 100% rename from arch/riscv/lib/crc-clmul-template.h rename to lib/crc/riscv/crc-clmul-template.h diff --git a/arch/riscv/lib/crc-clmul.h b/lib/crc/riscv/crc-clmul.h similarity index 100% rename from arch/riscv/lib/crc-clmul.h rename to lib/crc/riscv/crc-clmul.h diff --git a/arch/riscv/lib/crc-t10dif.c b/lib/crc/riscv/crc-t10dif.h similarity index 62% rename from arch/riscv/lib/crc-t10dif.c rename to lib/crc/riscv/crc-t10dif.h index e6b0051ccd86c..cd6136cbfda1c 100644 --- a/arch/riscv/lib/crc-t10dif.c +++ b/lib/crc/riscv/crc-t10dif.h @@ -5,20 +5,14 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc16_msb_clmul(crc, p, len, &crc16_msb_0x8bb7_consts); return crc_t10dif_generic(crc, p, len); } -EXPORT_SYMBOL(crc_t10dif_arch); - -MODULE_DESCRIPTION("RISC-V optimized CRC-T10DIF function"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc16_msb.c b/lib/crc/riscv/crc16_msb.c similarity index 100% rename from arch/riscv/lib/crc16_msb.c rename to lib/crc/riscv/crc16_msb.c diff --git a/arch/riscv/lib/crc32.c b/lib/crc/riscv/crc32.h similarity index 66% rename from arch/riscv/lib/crc32.c rename to lib/crc/riscv/crc32.h index a3188b7d9c403..08c33357d25cd 100644 --- a/arch/riscv/lib/crc32.c +++ b/lib/crc/riscv/crc32.h @@ -5,49 +5,43 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_lsb_clmul(crc, p, len, &crc32_lsb_0xedb88320_consts); return crc32_le_base(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) +#define crc32_be_arch crc32_be_arch +static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_msb_clmul(crc, p, len, &crc32_msb_0x04c11db7_consts); return crc32_be_base(crc, p, len); } -EXPORT_SYMBOL(crc32_be_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_lsb_clmul(crc, p, len, &crc32_lsb_0x82f63b78_consts); return crc32c_base(crc, p, len); } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return CRC32_LE_OPTIMIZATION | CRC32_BE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_DESCRIPTION("RISC-V optimized CRC32 functions"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc32_lsb.c b/lib/crc/riscv/crc32_lsb.c similarity index 100% rename from arch/riscv/lib/crc32_lsb.c rename to lib/crc/riscv/crc32_lsb.c diff --git a/arch/riscv/lib/crc32_msb.c b/lib/crc/riscv/crc32_msb.c similarity index 100% rename from arch/riscv/lib/crc32_msb.c rename to lib/crc/riscv/crc32_msb.c diff --git a/arch/riscv/lib/crc64.c b/lib/crc/riscv/crc64.h similarity index 65% rename from arch/riscv/lib/crc64.c rename to lib/crc/riscv/crc64.h index f0015a27836a4..a1b7873fde579 100644 --- a/arch/riscv/lib/crc64.c +++ b/lib/crc/riscv/crc64.h @@ -5,30 +5,23 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc64_msb_clmul(crc, p, len, &crc64_msb_0x42f0e1eba9ea3693_consts); return crc64_be_generic(crc, p, len); } -EXPORT_SYMBOL(crc64_be_arch); -u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc64_lsb_clmul(crc, p, len, &crc64_lsb_0x9a6c9329ac4bc9b5_consts); return crc64_nvme_generic(crc, p, len); } -EXPORT_SYMBOL(crc64_nvme_arch); - -MODULE_DESCRIPTION("RISC-V optimized CRC64 functions"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc64_lsb.c b/lib/crc/riscv/crc64_lsb.c similarity index 100% rename from arch/riscv/lib/crc64_lsb.c rename to lib/crc/riscv/crc64_lsb.c diff --git a/arch/riscv/lib/crc64_msb.c b/lib/crc/riscv/crc64_msb.c similarity index 100% rename from arch/riscv/lib/crc64_msb.c rename to lib/crc/riscv/crc64_msb.c From patchwork Sun Jun 1 22:44:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893893 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B68642417DE; Sun, 1 Jun 2025 22:45:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817959; cv=none; b=LYh8YxXWl1dDnwEo00EarVMe8LyXx7fF5B+xpszkAQdZsfHj1tzf5OyzF8yJTAFxxGKMTu4B98zzcc/TTjNnOw3skTUKrqxtLscPPwTXmPbOwqbZIrA5adBHOlOcW6Rq3Yf1Er1TecuEaH9EUevdN82sOtJUU3C2JRfOt4pXg/I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817959; c=relaxed/simple; bh=95cP740x9TPPdMrMRbzvOoF5hVV7Q6oui6O4DOa47nw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WxNBYn5li6hlRR+3dRf028oANyMHw91fRCoI3QQbPd3If0rbRM4mY//Al8NzGcC4NpoBuTA5cr5R20NfmN9KA/6gjZ9zON0p0uDMNzQvZQAOeTl6ssMhDUvAhO6Buh4+IDDZAlEAsyAvD9iHjaljTAKygFnhXzMUGCCUBMAY7zU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Mdw3TZwa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mdw3TZwa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B9938C4CEEE; Sun, 1 Jun 2025 22:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817959; bh=95cP740x9TPPdMrMRbzvOoF5hVV7Q6oui6O4DOa47nw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mdw3TZwasG6f4rAglr1W91xGPN+a4inpfdi2b0qnvd7sKuMz9nr1Dz+qGvhQhh6gh xPNU9z0LYQsyYV+JMiJan+IzMAWP4ZneVRqL8AzeD7GAv5eAoAOJ83DeAyF1As75RV USJO6+9kwlBew6TXngvXkWDl+9O/NyMuvI9riqJF6IWY+Dg4zSt5U4h80tHCB8I19W 4eWjLZ7k/moC2/rWv0ODHhbCnbPubEirpTbZXddZEA5g65ZOi3RgPzzKFT+PE6OXyN wrry2sFGzXhj6iprl7KYKXJ7VY+Im19So2xVHQe8nBBBXsjSaPdriP8g2OPUQWqMxx cilq53LID01vg== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 11/13] lib/crc/s390: migrate s390-optimized CRC code into lib/s390/ Date: Sun, 1 Jun 2025 15:44:39 -0700 Message-ID: <20250601224441.778374-12-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the s390-optimized CRC code from arch/s390/lib/crc* into its new location in lib/crc/s390/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/s390/Kconfig | 1 - arch/s390/lib/Makefile | 3 --- lib/crc/Kconfig | 1 + lib/crc/Makefile | 1 + {arch/s390/lib => lib/crc/s390}/crc32-vx.h | 0 arch/s390/lib/crc32.c => lib/crc/s390/crc32.h | 19 ++++++------------- {arch/s390/lib => lib/crc/s390}/crc32be-vx.c | 0 {arch/s390/lib => lib/crc/s390}/crc32le-vx.c | 0 8 files changed, 8 insertions(+), 17 deletions(-) rename {arch/s390/lib => lib/crc/s390}/crc32-vx.h (100%) rename arch/s390/lib/crc32.c => lib/crc/s390/crc32.h (81%) rename {arch/s390/lib => lib/crc/s390}/crc32be-vx.c (100%) rename {arch/s390/lib => lib/crc/s390}/crc32le-vx.c (100%) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 0c16dc443e2f6..22b90f6aa1a09 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -73,11 +73,10 @@ config S390 select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_HAS_CPU_FINALIZE_INIT - select ARCH_HAS_CRC32 select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX select ARCH_HAS_DEVMEM_IS_ALLOWED diff --git a/arch/s390/lib/Makefile b/arch/s390/lib/Makefile index cd35cdbfa8713..7c8583d46eca1 100644 --- a/arch/s390/lib/Makefile +++ b/arch/s390/lib/Makefile @@ -23,8 +23,5 @@ obj-$(CONFIG_S390_MODULES_SANITY_TEST) += test_modules.o obj-$(CONFIG_S390_MODULES_SANITY_TEST_HELPERS) += test_modules_helpers.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_EXPOLINE_EXTERN) += expoline.o - -obj-$(CONFIG_CRC32_ARCH) += crc32-s390.o -crc32-s390-y := crc32.o crc32le-vx.o crc32be-vx.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 4044e08cb9739..1ed6e11bef909 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -69,10 +69,11 @@ config CRC32_ARCH default y if ARM64 default y if LOONGARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC default y if RISCV && RISCV_ISA_ZBC + default y if S390 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index e46b35a2ffc04..1b83262b6987f 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -25,10 +25,11 @@ crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o +crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/s390/lib/crc32-vx.h b/lib/crc/s390/crc32-vx.h similarity index 100% rename from arch/s390/lib/crc32-vx.h rename to lib/crc/s390/crc32-vx.h diff --git a/arch/s390/lib/crc32.c b/lib/crc/s390/crc32.h similarity index 81% rename from arch/s390/lib/crc32.c rename to lib/crc/s390/crc32.h index 3c4b344417c11..8dbb07b9ea64c 100644 --- a/arch/s390/lib/crc32.c +++ b/lib/crc/s390/crc32.h @@ -3,16 +3,12 @@ * CRC-32 implemented with the z/Architecture Vector Extension Facility. * * Copyright IBM Corp. 2015 * Author(s): Hendrik Brueckner */ -#define KMSG_COMPONENT "crc32-vx" -#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt -#include #include -#include #include #include "crc32-vx.h" #define VX_MIN_LEN 64 #define VX_ALIGNMENT 16L @@ -25,11 +21,11 @@ * on the message buffer, the hardware-accelerated or software implementation * is used. Note that the message buffer is aligned to improve fetch * operations of VECTOR LOAD MULTIPLE instructions. */ #define DEFINE_CRC32_VX(___fname, ___crc32_vx, ___crc32_sw) \ - u32 ___fname(u32 crc, const u8 *data, size_t datalen) \ + static inline u32 ___fname(u32 crc, const u8 *data, size_t datalen) \ { \ unsigned long prealign, aligned, remaining; \ DECLARE_KERNEL_FPU_ONSTACK16(vxstate); \ \ if (datalen < VX_MIN_LEN + VX_ALIGN_MASK || !cpu_has_vx()) \ @@ -52,26 +48,23 @@ \ if (remaining) \ crc = ___crc32_sw(crc, data + aligned, remaining); \ \ return crc; \ - } \ - EXPORT_SYMBOL(___fname); + } +#define crc32_le_arch crc32_le_arch +#define crc32_be_arch crc32_be_arch +#define crc32c_arch crc32c_arch DEFINE_CRC32_VX(crc32_le_arch, crc32_le_vgfm_16, crc32_le_base) DEFINE_CRC32_VX(crc32_be_arch, crc32_be_vgfm_16, crc32_be_base) DEFINE_CRC32_VX(crc32c_arch, crc32c_le_vgfm_16, crc32c_base) -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (cpu_has_vx()) { return CRC32_LE_OPTIMIZATION | CRC32_BE_OPTIMIZATION | CRC32C_OPTIMIZATION; } return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Hendrik Brueckner "); -MODULE_DESCRIPTION("CRC-32 algorithms using z/Architecture Vector Extension Facility"); -MODULE_LICENSE("GPL"); diff --git a/arch/s390/lib/crc32be-vx.c b/lib/crc/s390/crc32be-vx.c similarity index 100% rename from arch/s390/lib/crc32be-vx.c rename to lib/crc/s390/crc32be-vx.c diff --git a/arch/s390/lib/crc32le-vx.c b/lib/crc/s390/crc32le-vx.c similarity index 100% rename from arch/s390/lib/crc32le-vx.c rename to lib/crc/s390/crc32le-vx.c From patchwork Sun Jun 1 22:44:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893690 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF49F241CA2; Sun, 1 Jun 2025 22:45:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817960; cv=none; b=WI1rIcfK8GRaSeGbg8PNtR6wYo9X9yZOOZKa4a4SmYe9NsSOZkQQoSWP+krXT56xl1tz8S8OxzOEUuOIMVsryDURR6y4Ybw4F23vxZCw7pAQtohEMUjdpBXWf+p9zd+oheNEe1V4d1v6wGm8rJz6Z069/ASBdxKDeDd0X5MvqHk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817960; c=relaxed/simple; bh=UTN9XJ3QNZMywLZq7NN5DWhOFROMdgLy2Y339vtWDb8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DPiwmLD0IrTAl3LDEOtnNZQXigJN3WRGdGlkqGCKPSPHeZstQOT3S0Jzt3Ky2KrOqKKJJvLKpscxnKMTfgo/q6k2oc9PDziBo5W9fHkfwLnrqHNgOJbhg8t1SP8UWzdGRHy4o0hP2htWYZacXyCu6+eRhHlQV+NsclXtU9HC0tA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QD27px5Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QD27px5Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 515FCC113CF; Sun, 1 Jun 2025 22:45:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817959; bh=UTN9XJ3QNZMywLZq7NN5DWhOFROMdgLy2Y339vtWDb8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QD27px5Y0+vwJcrCeIbV55N71S4ZUr2XHWbP/XxCLtPJPhpG3ODiMR1LOk5H8cgoT 5YQhq3ncwecKLxMzjM7LwH+dS1eCqaDl8vivncFNmMKvTLkZL9+NwKbTKOhhDZmat8 UfMmERij63jXqXkITGzIOYws4XBcJkjANeGRIkfh1Koit5MfKuYDzLs7NwpcPimj5I WSiWiysCjb1r4ccuZQOGBIO9vYxSCCfmC8TlFFSbAHWh4BZeO16c3mCc2RwT3F15ss tWhCmu6DEoGr2sO5Pkw8yp9DkP2dvVlhEoPp/tH97ZmjyC8mvMAPzmBSfkBgpAWUpQ 5whbFrzYkW5pQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 12/13] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:40 -0700 Message-ID: <20250601224441.778374-13-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the sparc-optimized CRC code from arch/sparc/lib/crc* into its new location in lib/crc/sparc/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/sparc/Kconfig | 1 - arch/sparc/lib/Makefile | 2 - lib/crc/Kconfig | 1 + lib/crc/Makefile | 1 + .../lib/crc32.c => lib/crc/sparc/crc32.h | 40 ++++--------------- .../sparc/lib => lib/crc/sparc}/crc32c_asm.S | 0 6 files changed, 9 insertions(+), 36 deletions(-) rename arch/sparc/lib/crc32.c => lib/crc/sparc/crc32.h (62%) rename {arch/sparc/lib => lib/crc/sparc}/crc32c_asm.S (100%) diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index 0f88123925a4f..dcfdb7f1dae97 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -108,11 +108,10 @@ config SPARC64 select ARCH_HAS_GIGANTIC_PAGE select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_SETUP_PER_CPU_AREA select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK - select ARCH_HAS_CRC32 config ARCH_PROC_KCORE_TEXT def_bool y config CPU_BIG_ENDIAN diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile index 5cf9781d68b40..2d6c3c5352734 100644 --- a/arch/sparc/lib/Makefile +++ b/arch/sparc/lib/Makefile @@ -52,7 +52,5 @@ lib-$(CONFIG_SPARC64) += copy_in_user.o memmove.o lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o xor.o hweight.o ffs.o obj-$(CONFIG_SPARC64) += iomap.o obj-$(CONFIG_SPARC32) += atomic32.o obj-$(CONFIG_SPARC64) += PeeCeeI.o -obj-$(CONFIG_CRC32_ARCH) += crc32-sparc.o -crc32-sparc-y := crc32.o crc32c_asm.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 1ed6e11bef909..c0b36d03b66dc 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -70,10 +70,11 @@ config CRC32_ARCH default y if LOONGARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC default y if RISCV && RISCV_ISA_ZBC default y if S390 + default y if SPARC64 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 1b83262b6987f..37dd4a776d1b4 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -26,10 +26,11 @@ ifeq ($(CONFIG_CRC32_ARCH),y) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o +crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/sparc/lib/crc32.c b/lib/crc/sparc/crc32.h similarity index 62% rename from arch/sparc/lib/crc32.c rename to lib/crc/sparc/crc32.h index 40d4720a42a1b..43d52e60b2960 100644 --- a/arch/sparc/lib/crc32.c +++ b/lib/crc/sparc/crc32.h @@ -8,28 +8,19 @@ * Kent Liu */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt -#include -#include -#include -#include #include #include static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32c_opcode); -u32 crc32_le_arch(u32 crc, const u8 *data, size_t len) -{ - return crc32_le_base(crc, data, len); -} -EXPORT_SYMBOL(crc32_le_arch); - void crc32c_sparc64(u32 *crcp, const u64 *data, size_t len); -u32 crc32c_arch(u32 crc, const u8 *data, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *data, size_t len) { size_t n = -(uintptr_t)data & 7; if (!static_branch_likely(&have_crc32c_opcode)) return crc32c_base(crc, data, len); @@ -49,45 +40,28 @@ u32 crc32c_arch(u32 crc, const u8 *data, size_t len) } if (len) crc = crc32c_base(crc, data, len); return crc; } -EXPORT_SYMBOL(crc32c_arch); - -u32 crc32_be_arch(u32 crc, const u8 *data, size_t len) -{ - return crc32_be_base(crc, data, len); -} -EXPORT_SYMBOL(crc32_be_arch); -static int __init crc32_sparc_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { unsigned long cfr; if (!(sparc64_elf_hwcap & HWCAP_SPARC_CRYPTO)) - return 0; + return; __asm__ __volatile__("rd %%asr26, %0" : "=r" (cfr)); if (!(cfr & CFR_CRC32C)) - return 0; + return; static_branch_enable(&have_crc32c_opcode); pr_info("Using sparc64 crc32c opcode optimized CRC32C implementation\n"); - return 0; } -subsys_initcall(crc32_sparc_init); -static void __exit crc32_sparc_exit(void) -{ -} -module_exit(crc32_sparc_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_crc32c_opcode)) return CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("CRC32c (Castagnoli), sparc64 crc32c opcode accelerated"); diff --git a/arch/sparc/lib/crc32c_asm.S b/lib/crc/sparc/crc32c_asm.S similarity index 100% rename from arch/sparc/lib/crc32c_asm.S rename to lib/crc/sparc/crc32c_asm.S From patchwork Sun Jun 1 22:44:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 893892 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3A4B242D92; Sun, 1 Jun 2025 22:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817961; cv=none; b=hKyLlSXN7zdYDbx3m/TZrt07H8wrJQeihzc5SHqbv/AJcCk/n/YaNeuhgPhpRSOmNojAMsOUbBOwsFeaO6BEfZCxy/0Nc2VWLp8B9TaBjewFmI/Q3MCm12cPmsh4MNn4rPaf/sYEzXI3cv2hefRwz6APIKdD6lIxLIjcTiVkXI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748817961; c=relaxed/simple; bh=JFnuSJVBOx88DMSk46a713Dd7WU0y0iy82Y5QlU+awI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nZszpNUjz2rFxmy0elEqL3NM5TqDUOrW9ampbDP/OVfj4RCGaAY+MZrM/a86BuDpXt91/RVI/d/HJ4HS4VGJ9az5bL8Z7YHlhAddpLOPdmiLbhWuRECROLXm2OxpfOvg/V4rU28pvEqzqlIL8v1XmDCWGYd27pAsYyXPVBLsXt8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WPtQlrtz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WPtQlrtz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF565C4CEF1; Sun, 1 Jun 2025 22:45:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748817960; bh=JFnuSJVBOx88DMSk46a713Dd7WU0y0iy82Y5QlU+awI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WPtQlrtz+SOyfkvQlUT25fbJdsAG45DaKURZPBmEbUlNAdPQL1iEzQ5yBA6ZW8+OJ VOU07tHdhZ66y4c0E0aA4PPTlXpYOa/+BcxIkze2c3+DJHGx9/13mxUvalZ4JA8ATG PO2VrwAEbkC9JpYloX1IkCS/erReit+leFHG+izBIir2KH3/sS4kcWO3BtGGIM768h Z07tc46YzrUFaHvmHHgdDHszbonUQwkM3dGu1Y4onVHQ4hQkLr6aoLHG7dXcSYkCy/ KGnLCOFFVCQyNrm4rNbDgzLKGgnjfUZUuKwXU7y1Cls2FQBKrLrBWBQosSphhK2mR5 +g1IR748+TWcw== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 13/13] lib/crc/x86: migrate x86-optimized CRC code into lib/crc/ Date: Sun, 1 Jun 2025 15:44:41 -0700 Message-ID: <20250601224441.778374-14-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250601224441.778374-1-ebiggers@kernel.org> References: <20250601224441.778374-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the x86-optimized CRC code from arch/x86/lib/crc* into its new location in lib/crc/x86/, and wire it up in the new way. For a detailed explanation of why this change is being made, see the commit that introduced the new way of integrating arch-specific code into lib/crc/. Signed-off-by: Eric Biggers --- arch/x86/Kconfig | 3 -- arch/x86/lib/Makefile | 10 ------ lib/crc/Kconfig | 3 ++ lib/crc/Makefile | 4 +++ .../lib => lib/crc/x86}/crc-pclmul-consts.h | 0 .../lib => lib/crc/x86}/crc-pclmul-template.S | 0 .../lib => lib/crc/x86}/crc-pclmul-template.h | 0 .../crc-t10dif.c => lib/crc/x86/crc-t10dif.h | 18 ++--------- .../lib => lib/crc/x86}/crc16-msb-pclmul.S | 0 {arch/x86/lib => lib/crc/x86}/crc32-pclmul.S | 0 arch/x86/lib/crc32.c => lib/crc/x86/crc32.h | 32 ++++--------------- {arch/x86/lib => lib/crc/x86}/crc32c-3way.S | 0 {arch/x86/lib => lib/crc/x86}/crc64-pclmul.S | 0 arch/x86/lib/crc64.c => lib/crc/x86/crc64.h | 21 +++--------- 14 files changed, 21 insertions(+), 70 deletions(-) rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-consts.h (100%) rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.S (100%) rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.h (100%) rename arch/x86/lib/crc-t10dif.c => lib/crc/x86/crc-t10dif.h (56%) rename {arch/x86/lib => lib/crc/x86}/crc16-msb-pclmul.S (100%) rename {arch/x86/lib => lib/crc/x86}/crc32-pclmul.S (100%) rename arch/x86/lib/crc32.c => lib/crc/x86/crc32.h (76%) rename {arch/x86/lib => lib/crc/x86}/crc32c-3way.S (100%) rename {arch/x86/lib => lib/crc/x86}/crc64-pclmul.S (100%) rename arch/x86/lib/crc64.c => lib/crc/x86/crc64.h (61%) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 340e5468980e0..6181f414f87ec 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -77,13 +77,10 @@ config X86 select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CPU_PASID if IOMMU_SVA - select ARCH_HAS_CRC32 - select ARCH_HAS_CRC64 if X86_64 - select ARCH_HAS_CRC_T10DIF select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE if !X86_PAE select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index 4fa5c4e1ba8a0..dc5ee2a6938c4 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -38,20 +38,10 @@ lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc.o copy_mc_64.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_MITIGATION_RETPOLINE) += retpoline.o -obj-$(CONFIG_CRC32_ARCH) += crc32-x86.o -crc32-x86-y := crc32.o crc32-pclmul.o -crc32-x86-$(CONFIG_64BIT) += crc32c-3way.o - -obj-$(CONFIG_CRC64_ARCH) += crc64-x86.o -crc64-x86-y := crc64.o crc64-pclmul.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-x86.o -crc-t10dif-x86-y := crc-t10dif.o crc16-msb-pclmul.o - obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o obj-y += iomem.o ifeq ($(CONFIG_X86_32),y) obj-y += atomic64_32.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index c0b36d03b66dc..450c8290cbc1f 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -52,10 +52,11 @@ config CRC_T10DIF_ARCH depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON default y if PPC64 && ALTIVEC default y if RISCV && RISCV_ISA_ZBC + default y if X86 config CRC32 tristate select BITREVERSE help @@ -71,10 +72,11 @@ config CRC32_ARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC default y if RISCV && RISCV_ISA_ZBC default y if S390 default y if SPARC64 + default y if X86 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of @@ -82,10 +84,11 @@ config CRC64 config CRC64_ARCH bool depends on CRC64 && CRC_OPTIMIZATIONS default y if RISCV && RISCV_ISA_ZBC && 64BIT + default y if X86_64 config CRC_OPTIMIZATIONS bool "Enable optimized CRC implementations" if EXPERT default y help diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 37dd4a776d1b4..80cfc94f5180d 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -16,10 +16,11 @@ crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o +crc-t10dif-$(CONFIG_X86) += x86/crc16-msb-pclmul.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) @@ -27,16 +28,19 @@ crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o +crc32-$(CONFIG_X86) += x86/crc32-pclmul.o +crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o +crc64-$(CONFIG_X86) += x86/crc64-pclmul.o endif obj-y += tests/ hostprogs := gen_crc32table gen_crc64table diff --git a/arch/x86/lib/crc-pclmul-consts.h b/lib/crc/x86/crc-pclmul-consts.h similarity index 100% rename from arch/x86/lib/crc-pclmul-consts.h rename to lib/crc/x86/crc-pclmul-consts.h diff --git a/arch/x86/lib/crc-pclmul-template.S b/lib/crc/x86/crc-pclmul-template.S similarity index 100% rename from arch/x86/lib/crc-pclmul-template.S rename to lib/crc/x86/crc-pclmul-template.S diff --git a/arch/x86/lib/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-template.h similarity index 100% rename from arch/x86/lib/crc-pclmul-template.h rename to lib/crc/x86/crc-pclmul-template.h diff --git a/arch/x86/lib/crc-t10dif.c b/lib/crc/x86/crc-t10dif.h similarity index 56% rename from arch/x86/lib/crc-t10dif.c rename to lib/crc/x86/crc-t10dif.h index db7ce59c31ace..eb1f23db4daa1 100644 --- a/arch/x86/lib/crc-t10dif.c +++ b/lib/crc/x86/crc-t10dif.h @@ -3,38 +3,26 @@ * CRC-T10DIF using [V]PCLMULQDQ instructions * * Copyright 2024 Google LLC */ -#include -#include #include "crc-pclmul-template.h" static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq); DECLARE_CRC_PCLMUL_FUNCS(crc16_msb, u16); -u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) { CRC_PCLMUL(crc, p, len, crc16_msb, crc16_msb_0x8bb7_consts, have_pclmulqdq); return crc_t10dif_generic(crc, p, len); } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_x86_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) { static_branch_enable(&have_pclmulqdq); INIT_CRC_PCLMUL(crc16_msb); } - return 0; } -subsys_initcall(crc_t10dif_x86_init); - -static void __exit crc_t10dif_x86_exit(void) -{ -} -module_exit(crc_t10dif_x86_exit); - -MODULE_DESCRIPTION("CRC-T10DIF using [V]PCLMULQDQ instructions"); -MODULE_LICENSE("GPL"); diff --git a/arch/x86/lib/crc16-msb-pclmul.S b/lib/crc/x86/crc16-msb-pclmul.S similarity index 100% rename from arch/x86/lib/crc16-msb-pclmul.S rename to lib/crc/x86/crc16-msb-pclmul.S diff --git a/arch/x86/lib/crc32-pclmul.S b/lib/crc/x86/crc32-pclmul.S similarity index 100% rename from arch/x86/lib/crc32-pclmul.S rename to lib/crc/x86/crc32-pclmul.S diff --git a/arch/x86/lib/crc32.c b/lib/crc/x86/crc32.h similarity index 76% rename from arch/x86/lib/crc32.c rename to lib/crc/x86/crc32.h index d09343e2cea93..ad7fc96fd6bc9 100644 --- a/arch/x86/lib/crc32.c +++ b/lib/crc/x86/crc32.h @@ -5,26 +5,24 @@ * Copyright (C) 2008 Intel Corporation * Copyright 2012 Xyratex Technology Limited * Copyright 2024 Google LLC */ -#include -#include #include "crc-pclmul-template.h" static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq); DECLARE_CRC_PCLMUL_FUNCS(crc32_lsb, u32); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +#define crc32_le_arch crc32_le_arch +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { CRC_PCLMUL(crc, p, len, crc32_lsb, crc32_lsb_0xedb88320_consts, have_pclmulqdq); return crc32_le_base(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); #ifdef CONFIG_X86_64 #define CRC32_INST "crc32q %1, %q0" #else #define CRC32_INST "crc32l %1, %0" @@ -36,11 +34,12 @@ EXPORT_SYMBOL(crc32_le_arch); */ #define CRC32C_PCLMUL_BREAKEVEN 512 asmlinkage u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +#define crc32c_arch crc32c_arch +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { size_t num_longs; if (!static_branch_likely(&have_crc32)) return crc32c_base(crc, p, len); @@ -68,44 +67,27 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) if (len & 1) asm("crc32b %1, %0" : "+r" (crc) : ASM_INPUT_RM (*p)); return crc; } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); - -static int __init crc32_x86_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_XMM4_2)) static_branch_enable(&have_crc32); if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) { static_branch_enable(&have_pclmulqdq); INIT_CRC_PCLMUL(crc32_lsb); } - return 0; } -subsys_initcall(crc32_x86_init); -static void __exit crc32_x86_exit(void) -{ -} -module_exit(crc32_x86_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { u32 optimizations = 0; if (static_key_enabled(&have_crc32)) optimizations |= CRC32C_OPTIMIZATION; if (static_key_enabled(&have_pclmulqdq)) optimizations |= CRC32_LE_OPTIMIZATION; return optimizations; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_DESCRIPTION("x86-optimized CRC32 functions"); -MODULE_LICENSE("GPL"); diff --git a/arch/x86/lib/crc32c-3way.S b/lib/crc/x86/crc32c-3way.S similarity index 100% rename from arch/x86/lib/crc32c-3way.S rename to lib/crc/x86/crc32c-3way.S diff --git a/arch/x86/lib/crc64-pclmul.S b/lib/crc/x86/crc64-pclmul.S similarity index 100% rename from arch/x86/lib/crc64-pclmul.S rename to lib/crc/x86/crc64-pclmul.S diff --git a/arch/x86/lib/crc64.c b/lib/crc/x86/crc64.h similarity index 61% rename from arch/x86/lib/crc64.c rename to lib/crc/x86/crc64.h index 351a09f5813e2..54aca3a9475c9 100644 --- a/arch/x86/lib/crc64.c +++ b/lib/crc/x86/crc64.h @@ -3,48 +3,35 @@ * CRC64 using [V]PCLMULQDQ instructions * * Copyright 2025 Google LLC */ -#include -#include #include "crc-pclmul-template.h" static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq); DECLARE_CRC_PCLMUL_FUNCS(crc64_msb, u64); DECLARE_CRC_PCLMUL_FUNCS(crc64_lsb, u64); -u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) { CRC_PCLMUL(crc, p, len, crc64_msb, crc64_msb_0x42f0e1eba9ea3693_consts, have_pclmulqdq); return crc64_be_generic(crc, p, len); } -EXPORT_SYMBOL_GPL(crc64_be_arch); -u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) { CRC_PCLMUL(crc, p, len, crc64_lsb, crc64_lsb_0x9a6c9329ac4bc9b5_consts, have_pclmulqdq); return crc64_nvme_generic(crc, p, len); } -EXPORT_SYMBOL_GPL(crc64_nvme_arch); -static int __init crc64_x86_init(void) +#define crc64_mod_init_arch crc64_mod_init_arch +static inline void crc64_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) { static_branch_enable(&have_pclmulqdq); INIT_CRC_PCLMUL(crc64_msb); INIT_CRC_PCLMUL(crc64_lsb); } - return 0; } -subsys_initcall(crc64_x86_init); - -static void __exit crc64_x86_exit(void) -{ -} -module_exit(crc64_x86_exit); - -MODULE_DESCRIPTION("CRC64 using [V]PCLMULQDQ instructions"); -MODULE_LICENSE("GPL");