From patchwork Sat Jun 7 20:04:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894845 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7991218821; Sat, 7 Jun 2025 20:07:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326849; cv=none; b=JIfWYoKCBddwnCEuc3Nd5eXeeCVVEvvCBcypij4hrjvUdfdyUotpaVCndBjjY7V0lZq+j4fpqV+Ig+7iUwT1TZXZcjl9dtSEMCgulDKHkACa67LC7+uOZVnT2Fq0gKkSwZCC51MObShiSLTdxi5lX5Wj568nlTm5rNLbYC8sEiM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326849; c=relaxed/simple; bh=yLUp/jnRDPRJYe2ElHSUe7azlBkiNfeDfNdSKo75+q8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b8hITkiMIFeh3VJ0CX5/a8aSX/sTM88uyGYLWxtiKj9EMd5HAC3sBhueRemT8eekGoiYnYAlVqosbRVDmCt38JkZRoVo32v944YLl6U0oJOvdcRB33jIJCYXh53v+54wZ+kIgVjpAKjoBUXnq1tKDPk651uzobHsfgEXpwnespE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f1FhhuAi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f1FhhuAi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8B85C4CEF4; Sat, 7 Jun 2025 20:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326849; bh=yLUp/jnRDPRJYe2ElHSUe7azlBkiNfeDfNdSKo75+q8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f1FhhuAiTrZ3n1+2RxOIsHAMGSJ/VjobP7NJfQNN0j/NBFbwNgGFUyusJmRjkKbXS sqWh1Vqgcop8SjWbs6LXUgNLslH9vJAaJtRSlUYVblMBS0RiVdEhJomtVaMvtV8ZOa PEHblLWQdMquqQZGguOsB3PGpaG1D6iMt3eqwjzhDYP/ljmb10RAveSRuslbu0TNpU aXvLzJN+nKuiGXJjGUMUMH4iyFPeV5RkG3XLHj627tTQZ05KelNWJFE8NxxO5DTTAN /pAmDXiq2dwYEPh4cjU29Xt59sFxxmxlJaZ8E/hzV8XcP6IZjjcVaxjEm3WyRFm4Z6 8pchieYF/5jfg== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 02/12] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/ Date: Sat, 7 Jun 2025 13:04:44 -0700 Message-ID: <20250607200454.73587-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Rework how lib/crc/ supports arch-optimized code. First, instead of the arch-optimized CRC code being in arch/$(SRCARCH)/lib/, it will now be in lib/crc/$(SRCARCH)/. Second, the API functions (e.g. crc32c()), arch-optimized functions (e.g. crc32c_arch()), and generic functions (e.g. crc32c_base()) will now be part of a single module for each CRC type, allowing better inlining and dead code elimination. The second change is made possible by the first. As an example, consider CONFIG_CRC32=m on x86. We'll now have just crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules were already coupled together and always both got loaded together via direct symbol dependency, so the separation provided no benefit. Note: later I'd like to apply the same design to lib/crypto/ too, where often the API functions are out-of-line so this will work even better. In those cases, for each algorithm we currently have 3 modules all coupled together, e.g. libsha256.ko, libsha256-generic.ko, and sha256-x86.ko. We should have just one, inline things properly, and rely on the compiler's dead code elimination to decide the inclusion of the generic code instead of manually setting it via kconfig. Having arch-specific code outside arch/ was somewhat controversial when Zinc proposed it back in 2018. But I don't think the concerns are warranted. It's better from a technical perspective, as it enables the improvements mentioned above. This model is already successfully used in other places in the kernel such as lib/raid6/. The community of each architecture still remains free to work on the code, even if it's not in arch/. At the time there was also a desire to put the library code in the same files as the old-school crypto API, but that was a mistake; now that the library is separate, that's no longer a constraint either. Signed-off-by: Eric Biggers --- Documentation/core-api/kernel-api.rst | 2 +- MAINTAINERS | 1 - include/linux/crc-t10dif.h | 10 +-- include/linux/crc32.h | 30 +-------- include/linux/crc64.h | 19 +----- lib/crc/Kconfig | 12 ++-- lib/crc/Makefile | 20 +++++- lib/crc/{crc-t10dif.c => crc-t10dif-main.c} | 37 ++++++++--- lib/crc/{crc32.c => crc32-main.c} | 69 +++++++++++++++++---- lib/crc/{crc64.c => crc64-main.c} | 47 +++++++++++--- 10 files changed, 158 insertions(+), 89 deletions(-) rename lib/crc/{crc-t10dif.c => crc-t10dif-main.c} (78%) rename lib/crc/{crc32.c => crc32-main.c} (58%) rename lib/crc/{crc64.c => crc64-main.c} (66%) diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index c4642d9f13a9c..9c8370891a39b 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -146,11 +146,11 @@ CRC Functions :export: .. kernel-doc:: lib/crc/crc16.c :export: -.. kernel-doc:: lib/crc/crc32.c +.. kernel-doc:: lib/crc/crc32-main.c .. kernel-doc:: lib/crc/crc-ccitt.c :export: .. kernel-doc:: lib/crc/crc-itu-t.c diff --git a/MAINTAINERS b/MAINTAINERS index 05ff204b016c5..a729d80594557 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6356,11 +6356,10 @@ M: Eric Biggers R: Ard Biesheuvel L: linux-crypto@vger.kernel.org S: Maintained T: git https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crc-next F: Documentation/staging/crc* -F: arch/*/lib/crc* F: include/linux/crc* F: lib/crc/ F: scripts/gen-crc-consts.py CREATIVE SB0540 diff --git a/include/linux/crc-t10dif.h b/include/linux/crc-t10dif.h index a559fdff3f7e2..ecc8bc2dd7f4c 100644 --- a/include/linux/crc-t10dif.h +++ b/include/linux/crc-t10dif.h @@ -2,19 +2,11 @@ #ifndef _LINUX_CRC_T10DIF_H #define _LINUX_CRC_T10DIF_H #include -u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len); -u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len); - -static inline u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC_T10DIF_ARCH)) - return crc_t10dif_arch(crc, p, len); - return crc_t10dif_generic(crc, p, len); -} +u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len); static inline u16 crc_t10dif(const u8 *p, size_t len) { return crc_t10dif_update(0, p, len); } diff --git a/include/linux/crc32.h b/include/linux/crc32.h index 36bbc0405aa04..22dbe7144eb44 100644 --- a/include/linux/crc32.h +++ b/include/linux/crc32.h @@ -3,37 +3,13 @@ #define _LINUX_CRC32_H #include #include -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len); -u32 crc32_le_base(u32 crc, const u8 *p, size_t len); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len); -u32 crc32_be_base(u32 crc, const u8 *p, size_t len); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len); -u32 crc32c_base(u32 crc, const u8 *p, size_t len); - -static inline u32 crc32_le(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32_le_arch(crc, p, len); - return crc32_le_base(crc, p, len); -} - -static inline u32 crc32_be(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32_be_arch(crc, p, len); - return crc32_be_base(crc, p, len); -} - -static inline u32 crc32c(u32 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32c_arch(crc, p, len); - return crc32c_base(crc, p, len); -} +u32 crc32_le(u32 crc, const void *p, size_t len); +u32 crc32_be(u32 crc, const void *p, size_t len); +u32 crc32c(u32 crc, const void *p, size_t len); /* * crc32_optimizations() returns flags that indicate which CRC32 library * functions are using architecture-specific optimizations. Unlike * IS_ENABLED(CONFIG_CRC32_ARCH) it takes into account the different CRC32 diff --git a/include/linux/crc64.h b/include/linux/crc64.h index b6aa290a79312..fc0c06ab1993c 100644 --- a/include/linux/crc64.h +++ b/include/linux/crc64.h @@ -2,28 +2,18 @@ #ifndef _LINUX_CRC64_H #define _LINUX_CRC64_H #include -u64 crc64_be_arch(u64 crc, const u8 *p, size_t len); -u64 crc64_be_generic(u64 crc, const u8 *p, size_t len); -u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len); -u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len); - /** * crc64_be - Calculate bitwise big-endian ECMA-182 CRC64 * @crc: seed value for computation. 0 or (u64)~0 for a new CRC calculation, * or the previous crc64 value if computing incrementally. * @p: pointer to buffer over which CRC64 is run * @len: length of buffer @p */ -static inline u64 crc64_be(u64 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC64_ARCH)) - return crc64_be_arch(crc, p, len); - return crc64_be_generic(crc, p, len); -} +u64 crc64_be(u64 crc, const void *p, size_t len); /** * crc64_nvme - Calculate CRC64-NVME * @crc: seed value for computation. 0 for a new CRC calculation, or the * previous crc64 value if computing incrementally. @@ -31,13 +21,8 @@ static inline u64 crc64_be(u64 crc, const void *p, size_t len) * @len: length of buffer @p * * This computes the CRC64 defined in the NVME NVM Command Set Specification, * *including the bitwise inversion at the beginning and end*. */ -static inline u64 crc64_nvme(u64 crc, const void *p, size_t len) -{ - if (IS_ENABLED(CONFIG_CRC64_ARCH)) - return ~crc64_nvme_arch(~crc, p, len); - return ~crc64_nvme_generic(~crc, p, len); -} +u64 crc64_nvme(u64 crc, const void *p, size_t len); #endif /* _LINUX_CRC64_H */ diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index e0e7168b74c75..e049f01a4d2c8 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -46,12 +46,12 @@ config CRC_T10DIF config ARCH_HAS_CRC_T10DIF bool config CRC_T10DIF_ARCH - tristate - default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS + bool + depends on CRC_T10DIF && CRC_OPTIMIZATIONS config CRC32 tristate select BITREVERSE help @@ -60,12 +60,12 @@ config CRC32 config ARCH_HAS_CRC32 bool config CRC32_ARCH - tristate - default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS + bool + depends on CRC32 && CRC_OPTIMIZATIONS config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of @@ -73,12 +73,12 @@ config CRC64 config ARCH_HAS_CRC64 bool config CRC64_ARCH - tristate - default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS + bool + depends on CRC64 && CRC_OPTIMIZATIONS config CRC_OPTIMIZATIONS bool "Enable optimized CRC implementations" if EXPERT default y help diff --git a/lib/crc/Makefile b/lib/crc/Makefile index ff4c30dda4528..926edc3b035f6 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -6,20 +6,36 @@ obj-$(CONFIG_CRC4) += crc4.o obj-$(CONFIG_CRC7) += crc7.o obj-$(CONFIG_CRC8) += crc8.o obj-$(CONFIG_CRC16) += crc16.o obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o + obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o +crc-t10dif-y := crc-t10dif-main.o +ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) +CFLAGS_crc-t10dif-main.o += -I$(src)/$(SRCARCH) +endif + obj-$(CONFIG_CRC32) += crc32.o +crc32-y := crc32-main.o +ifeq ($(CONFIG_CRC32_ARCH),y) +CFLAGS_crc32-main.o += -I$(src)/$(SRCARCH) +endif + obj-$(CONFIG_CRC64) += crc64.o +crc64-y := crc64-main.o +ifeq ($(CONFIG_CRC64_ARCH),y) +CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH) +endif + obj-y += tests/ hostprogs := gen_crc32table gen_crc64table clean-files := crc32table.h crc64table.h -$(obj)/crc32.o: $(obj)/crc32table.h -$(obj)/crc64.o: $(obj)/crc64table.h +$(obj)/crc32-main.o: $(obj)/crc32table.h +$(obj)/crc64-main.o: $(obj)/crc64table.h quiet_cmd_crc32 = GEN $@ cmd_crc32 = $< > $@ quiet_cmd_crc64 = GEN $@ diff --git a/lib/crc/crc-t10dif.c b/lib/crc/crc-t10dif-main.c similarity index 78% rename from lib/crc/crc-t10dif.c rename to lib/crc/crc-t10dif-main.c index 311c2ab829f15..08f1f1042a712 100644 --- a/lib/crc/crc-t10dif.c +++ b/lib/crc/crc-t10dif-main.c @@ -48,18 +48,41 @@ static const u16 t10_dif_crc_table[256] = { 0xA415, 0x2FA2, 0x38CC, 0xB37B, 0x1610, 0x9DA7, 0x8AC9, 0x017E, 0x1F65, 0x94D2, 0x83BC, 0x080B, 0xAD60, 0x26D7, 0x31B9, 0xBA0E, 0xF0D8, 0x7B6F, 0x6C01, 0xE7B6, 0x42DD, 0xC96A, 0xDE04, 0x55B3 }; -u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len) +static inline __maybe_unused u16 +crc_t10dif_generic(u16 crc, const u8 *p, size_t len) { - size_t i; + while (len--) + crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ *p++]; + return crc; +} - for (i = 0; i < len; i++) - crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ p[i]]; +#ifdef CONFIG_CRC_T10DIF_ARCH +#include "crc-t10dif.h" /* $(SRCARCH)/crc-t10dif.h */ +#else +#define crc_t10dif_arch crc_t10dif_generic +#endif - return crc; +u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len) +{ + return crc_t10dif_arch(crc, p, len); +} +EXPORT_SYMBOL(crc_t10dif_update); + +#ifdef crc_t10dif_mod_init_arch +static int __init crc_t10dif_mod_init(void) +{ + crc_t10dif_mod_init_arch(); + return 0; +} +subsys_initcall(crc_t10dif_mod_init); + +static void __exit crc_t10dif_mod_exit(void) +{ } -EXPORT_SYMBOL(crc_t10dif_generic); +module_exit(crc_t10dif_mod_exit); +#endif -MODULE_DESCRIPTION("T10 DIF CRC calculation"); +MODULE_DESCRIPTION("CRC-T10DIF library functions"); MODULE_LICENSE("GPL"); diff --git a/lib/crc/crc32.c b/lib/crc/crc32-main.c similarity index 58% rename from lib/crc/crc32.c rename to lib/crc/crc32-main.c index 6811b37df2aad..b86ee66075d0e 100644 --- a/lib/crc/crc32.c +++ b/lib/crc/crc32-main.c @@ -28,32 +28,77 @@ #include #include #include "crc32table.h" -MODULE_AUTHOR("Matt Domsch "); -MODULE_DESCRIPTION("Various CRC32 calculations"); -MODULE_LICENSE("GPL"); - -u32 crc32_le_base(u32 crc, const u8 *p, size_t len) +static inline __maybe_unused u32 +crc32_le_base(u32 crc, const u8 *p, size_t len) { while (len--) crc = (crc >> 8) ^ crc32table_le[(crc & 255) ^ *p++]; return crc; } -EXPORT_SYMBOL(crc32_le_base); -u32 crc32c_base(u32 crc, const u8 *p, size_t len) +static inline __maybe_unused u32 +crc32_be_base(u32 crc, const u8 *p, size_t len) { while (len--) - crc = (crc >> 8) ^ crc32ctable_le[(crc & 255) ^ *p++]; + crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++]; return crc; } -EXPORT_SYMBOL(crc32c_base); -u32 crc32_be_base(u32 crc, const u8 *p, size_t len) +static inline __maybe_unused u32 +crc32c_base(u32 crc, const u8 *p, size_t len) { while (len--) - crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++]; + crc = (crc >> 8) ^ crc32ctable_le[(crc & 255) ^ *p++]; return crc; } -EXPORT_SYMBOL(crc32_be_base); + +#ifdef CONFIG_CRC32_ARCH +#include "crc32.h" /* $(SRCARCH)/crc32.h */ + +u32 crc32_optimizations(void) +{ + return crc32_optimizations_arch(); +} +EXPORT_SYMBOL(crc32_optimizations); +#else +#define crc32_le_arch crc32_le_base +#define crc32_be_arch crc32_be_base +#define crc32c_arch crc32c_base +#endif + +u32 crc32_le(u32 crc, const void *p, size_t len) +{ + return crc32_le_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32_le); + +u32 crc32_be(u32 crc, const void *p, size_t len) +{ + return crc32_be_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32_be); + +u32 crc32c(u32 crc, const void *p, size_t len) +{ + return crc32c_arch(crc, p, len); +} +EXPORT_SYMBOL(crc32c); + +#ifdef crc32_mod_init_arch +static int __init crc32_mod_init(void) +{ + crc32_mod_init_arch(); + return 0; +} +subsys_initcall(crc32_mod_init); + +static void __exit crc32_mod_exit(void) +{ +} +module_exit(crc32_mod_exit); +#endif + +MODULE_DESCRIPTION("CRC32 library functions"); +MODULE_LICENSE("GPL"); diff --git a/lib/crc/crc64.c b/lib/crc/crc64-main.c similarity index 66% rename from lib/crc/crc64.c rename to lib/crc/crc64-main.c index 5b1b17057f0ae..e4a6d879e84c3 100644 --- a/lib/crc/crc64.c +++ b/lib/crc/crc64-main.c @@ -36,23 +36,56 @@ #include #include #include #include "crc64table.h" -MODULE_DESCRIPTION("CRC64 calculations"); -MODULE_LICENSE("GPL v2"); - -u64 crc64_be_generic(u64 crc, const u8 *p, size_t len) +static inline __maybe_unused u64 +crc64_be_generic(u64 crc, const u8 *p, size_t len) { while (len--) crc = (crc << 8) ^ crc64table[(crc >> 56) ^ *p++]; return crc; } -EXPORT_SYMBOL_GPL(crc64_be_generic); -u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len) +static inline __maybe_unused u64 +crc64_nvme_generic(u64 crc, const u8 *p, size_t len) { while (len--) crc = (crc >> 8) ^ crc64nvmetable[(crc & 0xff) ^ *p++]; return crc; } -EXPORT_SYMBOL_GPL(crc64_nvme_generic); + +#ifdef CONFIG_CRC64_ARCH +#include "crc64.h" /* $(SRCARCH)/crc64.h */ +#else +#define crc64_be_arch crc64_be_generic +#define crc64_nvme_arch crc64_nvme_generic +#endif + +u64 crc64_be(u64 crc, const void *p, size_t len) +{ + return crc64_be_arch(crc, p, len); +} +EXPORT_SYMBOL_GPL(crc64_be); + +u64 crc64_nvme(u64 crc, const void *p, size_t len) +{ + return ~crc64_nvme_arch(~crc, p, len); +} +EXPORT_SYMBOL_GPL(crc64_nvme); + +#ifdef crc64_mod_init_arch +static int __init crc64_mod_init(void) +{ + crc64_mod_init_arch(); + return 0; +} +subsys_initcall(crc64_mod_init); + +static void __exit crc64_mod_exit(void) +{ +} +module_exit(crc64_mod_exit); +#endif + +MODULE_DESCRIPTION("CRC64 library functions"); +MODULE_LICENSE("GPL"); From patchwork Sat Jun 7 20:04:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894844 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C474321A928; Sat, 7 Jun 2025 20:07:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326850; cv=none; b=BGWOd0y1YBgQhIYfAagiwRlyFiwjuMiJrn8STGHxhprgb3GZBP+4FaDoNbOTY4QSp2oKMU5DbNGLq/48pGUxSC/w3MlLyx7ZmE1+feldkCnzwTajq9DEHS02FZSAOJas8WWnDNluCzo6zQnx/WylGGGXJgVlJshmCrFvkZ2rp+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326850; c=relaxed/simple; bh=WdPZKPTFqdPtFVY3vxcYLn4JpXpcQFxK5xoZxw4oN6M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PW74lS86T7jg4ZRi3qhnXy9nZ0mBtDQ91Lg/tQYLrucjxdNHUrsvnqsyY8Z6o81ytHiuk+HttcxMHxlEyC5OOV/I+GVamtxIew+UpJlfk1m9kBnFvUg0J5vp13IBElmq9CRm5Ep88q5zI1JnZy6SchoZQYWsxT+aLTdjk9ABQjY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rYwbw18n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rYwbw18n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E608CC4CEF3; Sat, 7 Jun 2025 20:07:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326850; bh=WdPZKPTFqdPtFVY3vxcYLn4JpXpcQFxK5xoZxw4oN6M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rYwbw18nEbKgn50Q7dJjhLaOiApwovsGIwvol4hlE3ERmUUlQJY+VNQ/CnX2+rGSM c0YqjT0Th4xYfjnGQHFz/Sw5Z/ikWRamSc3unwHRfBVwFIjAhxW3i0m6tSQxaDEnvp xHg6+or6mz0ts6zU8NMMPKf7RzLSPbJyLmi0l614IUJ6MicSw9Gk14Y0i5HWuP/W93 x70mCt46NarWsMV8FOFk6YVKK1eOo5CxPH15DKrwBazQXQw1ckbrMhaNtdI9VZxxvX SHiVYR/cJZROszwFVf64VoMxta5X2NJsNqIhT7u7ipAeU4j0/7fkWcqESF/PWSm+3L SdKejv+mALrJg== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 04/12] lib/crc/arm64: migrate arm64-optimized CRC code into lib/crc/ Date: Sat, 7 Jun 2025 13:04:46 -0700 Message-ID: <20250607200454.73587-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the arm64-optimized CRC code from arch/arm64/lib/crc* into its new location in lib/crc/arm64/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: prepare for arch-optimized code in subdirs of lib/crc/". Signed-off-by: Eric Biggers --- arch/arm64/Kconfig | 2 -- arch/arm64/lib/Makefile | 6 ----- lib/crc/Kconfig | 2 ++ lib/crc/Makefile | 2 ++ .../lib => lib/crc/arm64}/crc-t10dif-core.S | 0 .../crc/arm64/crc-t10dif.h | 22 +++---------------- .../arm64/lib => lib/crc/arm64}/crc32-core.S | 0 .../lib/crc32.c => lib/crc/arm64/crc32.h | 19 ++++------------ 8 files changed, 11 insertions(+), 42 deletions(-) rename {arch/arm64/lib => lib/crc/arm64}/crc-t10dif-core.S (100%) rename arch/arm64/lib/crc-t10dif.c => lib/crc/arm64/crc-t10dif.h (70%) rename {arch/arm64/lib => lib/crc/arm64}/crc32-core.S (100%) rename arch/arm64/lib/crc32.c => lib/crc/arm64/crc32.h (81%) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 55fc331af3371..cbeac225903f7 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -19,12 +19,10 @@ config ARM64 select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CC_PLATFORM - select ARCH_HAS_CRC32 - select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DMA_OPS if XEN select ARCH_HAS_DMA_PREP_COHERENT diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 027bfa9689c6a..9b255d9332479 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -14,16 +14,10 @@ CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU) endif lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o -obj-$(CONFIG_CRC32_ARCH) += crc32-arm64.o -crc32-arm64-y := crc32.o crc32-core.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm64.o -crc-t10dif-arm64-y := crc-t10dif.o crc-t10dif-core.o - obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index edd1b99098003..63edb487daff8 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -49,10 +49,11 @@ config ARCH_HAS_CRC_T10DIF config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON + default y if ARM64 && KERNEL_MODE_NEON config CRC32 tristate select BITREVERSE help @@ -64,10 +65,11 @@ config ARCH_HAS_CRC32 config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON + default y if ARM64 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index c72d351be6cb8..8adff4ae1ba63 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -12,17 +12,19 @@ obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) CFLAGS_crc-t10dif-main.o += -I$(src)/$(SRCARCH) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o +crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) CFLAGS_crc32-main.o += -I$(src)/$(SRCARCH) crc32-$(CONFIG_ARM) += arm/crc32-core.o +crc32-$(CONFIG_ARM64) += arm64/crc32-core.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/arm64/lib/crc-t10dif-core.S b/lib/crc/arm64/crc-t10dif-core.S similarity index 100% rename from arch/arm64/lib/crc-t10dif-core.S rename to lib/crc/arm64/crc-t10dif-core.S diff --git a/arch/arm64/lib/crc-t10dif.c b/lib/crc/arm64/crc-t10dif.h similarity index 70% rename from arch/arm64/lib/crc-t10dif.c rename to lib/crc/arm64/crc-t10dif.h index c2ffe4fdb59d1..c4521a7f1ee9b 100644 --- a/arch/arm64/lib/crc-t10dif.c +++ b/lib/crc/arm64/crc-t10dif.h @@ -4,15 +4,10 @@ * * Copyright (C) 2016 - 2017 Linaro Ltd */ #include -#include -#include -#include -#include -#include #include #include #include @@ -24,11 +19,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull); asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len, u8 out[16]); asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len); -u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) { if (static_branch_likely(&have_pmull)) { if (crypto_simd_usable()) { kernel_neon_begin(); @@ -48,26 +43,15 @@ u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) return crc_t10dif_generic(0, buf, sizeof(buf)); } } return crc_t10dif_generic(crc, data, length); } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_arm64_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (cpu_have_named_feature(ASIMD)) { static_branch_enable(&have_asimd); if (cpu_have_named_feature(PMULL)) static_branch_enable(&have_pmull); } - return 0; } -subsys_initcall(crc_t10dif_arm64_init); - -static void __exit crc_t10dif_arm64_exit(void) -{ -} -module_exit(crc_t10dif_arm64_exit); - -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_DESCRIPTION("CRC-T10DIF using arm64 NEON and Crypto Extensions"); -MODULE_LICENSE("GPL v2"); diff --git a/arch/arm64/lib/crc32-core.S b/lib/crc/arm64/crc32-core.S similarity index 100% rename from arch/arm64/lib/crc32-core.S rename to lib/crc/arm64/crc32-core.S diff --git a/arch/arm64/lib/crc32.c b/lib/crc/arm64/crc32.h similarity index 81% rename from arch/arm64/lib/crc32.c rename to lib/crc/arm64/crc32.h index ed3acd71178f8..6e5dec45f05d2 100644 --- a/arch/arm64/lib/crc32.c +++ b/lib/crc/arm64/crc32.h @@ -1,11 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only -#include -#include -#include - #include #include #include #include @@ -20,11 +16,11 @@ asmlinkage u32 crc32_be_arm64(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32_le_arm64_4way(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32c_le_arm64_4way(u32 crc, unsigned char const *p, size_t len); asmlinkage u32 crc32_be_arm64_4way(u32 crc, unsigned char const *p, size_t len); -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_le_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -39,13 +35,12 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32_le_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32c_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -60,13 +55,12 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32c_le_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) { if (!alternative_has_cap_likely(ARM64_HAS_CRC32)) return crc32_be_base(crc, p, len); if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) { @@ -81,19 +75,14 @@ u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) return crc; } return crc32_be_arm64(crc, p, len); } -EXPORT_SYMBOL(crc32_be_arch); -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (alternative_has_cap_likely(ARM64_HAS_CRC32)) return CRC32_LE_OPTIMIZATION | CRC32_BE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("arm64-optimized CRC32 functions"); From patchwork Sat Jun 7 20:04:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894843 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3A4321B9C3; Sat, 7 Jun 2025 20:07:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326852; cv=none; b=PLapWlZoI9xW8RY3B2qPG6lZrWCl2DW0CIgVIXKSysLyjOZn/bnMly5BIGvG3P3dRDLV2a8BbC/AhEY5yJ8rRPy7tn79UkpV65zPWSXN/Gn7eR50R+pgm367tiQH5VLkM4KSoZSJwAHz8Rvt0vpEnbaJ6CulgRBq/9ztbV71vyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326852; c=relaxed/simple; bh=hXWT8G8Ba/3m7q18Loqe+RpZLh/rS7DlIty2E8Dsl/I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B25320/i21sTFjqkGfiQX4xNWyJUBnnpBujkKhgW6BBTKncEe7Boadi/MsvHhRIBR6cOhjaU5nSJrCn7s5bLIZBSOZjU/ZVmeL0ihriB3kbHARyZin+O6wBheQVbj6yFgMIeQgWS7ylm0jFbFBfOg6zj5FUj2wCSYuSGBgQwYUY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OEO6mVXc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OEO6mVXc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 79452C4CEE4; Sat, 7 Jun 2025 20:07:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326851; bh=hXWT8G8Ba/3m7q18Loqe+RpZLh/rS7DlIty2E8Dsl/I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OEO6mVXcE91YyKwFjzGXi6irTVUA1tBomwmunmGf1Du+jIfCHuAy5wElQKbGfewk2 cqt+/pVHofIk25s+7+z+vHs/AJUB6wf/SUlcpg28mmvvLHOLI/Nk6RBi4WgacHd+fI 8Ry8eQfLXJJXs+JuxPy8DSdiY9kmSJgWdEg7yVYwlzoS3PXBCzEJ0XTyC8wfCuAxf9 sReeyz+BcE2Ei41nTT3Qjcca+n2PV3KWkrOmwxJoIbv878RTzGgxk0mjxZHgmCWyPJ xbZ71jtzvDPmm+Ri2cv1Z8To5NdLFIrrZL4k/5XirGm9T0oJLkVy3ZHhGj0PjAaVjb G1L7OksnUwNSQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 07/12] lib/crc/powerpc: migrate powerpc-optimized CRC code into lib/crc/ Date: Sat, 7 Jun 2025 13:04:49 -0700 Message-ID: <20250607200454.73587-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the powerpc-optimized CRC code from arch/powerpc/lib/crc* into its new location in lib/crc/powerpc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: prepare for arch-optimized code in subdirs of lib/crc/". Signed-off-by: Eric Biggers --- arch/powerpc/Kconfig | 2 - arch/powerpc/lib/Makefile | 6 --- lib/crc/Kconfig | 2 + lib/crc/Makefile | 2 + .../crc/powerpc/crc-t10dif.h | 20 ++-------- .../crc/powerpc}/crc-vpmsum-template.S | 0 .../lib/crc32.c => lib/crc/powerpc/crc32.h | 38 ++++--------------- .../crc/powerpc}/crc32c-vpmsum_asm.S | 0 .../crc/powerpc}/crct10dif-vpmsum_asm.S | 0 9 files changed, 14 insertions(+), 56 deletions(-) rename arch/powerpc/lib/crc-t10dif.c => lib/crc/powerpc/crc-t10dif.h (75%) rename {arch/powerpc/lib => lib/crc/powerpc}/crc-vpmsum-template.S (100%) rename arch/powerpc/lib/crc32.c => lib/crc/powerpc/crc32.h (64%) rename {arch/powerpc/lib => lib/crc/powerpc}/crc32c-vpmsum_asm.S (100%) rename {arch/powerpc/lib => lib/crc/powerpc}/crct10dif-vpmsum_asm.S (100%) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c3e0cc83f1205..45b4fa7b9b02f 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -125,12 +125,10 @@ config PPC select ARCH_DISABLE_KASAN_INLINE if PPC_RADIX_MMU select ARCH_DMA_DEFAULT_COHERENT if !NOT_COHERENT_CACHE select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_HAS_COPY_MC if PPC64 - select ARCH_HAS_CRC32 if PPC64 && ALTIVEC - select ARCH_HAS_CRC_T10DIF if PPC64 && ALTIVEC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX if STRICT_KERNEL_RWX select ARCH_HAS_DEVMEM_IS_ALLOWED diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index 481f968e42c7b..59de2e2232df6 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -78,12 +78,6 @@ obj-$(CONFIG_FTR_FIXUP_SELFTEST) += feature-fixups-test.o obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o CFLAGS_xor_vmx.o += -mhard-float -maltivec $(call cc-option,-mabi=altivec) # Enable CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include) -obj-$(CONFIG_CRC32_ARCH) += crc32-powerpc.o -crc32-powerpc-y := crc32.o crc32c-vpmsum_asm.o - -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-powerpc.o -crc-t10dif-powerpc-y := crc-t10dif.o crct10dif-vpmsum_asm.o - obj-$(CONFIG_PPC64) += $(obj64-y) diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 3f534fbfba951..c3f9091ee512e 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -50,10 +50,11 @@ config ARCH_HAS_CRC_T10DIF config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON + default y if PPC64 && ALTIVEC config CRC32 tristate select BITREVERSE help @@ -68,10 +69,11 @@ config CRC32_ARCH depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 default y if LOONGARCH default y if MIPS && CPU_MIPSR6 + default y if PPC64 && ALTIVEC config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 8adff4ae1ba63..555fd3fb6d197 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -13,18 +13,20 @@ obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) CFLAGS_crc-t10dif-main.o += -I$(src)/$(SRCARCH) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o +crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) CFLAGS_crc32-main.o += -I$(src)/$(SRCARCH) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o +crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/powerpc/lib/crc-t10dif.c b/lib/crc/powerpc/crc-t10dif.h similarity index 75% rename from arch/powerpc/lib/crc-t10dif.c rename to lib/crc/powerpc/crc-t10dif.h index be23ded3a9df6..59e16804a6eae 100644 --- a/arch/powerpc/lib/crc-t10dif.c +++ b/lib/crc/powerpc/crc-t10dif.h @@ -7,14 +7,11 @@ */ #include #include #include -#include #include -#include -#include #include #include #define VMX_ALIGN 16 #define VMX_ALIGN_MASK (VMX_ALIGN-1) @@ -23,11 +20,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto); u32 __crct10dif_vpmsum(u32 crc, unsigned char const *p, size_t len); -u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) +static inline u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) { unsigned int prealign; unsigned int tail; u32 crc = crci; @@ -60,24 +57,13 @@ u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len) crc = crc_t10dif_generic(crc, p, tail); } return crc & 0xffff; } -EXPORT_SYMBOL(crc_t10dif_arch); -static int __init crc_t10dif_powerpc_init(void) +#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch +static inline void crc_t10dif_mod_init_arch(void) { if (cpu_has_feature(CPU_FTR_ARCH_207S) && (cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO)) static_branch_enable(&have_vec_crypto); - return 0; } -subsys_initcall(crc_t10dif_powerpc_init); - -static void __exit crc_t10dif_powerpc_exit(void) -{ -} -module_exit(crc_t10dif_powerpc_exit); - -MODULE_AUTHOR("Daniel Axtens "); -MODULE_DESCRIPTION("CRCT10DIF using vector polynomial multiply-sum instructions"); -MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/lib/crc-vpmsum-template.S b/lib/crc/powerpc/crc-vpmsum-template.S similarity index 100% rename from arch/powerpc/lib/crc-vpmsum-template.S rename to lib/crc/powerpc/crc-vpmsum-template.S diff --git a/arch/powerpc/lib/crc32.c b/lib/crc/powerpc/crc32.h similarity index 64% rename from arch/powerpc/lib/crc32.c rename to lib/crc/powerpc/crc32.h index 0d9befb6e7b83..811cc2e6ed24d 100644 --- a/arch/powerpc/lib/crc32.c +++ b/lib/crc/powerpc/crc32.h @@ -1,32 +1,26 @@ // SPDX-License-Identifier: GPL-2.0-only #include #include #include -#include #include -#include -#include #include #include #define VMX_ALIGN 16 #define VMX_ALIGN_MASK (VMX_ALIGN-1) #define VECTOR_BREAKPOINT 512 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto); -u32 __crc32c_vpmsum(u32 crc, const u8 *p, size_t len); +#define crc32_le_arch crc32_le_base /* not implemented on this arch */ +#define crc32_be_arch crc32_be_base /* not implemented on this arch */ -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_le_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_le_arch); +u32 __crc32c_vpmsum(u32 crc, const u8 *p, size_t len); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { unsigned int prealign; unsigned int tail; if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || @@ -56,38 +50,20 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len) crc = crc32c_base(crc, p, tail); } return crc; } -EXPORT_SYMBOL(crc32c_arch); - -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) -{ - return crc32_be_base(crc, p, len); -} -EXPORT_SYMBOL(crc32_be_arch); -static int __init crc32_powerpc_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { if (cpu_has_feature(CPU_FTR_ARCH_207S) && (cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO)) static_branch_enable(&have_vec_crypto); - return 0; -} -subsys_initcall(crc32_powerpc_init); - -static void __exit crc32_powerpc_exit(void) -{ } -module_exit(crc32_powerpc_exit); -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_vec_crypto)) return CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_AUTHOR("Anton Blanchard "); -MODULE_DESCRIPTION("CRC32C using vector polynomial multiply-sum instructions"); -MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/lib/crc32c-vpmsum_asm.S b/lib/crc/powerpc/crc32c-vpmsum_asm.S similarity index 100% rename from arch/powerpc/lib/crc32c-vpmsum_asm.S rename to lib/crc/powerpc/crc32c-vpmsum_asm.S diff --git a/arch/powerpc/lib/crct10dif-vpmsum_asm.S b/lib/crc/powerpc/crct10dif-vpmsum_asm.S similarity index 100% rename from arch/powerpc/lib/crct10dif-vpmsum_asm.S rename to lib/crc/powerpc/crct10dif-vpmsum_asm.S From patchwork Sat Jun 7 20:04:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894842 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA58621C9F6; Sat, 7 Jun 2025 20:07:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326853; cv=none; b=cClZlyS0c45NPf53DLSqdhUP+qVPYwwAFnDzyLuyDQgJBpxBwmQ5PJwN/tVqH0ymqNrFS/WKV99OQ8weUiq+tN2qjllJkrgcHJkRIN9WT0w/PYtuUX/EIbL62F9dl8sfBE+9/pEKOIEafSozQkc6wOxYkgBQsoP+J13CxLHUIpY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326853; c=relaxed/simple; bh=UhWtk6TkzpgWbad69ugt2GMpM8wI/AOjLVW/1+GRCZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aWAUNOKkSiQLo4vOT5Fn1lE2erMvmaSxCMHz/4hziDfJedlXxE6ZhNuJmfT9ej9JqIRM/7iLMMUs8lFo0yxkHpRRKBn4lLkKlAzC4Lp7amgYkHYGotk7I2JLyFZFPQMVrT80sLGti875f7wHJ3Bn7vZcP2jyLpM7sZ0mgwCQK1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pTj4CUvS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pTj4CUvS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 081C2C4CEFE; Sat, 7 Jun 2025 20:07:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326852; bh=UhWtk6TkzpgWbad69ugt2GMpM8wI/AOjLVW/1+GRCZM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pTj4CUvSbhajPr41bWro0qvHnYDm1wwoVE3HzOEhFvE9Wg8gJs54AypdZVwebNAc+ o7wZ6Rh7vv1xZwENmbDBbEl+75t8pskCZwchC38d1qwpq65rU2eHcmy72uzoZxLy/K u3pUA71njZJv5rj6gFTaoiTE0XxXyBWHhpfGFFPBcVnl1WE7jhM8VphGM9/TB9mTEm P2vg+92ZfulqYd8eoyrQZBgDFysfjDyS0Uzw0TTNvPjuEVdoBdqam0nhCguCyBifTk wAMgc4MWzDADFR3rCo5N4HsYK7IcJ9evOG4lUyf1WV5NgsZl8U9B6EkO0nqsbLhjjq YhCKW0G1L4U5g== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 08/12] lib/crc/riscv: migrate riscv-optimized CRC code into lib/crc/ Date: Sat, 7 Jun 2025 13:04:50 -0700 Message-ID: <20250607200454.73587-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the riscv-optimized CRC code from arch/riscv/lib/crc* into its new location in lib/crc/riscv/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: prepare for arch-optimized code in subdirs of lib/crc/". Signed-off-by: Eric Biggers --- arch/riscv/Kconfig | 3 --- arch/riscv/lib/Makefile | 6 ------ lib/crc/Kconfig | 3 +++ lib/crc/Makefile | 3 +++ .../lib => lib/crc/riscv}/crc-clmul-consts.h | 0 .../lib => lib/crc/riscv}/crc-clmul-template.h | 0 {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h | 0 .../crc-t10dif.c => lib/crc/riscv/crc-t10dif.h | 8 +------- {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c | 0 arch/riscv/lib/crc32.c => lib/crc/riscv/crc32.h | 17 ++++------------- {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c | 0 {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c | 0 arch/riscv/lib/crc64.c => lib/crc/riscv/crc64.h | 11 ++--------- {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c | 0 {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c | 0 15 files changed, 13 insertions(+), 38 deletions(-) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-consts.h (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-template.h (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h (100%) rename arch/riscv/lib/crc-t10dif.c => lib/crc/riscv/crc-t10dif.h (62%) rename {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c (100%) rename arch/riscv/lib/crc32.c => lib/crc/riscv/crc32.h (66%) rename {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c (100%) rename arch/riscv/lib/crc64.c => lib/crc/riscv/crc64.h (65%) rename {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c (100%) rename {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c (100%) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 36061f4732b74..d963d8faf2aeb 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -22,13 +22,10 @@ config RISCV select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT - select ARCH_HAS_CRC32 if RISCV_ISA_ZBC - select ARCH_HAS_CRC64 if 64BIT && RISCV_ISA_ZBC - select ARCH_HAS_CRC_T10DIF if RISCV_ISA_ZBC select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX select ARCH_HAS_FAST_MULTIPLIER diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 0baec92d2f55b..a4f4b48ed3a47 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -14,14 +14,8 @@ ifeq ($(CONFIG_MMU), y) lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o endif lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o -obj-$(CONFIG_CRC32_ARCH) += crc32-riscv.o -crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o -obj-$(CONFIG_CRC64_ARCH) += crc64-riscv.o -crc64-riscv-y := crc64.o crc64_msb.o crc64_lsb.o -obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-riscv.o -crc-t10dif-riscv-y := crc-t10dif.o crc16_msb.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index c3f9091ee512e..d0c293b1a4182 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -51,10 +51,11 @@ config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON default y if PPC64 && ALTIVEC + default y if RISCV && RISCV_ISA_ZBC config CRC32 tristate select BITREVERSE help @@ -70,10 +71,11 @@ config CRC32_ARCH default y if ARM && KERNEL_MODE_NEON default y if ARM64 default y if LOONGARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC + default y if RISCV && RISCV_ISA_ZBC config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of @@ -83,10 +85,11 @@ config ARCH_HAS_CRC64 bool config CRC64_ARCH bool depends on CRC64 && CRC_OPTIMIZATIONS + default y if RISCV && RISCV_ISA_ZBC && 64BIT config CRC_OPTIMIZATIONS bool "Enable optimized CRC implementations" if EXPERT default y help diff --git a/lib/crc/Makefile b/lib/crc/Makefile index 555fd3fb6d197..190f889adf556 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -14,25 +14,28 @@ crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y) CFLAGS_crc-t10dif-main.o += -I$(src)/$(SRCARCH) crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o +crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o endif obj-$(CONFIG_CRC32) += crc32.o crc32-y := crc32-main.o ifeq ($(CONFIG_CRC32_ARCH),y) CFLAGS_crc32-main.o += -I$(src)/$(SRCARCH) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o +crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH) +crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o endif obj-y += tests/ hostprogs := gen_crc32table gen_crc64table diff --git a/arch/riscv/lib/crc-clmul-consts.h b/lib/crc/riscv/crc-clmul-consts.h similarity index 100% rename from arch/riscv/lib/crc-clmul-consts.h rename to lib/crc/riscv/crc-clmul-consts.h diff --git a/arch/riscv/lib/crc-clmul-template.h b/lib/crc/riscv/crc-clmul-template.h similarity index 100% rename from arch/riscv/lib/crc-clmul-template.h rename to lib/crc/riscv/crc-clmul-template.h diff --git a/arch/riscv/lib/crc-clmul.h b/lib/crc/riscv/crc-clmul.h similarity index 100% rename from arch/riscv/lib/crc-clmul.h rename to lib/crc/riscv/crc-clmul.h diff --git a/arch/riscv/lib/crc-t10dif.c b/lib/crc/riscv/crc-t10dif.h similarity index 62% rename from arch/riscv/lib/crc-t10dif.c rename to lib/crc/riscv/crc-t10dif.h index e6b0051ccd86c..cd6136cbfda1c 100644 --- a/arch/riscv/lib/crc-t10dif.c +++ b/lib/crc/riscv/crc-t10dif.h @@ -5,20 +5,14 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) +static inline u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc16_msb_clmul(crc, p, len, &crc16_msb_0x8bb7_consts); return crc_t10dif_generic(crc, p, len); } -EXPORT_SYMBOL(crc_t10dif_arch); - -MODULE_DESCRIPTION("RISC-V optimized CRC-T10DIF function"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc16_msb.c b/lib/crc/riscv/crc16_msb.c similarity index 100% rename from arch/riscv/lib/crc16_msb.c rename to lib/crc/riscv/crc16_msb.c diff --git a/arch/riscv/lib/crc32.c b/lib/crc/riscv/crc32.h similarity index 66% rename from arch/riscv/lib/crc32.c rename to lib/crc/riscv/crc32.h index a3188b7d9c403..3ec6eee98afa8 100644 --- a/arch/riscv/lib/crc32.c +++ b/lib/crc/riscv/crc32.h @@ -5,49 +5,40 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_lsb_clmul(crc, p, len, &crc32_lsb_0xedb88320_consts); return crc32_le_base(crc, p, len); } -EXPORT_SYMBOL(crc32_le_arch); -u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_msb_clmul(crc, p, len, &crc32_msb_0x04c11db7_consts); return crc32_be_base(crc, p, len); } -EXPORT_SYMBOL(crc32_be_arch); -u32 crc32c_arch(u32 crc, const u8 *p, size_t len) +static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc32_lsb_clmul(crc, p, len, &crc32_lsb_0x82f63b78_consts); return crc32c_base(crc, p, len); } -EXPORT_SYMBOL(crc32c_arch); -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return CRC32_LE_OPTIMIZATION | CRC32_BE_OPTIMIZATION | CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_DESCRIPTION("RISC-V optimized CRC32 functions"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc32_lsb.c b/lib/crc/riscv/crc32_lsb.c similarity index 100% rename from arch/riscv/lib/crc32_lsb.c rename to lib/crc/riscv/crc32_lsb.c diff --git a/arch/riscv/lib/crc32_msb.c b/lib/crc/riscv/crc32_msb.c similarity index 100% rename from arch/riscv/lib/crc32_msb.c rename to lib/crc/riscv/crc32_msb.c diff --git a/arch/riscv/lib/crc64.c b/lib/crc/riscv/crc64.h similarity index 65% rename from arch/riscv/lib/crc64.c rename to lib/crc/riscv/crc64.h index f0015a27836a4..a1b7873fde579 100644 --- a/arch/riscv/lib/crc64.c +++ b/lib/crc/riscv/crc64.h @@ -5,30 +5,23 @@ * Copyright 2025 Google LLC */ #include #include -#include -#include #include "crc-clmul.h" -u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_be_arch(u64 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc64_msb_clmul(crc, p, len, &crc64_msb_0x42f0e1eba9ea3693_consts); return crc64_be_generic(crc, p, len); } -EXPORT_SYMBOL(crc64_be_arch); -u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) +static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) { if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC)) return crc64_lsb_clmul(crc, p, len, &crc64_lsb_0x9a6c9329ac4bc9b5_consts); return crc64_nvme_generic(crc, p, len); } -EXPORT_SYMBOL(crc64_nvme_arch); - -MODULE_DESCRIPTION("RISC-V optimized CRC64 functions"); -MODULE_LICENSE("GPL"); diff --git a/arch/riscv/lib/crc64_lsb.c b/lib/crc/riscv/crc64_lsb.c similarity index 100% rename from arch/riscv/lib/crc64_lsb.c rename to lib/crc/riscv/crc64_lsb.c diff --git a/arch/riscv/lib/crc64_msb.c b/lib/crc/riscv/crc64_msb.c similarity index 100% rename from arch/riscv/lib/crc64_msb.c rename to lib/crc/riscv/crc64_msb.c From patchwork Sat Jun 7 20:04:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894841 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C38521CC7F; Sat, 7 Jun 2025 20:07:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326853; cv=none; b=FGqIi1YS6JJF8hkAIRr0ymbTdtTmlmA9SnG4VOjv5EmrGbOu3j9ECu0nqKF4bhLR09c8J8koKuUWsigdBLdxbUQQsxK24wkw1jzD8LyeL/aAkKYujw1R4c0PeUFTusBY4981yKraIG+WuvFq3bW20iq8d9KXDr33RVSWwajQeSo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326853; c=relaxed/simple; bh=ml00Eu6L3MCivT8VcjfT2RgfiZJS9znV2CRfUv6dB7E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ir3ui3SN3I9ZqcEKKMg+OL7zpG8YhxDco0cWHqq1bqXXurY8nYEK5iXSeoY4ANq8+pZdC611fCFTPIDMNZO4zD2nE1Ef9Ym9W6gGiVDZ5iNCJs3OVEVwxeAKR3Xbk7minAMzSif9m78t+vwd1K2Kkg4IIxiULEsiYhqy1eOCyjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Hf6I+Ges; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Hf6I+Ges" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10C30C4CEE4; Sat, 7 Jun 2025 20:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326853; bh=ml00Eu6L3MCivT8VcjfT2RgfiZJS9znV2CRfUv6dB7E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hf6I+GessssiRha4MpA6RFI+58URioJcA8hkf18voXD1e9otmRrh6ut6f37cDXBp+ t02pRBAP/ZlXisnR/Q2kSIylUzzyohdF4xc+dYSaCvPizsqHiblpXpH3M/AcB4DwZm 1gsBgk8eLNYcmlAGFZLxK2WApGsMmbHKpUBzmx0pNNo3RVKGFzhTcZ7ZCx//ZzZdDI hLEhV+ur56mvrGVIPD3EuNCPvrKW9CQHIaAhExDe1uBKmjOUWdJvccInA7Py7CALlr ISUzxfUqOE8bsLzp6szFM0WEjwnQYDirYRxe83hqmv+e8gbpJV65fpLADWeNNXS6YC lNZw0bHLVTzlA== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 10/12] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/ Date: Sat, 7 Jun 2025 13:04:52 -0700 Message-ID: <20250607200454.73587-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Move the sparc-optimized CRC code from arch/sparc/lib/crc* into its new location in lib/crc/sparc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: prepare for arch-optimized code in subdirs of lib/crc/". Signed-off-by: Eric Biggers --- arch/sparc/Kconfig | 1 - arch/sparc/lib/Makefile | 2 - lib/crc/Kconfig | 1 + lib/crc/Makefile | 1 + .../lib/crc32.c => lib/crc/sparc/crc32.h | 42 ++++--------------- .../sparc/lib => lib/crc/sparc}/crc32c_asm.S | 0 6 files changed, 10 insertions(+), 37 deletions(-) rename arch/sparc/lib/crc32.c => lib/crc/sparc/crc32.h (60%) rename {arch/sparc/lib => lib/crc/sparc}/crc32c_asm.S (100%) diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index 0f88123925a4f..dcfdb7f1dae97 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -108,11 +108,10 @@ config SPARC64 select ARCH_HAS_GIGANTIC_PAGE select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_SETUP_PER_CPU_AREA select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK - select ARCH_HAS_CRC32 config ARCH_PROC_KCORE_TEXT def_bool y config CPU_BIG_ENDIAN diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile index 5cf9781d68b40..2d6c3c5352734 100644 --- a/arch/sparc/lib/Makefile +++ b/arch/sparc/lib/Makefile @@ -52,7 +52,5 @@ lib-$(CONFIG_SPARC64) += copy_in_user.o memmove.o lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o xor.o hweight.o ffs.o obj-$(CONFIG_SPARC64) += iomap.o obj-$(CONFIG_SPARC32) += atomic32.o obj-$(CONFIG_SPARC64) += PeeCeeI.o -obj-$(CONFIG_CRC32_ARCH) += crc32-sparc.o -crc32-sparc-y := crc32.o crc32c_asm.o diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index 1b69a8bef4a85..af4fa857a81f8 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -73,10 +73,11 @@ config CRC32_ARCH default y if LOONGARCH default y if MIPS && CPU_MIPSR6 default y if PPC64 && ALTIVEC default y if RISCV && RISCV_ISA_ZBC default y if S390 + default y if SPARC64 config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of diff --git a/lib/crc/Makefile b/lib/crc/Makefile index bec58266251f8..81e176db0947c 100644 --- a/lib/crc/Makefile +++ b/lib/crc/Makefile @@ -26,10 +26,11 @@ CFLAGS_crc32-main.o += -I$(src)/$(SRCARCH) crc32-$(CONFIG_ARM) += arm/crc32-core.o crc32-$(CONFIG_ARM64) += arm64/crc32-core.o crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o +crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o endif obj-$(CONFIG_CRC64) += crc64.o crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) diff --git a/arch/sparc/lib/crc32.c b/lib/crc/sparc/crc32.h similarity index 60% rename from arch/sparc/lib/crc32.c rename to lib/crc/sparc/crc32.h index 40d4720a42a1b..60f2765ac0157 100644 --- a/arch/sparc/lib/crc32.c +++ b/lib/crc/sparc/crc32.h @@ -6,30 +6,21 @@ * Copyright (C) 2008 Intel Corporation * Authors: Austin Zhang * Kent Liu */ -#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt - -#include -#include -#include -#include #include #include static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32c_opcode); -u32 crc32_le_arch(u32 crc, const u8 *data, size_t len) -{ - return crc32_le_base(crc, data, len); -} -EXPORT_SYMBOL(crc32_le_arch); +#define crc32_le_arch crc32_le_base /* not implemented on this arch */ +#define crc32_be_arch crc32_be_base /* not implemented on this arch */ void crc32c_sparc64(u32 *crcp, const u64 *data, size_t len); -u32 crc32c_arch(u32 crc, const u8 *data, size_t len) +static inline u32 crc32c_arch(u32 crc, const u8 *data, size_t len) { size_t n = -(uintptr_t)data & 7; if (!static_branch_likely(&have_crc32c_opcode)) return crc32c_base(crc, data, len); @@ -49,45 +40,28 @@ u32 crc32c_arch(u32 crc, const u8 *data, size_t len) } if (len) crc = crc32c_base(crc, data, len); return crc; } -EXPORT_SYMBOL(crc32c_arch); - -u32 crc32_be_arch(u32 crc, const u8 *data, size_t len) -{ - return crc32_be_base(crc, data, len); -} -EXPORT_SYMBOL(crc32_be_arch); -static int __init crc32_sparc_init(void) +#define crc32_mod_init_arch crc32_mod_init_arch +static inline void crc32_mod_init_arch(void) { unsigned long cfr; if (!(sparc64_elf_hwcap & HWCAP_SPARC_CRYPTO)) - return 0; + return; __asm__ __volatile__("rd %%asr26, %0" : "=r" (cfr)); if (!(cfr & CFR_CRC32C)) - return 0; + return; static_branch_enable(&have_crc32c_opcode); pr_info("Using sparc64 crc32c opcode optimized CRC32C implementation\n"); - return 0; } -subsys_initcall(crc32_sparc_init); -static void __exit crc32_sparc_exit(void) -{ -} -module_exit(crc32_sparc_exit); - -u32 crc32_optimizations(void) +static inline u32 crc32_optimizations_arch(void) { if (static_key_enabled(&have_crc32c_opcode)) return CRC32C_OPTIMIZATION; return 0; } -EXPORT_SYMBOL(crc32_optimizations); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("CRC32c (Castagnoli), sparc64 crc32c opcode accelerated"); diff --git a/arch/sparc/lib/crc32c_asm.S b/lib/crc/sparc/crc32c_asm.S similarity index 100% rename from arch/sparc/lib/crc32c_asm.S rename to lib/crc/sparc/crc32c_asm.S From patchwork Sat Jun 7 20:04:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 894840 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95CD021FF3D; Sat, 7 Jun 2025 20:07:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326854; cv=none; b=oJYwXHAi4CV7uQLUdu8p7PSmILQy96wboUnQBE4umnMByR6F2+a/CG7Yc/Y0giYpuUtT5O2Eq9sICMOSDGb5jFgplksZq1lGg9fmHqSAbxb0ImwKbbECPuVliVBjclKBKLWyX6zmeP5LuHBjlEdsGXLIWBx6ru/ce7icTzJkftM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749326854; c=relaxed/simple; bh=ZDBXUza5mYKN0G+LEgQAmbkZEIUs7iZW79q0Idl2Apk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f8bMCWEHJi3GaHGjHZ91WvkCVuer56u7SvzmFekeRp4gfWBCLYuFycACl2sT//O0uGPr1YJD1nULl74cta1UbZl8z1qZLf/3yY3ZgqCZoBjIJuyZaR1mhzKHwWPWZ98ab1oml2DE6D+RMNMpm6+YjNEK80JMSVdQvz8nhXyOSjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ajEhG5/8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ajEhG5/8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E744C4CEE4; Sat, 7 Jun 2025 20:07:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749326854; bh=ZDBXUza5mYKN0G+LEgQAmbkZEIUs7iZW79q0Idl2Apk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ajEhG5/8HVrfaad1n+slUIYvzE1FkTzDHcl7TWi7cGlPQnDlA04WdKniPoxYQOMYE NFPprqJK5IW0ur0fiG3+y7GcPTyQuszc7j+8AfGYwPfJ+iPIu+ZTxAE0RYDH2G8FKm BJJAZ7QkJFPdzFyT6dT+BjT8tLQ1siQLlN12X59hwHDp/q3vgGfG3SBu0/JsI+taKk yOMBHSm3WUeEOXPURUAoEtz4nwBTplDDJdpMj/2dInexP90CK4TPNO12gzd+vmDwg1 x3S8BGN0sPA3MXigsw27iqzjq/8mJ/jiCprDcU9qVKaUU6hKLM3CwKWLsWamrSVvTN 6UALswZ1OpPdQ== From: Eric Biggers To: linux-kernel@vger.kernel.org Cc: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Linus Torvalds Subject: [PATCH v2 12/12] lib/crc: remove ARCH_HAS_* kconfig symbols Date: Sat, 7 Jun 2025 13:04:54 -0700 Message-ID: <20250607200454.73587-13-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250607200454.73587-1-ebiggers@kernel.org> References: <20250607200454.73587-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers These symbols are no longer used, so remove them. Signed-off-by: Eric Biggers --- lib/crc/Kconfig | 9 --------- 1 file changed, 9 deletions(-) diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig index de4b2182ae7ff..5858b3acc6630 100644 --- a/lib/crc/Kconfig +++ b/lib/crc/Kconfig @@ -42,13 +42,10 @@ config CRC_T10DIF tristate help The CRC-T10DIF library functions. Select this if your module uses any of the functions from . -config ARCH_HAS_CRC_T10DIF - bool - config CRC_T10DIF_ARCH bool depends on CRC_T10DIF && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 && KERNEL_MODE_NEON @@ -61,13 +58,10 @@ config CRC32 select BITREVERSE help The CRC32 library functions. Select this if your module uses any of the functions from or . -config ARCH_HAS_CRC32 - bool - config CRC32_ARCH bool depends on CRC32 && CRC_OPTIMIZATIONS default y if ARM && KERNEL_MODE_NEON default y if ARM64 @@ -83,13 +77,10 @@ config CRC64 tristate help The CRC64 library functions. Select this if your module uses any of the functions from . -config ARCH_HAS_CRC64 - bool - config CRC64_ARCH bool depends on CRC64 && CRC_OPTIMIZATIONS default y if RISCV && RISCV_ISA_ZBC && 64BIT default y if X86_64