From patchwork Fri Jun 2 10:12:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20B7DC7EE2E for ; Fri, 2 Jun 2023 10:22:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234279AbjFBKWs (ORCPT ); Fri, 2 Jun 2023 06:22:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235377AbjFBKWO (ORCPT ); Fri, 2 Jun 2023 06:22:14 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E5E51A7; Fri, 2 Jun 2023 03:22:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3BA4364E77; Fri, 2 Jun 2023 10:22:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC33DC433D2; Fri, 2 Jun 2023 10:22:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701331; bh=2ugw2aI3WS8XFLpq392iX/qheNdS02iXQMzmuFjsdqI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K2u71tj2CtRCMgj8JKI0rybfZ5Qn7hS9qJVCdQZcd2kt0ypXAbHBekGJyMIRwRSr1 1iQqYieUF7zYnbAlhIz+2cDP6gMjjdZRTiKVvCMH+FLd+FL/kU6avIoWPCKKEDpEXC /lXwDojb2VykTcf2Vu6lQpUVQ4A7uOIqAmO0kuhmnReuuN8rrDfKw46IcYHHRjCVyu 1C1pQZrCCOPQNwiMTbxRr+kCaxCr+HRf18Slv934O4rcNhhIIZMe/XTwBU8DuHCOTx bd/nZ0BIHtmiptb3okFsRCbflPMOLYD0oo5qEnY/rZjIhjtDXbz+PiLFuVSnvn0OMx nLx+Cq4Fiv3jA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 01/21] x86/efistub: Branch straight to kernel entry point from C code Date: Fri, 2 Jun 2023 12:12:53 +0200 Message-Id: <20230602101313.3557775-2-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1778; i=ardb@kernel.org; h=from:subject; bh=2ugw2aI3WS8XFLpq392iX/qheNdS02iXQMzmuFjsdqI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywIKrhi7XVpdfTgu8p1a5ddOvKrn5p+fWbTFQUbgaP PEPz4M/HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAi+XcZGdZHROwtufPF4oSU 4hadjQKn7+TV9U/9oNL/7sHMEyWBBpEMf+XsN5qEaiwT5NSzecemE+4q/p7pTxxjUG32wqctL19 P4gQA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Instead of returning to the calling code in assembler that does nothing more than perform an indirect call with the boot_params pointer in register ESI/RSI, perform the jump directly from the EFI stub C code. This will allow the asm entrypoint code to be dropped entirely in subsequent patches. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/x86-stub.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 8d17cee8b98e1a63..095aaa8b0ee30fb9 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -769,9 +769,17 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle) return EFI_SUCCESS; } +static void __noreturn enter_kernel(unsigned long kernel_addr, + struct boot_params *boot_params) +{ + /* enter decompressed kernel with boot_params pointer in RSI/ESI */ + asm("jmp *%0"::"r"(kernel_addr), "S"(boot_params)); + + unreachable(); +} + /* - * On success, we return the address of startup_32, which has potentially been - * relocated by efi_relocate_kernel. + * On success, we jump to the relocated kernel directly and never return. * On failure, we exit to the firmware via efi_exit instead of returning. */ asmlinkage unsigned long efi_main(efi_handle_t handle, @@ -914,7 +922,11 @@ asmlinkage unsigned long efi_main(efi_handle_t handle, goto fail; } - return bzimage_addr; + if (IS_ENABLED(CONFIG_X86_64)) + /* add offset of startup_64() */ + bzimage_addr += 0x200; + + enter_kernel(bzimage_addr, boot_params); fail: efi_err("efi_main() failed!\n"); From patchwork Fri Jun 2 10:12:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B738BC7EE29 for ; Fri, 2 Jun 2023 10:22:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235319AbjFBKWr (ORCPT ); Fri, 2 Jun 2023 06:22:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235151AbjFBKWS (ORCPT ); Fri, 2 Jun 2023 06:22:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11A971A2; Fri, 2 Jun 2023 03:22:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9732364E71; Fri, 2 Jun 2023 10:22:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DCE8C433A1; Fri, 2 Jun 2023 10:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701336; bh=6vIoJTudACrMHaUZpEmA9kKzuuAA0lBBUtfzLDgP/Vg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FwZc6n7DPdapDp2Ze4CXOM31IKbh0eg+q4IJAck/FYqJ6Pz4jl6R/9drIZeK4FZ/7 PqCGsVkq+GgAU+9RX53A/RYcc7yJCpN9flRRXsfEICAkAHJCCNAyf+Q4r/b7Gi42mB gKkLWw3hcIu/Ocu6mMro2rN4WnuJoBNPa3HHU/a/f+n2q8Qrcr3bMhvXg9tZaVPQJk pLQl1nE5+isCzjxCq0327s91qm2sNU5Awqvug+S8dUaMj8il0sgoiP4hCjP3GbwOaB SrnKqvGsDaSe9y5zxOQ0ER/nYjhVfc3O0zW1c+P1DesA/U2Uv0N1vGHQ7eoGh1Qd7j noy2vRtlcTjfw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 02/21] x86/efistub: Simplify and clean up handover entry code Date: Fri, 2 Jun 2023 12:12:54 +0200 Message-Id: <20230602101313.3557775-3-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7508; i=ardb@kernel.org; h=from:subject; bh=6vIoJTudACrMHaUZpEmA9kKzuuAA0lBBUtfzLDgP/Vg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywMI/xWdftd5W1Ti0skFl44qdD6xDGlvOaew6xSowt /1M7bKVHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiMucYGW7nBm2VDizK9tJo 0rxft+L47YZdLmv54j4nHXx7IovdQoGRYU5ikSNT4RX+S3fOf7Ooda2+btKRHtksrPvvpP3JT6q XeAA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Now that the EFI entry code in assembler is only used by the optional and deprecated EFI handover protocol, and given that the EFI stub C code no longer returns to it, most of it can simply be dropped. While at it, clarify the symbol naming, by merging efi_main() and efi_stub_entry(), making the latter the shared entry point for all different boot modes that enter via the EFI stub. The efi32_stub_entry() and efi64_stub_entry() names are referenced explicitly by the tooling that populates the setup header, so these must be retained, but can be emitted as aliases of efi_stub_entry() where appropriate. Signed-off-by: Ard Biesheuvel --- Documentation/arch/x86/boot.rst | 2 +- arch/x86/boot/compressed/efi_mixed.S | 22 +++++++++++--------- arch/x86/boot/compressed/head_32.S | 11 ---------- arch/x86/boot/compressed/head_64.S | 12 ++--------- drivers/firmware/efi/libstub/x86-stub.c | 20 ++++++++++++++---- 5 files changed, 31 insertions(+), 36 deletions(-) diff --git a/Documentation/arch/x86/boot.rst b/Documentation/arch/x86/boot.rst index 33520ecdb37abfda..cdbca15a4fc23833 100644 --- a/Documentation/arch/x86/boot.rst +++ b/Documentation/arch/x86/boot.rst @@ -1417,7 +1417,7 @@ execution context provided by the EFI firmware. The function prototype for the handover entry point looks like this:: - efi_main(void *handle, efi_system_table_t *table, struct boot_params *bp) + efi_stub_entry(void *handle, efi_system_table_t *table, struct boot_params *bp) 'handle' is the EFI image handle passed to the boot loader by the EFI firmware, 'table' is the EFI system table - these are the first two diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S index 4ca70bf93dc0bdcd..dcc562c8f7f35162 100644 --- a/arch/x86/boot/compressed/efi_mixed.S +++ b/arch/x86/boot/compressed/efi_mixed.S @@ -26,8 +26,8 @@ * When booting in 64-bit mode on 32-bit EFI firmware, startup_64_mixed_mode() * is the first thing that runs after switching to long mode. Depending on * whether the EFI handover protocol or the compat entry point was used to - * enter the kernel, it will either branch to the 64-bit EFI handover - * entrypoint at offset 0x390 in the image, or to the 64-bit EFI PE/COFF + * enter the kernel, it will either branch to the common 64-bit EFI stub + * entrypoint efi_stub_entry() directly, or via the 64-bit EFI PE/COFF * entrypoint efi_pe_entry(). In the former case, the bootloader must provide a * struct bootparams pointer as the third argument, so the presence of such a * pointer is used to disambiguate. @@ -37,21 +37,23 @@ * | efi32_pe_entry |---->| | | +-----------+--+ * +------------------+ | | +------+----------------+ | * | startup_32 |---->| startup_64_mixed_mode | | - * +------------------+ | | +------+----------------+ V - * | efi32_stub_entry |---->| | | +------------------+ - * +------------------+ +------------+ +---->| efi64_stub_entry | - * +-------------+----+ - * +------------+ +----------+ | - * | startup_64 |<----| efi_main |<--------------+ - * +------------+ +----------+ + * +------------------+ | | +------+----------------+ | + * | efi32_stub_entry |---->| | | | + * +------------------+ +------------+ | | + * V | + * +------------+ +----------------+ | + * | startup_64 |<----| efi_stub_entry |<--------+ + * +------------+ +----------------+ */ SYM_FUNC_START(startup_64_mixed_mode) lea efi32_boot_args(%rip), %rdx mov 0(%rdx), %edi mov 4(%rdx), %esi +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL mov 8(%rdx), %edx // saved bootparams pointer test %edx, %edx - jnz efi64_stub_entry + jnz efi_stub_entry +#endif /* * efi_pe_entry uses MS calling convention, which requires 32 bytes of * shadow space on the stack even if all arguments are passed in diff --git a/arch/x86/boot/compressed/head_32.S b/arch/x86/boot/compressed/head_32.S index 987ae727cf9f0d04..8876ffe30e9a4819 100644 --- a/arch/x86/boot/compressed/head_32.S +++ b/arch/x86/boot/compressed/head_32.S @@ -150,17 +150,6 @@ SYM_FUNC_START(startup_32) jmp *%eax SYM_FUNC_END(startup_32) -#ifdef CONFIG_EFI_STUB -SYM_FUNC_START(efi32_stub_entry) - add $0x4, %esp - movl 8(%esp), %esi /* save boot_params pointer */ - call efi_main - /* efi_main returns the possibly relocated address of startup_32 */ - jmp *%eax -SYM_FUNC_END(efi32_stub_entry) -SYM_FUNC_ALIAS(efi_stub_entry, efi32_stub_entry) -#endif - .text SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 03c4328a88cbd5d0..71c1f40a7ac067b9 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -523,19 +523,11 @@ trampoline_return: jmp *%rax SYM_CODE_END(startup_64) -#ifdef CONFIG_EFI_STUB -#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) .org 0x390 -#endif SYM_FUNC_START(efi64_stub_entry) - and $~0xf, %rsp /* realign the stack */ - movq %rdx, %rbx /* save boot_params pointer */ - call efi_main - movq %rbx,%rsi - leaq rva(startup_64)(%rax), %rax - jmp *%rax + jmp efi_stub_entry SYM_FUNC_END(efi64_stub_entry) -SYM_FUNC_ALIAS(efi_stub_entry, efi64_stub_entry) #endif .text diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 095aaa8b0ee30fb9..d6a376e52cbe1399 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -782,9 +782,9 @@ static void __noreturn enter_kernel(unsigned long kernel_addr, * On success, we jump to the relocated kernel directly and never return. * On failure, we exit to the firmware via efi_exit instead of returning. */ -asmlinkage unsigned long efi_main(efi_handle_t handle, - efi_system_table_t *sys_table_arg, - struct boot_params *boot_params) +void __noreturn efi_stub_entry(efi_handle_t handle, + efi_system_table_t *sys_table_arg, + struct boot_params *boot_params) { unsigned long bzimage_addr = (unsigned long)startup_32; unsigned long buffer_start, buffer_end; @@ -928,7 +928,19 @@ asmlinkage unsigned long efi_main(efi_handle_t handle, enter_kernel(bzimage_addr, boot_params); fail: - efi_err("efi_main() failed!\n"); + efi_err("efi_stub_entry() failed!\n"); efi_exit(handle, status); } + +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +#ifndef CONFIG_EFI_MIXED +extern __alias(efi_stub_entry) +void efi32_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, + struct boot_params *boot_params); + +extern __alias(efi_stub_entry) +void efi64_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, + struct boot_params *boot_params); +#endif +#endif From patchwork Fri Jun 2 10:12:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEF49C7EE24 for ; Fri, 2 Jun 2023 10:22:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235357AbjFBKWu (ORCPT ); Fri, 2 Jun 2023 06:22:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235342AbjFBKWX (ORCPT ); Fri, 2 Jun 2023 06:22:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83AD11AE; Fri, 2 Jun 2023 03:22:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 05C0464E71; Fri, 2 Jun 2023 10:22:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A312C4339C; Fri, 2 Jun 2023 10:22:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701340; bh=+pOeFG4I2M0NxWTqt2DlFeFWBXmc5I0TflwHlC5RHXA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZHQQ8iiabB8LyufShp63/BFbUFbzOsYVWTKUi65Muc89+1LlzSmC4seQ4t5NnekYW q3amSyw7qDJpJgAyzuClLUQ7PNqWnsuOwFCAS6/xoEyBBmv9gb+Q5OKkmYLrUAft1m +RSn0pbgLQ82JlXs70oJnJXuOpa8EyTaBm5N45QdyDML95rmNZa/Rchlvv36h1n3bX UYMM8V/chWu+W8URIXqRvjD1ZXRVbayKKYZxOklXL/wBgc+iVr6Ayn/NIXny2SCBi3 kQ5ypVcJWllJpClE0XKIB57ucg9ULU++xnhhQ3pRG8UnBLugQiySiwAraJCOAm78s/ 97tKMrlBS55lg== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 03/21] x86/decompressor: Avoid magic offsets for EFI handover entrypoint Date: Fri, 2 Jun 2023 12:12:55 +0200 Message-Id: <20230602101313.3557775-4-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3376; i=ardb@kernel.org; h=from:subject; bh=+pOeFG4I2M0NxWTqt2DlFeFWBXmc5I0TflwHlC5RHXA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywOI3HT6SmyscomyMjU58fXF7TUzazdYEE8dZQYfXb 9MJ36HZUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbyjJvhD//d1w33X2q9MVm8 xXJS1KQJbA3zJ/9Ss5v14eNruejeKZMY/tfet08tOHyzIObI+l2dwpuNLC4pM+6Z6me5JWVaT8W RN0wA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The native 32-bit or 64-bit EFI handover protocol entrypoint offset relative to the respective startup_32/64 address is described in boot_params as handover_offset, so that the special Linux/x86 aware EFI loader can find it there. When mixed mode is enabled, this single field has to describe this offset for both the 32-bit and 64-bit entrypoints, so their respective relative offsets have to be identical. Given that startup_32 and startup_64 are 0x200 bytes apart, and the EFI handover entrypoint resides at a fixed offset, the 32-bit and 64-bit versions of those entrypoints must be exactly 0x200 bytes apart as well. Currently, hard-coded fixed offsets are used to ensure this, but it is sufficient to emit the 64-bit entrypoint 0x200 bytes after the 32-bit one, wherever it happens to reside. This allows this code (which is now EFI mixed mode specific) to be moved into efi_mixed.S and out of the startup code in head_64.S. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/efi_mixed.S | 20 +++++++++++++++++++- arch/x86/boot/compressed/head_64.S | 18 ------------------ 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S index dcc562c8f7f35162..9308b595f6f0a5de 100644 --- a/arch/x86/boot/compressed/efi_mixed.S +++ b/arch/x86/boot/compressed/efi_mixed.S @@ -140,6 +140,16 @@ SYM_FUNC_START(__efi64_thunk) SYM_FUNC_END(__efi64_thunk) .code32 +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +SYM_FUNC_START(efi32_stub_entry) + add $0x4, %esp /* Discard return address */ + popl %ecx + popl %edx + popl %esi + jmp efi32_entry +SYM_FUNC_END(efi32_stub_entry) +#endif + /* * EFI service pointer must be in %edi. * @@ -220,7 +230,7 @@ SYM_FUNC_END(efi_enter32) * stub may still exit and return to the firmware using the Exit() EFI boot * service.] */ -SYM_FUNC_START(efi32_entry) +SYM_FUNC_START_LOCAL(efi32_entry) call 1f 1: pop %ebx @@ -320,6 +330,14 @@ SYM_FUNC_START(efi32_pe_entry) RET SYM_FUNC_END(efi32_pe_entry) +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL + .org efi32_stub_entry + 0x200 + .code64 +SYM_FUNC_START_NOALIGN(efi64_stub_entry) + jmp efi_stub_entry +SYM_FUNC_END(efi64_stub_entry) +#endif + .section ".rodata" /* EFI loaded image protocol GUID */ .balign 4 diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 71c1f40a7ac067b9..9f90661744741210 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -294,17 +294,6 @@ SYM_FUNC_START(startup_32) lret SYM_FUNC_END(startup_32) -#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) - .org 0x190 -SYM_FUNC_START(efi32_stub_entry) - add $0x4, %esp /* Discard return address */ - popl %ecx - popl %edx - popl %esi - jmp efi32_entry -SYM_FUNC_END(efi32_stub_entry) -#endif - .code64 .org 0x200 SYM_CODE_START(startup_64) @@ -523,13 +512,6 @@ trampoline_return: jmp *%rax SYM_CODE_END(startup_64) -#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) - .org 0x390 -SYM_FUNC_START(efi64_stub_entry) - jmp efi_stub_entry -SYM_FUNC_END(efi64_stub_entry) -#endif - .text SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) From patchwork Fri Jun 2 10:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63596C7EE2C for ; Fri, 2 Jun 2023 10:22:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235387AbjFBKWv (ORCPT ); Fri, 2 Jun 2023 06:22:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46064 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235346AbjFBKW0 (ORCPT ); Fri, 2 Jun 2023 06:22:26 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5EB61AD; Fri, 2 Jun 2023 03:22:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6245364D72; Fri, 2 Jun 2023 10:22:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D7F09C433EF; Fri, 2 Jun 2023 10:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701344; bh=GI53KZcq3ZI3RxFd9aG0XytLXRdQ1EhHXwVomNHohgI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J8XIym0Lyhfz4AvAyKuXkqdvzHiAvwTYsNh8b2C+V5KUAfkWwbPccxBAepB/jmwGx CKwDMLbaZBnojettYP09S/iRurmuFOUzshOLeTs4x8Mgmk6i3Tr8sLLfLkunMbYw6q 4Y5XFHNAOhfRyh4jTLSsKw6B8T006NY4dPwAq1ePcMYOD0hnAVad4YdbWTRG9StB9q gr1DJffn+Cah5RqWIKrFIwvQSSwEGEkACput+uX05JOhaWcU9Uh+KkDMy/3/5vukFw 29KkC8Uv0hj/6LqsQ252KOQQ3RheQKfktbz9hhdDvb5szqrnnnWEYQoaCXPER3VxOq iRy23vVmIkhVA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 04/21] x86/efistub: Clear BSS in EFI handover protocol entrypoint Date: Fri, 2 Jun 2023 12:12:56 +0200 Message-Id: <20230602101313.3557775-5-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2971; i=ardb@kernel.org; h=from:subject; bh=GI53KZcq3ZI3RxFd9aG0XytLXRdQ1EhHXwVomNHohgI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywJL4TY7ZNU/ZV6cpv2uJT3g7L0reK014z1yz8u1LG rd8NkzsKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPJXsnI8OpnWZB8rkj81fyI O0z8Dw/M4nR6NtFywRGBtW9N572qs2FkeLT00Yncj/E/xAPyZcQPL/New+NT3Ct1cEn2SsmChL/ J3AA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The so-called EFI handover protocol is value-add from the distros that permits a loader to simply copy a PE kernel image into memory and call an alternative entrypoint that is described by an embedded boot_params structure. Most implementations of this protocol do not bother to check the PE header for minimum alignment, section placement, etc, and therefore also don't clear the image's BSS, or even allocate enough memory for it. Allocating more memory on the fly is rather difficult, but at least clear the BSS region explicitly when entering in this manner, so that the EFI stub code does not get confused by global variables that were not zero-initialized correctly. When booting in mixed mode, this BSS clearing must occur before any global state is created, so clear it in the 32-bit asm entry point. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/efi_mixed.S | 14 +++++++++++++- drivers/firmware/efi/libstub/x86-stub.c | 13 +++++++++++-- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S index 9308b595f6f0a5de..8a02a151806df14c 100644 --- a/arch/x86/boot/compressed/efi_mixed.S +++ b/arch/x86/boot/compressed/efi_mixed.S @@ -142,6 +142,18 @@ SYM_FUNC_END(__efi64_thunk) .code32 #ifdef CONFIG_EFI_HANDOVER_PROTOCOL SYM_FUNC_START(efi32_stub_entry) + call 1f +1: popl %ecx + + /* Clear BSS */ + xorl %eax, %eax + leal (_bss - 1b)(%ecx), %edi + leal (_ebss - 1b)(%ecx), %ecx + subl %edi, %ecx + shrl $2, %ecx + cld + rep stosl + add $0x4, %esp /* Discard return address */ popl %ecx popl %edx @@ -334,7 +346,7 @@ SYM_FUNC_END(efi32_pe_entry) .org efi32_stub_entry + 0x200 .code64 SYM_FUNC_START_NOALIGN(efi64_stub_entry) - jmp efi_stub_entry + jmp efi_handover_entry SYM_FUNC_END(efi64_stub_entry) #endif diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index d6a376e52cbe1399..d010448dffb12cb8 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -934,12 +934,21 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } #ifdef CONFIG_EFI_HANDOVER_PROTOCOL +void efi_handover_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, + struct boot_params *boot_params) +{ + extern char _bss[], _ebss[]; + + memset(_bss, 0, _ebss - _bss); + efi_stub_entry(handle, sys_table_arg, boot_params); +} + #ifndef CONFIG_EFI_MIXED -extern __alias(efi_stub_entry) +extern __alias(efi_handover_entry) void efi32_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, struct boot_params *boot_params); -extern __alias(efi_stub_entry) +extern __alias(efi_handover_entry) void efi64_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, struct boot_params *boot_params); #endif From patchwork Fri Jun 2 10:12:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57653C7EE2F for ; Fri, 2 Jun 2023 10:22:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235417AbjFBKWx (ORCPT ); Fri, 2 Jun 2023 06:22:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235110AbjFBKWj (ORCPT ); Fri, 2 Jun 2023 06:22:39 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43F8AE4A; Fri, 2 Jun 2023 03:22:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BC4DE6150D; Fri, 2 Jun 2023 10:22:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40178C4339E; Fri, 2 Jun 2023 10:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701349; bh=ygI81p6SqewQoIn6tCgoZLUH2Qo5tPVgo3BNvwGSS3U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GZ1i4lSNKD6ujUUsHx+hPnCpXcwIdXt7P5tqVeJtNUBNzyDpooOlOTw+T21o+TsZv naJVHYDHvDFpzoY0b86GkXMYO5xTC/ZyTyTEqTpYFvgVj0WUt6vCGCLzH9QWjSzgJ+ /zlHY32uUjWUK6/K83Npl3OFBknz761RAdthFdtMKVkrDh8ngOd7itHF8mjqPA3pic oqJZSI30+M/doTqE8bM3PhF/Z88E+6P0U47FmAiCuWuIfAj25VzsJnJQSUJFw3uJmq IKHAMrwJWUWcpgcNxCn2MWywcOTMvNtaTOdSUWlYuL7/29Z3XWZDXQkyH3O+yc6HYn K+ej+axOgFS6A== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 05/21] x86/decompressor: Use proper sequence to take the address of the GOT Date: Fri, 2 Jun 2023 12:12:57 +0200 Message-Id: <20230602101313.3557775-6-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1770; i=ardb@kernel.org; h=from:subject; bh=ygI81p6SqewQoIn6tCgoZLUH2Qo5tPVgo3BNvwGSS3U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywNJ9MbLVZ4udDPN2ps4Pn9SSvGH2vsUaWfVlujOqd hx/UanRUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbydRbD/9Tq5ZsP/5x+825W lvFh/z0C51+52kyxidQ9pZbMz1SrmcnwPyQ3hI/BZYrdas6alLLsTyGux3ZJ6PLsFVASevzi650 2LgA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The 32-bit decompressor does not actually use a global offset table (GOT), but as is common for 32-bit position independent code, it uses the magic symbol _GLOBAL_OFFSET_TABLE_ as an anchor from which to derive the actual runtime addresses of other symbols, using special @GOTOFF symbol references that are resolved at link time, and populated with the distance between the address of the magic _GLOBAL_OFFSET_TABLE_ anchor and the address of the symbol in question. This means _GLOBAL_OFFSET_TABLE_ is the only symbol whose actual runtime address needs to be determined explicitly, which is one of the first things that happens in startup_32. However, it does so by taking the absolute address via the immediate field of an ADD instruction (plus a small offset), which seems to defeat the point. Fortunately, the assembler knows that _GLOBAL_OFFSET_TABLE_ is magic, and emits a special relative relocation instead, and so the resulting code works as expected. However, this is not obvious for someone reading the code, and the use of LEA with an explicit relative addend is more idiomatic so use that instead. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_32.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/boot/compressed/head_32.S b/arch/x86/boot/compressed/head_32.S index 8876ffe30e9a4819..3530465b5b85ccf3 100644 --- a/arch/x86/boot/compressed/head_32.S +++ b/arch/x86/boot/compressed/head_32.S @@ -58,7 +58,7 @@ SYM_FUNC_START(startup_32) leal (BP_scratch+4)(%esi), %esp call 1f 1: popl %edx - addl $_GLOBAL_OFFSET_TABLE_+(.-1b), %edx + leal (_GLOBAL_OFFSET_TABLE_ - 1b)(%edx), %edx /* Load new GDT */ leal gdt@GOTOFF(%edx), %eax From patchwork Fri Jun 2 10:12:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5999DC7EE24 for ; Fri, 2 Jun 2023 10:23:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235361AbjFBKXR (ORCPT ); Fri, 2 Jun 2023 06:23:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235397AbjFBKWl (ORCPT ); Fri, 2 Jun 2023 06:22:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99E53E6B; Fri, 2 Jun 2023 03:22:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2065064E80; Fri, 2 Jun 2023 10:22:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E721C433D2; Fri, 2 Jun 2023 10:22:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701353; bh=XH7auhXZovyO0ZjhIGuK6rXf321+XRWDmJGikLNf51o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=srYbptzQBzYV3/OxXwnNT/jdduhoVxDUDCfoZipJ4I3dR5FsBOg+xMN5NP/esZqNe KgaYwWj2BdXEqRO2k7MhTX23ywnGyatZaIhhh2oIQ8CzhqC/rCpsmuLfjTDmfQ8epM rjFH6gimB/KKWYakSB5KMA4UUKL7RcvRtEBeU2qszAfUx5v608d54yF4jnXkneqfMV zvux2ocAAf95TYFHt7mchz8Sx/waq5mFSe8BHpVaZTRMk3mESEy8Zmr+eCsA1Pg+ao whiAk6oQh7twHuuFCdtjE/Ve9ErQbXaZrRHJK87Df67IOl0K+jcsi64mGu56MU298h zCy6IifwYUujA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 06/21] x86/decompressor: Store boot_params pointer in callee save register Date: Fri, 2 Jun 2023 12:12:58 +0200 Message-Id: <20230602101313.3557775-7-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3619; i=ardb@kernel.org; h=from:subject; bh=XH7auhXZovyO0ZjhIGuK6rXf321+XRWDmJGikLNf51o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywLJFkUZ2V2dYbxSdMO9/QNCrrs87k70vJr201T0i4 KA2X924o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykPIOR4VTCcq7gj64idQ5T tkel7vc7PJVHsvVheWn50e0zJpS+k2NkuFW8tWtleNjlzMez98+OXsar+8nuwt/n9rU8/1NtFLk imAE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Instead of pushing and popping %RSI several times to preserve the struct boot_params pointer across the execution of the startup code, move it into a callee save register before the first call into C, and copy it back when needed. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 34 +++++++------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 9f90661744741210..2d1b0ee94929f7ec 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -405,10 +405,14 @@ SYM_CODE_START(startup_64) lretq .Lon_kernel_cs: + /* + * RSI holds a pointer to a boot_params structure provided by the + * loader, and this needs to be preserved across C function calls. So + * move it into a callee saved register. + */ + movq %rsi, %r15 - pushq %rsi call load_stage1_idt - popq %rsi #ifdef CONFIG_AMD_MEM_ENCRYPT /* @@ -421,10 +425,8 @@ SYM_CODE_START(startup_64) * detection/setup to ensure that has been done in advance of any dependent * code. */ - pushq %rsi - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ call sev_enable - popq %rsi #endif /* @@ -437,13 +439,9 @@ SYM_CODE_START(startup_64) * - Non zero RDX means trampoline needs to enable 5-level * paging. * - * RSI holds real mode data and needs to be preserved across - * this function call. */ - pushq %rsi - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - popq %rsi /* Save the trampoline address in RCX */ movq %rax, %rcx @@ -468,14 +466,9 @@ trampoline_return: * * RDI is address of the page table to use instead of page table * in trampoline memory (if required). - * - * RSI holds real mode data and needs to be preserved across - * this function call. */ - pushq %rsi leaq rva(top_pgtable)(%rbx), %rdi call cleanup_trampoline - popq %rsi /* Zero EFLAGS */ pushq $0 @@ -485,7 +478,6 @@ trampoline_return: * Copy the compressed kernel to the end of our buffer * where decompression in place becomes safe. */ - pushq %rsi leaq (_bss-8)(%rip), %rsi leaq rva(_bss-8)(%rbx), %rdi movl $(_bss - startup_32), %ecx @@ -493,7 +485,6 @@ trampoline_return: std rep movsq cld - popq %rsi /* * The GDT may get overwritten either during the copy we just did or @@ -525,30 +516,27 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) shrq $3, %rcx rep stosq - pushq %rsi call load_stage2_idt /* Pass boot_params to initialize_identity_maps() */ - movq (%rsp), %rdi + movq %r15, %rdi /* pass struct boot_params pointer */ call initialize_identity_maps - popq %rsi /* * Do the extraction, and jump to the new kernel.. */ - pushq %rsi /* Save the real mode argument */ - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ leaq boot_heap(%rip), %rsi /* malloc area for uncompression */ leaq input_data(%rip), %rdx /* input_data */ movl input_len(%rip), %ecx /* input_len */ movq %rbp, %r8 /* output target address */ movl output_len(%rip), %r9d /* decompressed length, end of relocs */ call extract_kernel /* returns kernel entry point in %rax */ - popq %rsi /* * Jump to the decompressed kernel. */ + movq %r15, %rsi jmp *%rax SYM_FUNC_END(.Lrelocated) From patchwork Fri Jun 2 10:12:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC1C6C7EE24 for ; Fri, 2 Jun 2023 10:23:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235379AbjFBKXr (ORCPT ); Fri, 2 Jun 2023 06:23:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235275AbjFBKWt (ORCPT ); Fri, 2 Jun 2023 06:22:49 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F215210C6; Fri, 2 Jun 2023 03:22:38 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7FC7D64E4E; Fri, 2 Jun 2023 10:22:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06598C4339E; Fri, 2 Jun 2023 10:22:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701357; bh=4Bh3UcI6mBb60v5tgLvoAeGffUld7BrhFCXiRmmP3sE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=geZ/QPOOKIpEeGzmFOkHVZ1D8zL1xKTkxEVDKssO4q1c7HqrdjTON3rt+Sy2PqtCf stx9KDftl3r7NLYiVvEpL09K/njN+Qzhbm2tULVn0cXX1TuVTDfKbJ9zPDUwfA8juj ruitBx7gywBEfdCqPDTFyRGMeyYNR46Hfra1dOBGlQKdPaTso0QAcaXZaB92/mPq5K CtdM5lhmpSUQ1tkgeYaGjmIaXhTPPZX28lS2hJWAPXrKAome8t/GVxwW5u4qWCfEPB j2oFsEB08YZWTlbx52KXqF5ZLw70By4GIYoqQN3XpqiTB3/tTdAlMSQXr2+Gnb15ll fILd++h7RhN2A== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 07/21] x86/decompressor: Call trampoline as a normal function Date: Fri, 2 Jun 2023 12:12:59 +0200 Message-Id: <20230602101313.3557775-8-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2554; i=ardb@kernel.org; h=from:subject; bh=4Bh3UcI6mBb60v5tgLvoAeGffUld7BrhFCXiRmmP3sE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywIrLdwW/ut2TuTv3OdMUt6AgAyV9Lx3bgpTZ36+3b mVxKwnpKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABO5YM3wV/yCv0jOD32zyzGW NT//+SmuNHt0vb6u+Iz0LY8dEzL3xzMyLJxyS/unhU/uq1VyDMvSQ10nTAj75LBjX1Ifk/uLmqL NXAA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Move the long return to switch to 32-bit mode into the trampoline code so it can be called as an ordinary function. This will allow it to be called directly from C code in a subsequent patch. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 25 +++++++++----------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 2d1b0ee94929f7ec..af45ddd8297a4a07 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -446,18 +446,9 @@ SYM_CODE_START(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* - * Load the address of trampoline_return() into RDI. - * It will be used by the trampoline to return to the main code. - */ - leaq trampoline_return(%rip), %rdi - - /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ - pushq $__KERNEL32_CS leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax - pushq %rax - lretq -trampoline_return: + call *%rax + /* Restore the stack, the 32-bit trampoline uses its own stack */ leaq rva(boot_stack_end)(%rbx), %rsp @@ -540,16 +531,22 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) jmp *%rax SYM_FUNC_END(.Lrelocated) - .code32 /* * This is the 32-bit trampoline that will be copied over to low memory. * - * RDI contains the return address (might be above 4G). * ECX contains the base address of the trampoline memory. * Non zero RDX means trampoline needs to enable 5-level paging. */ SYM_CODE_START(trampoline_32bit_src) - /* Set up data and stack segments */ + popq %rdi + /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ + pushq $__KERNEL32_CS + leaq 0f(%rip), %rax + pushq %rax + lretq + + .code32 +0: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss diff --git a/arch/x86/boot/compressed/pgtable.h b/arch/x86/boot/compressed/pgtable.h index cc9b2529a08634b4..91dbb99203fbce2d 100644 --- a/arch/x86/boot/compressed/pgtable.h +++ b/arch/x86/boot/compressed/pgtable.h @@ -6,7 +6,7 @@ #define TRAMPOLINE_32BIT_PGTABLE_OFFSET 0 #define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE -#define TRAMPOLINE_32BIT_CODE_SIZE 0x80 +#define TRAMPOLINE_32BIT_CODE_SIZE 0xA0 #define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE From patchwork Fri Jun 2 10:13:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D541C7EE2C for ; Fri, 2 Jun 2023 10:23:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235429AbjFBKXt (ORCPT ); Fri, 2 Jun 2023 06:23:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235424AbjFBKWy (ORCPT ); Fri, 2 Jun 2023 06:22:54 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52C1B194; Fri, 2 Jun 2023 03:22:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DB2FA64D72; Fri, 2 Jun 2023 10:22:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62616C433D2; Fri, 2 Jun 2023 10:22:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701362; bh=6LI2p3tMGpqgjf5Vh98U5/dlrPhBIFsp0XryDg0+x/4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OcyfFAQ6RVV+8Lu4USq6OoJtHmakXPe0UpZ5D5MSPgjEPNyDgGPzD7FDo/+AnqzRv 71onc3n/g25MjzXII0oQMdweJta0Q5vHBnNIcJPnc3udd0D/5f3bu3zsBQIXPWaWa1 mOufZDRojnZ/XXzxjT53CrfmzYFOIczhBNFi4kmXHBTrSo3YpmgRBs9IwWk9qqTLlQ kB8NZDU705pyAiNh9Ps73zgdZ4cdIIThOcZYP8B6nupWPdDgwWMTIKqLiu2CkfhXOg zStzpM/NfVUPbY7u+l5jkbnUgWN3nyB6XaWvMRywkNeK0ZmlT0hSkuFYTWwqsNAOPL IeGuyDXIzRpkg== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 08/21] x86/decompressor: Use standard calling convention for trampoline Date: Fri, 2 Jun 2023 12:13:00 +0200 Message-Id: <20230602101313.3557775-9-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4143; i=ardb@kernel.org; h=from:subject; bh=6LI2p3tMGpqgjf5Vh98U5/dlrPhBIFsp0XryDg0+x/4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywMqVqRclztc4ObWydZ06ZdprG9f6szVTZddMnxI7i eA3fLYdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCI35jD84T2fOUOZ7Umtt5Di 2vUfHLYrVuxKsyk8afpbUN/cKrRzDSPDxd1CX1//vuPOuf6jtUFfn0xvrfdHk4Ib1goLtl3mTWh mAQA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Update the trampoline code so its arguments are passed via RDI and RSI, which matches the ordinary SysV calling convention for x86_64. This will allow this code to be called directly from C. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 30 +++++++++----------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index af45ddd8297a4a07..a387cd80964e1a1e 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -443,9 +443,9 @@ SYM_CODE_START(startup_64) movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - /* Save the trampoline address in RCX */ - movq %rax, %rcx - + /* Pass the trampoline address and boolean flag as args #1 and #2 */ + movq %rax, %rdi + movq %rdx, %rsi leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax @@ -534,11 +534,11 @@ SYM_FUNC_END(.Lrelocated) /* * This is the 32-bit trampoline that will be copied over to low memory. * - * ECX contains the base address of the trampoline memory. - * Non zero RDX means trampoline needs to enable 5-level paging. + * EDI contains the base address of the trampoline memory. + * Non-zero ESI means trampoline needs to enable 5-level paging. */ SYM_CODE_START(trampoline_32bit_src) - popq %rdi + popq %r8 /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS leaq 0f(%rip), %rax @@ -552,7 +552,7 @@ SYM_CODE_START(trampoline_32bit_src) movl %eax, %ss /* Set up new stack */ - leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + leal TRAMPOLINE_32BIT_STACK_END(%edi), %esp /* Disable paging */ movl %cr0, %eax @@ -560,7 +560,7 @@ SYM_CODE_START(trampoline_32bit_src) movl %eax, %cr0 /* Check what paging mode we want to be in after the trampoline */ - testl %edx, %edx + testl %esi, %esi jz 1f /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ @@ -575,21 +575,17 @@ SYM_CODE_START(trampoline_32bit_src) jz 3f 2: /* Point CR3 to the trampoline's new top level page table */ - leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%edi), %eax movl %eax, %cr3 3: /* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */ - pushl %ecx - pushl %edx movl $MSR_EFER, %ecx rdmsr btsl $_EFER_LME, %eax /* Avoid writing EFER if no change was made (for TDX guest) */ jc 1f wrmsr -1: popl %edx - popl %ecx - +1: #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -606,14 +602,14 @@ SYM_CODE_START(trampoline_32bit_src) /* Enable PAE and LA57 (if required) paging modes */ orl $X86_CR4_PAE, %eax - testl %edx, %edx + testl %esi, %esi jz 1f orl $X86_CR4_LA57, %eax 1: movl %eax, %cr4 /* Calculate address of paging_enabled() once we are executing in the trampoline */ - leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax + leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%edi), %eax /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS @@ -630,7 +626,7 @@ SYM_CODE_END(trampoline_32bit_src) .code64 SYM_FUNC_START_LOCAL_NOALIGN(.Lpaging_enabled) /* Return from the trampoline */ - jmp *%rdi + jmp *%r8 SYM_FUNC_END(.Lpaging_enabled) /* diff --git a/arch/x86/boot/compressed/pgtable.h b/arch/x86/boot/compressed/pgtable.h index 91dbb99203fbce2d..4e8cef135226bcbb 100644 --- a/arch/x86/boot/compressed/pgtable.h +++ b/arch/x86/boot/compressed/pgtable.h @@ -14,7 +14,7 @@ extern unsigned long *trampoline_32bit; -extern void trampoline_32bit_src(void *return_ptr); +extern void trampoline_32bit_src(void *trampoline, bool enable_5lvl); #endif /* __ASSEMBLER__ */ #endif /* BOOT_COMPRESSED_PAGETABLE_H */ From patchwork Fri Jun 2 10:13:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24FC9C7EE29 for ; Fri, 2 Jun 2023 10:23:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235515AbjFBKXu (ORCPT ); Fri, 2 Jun 2023 06:23:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235388AbjFBKWz (ORCPT ); Fri, 2 Jun 2023 06:22:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C02A71A2; Fri, 2 Jun 2023 03:22:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 479C764E6C; Fri, 2 Jun 2023 10:22:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0A38C4339E; Fri, 2 Jun 2023 10:22:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701366; bh=opd5H6IVQawxAyaERFSxkNMBcud1PLTZeWa/VKUckUQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HMziCQIVqQGPlxVhREtKOQUKNdUzCNaj8norH3m8d9ok64s2AgSIVPGbFwGC8pTNZ 8fBlaSBOkbDaNKchiVuUcX2LI96YhWaKKvNkP+59H1H+RskBg0f7qqZ7lDrefC7VVF mHhH+6of6LC/Nrcoofc3CB7IFfsztqW1VUnc1iBNsQHguqko7vpVJFVad6h/ghnWc4 bAaNQqvhisciRrf0UBwxtXappXKZJqaHyQLE4D+QYwRlmE2n6084qz0ER8eLxmOYqF rPKZNXpQYn0aiIQdNj0JdD1cahzAACVGXITO+7+1/PFvxxGSN4drTlzYx4vJE1mLZB kLq95up9Nk59g== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 09/21] x86/decompressor: Avoid the need for a stack in the 32-bit trampoline Date: Fri, 2 Jun 2023 12:13:01 +0200 Message-Id: <20230602101313.3557775-10-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6184; i=ardb@kernel.org; h=from:subject; bh=opd5H6IVQawxAyaERFSxkNMBcud1PLTZeWa/VKUckUQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywKrD16I62NpYrdcd4c0/OFVmC+uFDfX//6yR3fBXM Kcl3reuo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzEMomRYYHD/GJBbqcbry4d D+vusGev3NSrOXWX8qPTTV8tFp3bpsfwT1tiIa9iwh7tqdET1x2dzqHVVd2+/7bt/QUP4x7NWsP 6gh8A X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The 32-bit trampoline no longer uses the stack for anything except performing a far return back to long mode. Currently, this stack is placed in the same page that carries the trampoline code, which means this page must be mapped writable and executable, and the stack is therefore executable as well. Replace the far return with a far jump, so that the return address can be pre-calculated and patched into the code before it is called. This removes the need for a stack entirely, and in a later patch, this will be taken advantage of by removing writable permissions from (and adding executable permissions to) this code page explicitly when booting via the EFI stub. Not touching the stack pointer also makes it more straight-forward to call the trampoline code as an ordinary 64-bit function from C code. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 45 ++++++++------------ arch/x86/boot/compressed/pgtable.h | 6 +-- arch/x86/boot/compressed/pgtable_64.c | 12 +++++- 3 files changed, 32 insertions(+), 31 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index a387cd80964e1a1e..741b4e8fefc915ea 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -449,9 +449,6 @@ SYM_CODE_START(startup_64) leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax - /* Restore the stack, the 32-bit trampoline uses its own stack */ - leaq rva(boot_stack_end)(%rbx), %rsp - /* * cleanup_trampoline() would restore trampoline memory. * @@ -537,24 +534,22 @@ SYM_FUNC_END(.Lrelocated) * EDI contains the base address of the trampoline memory. * Non-zero ESI means trampoline needs to enable 5-level paging. */ + .section ".rodata", "a", @progbits SYM_CODE_START(trampoline_32bit_src) - popq %r8 /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS leaq 0f(%rip), %rax pushq %rax lretq + /* + * The 32-bit code below will do a far jump back to long mode and end + * up here after reconfiguring the number of paging levels. + */ +.Lret: retq + .code32 -0: /* Set up data and stack segments */ - movl $__KERNEL_DS, %eax - movl %eax, %ds - movl %eax, %ss - - /* Set up new stack */ - leal TRAMPOLINE_32BIT_STACK_END(%edi), %esp - - /* Disable paging */ +0: /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 @@ -608,26 +603,22 @@ SYM_CODE_START(trampoline_32bit_src) 1: movl %eax, %cr4 - /* Calculate address of paging_enabled() once we are executing in the trampoline */ - leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%edi), %eax - - /* Prepare the stack for far return to Long Mode */ - pushl $__KERNEL_CS - pushl %eax - /* Enable paging again. */ movl %cr0, %eax btsl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - lret + /* + * Return to the 64-bit calling code using LJMP rather than LRET, to + * avoid the need for a 32-bit addressable stack. The destination + * address will be adjusted after the template code is copied into a + * 32-bit addressable buffer. + */ +.Ljmp: ljmpl $__KERNEL_CS, $(.Lret - trampoline_32bit_src) SYM_CODE_END(trampoline_32bit_src) - .code64 -SYM_FUNC_START_LOCAL_NOALIGN(.Lpaging_enabled) - /* Return from the trampoline */ - jmp *%r8 -SYM_FUNC_END(.Lpaging_enabled) +/* keep this right after trampoline_32bit_src() so we can infer its size */ +SYM_DATA(trampoline_ljmp_imm_offset, .word .Ljmp + 1 - trampoline_32bit_src) /* * The trampoline code has a size limit. @@ -636,7 +627,7 @@ SYM_FUNC_END(.Lpaging_enabled) */ .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE - .code32 + .text SYM_FUNC_START_LOCAL_NOALIGN(.Lno_longmode) /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: diff --git a/arch/x86/boot/compressed/pgtable.h b/arch/x86/boot/compressed/pgtable.h index 4e8cef135226bcbb..131488f50af55d0a 100644 --- a/arch/x86/boot/compressed/pgtable.h +++ b/arch/x86/boot/compressed/pgtable.h @@ -6,9 +6,7 @@ #define TRAMPOLINE_32BIT_PGTABLE_OFFSET 0 #define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE -#define TRAMPOLINE_32BIT_CODE_SIZE 0xA0 - -#define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE +#define TRAMPOLINE_32BIT_CODE_SIZE 0x80 #ifndef __ASSEMBLER__ @@ -16,5 +14,7 @@ extern unsigned long *trampoline_32bit; extern void trampoline_32bit_src(void *trampoline, bool enable_5lvl); +extern const u16 trampoline_ljmp_imm_offset; + #endif /* __ASSEMBLER__ */ #endif /* BOOT_COMPRESSED_PAGETABLE_H */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 2ac12ff4111bf8c0..09fc18180929fab3 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -109,6 +109,7 @@ static unsigned long find_trampoline_placement(void) struct paging_config paging_prepare(void *rmode) { struct paging_config paging_config = {}; + void *tramp_code; /* Initialize boot_params. Required for cmdline_find_option_bool(). */ boot_params = rmode; @@ -143,9 +144,18 @@ struct paging_config paging_prepare(void *rmode) memset(trampoline_32bit, 0, TRAMPOLINE_32BIT_SIZE); /* Copy trampoline code in place */ - memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long), + tramp_code = memcpy(trampoline_32bit + + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long), &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE); + /* + * Avoid the need for a stack in the 32-bit trampoline code, by using + * LJMP rather than LRET to return back to long mode. LJMP takes an + * immediate absolute address, so we have to adjust that based on the + * placement of the trampoline. + */ + *(u32 *)(tramp_code + trampoline_ljmp_imm_offset) += (unsigned long)tramp_code; + /* * The code below prepares page table in trampoline memory. * From patchwork Fri Jun 2 10:13:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A107C7EE29 for ; Fri, 2 Jun 2023 10:24:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235481AbjFBKYG (ORCPT ); Fri, 2 Jun 2023 06:24:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235470AbjFBKXU (ORCPT ); Fri, 2 Jun 2023 06:23:20 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55397E62; Fri, 2 Jun 2023 03:22:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A6F5164E4E; Fri, 2 Jun 2023 10:22:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29891C433A1; Fri, 2 Jun 2023 10:22:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701371; bh=hFZ/SCviIEc1YZbiLBLlEDyweQ6PGSxYwFHksamIzHs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Nfff8boM+8RvgzsdhvHcWIJlC+g8g4Wq7plDTW+pSwMYIQ+FA/pQdAwR9wmQEH9Hu tozvIv+dHkPk86T9EBsrWvOOuGTPKW7gZ1ZHFUVu2k6Ovt3PHh+gxoR8+qQPsVhlDW I6Y/rXYNtS0G3JWI7pYAWi7AWH91sSo+67RvGTNLsw5fRCo9oURHj2UeXSnUhWALgD JPnuP1Y3H72MpzNEBpqLfzxVt5VauFyizJ1+959HL6IdT4uP5VmAQ8c1oz24VJVXBI nCzjaghvF4ZqJNjwJRi87HnpiVPxe0xMZ/kWrnWUBOv80HiGGxF9Se1bYlP7zMhhrY WEN/khJAwidrA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 10/21] x86/decompressor: Call trampoline directly from C code Date: Fri, 2 Jun 2023 12:13:02 +0200 Message-Id: <20230602101313.3557775-11-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4966; i=ardb@kernel.org; h=from:subject; bh=hFZ/SCviIEc1YZbiLBLlEDyweQ6PGSxYwFHksamIzHs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywJrzy7b+fukf7pKxUXarwbGeLatCFi5a9qu7hWG6Y BHjqmPzO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBE2NYwMvQlzjl3wOv7IbPb G1zn37z3xyPONlq48lbd/YbFa+5dmvWK4X+69v0z8yrf/xZ/78K/z/ENe0idEl+Imabpz9+VvoJ OAewA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Instead of returning to the asm calling code to invoke the trampoline, call it straight from the C code that sets the scene. That way, the struct return type is no longer needed for returning two values, and the call can be made conditional more cleanly in a subsequent patch. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 20 +++----------- arch/x86/boot/compressed/pgtable_64.c | 28 ++++++++------------ 2 files changed, 15 insertions(+), 33 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 741b4e8fefc915ea..a60ec9283bd760e3 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -430,24 +430,12 @@ SYM_CODE_START(startup_64) #endif /* - * paging_prepare() sets up the trampoline and checks if we need to - * enable 5-level paging. - * - * paging_prepare() returns a two-quadword structure which lands - * into RDX:RAX: - * - Address of the trampoline is returned in RAX. - * - Non zero RDX means trampoline needs to enable 5-level - * paging. - * + * set_paging_levels() updates the number of paging levels using a + * trampoline in 32-bit addressable memory if the current number does + * not match the desired number. */ movq %r15, %rdi /* pass struct boot_params pointer */ - call paging_prepare - - /* Pass the trampoline address and boolean flag as args #1 and #2 */ - movq %rax, %rdi - movq %rdx, %rsi - leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax - call *%rax + call set_paging_levels /* * cleanup_trampoline() would restore trampoline memory. diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 09fc18180929fab3..b62b6819dcdd01be 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -16,11 +16,6 @@ unsigned int __section(".data") pgdir_shift = 39; unsigned int __section(".data") ptrs_per_p4d = 1; #endif -struct paging_config { - unsigned long trampoline_start; - unsigned long l5_required; -}; - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -106,10 +101,10 @@ static unsigned long find_trampoline_placement(void) return bios_start - TRAMPOLINE_32BIT_SIZE; } -struct paging_config paging_prepare(void *rmode) +asmlinkage void set_paging_levels(void *rmode) { - struct paging_config paging_config = {}; - void *tramp_code; + void (*toggle_la57)(void *trampoline, bool enable_5lvl); + bool l5_required = false; /* Initialize boot_params. Required for cmdline_find_option_bool(). */ boot_params = rmode; @@ -130,12 +125,10 @@ struct paging_config paging_prepare(void *rmode) !cmdline_find_option_bool("no5lvl") && native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { - paging_config.l5_required = 1; + l5_required = true; } - paging_config.trampoline_start = find_trampoline_placement(); - - trampoline_32bit = (unsigned long *)paging_config.trampoline_start; + trampoline_32bit = (unsigned long *)find_trampoline_placement(); /* Preserve trampoline memory */ memcpy(trampoline_save, trampoline_32bit, TRAMPOLINE_32BIT_SIZE); @@ -144,7 +137,7 @@ struct paging_config paging_prepare(void *rmode) memset(trampoline_32bit, 0, TRAMPOLINE_32BIT_SIZE); /* Copy trampoline code in place */ - tramp_code = memcpy(trampoline_32bit + + toggle_la57 = memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long), &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE); @@ -154,7 +147,8 @@ struct paging_config paging_prepare(void *rmode) * immediate absolute address, so we have to adjust that based on the * placement of the trampoline. */ - *(u32 *)(tramp_code + trampoline_ljmp_imm_offset) += (unsigned long)tramp_code; + *(u32 *)((u8 *)toggle_la57 + trampoline_ljmp_imm_offset) += + (unsigned long)toggle_la57; /* * The code below prepares page table in trampoline memory. @@ -170,10 +164,10 @@ struct paging_config paging_prepare(void *rmode) * We are not going to use the page table in trampoline memory if we * are already in the desired paging mode. */ - if (paging_config.l5_required == !!(native_read_cr4() & X86_CR4_LA57)) + if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) goto out; - if (paging_config.l5_required) { + if (l5_required) { /* * For 4- to 5-level paging transition, set up current CR3 as * the first and the only entry in a new top-level page table. @@ -196,7 +190,7 @@ struct paging_config paging_prepare(void *rmode) } out: - return paging_config; + toggle_la57(trampoline_32bit, l5_required); } void cleanup_trampoline(void *pgtable) From patchwork Fri Jun 2 10:13:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7FDDC7EE24 for ; Fri, 2 Jun 2023 10:24:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235043AbjFBKYI (ORCPT ); Fri, 2 Jun 2023 06:24:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235404AbjFBKXd (ORCPT ); Fri, 2 Jun 2023 06:23:33 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F94AE49; Fri, 2 Jun 2023 03:22:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0C21461626; Fri, 2 Jun 2023 10:22:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 888DCC4339C; Fri, 2 Jun 2023 10:22:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701375; bh=u+mBV9LASMmynY9CQFGbcEYb3DED+j3/4fMymyiLQXA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SEw5WzIvMZEi1C6SDbCrEwfXPL4dMy5g6DuXLSNjQTOMLr0LYqwl/gLdfVE9oiTHy 7ltF6fpA+ViZ/h0E8bHq15dKnJ18g3g/rHoUg7S1SbJnvickMwjo817z920zium54A kD/JWQlvWMUaF7WZgle0hq4hOiP2sG4IIfykhRW0pjDZfte5cvwyxDgVX6pGDlnGmy J4fxmv1zG7qKIjIHsaOnj52gk6ooneg0CKvDs3lfMhz3bqRor8LkSPejWrs1Sgl/7h d5o9rJ3Ccogv/VuqybG+UHOzdwgp9mWJ71aBhNRHdTBnZ1OWbajAr3NDBYbXgOY6fe FBGb0E60gLGEA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 11/21] x86/decompressor: Only call the trampoline when changing paging levels Date: Fri, 2 Jun 2023 12:13:03 +0200 Message-Id: <20230602101313.3557775-12-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3739; i=ardb@kernel.org; h=from:subject; bh=u+mBV9LASMmynY9CQFGbcEYb3DED+j3/4fMymyiLQXA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywFoL2/M9ifYK4fo86pwGr3K0tW+xTTjFrfHV0uPNx kvdj107SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQijjD8d9htIMtz8Mu1JUxd 29Qqlx6LNWJvMz0mffD/nMA53B95ZzIytFsYxCXG37x4Zrqd8waB/1r+f1q5VC/cDpGv2fbaW2A 1JwA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Since the current and desired number of paging levels are known when the trampoline is being prepared, avoid calling the trampoline at all if it is clear that calling it is not going to result in a change to the number of paging levels. Given that the CPU is already running in long mode, the PAE and LA57 settings are necessarily consistent with the currently active page tables - the only difference is that CR4.MCE will always be preserved in this case, but it will be cleared by the real kernel startup code if CONFIG_X86_MCE is not enabled. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 21 +------------------- arch/x86/boot/compressed/pgtable_64.c | 18 +++++++---------- 2 files changed, 8 insertions(+), 31 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index a60ec9283bd760e3..403c96dae34d9c6d 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -387,10 +387,6 @@ SYM_CODE_START(startup_64) * For the trampoline, we need the top page table to reside in lower * memory as we don't have a way to load 64-bit values into CR3 in * 32-bit mode. - * - * We go though the trampoline even if we don't have to: if we're - * already in a desired paging mode. This way the trampoline code gets - * tested on every boot. */ /* Make sure we have GDT with 32-bit code segment */ @@ -542,25 +538,10 @@ SYM_CODE_START(trampoline_32bit_src) btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Check what paging mode we want to be in after the trampoline */ - testl %esi, %esi - jz 1f - - /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ - movl %cr4, %eax - testl $X86_CR4_LA57, %eax - jnz 3f - jmp 2f -1: - /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ - movl %cr4, %eax - testl $X86_CR4_LA57, %eax - jz 3f -2: /* Point CR3 to the trampoline's new top level page table */ leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%edi), %eax movl %eax, %cr3 -3: + /* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */ movl $MSR_EFER, %ecx rdmsr diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index b62b6819dcdd01be..b92cf1d6e156d5f6 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -128,6 +128,13 @@ asmlinkage void set_paging_levels(void *rmode) l5_required = true; } + /* + * We are not going to use the trampoline if we + * are already in the desired paging mode. + */ + if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) + return; + trampoline_32bit = (unsigned long *)find_trampoline_placement(); /* Preserve trampoline memory */ @@ -155,18 +162,8 @@ asmlinkage void set_paging_levels(void *rmode) * * The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. - * - * If switching is not required, the page table is unused: trampoline - * code wouldn't touch CR3. */ - /* - * We are not going to use the page table in trampoline memory if we - * are already in the desired paging mode. - */ - if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) - goto out; - if (l5_required) { /* * For 4- to 5-level paging transition, set up current CR3 as @@ -189,7 +186,6 @@ asmlinkage void set_paging_levels(void *rmode) (void *)src, PAGE_SIZE); } -out: toggle_la57(trampoline_32bit, l5_required); } From patchwork Fri Jun 2 10:13:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78C0AC7EE24 for ; Fri, 2 Jun 2023 10:24:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235439AbjFBKYM (ORCPT ); Fri, 2 Jun 2023 06:24:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235493AbjFBKXo (ORCPT ); Fri, 2 Jun 2023 06:23:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC9BF10D2; Fri, 2 Jun 2023 03:23:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6BF0E64D72; Fri, 2 Jun 2023 10:23:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E63D4C433EF; Fri, 2 Jun 2023 10:22:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701379; bh=CnFv45/Vh35jv2ecaVpF4w7431pbCKxy7DMiPDdcENI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iUQjx7k13wrKSTU1kAQSwHaPTJ3vJ4TbY6F0wPnBR9oKHGQa9a4vablgt2la4spVl Gn8hOZQstppOvz+pGJRjqID6yolYtv/9GGOl6lv04CvOI+/0hZKi924ZLPhcHnjGaz HgvP1reOu2hl29sNFXl3oVz9alo/BgaLzJc3ksXstftmI2gDkX5p9B2RlYEa3YAP9Z 9wjXXWnSA7nFhFo1MHAHfjl/HaYRh+vX7T0nvMTJTNr2DWwZam/hx0QlwF6ysc27bq RyzHGM4YO4ybNwGOtUP/g4hE39gro4Z2SDqGvHxeJlYy6yJ6if5icImBB5cw/zcTSv yVQ5i6HQP8N7A== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 12/21] x86/decompressor: Merge trampoline cleanup with switching code Date: Fri, 2 Jun 2023 12:13:04 +0200 Message-Id: <20230602101313.3557775-13-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4166; i=ardb@kernel.org; h=from:subject; bh=CnFv45/Vh35jv2ecaVpF4w7431pbCKxy7DMiPDdcENI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywLpmfZ+JtbF2N7ZGfVrba5ei/557++WT3HeNa+4/O FKycu3djlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRuuMM/xP16rc73b79aO/O gsmcTYs35zOa78u4cPeO7qKvk/Xqrs1k+M2WO7M5R/7Qs9Q5O/XOlfxXV1NP6Ei/vnzB5n41mbs f3VgA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Now that the trampoline setup code and the actual invocation of it are all done from the C routine, the trampoline cleanup can be merged into it as well, instead of returning to asm just to call another C function. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_64.S | 13 +++------ arch/x86/boot/compressed/pgtable_64.c | 28 ++++++++------------ 2 files changed, 15 insertions(+), 26 deletions(-) diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 403c96dae34d9c6d..b5bd6be035a7b7ec 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -429,19 +429,14 @@ SYM_CODE_START(startup_64) * set_paging_levels() updates the number of paging levels using a * trampoline in 32-bit addressable memory if the current number does * not match the desired number. + * + * RSI is the relocated address of the page table to use instead of + * page table in trampoline memory (if required). */ movq %r15, %rdi /* pass struct boot_params pointer */ + leaq rva(top_pgtable)(%rbx), %rsi call set_paging_levels - /* - * cleanup_trampoline() would restore trampoline memory. - * - * RDI is address of the page table to use instead of page table - * in trampoline memory (if required). - */ - leaq rva(top_pgtable)(%rbx), %rdi - call cleanup_trampoline - /* Zero EFLAGS */ pushq $0 popfq diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index b92cf1d6e156d5f6..eeddad8c8335655e 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -101,9 +101,10 @@ static unsigned long find_trampoline_placement(void) return bios_start - TRAMPOLINE_32BIT_SIZE; } -asmlinkage void set_paging_levels(void *rmode) +asmlinkage void set_paging_levels(void *rmode, void *pgtable) { void (*toggle_la57)(void *trampoline, bool enable_5lvl); + void *trampoline_pgtable; bool l5_required = false; /* Initialize boot_params. Required for cmdline_find_option_bool(). */ @@ -133,7 +134,7 @@ asmlinkage void set_paging_levels(void *rmode) * are already in the desired paging mode. */ if (l5_required == !!(native_read_cr4() & X86_CR4_LA57)) - return; + goto out; trampoline_32bit = (unsigned long *)find_trampoline_placement(); @@ -163,6 +164,8 @@ asmlinkage void set_paging_levels(void *rmode) * The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. */ + trampoline_pgtable = trampoline_32bit + + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long); if (l5_required) { /* @@ -182,31 +185,21 @@ asmlinkage void set_paging_levels(void *rmode) * may be above 4G. */ src = *(unsigned long *)__native_read_cr3() & PAGE_MASK; - memcpy(trampoline_32bit + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long), - (void *)src, PAGE_SIZE); + memcpy(trampoline_pgtable, (void *)src, PAGE_SIZE); } toggle_la57(trampoline_32bit, l5_required); -} - -void cleanup_trampoline(void *pgtable) -{ - void *trampoline_pgtable; - - trampoline_pgtable = trampoline_32bit + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long); /* - * Move the top level page table out of trampoline memory, - * if it's there. + * Move the top level page table out of trampoline memory. */ - if ((void *)__native_read_cr3() == trampoline_pgtable) { - memcpy(pgtable, trampoline_pgtable, PAGE_SIZE); - native_write_cr3((unsigned long)pgtable); - } + memcpy(pgtable, trampoline_pgtable, PAGE_SIZE); + native_write_cr3((unsigned long)pgtable); /* Restore trampoline memory */ memcpy(trampoline_32bit, trampoline_save, TRAMPOLINE_32BIT_SIZE); +out: /* Initialize variables for 5-level paging */ #ifdef CONFIG_X86_5LEVEL if (__read_cr4() & X86_CR4_LA57) { @@ -215,4 +208,5 @@ void cleanup_trampoline(void *pgtable) ptrs_per_p4d = 512; } #endif + return; } From patchwork Fri Jun 2 10:13:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E4E3C7EE24 for ; Fri, 2 Jun 2023 10:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235542AbjFBKYS (ORCPT ); Fri, 2 Jun 2023 06:24:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235170AbjFBKXp (ORCPT ); Fri, 2 Jun 2023 06:23:45 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4530E10D7; Fri, 2 Jun 2023 03:23:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF40764E6C; Fri, 2 Jun 2023 10:23:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 512DBC433A0; Fri, 2 Jun 2023 10:23:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701384; bh=qSgmzewWVKnnUee0WJqteVdeET11EUDulO4v6yJCdIc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FQVut3s5U9T+3LYhk66bKvpmn6FqcxlkNLEn/8Nh6YzIjqtDebcucSUh++WaKzoDe /dcyEM3ksvmUCUDWdDC9hAgHhfA66pPh17miBS7H9aoNjAobsa6nOfhUCtlAcTLjsA vjndQRELfccd7QcI3emzayAoF8aZY7fuvV73omVWXzV0aXCZJtbEYy3kucH8ClkMCu +ftHmqNUhAkZtGH159xqpt7mGANfGkOqeuEqTAMs3Orkk54A18DyCSHQKi+y+vu8R4 NgdcUgUWAoBle4h+HcqEKH7cIWQlcnjSvkcHTdhGgwp6NT/bVN1Jg9w9oZQzKodWVC Mdv93Ri5t8FEQ== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 13/21] x86/efistub: Perform 4/5 level paging switch from the stub Date: Fri, 2 Jun 2023 12:13:05 +0200 Message-Id: <20230602101313.3557775-14-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10070; i=ardb@kernel.org; h=from:subject; bh=qSgmzewWVKnnUee0WJqteVdeET11EUDulO4v6yJCdIc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywPr78zd3BNqtP3xL96b0OtaeuL++Nt1HWk7enfiPp V0qbBd/RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhI0F+G/4EZS9Lfn681lQnb tvCu2Z0bwnvfRf/gLZ/mnay58dLpk1GMDH+4ZdjqEroFVmebT8g5bNW271F38ubAJjdjbZlJy5f MYgUA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org In preparation for updating the EFI stub boot flow to avoid the bare metal decompressor code altogether, implement the support code for switching between 4 and 5 levels of paging before jumping to the kernel proper. This reuses the newly refactored trampoline that the bare metal decompressor uses, but relies on EFI APIs to allocate 32-bit addressable memory and remap it with the appropriate permissions. Given that the bare metal decompressor will no longer call into the trampoline if the number of paging levels is already set correctly, it is no longer needed to remove NX restrictions from the memory range where this trampoline may end up. Acked-by: Kirill A. Shutemov Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/Makefile | 1 + drivers/firmware/efi/libstub/efi-stub-helper.c | 2 + drivers/firmware/efi/libstub/efistub.h | 1 + drivers/firmware/efi/libstub/x86-5lvl.c | 94 ++++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 40 +++------ drivers/firmware/efi/libstub/x86-stub.h | 17 ++++ 6 files changed, 129 insertions(+), 26 deletions(-) diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index 16d64a34d1e19465..ae8874401a9f1490 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -88,6 +88,7 @@ lib-$(CONFIG_EFI_GENERIC_STUB) += efi-stub.o string.o intrinsics.o systable.o \ lib-$(CONFIG_ARM) += arm32-stub.o lib-$(CONFIG_ARM64) += arm64.o arm64-stub.o smbios.o lib-$(CONFIG_X86) += x86-stub.o +lib-$(CONFIG_X86_64) += x86-5lvl.o lib-$(CONFIG_RISCV) += riscv.o riscv-stub.o lib-$(CONFIG_LOONGARCH) += loongarch.o loongarch-stub.o diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index 1e0203d74691ffcc..51779279fbff21b5 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -73,6 +73,8 @@ efi_status_t efi_parse_options(char const *cmdline) efi_loglevel = CONSOLE_LOGLEVEL_QUIET; } else if (!strcmp(param, "noinitrd")) { efi_noinitrd = true; + } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { + efi_no5lvl = true; } else if (!strcmp(param, "efi") && val) { efi_nochunk = parse_option_str(val, "nochunk"); efi_novamap |= parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 8659a01664b85d95..191698e8489d82e7 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -33,6 +33,7 @@ #define EFI_ALLOC_LIMIT ULONG_MAX #endif +extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; diff --git a/drivers/firmware/efi/libstub/x86-5lvl.c b/drivers/firmware/efi/libstub/x86-5lvl.c new file mode 100644 index 0000000000000000..f7284f6270abcc18 --- /dev/null +++ b/drivers/firmware/efi/libstub/x86-5lvl.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include + +#include +#include +#include + +#include "efistub.h" +#include "x86-stub.h" + +bool efi_no5lvl; + +static void (*la57_toggle)(void *trampoline, bool enable_5lvl); + +static const struct desc_struct gdt[] = { + [GDT_ENTRY_KERNEL32_CS] = GDT_ENTRY_INIT(0xc09b, 0, 0xfffff), + [GDT_ENTRY_KERNEL_CS] = GDT_ENTRY_INIT(0xa09b, 0, 0xfffff), +}; + +/* + * Enabling (or disabling) 5 level paging is tricky, because it can only be + * done from 32-bit mode with paging disabled. This means not only that the + * code itself must be running from 32-bit addressable physical memory, but + * also that the root page table must be 32-bit addressable, as we cannot + * program a 64-bit value into CR3 when running in 32-bit mode. + */ +efi_status_t efi_setup_5level_paging(void) +{ + u8 tmpl_size = (u8 *)&trampoline_ljmp_imm_offset - (u8 *)&trampoline_32bit_src; + efi_status_t status; + u8 *la57_code; + + if (!efi_is_64bit()) + return EFI_SUCCESS; + + /* check for 5 level paging support */ + if (native_cpuid_eax(0) < 7 || + !(native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) + return EFI_SUCCESS; + + /* allocate some 32-bit addressable memory for code and a page table */ + status = efi_allocate_pages(2 * PAGE_SIZE, (unsigned long *)&la57_code, + U32_MAX); + if (status != EFI_SUCCESS) + return status; + + la57_toggle = memcpy(la57_code, trampoline_32bit_src, tmpl_size); + memset(la57_code + tmpl_size, 0x90, PAGE_SIZE - tmpl_size); + + /* + * To avoid having to allocate a 32-bit addressable stack, we use a + * ljmp to switch back to long mode. However, this takes an absolute + * address, so we have to poke it in at runtime. + */ + *(u32 *)&la57_code[trampoline_ljmp_imm_offset] += (unsigned long)la57_code; + + efi_adjust_memory_range_protection((unsigned long)la57_toggle, PAGE_SIZE); + + return EFI_SUCCESS; +} + +void efi_5level_switch(void) +{ + bool want_la57 = IS_ENABLED(CONFIG_X86_5LEVEL) && !efi_no5lvl; + bool have_la57 = native_read_cr4() & X86_CR4_LA57; + bool need_toggle = want_la57 ^ have_la57; + u64 *pgt = (void *)la57_toggle + PAGE_SIZE; + u64 *cr3 = (u64 *)__native_read_cr3(); + u64 *new_cr3; + + if (!la57_toggle || !need_toggle) + return; + + if (!have_la57) { + /* + * We are going to enable 5 level paging, so we need to + * allocate a root level page from the 32-bit addressable + * physical region, and plug the existing hierarchy into it. + */ + new_cr3 = memset(pgt, 0, PAGE_SIZE); + new_cr3[0] = (u64)cr3 | _PAGE_TABLE_NOENC; + } else { + // take the new root table pointer from the current entry #0 + new_cr3 = (u64 *)(cr3[0] & PAGE_MASK); + + // copy the new root level table if it is not 32-bit addressable + if ((u64)new_cr3 > U32_MAX) + new_cr3 = memcpy(pgt, new_cr3, PAGE_SIZE); + } + + native_load_gdt(&(struct desc_ptr){ sizeof(gdt) - 1, (u64)gdt }); + + la57_toggle(new_cr3, want_la57); +} diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index d010448dffb12cb8..c55c028cf911bce0 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -17,6 +17,7 @@ #include #include "efistub.h" +#include "x86-stub.h" /* Maximum physical address for 64-bit kernel with 4-level paging */ #define MAXMEM_X86_64_4LEVEL (1ull << 46) @@ -212,8 +213,8 @@ static void retrieve_apple_device_properties(struct boot_params *boot_params) } } -static void -adjust_memory_range_protection(unsigned long start, unsigned long size) +void efi_adjust_memory_range_protection(unsigned long start, + unsigned long size) { efi_status_t status; efi_gcd_memory_space_desc_t desc; @@ -267,35 +268,14 @@ adjust_memory_range_protection(unsigned long start, unsigned long size) } } -/* - * Trampoline takes 2 pages and can be loaded in first megabyte of memory - * with its end placed between 128k and 640k where BIOS might start. - * (see arch/x86/boot/compressed/pgtable_64.c) - * - * We cannot find exact trampoline placement since memory map - * can be modified by UEFI, and it can alter the computed address. - */ - -#define TRAMPOLINE_PLACEMENT_BASE ((128 - 8)*1024) -#define TRAMPOLINE_PLACEMENT_SIZE (640*1024 - (128 - 8)*1024) - void startup_32(struct boot_params *boot_params); static void setup_memory_protection(unsigned long image_base, unsigned long image_size) { - /* - * Allow execution of possible trampoline used - * for switching between 4- and 5-level page tables - * and relocated kernel image. - */ - - adjust_memory_range_protection(TRAMPOLINE_PLACEMENT_BASE, - TRAMPOLINE_PLACEMENT_SIZE); - #ifdef CONFIG_64BIT if (image_base != (unsigned long)startup_32) - adjust_memory_range_protection(image_base, image_size); + efi_adjust_memory_range_protection(image_base, image_size); #else /* * Clear protection flags on a whole range of possible @@ -305,8 +285,8 @@ setup_memory_protection(unsigned long image_base, unsigned long image_size) * need to remove possible protection on relocated image * itself disregarding further relocations. */ - adjust_memory_range_protection(LOAD_PHYSICAL_ADDR, - KERNEL_IMAGE_SIZE - LOAD_PHYSICAL_ADDR); + efi_adjust_memory_range_protection(LOAD_PHYSICAL_ADDR, + KERNEL_IMAGE_SIZE - LOAD_PHYSICAL_ADDR); #endif } @@ -804,6 +784,12 @@ void __noreturn efi_stub_entry(efi_handle_t handle, efi_dxe_table = NULL; } + status = efi_setup_5level_paging(); + if (status != EFI_SUCCESS) { + efi_err("efi_setup_5level_paging() failed!\n"); + goto fail; + } + /* * If the kernel isn't already loaded at a suitable address, * relocate it. @@ -922,6 +908,8 @@ void __noreturn efi_stub_entry(efi_handle_t handle, goto fail; } + efi_5level_switch(); + if (IS_ENABLED(CONFIG_X86_64)) /* add offset of startup_64() */ bzimage_addr += 0x200; diff --git a/drivers/firmware/efi/libstub/x86-stub.h b/drivers/firmware/efi/libstub/x86-stub.h new file mode 100644 index 0000000000000000..37c5a36b9d8cf9b2 --- /dev/null +++ b/drivers/firmware/efi/libstub/x86-stub.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include + +extern void trampoline_32bit_src(void *, bool); +extern const u16 trampoline_ljmp_imm_offset; + +void efi_adjust_memory_range_protection(unsigned long start, + unsigned long size); + +#ifdef CONFIG_X86_64 +efi_status_t efi_setup_5level_paging(void); +void efi_5level_switch(void); +#else +static inline efi_status_t efi_setup_5level_paging(void) { return EFI_SUCCESS; } +static inline void efi_5level_switch(void) {} +#endif From patchwork Fri Jun 2 10:13:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E22FC7EE24 for ; Fri, 2 Jun 2023 10:24:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235552AbjFBKYY (ORCPT ); Fri, 2 Jun 2023 06:24:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235353AbjFBKXq (ORCPT ); Fri, 2 Jun 2023 06:23:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BFF710E0; Fri, 2 Jun 2023 03:23:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3AB7564D72; Fri, 2 Jun 2023 10:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF6D6C4339C; Fri, 2 Jun 2023 10:23:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701388; bh=ILOYseVDB35z5GQ5Qwen/Xris3QVBcVN/ymf61ysRYw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W02pzc04ErlYysGiY4iN2FSYtKQ1szh/aHdjkX82grtpZn+Jg6s+ctlE72YdTMaeX 2yi/5H4XPP+qlAZe2v0I94rT5sCOXQpz65tAXOkA5AWBT20lCOVviL2g/HGj3FDhPl MqKzO9ldIkIRUtBtvx7U8WxWiHfFCP7Sh02wpaAJ84J6DMa0LpFW+fLUkGZInh5M5I pt5J/2YKHN7n08ZQe2X3zeapqlHDITDXsWWc5RYt1UJvYd0lklToxtAUpxdGKW6Ew1 FcUAjrak44D1VAl/KJgoccKRnYeZTJvByOv2gPbifU/ZwrDOCS2cTofOxeuNC2izfq 6RLK8qMx237Gg== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 14/21] x86/efistub: Prefer EFI memory attributes protocol over DXE services Date: Fri, 2 Jun 2023 12:13:06 +0200 Message-Id: <20230602101313.3557775-15-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3307; i=ardb@kernel.org; h=from:subject; bh=ILOYseVDB35z5GQ5Qwen/Xris3QVBcVN/ymf61ysRYw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywIaABb6yz/Z+ZPMSMXkxSV5dfOv+sJwzdbtvLp3Ef m6eOHNDRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiI4wpGhpkWfmmvej89nlIp s7J9caeN7I2V5eFtD++6tXsfevOdo5WR4eSOc5ctPJ1aKqRaOrLNNzRu2eX9zCFivdSWK3Lzjr9 3ZAIA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Currently, the EFI stub relies on DXE services in some cases to clear non-execute restrictions from page allocations that need to be executable. This is dodgy, because DXE services are not specified by UEFI but by PI, and they are not intended for consumption by OS loaders. However, no alternative existed at the time. Now, there is a new UEFI protocol that should be used instead, so if it exists, prefer it over the DXE services calls. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/x86-stub.c | 29 ++++++++++++++------ 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index c55c028cf911bce0..2d3282d2ed6eb756 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -26,6 +26,7 @@ const efi_system_table_t *efi_system_table; const efi_dxe_services_table_t *efi_dxe_table; u32 image_offset __section(".data"); static efi_loaded_image_t *image = NULL; +static efi_memory_attribute_protocol_t *memattr; static efi_status_t preserve_pci_rom_image(efi_pci_io_protocol_t *pci, struct pci_setup_rom **__rom) @@ -222,12 +223,18 @@ void efi_adjust_memory_range_protection(unsigned long start, unsigned long rounded_start, rounded_end; unsigned long unprotect_start, unprotect_size; - if (efi_dxe_table == NULL) - return; - rounded_start = rounddown(start, EFI_PAGE_SIZE); rounded_end = roundup(start + size, EFI_PAGE_SIZE); + if (memattr != NULL) { + efi_call_proto(memattr, clear_memory_attributes, rounded_start, + rounded_end - rounded_start, EFI_MEMORY_XP); + return; + } + + if (efi_dxe_table == NULL) + return; + /* * Don't modify memory region attributes, they are * already suitable, to lower the possibility to @@ -766,6 +773,7 @@ void __noreturn efi_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, struct boot_params *boot_params) { + efi_guid_t guid = EFI_MEMORY_ATTRIBUTE_PROTOCOL_GUID; unsigned long bzimage_addr = (unsigned long)startup_32; unsigned long buffer_start, buffer_end; struct setup_header *hdr = &boot_params->hdr; @@ -777,13 +785,18 @@ void __noreturn efi_stub_entry(efi_handle_t handle, if (efi_system_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE) efi_exit(handle, EFI_INVALID_PARAMETER); - efi_dxe_table = get_efi_config_table(EFI_DXE_SERVICES_TABLE_GUID); - if (efi_dxe_table && - efi_dxe_table->hdr.signature != EFI_DXE_SERVICES_TABLE_SIGNATURE) { - efi_warn("Ignoring DXE services table: invalid signature\n"); - efi_dxe_table = NULL; + if (IS_ENABLED(CONFIG_EFI_DXE_MEM_ATTRIBUTES)) { + efi_dxe_table = get_efi_config_table(EFI_DXE_SERVICES_TABLE_GUID); + if (efi_dxe_table && + efi_dxe_table->hdr.signature != EFI_DXE_SERVICES_TABLE_SIGNATURE) { + efi_warn("Ignoring DXE services table: invalid signature\n"); + efi_dxe_table = NULL; + } } + /* grab the memory attributes protocol if it exists */ + efi_bs_call(locate_protocol, &guid, NULL, (void **)&memattr); + status = efi_setup_5level_paging(); if (status != EFI_SUCCESS) { efi_err("efi_setup_5level_paging() failed!\n"); From patchwork Fri Jun 2 10:13:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1514FC7EE29 for ; Fri, 2 Jun 2023 10:24:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235316AbjFBKY0 (ORCPT ); Fri, 2 Jun 2023 06:24:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235182AbjFBKXq (ORCPT ); Fri, 2 Jun 2023 06:23:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F11E10E4; Fri, 2 Jun 2023 03:23:14 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A89A664E63; Fri, 2 Jun 2023 10:23:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1853FC4339E; Fri, 2 Jun 2023 10:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701393; bh=TDhagb7/1BjVNI9fsBOdvguiTH1+pJKnlE7Km5AKGt0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JDWx2x8NS3EXWONbS4ao2cLywwjBkOxSVub0aB4TdyV8RL9twr4aztr/IY/zCAljA K2quBW9clDp8TUhlmV9LD7TzM4L//8fZFYHZxvLElvz+tgnqe5I6lJvMhD7jblttSS 1N1vZZRsCxgUl/E46/JyddBynvcCdonJFVvlc+uuAJM52e6vBet/2QZaRFHfx+Cut9 kDEFCXx3l8l31o6QLZXmYpe6EdwwX2LiIBnkKJ+Ez4/OIGufEmZEr10tw3rXyWY14F UCzGLja/1kx6eg7oMFgPqx5tOR1ZsTRI6M8pmsxCLN/hL0tsJdjZm0+enwFpL9tMHp MLatLcXPzWj8Q== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 15/21] decompress: Use 8 byte alignment Date: Fri, 2 Jun 2023 12:13:07 +0200 Message-Id: <20230602101313.3557775-16-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=737; i=ardb@kernel.org; h=from:subject; bh=TDhagb7/1BjVNI9fsBOdvguiTH1+pJKnlE7Km5AKGt0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywCbH4HPMVjN2iPOzhUuqy3Wvr7hRdEN69SXmiLqDH FP5A+50lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIlE32b4pxNVkM2cIBzz1En+ xx4jBomCA/ICNzRFHvntseObOCV2CyPD4fyFE/jqUv6/lGutyexOcKy+msrx/7ZPzWwdR2abMxq sAA== X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The ZSTD decompressor requires malloc() allocations to be 8 byte aligned, so ensure that this the case. Signed-off-by: Ard Biesheuvel --- include/linux/decompress/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h index 9192986b1a731323..ac862422df158bef 100644 --- a/include/linux/decompress/mm.h +++ b/include/linux/decompress/mm.h @@ -48,7 +48,7 @@ MALLOC_VISIBLE void *malloc(int size) if (!malloc_ptr) malloc_ptr = free_mem_ptr; - malloc_ptr = (malloc_ptr + 3) & ~3; /* Align */ + malloc_ptr = (malloc_ptr + 7) & ~7; /* Align */ p = (void *)malloc_ptr; malloc_ptr += size; From patchwork Fri Jun 2 10:13:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03714C7EE29 for ; Fri, 2 Jun 2023 10:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235504AbjFBKYo (ORCPT ); Fri, 2 Jun 2023 06:24:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235420AbjFBKXs (ORCPT ); Fri, 2 Jun 2023 06:23:48 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F32010EB; Fri, 2 Jun 2023 03:23:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 01D2364E63; Fri, 2 Jun 2023 10:23:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 76E3FC433D2; Fri, 2 Jun 2023 10:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701397; bh=g3nSTumb/KYsp4GqRL0zJ1a7x2cI2HwxP02u+R5/mbI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NBKBOfWv0jG5QaD+uhn1eNfnePlAIv9FspYCYiwSE/yvaw5zGyKDvWW8yYW2na42/ bTVAMTliii6hsUXd6P4tiPR69wJqTkvBYhvAzkJpRf2j2ARIxRg86WzubzBviwg8ws 2JBxVKNT/IfMkDqvbdb04mJVKiem5nKawkp8TXkl+uTnN4++DJbWG1uZqEHm4O5wpR gOKplx6OOltL7RPaTRZmXoO5Z6IU9VI1rwn7cEQrRzuMNR/FUuIOIodG/Aq53nrErF VqrdpHKz0Eh8lOolqeK7CSND6MOvNmhmsMBGGc5DmvSS6RtQEMEbwNY16RKuswjeze ZkBUiKkNQdckw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 16/21] x86/decompressor: Move global symbol references to C code Date: Fri, 2 Jun 2023 12:13:08 +0200 Message-Id: <20230602101313.3557775-17-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5078; i=ardb@kernel.org; h=from:subject; bh=g3nSTumb/KYsp4GqRL0zJ1a7x2cI2HwxP02u+R5/mbI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywOaiaX/FNbXq3vm+2fMiODti3b04m93//ps0VaS6T jil1snTUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYiGcXI8EyOrTFdqrL5h9PG r9lzXaqOerSuXXDv+rETL4v933KIf2b4HyGVMkdz2YpnYne3/T4vGio+S+hyevjxquY3rRMyElT qOAA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org It is no longer necessary to be cautious when referring to global variables in the position independent decompressor code, now that it is built using PIE codegen and makes an assertion in the linker script that no GOT entries exist (which would require adjustment for the actual runtime load address of the decompressor binary). This means global variables can be referenced directly from C code, instead of having to pass their runtime addresses into C routines from asm code, which needs to happen at each call site. Do so for the code that will be called directly from the EFI stub after a subsequent patch, and avoid the need to duplicate this logic a third time. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/head_32.S | 8 -------- arch/x86/boot/compressed/head_64.S | 8 +------- arch/x86/boot/compressed/misc.c | 16 +++++++++------- 3 files changed, 10 insertions(+), 22 deletions(-) diff --git a/arch/x86/boot/compressed/head_32.S b/arch/x86/boot/compressed/head_32.S index 3530465b5b85ccf3..beee858058df4403 100644 --- a/arch/x86/boot/compressed/head_32.S +++ b/arch/x86/boot/compressed/head_32.S @@ -168,13 +168,7 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) */ /* push arguments for extract_kernel: */ - pushl output_len@GOTOFF(%ebx) /* decompressed length, end of relocs */ pushl %ebp /* output address */ - pushl input_len@GOTOFF(%ebx) /* input_len */ - leal input_data@GOTOFF(%ebx), %eax - pushl %eax /* input_data */ - leal boot_heap@GOTOFF(%ebx), %eax - pushl %eax /* heap area */ pushl %esi /* real mode pointer */ call extract_kernel /* returns kernel entry point in %eax */ addl $24, %esp @@ -202,8 +196,6 @@ SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end) */ .bss .balign 4 -boot_heap: - .fill BOOT_HEAP_SIZE, 1, 0 boot_stack: .fill BOOT_STACK_SIZE, 1, 0 boot_stack_end: diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index b5bd6be035a7b7ec..3074d278c7e665d8 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -493,11 +493,7 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) * Do the extraction, and jump to the new kernel.. */ movq %r15, %rdi /* pass struct boot_params pointer */ - leaq boot_heap(%rip), %rsi /* malloc area for uncompression */ - leaq input_data(%rip), %rdx /* input_data */ - movl input_len(%rip), %ecx /* input_len */ - movq %rbp, %r8 /* output target address */ - movl output_len(%rip), %r9d /* decompressed length, end of relocs */ + movq %rbp, %rsi /* output target address */ call extract_kernel /* returns kernel entry point in %rax */ /* @@ -636,8 +632,6 @@ SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end) */ .bss .balign 4 -SYM_DATA_LOCAL(boot_heap, .fill BOOT_HEAP_SIZE, 1, 0) - SYM_DATA_START_LOCAL(boot_stack) .fill BOOT_STACK_SIZE, 1, 0 .balign 16 diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index 36535a3753f5d5fa..ad7a2297c9e186df 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -330,6 +330,11 @@ static size_t parse_elf(void *output) return ehdr.e_entry - LOAD_PHYSICAL_ADDR; } +static u8 boot_heap[BOOT_HEAP_SIZE] __aligned(4); + +extern unsigned char input_data[]; +extern unsigned int input_len, output_len; + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -347,14 +352,11 @@ static size_t parse_elf(void *output) * |-------uncompressed kernel image---------| * */ -asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, - unsigned char *input_data, - unsigned long input_len, - unsigned char *output, - unsigned long output_len) +asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) { const unsigned long kernel_total_size = VO__end - VO__text; unsigned long virt_addr = LOAD_PHYSICAL_ADDR; + memptr heap = (memptr)boot_heap; unsigned long needed_size; size_t entry_offset; @@ -412,7 +414,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, * entries. This ensures the full mapped area is usable RAM * and doesn't include any reserved areas. */ - needed_size = max(output_len, kernel_total_size); + needed_size = max((unsigned long)output_len, kernel_total_size); #ifdef CONFIG_X86_64 needed_size = ALIGN(needed_size, MIN_KERNEL_ALIGN); #endif @@ -443,7 +445,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, #ifdef CONFIG_X86_64 if (heap > 0x3fffffffffffUL) error("Destination address too large"); - if (virt_addr + max(output_len, kernel_total_size) > KERNEL_IMAGE_SIZE) + if (virt_addr + needed_size > KERNEL_IMAGE_SIZE) error("Destination virtual address is beyond the kernel mapping area"); #else if (heap > ((-__PAGE_OFFSET-(128<<20)-1) & 0x7fffffff)) From patchwork Fri Jun 2 10:13:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88444C7EE29 for ; Fri, 2 Jun 2023 10:24:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235514AbjFBKYr (ORCPT ); Fri, 2 Jun 2023 06:24:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235389AbjFBKXu (ORCPT ); Fri, 2 Jun 2023 06:23:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE4FA10F1; Fri, 2 Jun 2023 03:23:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5B5A664E63; Fri, 2 Jun 2023 10:23:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3C10C4339C; Fri, 2 Jun 2023 10:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701401; bh=U1WWTB8Ed6jF5PagzQzhXVR5oyHC5bjwVmmRHfA2IP0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HJXId3a8i18aoMDC6f7F7nH7CSVy8FgoZIaU6lLBaNsLeOpllBOWqUOmwRwz0Ok+q vg+ynEoLQlKVXG5DTE4vJYpBgYoVBbr+ZG8NyT+9NOV7FwJFbdqKVBWF+XEoTYADwJ HIq4C3KDpQV+jSOv61iGLifJ+FvXTCs2KPY8vNyus4h0ZHXkX3dvdW9J7crnUScUw6 QyKYB507rOC65FsX4ZIWPpGUbFRGQSc/VkdOhTd/91IDGrflqp5oJZDxANpJigC4jB PKwGjNSfKZBU3qw7XsEunHVzQ5PPgX44aBoaqFAhFyZpAPsBcCigyU0mFXQNcUbBxe +bl5DpMPH/A3w== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 17/21] x86/decompressor: Factor out kernel decompression and relocation Date: Fri, 2 Jun 2023 12:13:09 +0200 Message-Id: <20230602101313.3557775-18-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2968; i=ardb@kernel.org; h=from:subject; bh=U1WWTB8Ed6jF5PagzQzhXVR5oyHC5bjwVmmRHfA2IP0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywBama5deh/9pYD/+7o20UPilU0cYGPcu8vpRorXs7 ryU+4VfO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEVH8yMuz8u3q9ROlEpuip d573LL8bY3g3rqxZZPlM9U5VMfZrskmMDG8Cci5/Z7aIk2c/mS94a/8Gk/623aE7LVQ3ffhxXWK RNi8A X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Factor out the decompressor sequence that invokes the decompressor, parses the ELF and applies the relocations so that it can be called directly from the EFI stub. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.c | 28 ++++++++++++++++---- arch/x86/include/asm/boot.h | 8 ++++++ 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index ad7a2297c9e186df..831485a66300319e 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -330,11 +330,33 @@ static size_t parse_elf(void *output) return ehdr.e_entry - LOAD_PHYSICAL_ADDR; } +const unsigned long kernel_total_size = VO__end - VO__text; + static u8 boot_heap[BOOT_HEAP_SIZE] __aligned(4); extern unsigned char input_data[]; extern unsigned int input_len, output_len; +unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, + void (*error)(char *x)) +{ + unsigned long entry; + + if (!free_mem_ptr) { + free_mem_ptr = (unsigned long)boot_heap; + free_mem_end_ptr = (unsigned long)boot_heap + sizeof(boot_heap); + } + + if (__decompress(input_data, input_len, NULL, NULL, outbuf, output_len, + NULL, error) < 0) + return ULONG_MAX; + + entry = parse_elf(outbuf); + handle_relocations(outbuf, output_len, virt_addr); + + return entry; +} + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -354,7 +376,6 @@ extern unsigned int input_len, output_len; */ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) { - const unsigned long kernel_total_size = VO__end - VO__text; unsigned long virt_addr = LOAD_PHYSICAL_ADDR; memptr heap = (memptr)boot_heap; unsigned long needed_size; @@ -464,10 +485,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) accept_memory(__pa(output), __pa(output) + needed_size); } - __decompress(input_data, input_len, NULL, NULL, output, output_len, - NULL, error); - entry_offset = parse_elf(output); - handle_relocations(output, output_len, virt_addr); + entry_offset = decompress_kernel(output, virt_addr, error); debug_putstr("done.\nBooting the kernel (entry_offset: 0x"); debug_puthex(entry_offset); diff --git a/arch/x86/include/asm/boot.h b/arch/x86/include/asm/boot.h index 9191280d9ea3160d..4ae14339cb8cc72d 100644 --- a/arch/x86/include/asm/boot.h +++ b/arch/x86/include/asm/boot.h @@ -62,4 +62,12 @@ # define BOOT_STACK_SIZE 0x1000 #endif +#ifndef __ASSEMBLY__ +extern unsigned int output_len; +extern const unsigned long kernel_total_size; + +unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, + void (*error)(char *x)); +#endif + #endif /* _ASM_X86_BOOT_H */ From patchwork Fri Jun 2 10:13:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B4FFC7EE29 for ; Fri, 2 Jun 2023 10:25:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235526AbjFBKYu (ORCPT ); Fri, 2 Jun 2023 06:24:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235527AbjFBKYD (ORCPT ); Fri, 2 Jun 2023 06:24:03 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 624CC1702; Fri, 2 Jun 2023 03:23:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B4E3D64E70; Fri, 2 Jun 2023 10:23:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D0E9C4339E; Fri, 2 Jun 2023 10:23:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701406; bh=XRnDiVEqXvkUypxOdLrzhfuiI1tjnMjnlxv76SViP7E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nb8cdG+r1sdlV0apRdvce02oohM8CR0EvHFvqpfm3DH1KVSVehy/Y61E+mDv5bwSk 1QJLdJLsskRV37lIubKeA+HS+dVI1mbbqM8u0VI4gBsDOBBUkjVZPIEk8KR7F37QJy URkBtMeAdd2UWM3tRVGrrOvgNZSnqK4dDNQwwJYoiDdKr1qRl1a6hGlIfT/h/CZGnm W+qVrZ5ShzYgkKdWTG+okyUaeWW1hqKngGSVk+AcRiVuOYly6dOzXTQsMygjnF2JnS RyZgCnMP01HhEshdc0KlTW+ZqtC1Pm8qEZ5mNn8WeodO78hLyudU5Z9LUk0bl9J5LM 47cC7cpCVk/vA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 18/21] x86/head_64: Store boot_params pointer in callee-preserved register Date: Fri, 2 Jun 2023 12:13:10 +0200 Message-Id: <20230602101313.3557775-19-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2440; i=ardb@kernel.org; h=from:subject; bh=XRnDiVEqXvkUypxOdLrzhfuiI1tjnMjnlxv76SViP7E=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywNYmuWOdi4WEwk+wdhR3/nt6SW2CpJWp+IZwqfNP7 yhsP3+ho5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyk+hQjw9vOUIOu2tWv0vp/ r5AxM1CI1Tl36LvWq2BROx0XY/6ynwz/bHs3vOC5Pbt34irHu60P2mcfEueL2Xs65sCDF2Wn79S tYQQA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Instead of pushing/popping %RSI to/from the stack every time a function is called from startup_64(), store it in a callee preserved register and grab it from there when its value is actualled needed. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index a5df3e994f04f10f..95b12fdae10e1dc9 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -60,6 +60,7 @@ SYM_CODE_START_NOALIGN(startup_64) * compiled to run at we first fixup the physical addresses in our page * tables and then reload them. */ + mov %rsi, %r15 /* Preserve boot_params pointer */ /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp @@ -73,9 +74,7 @@ SYM_CODE_START_NOALIGN(startup_64) shrq $32, %rdx wrmsr - pushq %rsi call startup_64_setup_env - popq %rsi #ifdef CONFIG_AMD_MEM_ENCRYPT /* @@ -84,10 +83,8 @@ SYM_CODE_START_NOALIGN(startup_64) * which needs to be done before any CPUID instructions are executed in * subsequent code. */ - movq %rsi, %rdi - pushq %rsi + movq %r15, %rdi call sme_enable - popq %rsi #endif /* Now switch to __KERNEL_CS so IRET works reliably */ @@ -109,9 +106,7 @@ SYM_CODE_START_NOALIGN(startup_64) * programmed into CR3. */ leaq _text(%rip), %rdi - pushq %rsi call __startup_64 - popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ addq $(early_top_pgt - __START_KERNEL_map), %rax @@ -200,10 +195,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * %rsi carries pointer to realmode data and is callee-clobbered. Save * and restore it. */ - pushq %rsi movq %rax, %rdi call sev_verify_cbit - popq %rsi /* * Switch to new page-table @@ -294,9 +287,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) wrmsr /* Setup and Load IDT */ - pushq %rsi call early_setup_idt - popq %rsi /* Check if nx is implemented */ movl $0x80000001, %eax @@ -334,7 +325,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* rsi is pointer to real mode structure with interesting info. pass it to C */ - movq %rsi, %rdi + movq %r15, %rdi .Ljump_to_C_code: /* From patchwork Fri Jun 2 10:13:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 805DFC7EE2C for ; Fri, 2 Jun 2023 10:25:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235350AbjFBKZR (ORCPT ); Fri, 2 Jun 2023 06:25:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235530AbjFBKYH (ORCPT ); Fri, 2 Jun 2023 06:24:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 902951709; Fri, 2 Jun 2023 03:23:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2433A64E6C; Fri, 2 Jun 2023 10:23:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A6B6C433D2; Fri, 2 Jun 2023 10:23:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701410; bh=H6tv4XyD73jIrY44u3Vc1zagcnctlaNlh/vBGAcqcQA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P/gwLBIzNH+NElYX2XCXlc0mfqavS4Z9lEpjyRWc4yupGqJalLdp/xVB2IIta23p7 iCXQZa1XpC2u93fYEToTQBUhyYpw6uY0QK8mXpv/sXc0wW4Lpo7kM3ilZATRgSxxNK oJ/vP8O5iG+LNoBBD9BiP3zezS5nKcFR/xcZ5lL5v6Uz1JUCg1VWpSZr15qSrme4+i ZoSHamEkcbGlYsigt+cHNCtb4WHkcwYsUaHUHA2dEKgMHD/zDqZZS9ayeAxC1aMIJ0 +CE3OaNoxNbgsFsF5sDryALj4lGaC1pUhfX2D2NkDF9PBUwgsLHpcjuCAezof+S0P8 A/0bXSf6bmf6w== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 19/21] efi/libstub: Add limit argument to efi_random_alloc() Date: Fri, 2 Jun 2023 12:13:11 +0200 Message-Id: <20230602101313.3557775-20-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3943; i=ardb@kernel.org; h=from:subject; bh=H6tv4XyD73jIrY44u3Vc1zagcnctlaNlh/vBGAcqcQA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywLbJCyR6Dz2PmK6q6BH/Yd9x/4dF2rM9P1xfHvTCf I+SuO6+jlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjAR79sMf0VZTnhx6DbIbfMU 847K2bdpk6G63bwpixfOFvH0EfXl/cDIsLJOmMmluPuNttwtxf6nz7bd5hM6oHRSTnyZuNWHCd6 iPAA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org x86 will need to limit the kernel memory allocation to the lowest 512 MiB of memory, to match the behavior of the existing bare metal KASLR physical randomization logic. So in preparation for that, add a limit parameter to efi_random_alloc() and wire it up. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/arm64-stub.c | 2 +- drivers/firmware/efi/libstub/efistub.h | 2 +- drivers/firmware/efi/libstub/randomalloc.c | 10 ++++++---- drivers/firmware/efi/libstub/zboot.c | 2 +- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 770b8ecb73984c61..8c40fc89f5f99209 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -106,7 +106,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, */ status = efi_random_alloc(*reserve_size, min_kimg_align, reserve_addr, phys_seed, - EFI_LOADER_CODE); + EFI_LOADER_CODE, EFI_ALLOC_LIMIT); if (status != EFI_SUCCESS) efi_warn("efi_random_alloc() failed: 0x%lx\n", status); } else { diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 191698e8489d82e7..e90b8d1d5c7e8fd4 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -956,7 +956,7 @@ efi_status_t efi_get_random_bytes(unsigned long size, u8 *out); efi_status_t efi_random_alloc(unsigned long size, unsigned long align, unsigned long *addr, unsigned long random_seed, - int memory_type); + int memory_type, unsigned long alloc_limit); efi_status_t efi_random_get_seed(void); diff --git a/drivers/firmware/efi/libstub/randomalloc.c b/drivers/firmware/efi/libstub/randomalloc.c index 32c7a54923b4c127..674a064b8f7adc68 100644 --- a/drivers/firmware/efi/libstub/randomalloc.c +++ b/drivers/firmware/efi/libstub/randomalloc.c @@ -16,7 +16,8 @@ */ static unsigned long get_entry_num_slots(efi_memory_desc_t *md, unsigned long size, - unsigned long align_shift) + unsigned long align_shift, + u64 alloc_limit) { unsigned long align = 1UL << align_shift; u64 first_slot, last_slot, region_end; @@ -29,7 +30,7 @@ static unsigned long get_entry_num_slots(efi_memory_desc_t *md, return 0; region_end = min(md->phys_addr + md->num_pages * EFI_PAGE_SIZE - 1, - (u64)EFI_ALLOC_LIMIT); + alloc_limit); if (region_end < size) return 0; @@ -54,7 +55,8 @@ efi_status_t efi_random_alloc(unsigned long size, unsigned long align, unsigned long *addr, unsigned long random_seed, - int memory_type) + int memory_type, + unsigned long alloc_limit) { unsigned long total_slots = 0, target_slot; unsigned long total_mirrored_slots = 0; @@ -76,7 +78,7 @@ efi_status_t efi_random_alloc(unsigned long size, efi_memory_desc_t *md = (void *)map->map + map_offset; unsigned long slots; - slots = get_entry_num_slots(md, size, ilog2(align)); + slots = get_entry_num_slots(md, size, ilog2(align), alloc_limit); MD_NUM_SLOTS(md) = slots; total_slots += slots; if (md->attribute & EFI_MEMORY_MORE_RELIABLE) diff --git a/drivers/firmware/efi/libstub/zboot.c b/drivers/firmware/efi/libstub/zboot.c index e5d7fa1f1d8fd160..bdb17eac0cb401be 100644 --- a/drivers/firmware/efi/libstub/zboot.c +++ b/drivers/firmware/efi/libstub/zboot.c @@ -119,7 +119,7 @@ efi_zboot_entry(efi_handle_t handle, efi_system_table_t *systab) } status = efi_random_alloc(alloc_size, min_kimg_align, &image_base, - seed, EFI_LOADER_CODE); + seed, EFI_LOADER_CODE, EFI_ALLOC_LIMIT); if (status != EFI_SUCCESS) { efi_err("Failed to allocate memory\n"); goto free_cmdline; From patchwork Fri Jun 2 10:13:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A0E2C7EE24 for ; Fri, 2 Jun 2023 10:25:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234799AbjFBKZR (ORCPT ); Fri, 2 Jun 2023 06:25:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235485AbjFBKYJ (ORCPT ); Fri, 2 Jun 2023 06:24:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 017E01710; Fri, 2 Jun 2023 03:23:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7E7DD64E63; Fri, 2 Jun 2023 10:23:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02497C4339C; Fri, 2 Jun 2023 10:23:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701414; bh=Uzc6TTIkVLCBtkTdTqlEAm179xBBwMbrhTFacsgdIgc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uYk1Zs4tYhSRbRmPt6QrbkWNXwtU/5wEasqMtMC8VqgbAXOCdhpbpGfJDr7UIH/UL DAG4Qr2okHRoXHaBooODGt+jY8vdrofyLbC8MAA7e+JqMQoJqNE6tn6b1molzC4kKf p4GqArNxm3NOKBiQLhq4S2692DhKyyPTd30gNrzGMAcnnU2fJmjlfoRC2eovkjt+1T UUTJgcj0m+P5T7gf5dewMHv2EJ4l0A/6MOeF/g+BHgSTXnjVlpZZnvEV1H8nHRlcxV agDzt81CGGipQSzrDszyvIcyNqClz8vqn3qkJWnn2NU6TxkI00f7CtDUYjiNU/JJIi a4qBd7O6uhyKw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 20/21] x86/efistub: Perform SNP feature test while running in the firmware Date: Fri, 2 Jun 2023 12:13:12 +0200 Message-Id: <20230602101313.3557775-21-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6505; i=ardb@kernel.org; h=from:subject; bh=Uzc6TTIkVLCBtkTdTqlEAm179xBBwMbrhTFacsgdIgc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywI6G13uSLty0ypBXNL8/nX/CnRtbrQvfTFeMO2w5j ylcp0ezo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwk/DjD/7xn6vPNjk145idQ LZXgkXdllcjlTatvv5LcnDt1asiWTamMDA2889Td5+4M+CHXtrlBLVv4kpKnZt7sI855My4wx3z J4AQA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Before refactoring the EFI stub boot flow to avoid the legacy bare metal decompressor, duplicate the SNP feature check in the EFI stub before handing over to the kernel proper. The SNP feature check can be performed while running under the EFI boot services, which means we can fail gracefully and return an error to the bootloader if the loaded kernel does not implement support for all the features that the hypervisor enabled. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 74 ++++++++++++-------- arch/x86/include/asm/sev.h | 4 ++ drivers/firmware/efi/libstub/x86-stub.c | 17 +++++ 3 files changed, 67 insertions(+), 28 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 014b89c890887b9a..be021e24f1ece421 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -315,6 +315,11 @@ static void enforce_vmpl0(void) */ #define SNP_FEATURES_PRESENT (0) +u64 snp_get_unsupported_features(u64 status) +{ + return status & SNP_FEATURES_IMPL_REQ & ~SNP_FEATURES_PRESENT; +} + void snp_check_features(void) { u64 unsupported; @@ -328,7 +333,7 @@ void snp_check_features(void) * EXIT_INFO_2 of the GHCB protocol so that those features can be reported * as part of the guest boot failure. */ - unsupported = sev_status & SNP_FEATURES_IMPL_REQ & ~SNP_FEATURES_PRESENT; + unsupported = snp_get_unsupported_features(sev_status); if (unsupported) { if (ghcb_version < 2 || (!boot_ghcb && !early_setup_ghcb())) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); @@ -338,10 +343,38 @@ void snp_check_features(void) } } -void sev_enable(struct boot_params *bp) +u64 sev_get_status(void) { unsigned int eax, ebx, ecx, edx; struct msr m; + + /* Check for the SME/SEV support leaf */ + eax = 0x80000000; + ecx = 0; + native_cpuid(&eax, &ebx, &ecx, &edx); + if (eax < 0x8000001f) + return 0; + + /* + * Check for the SME/SEV feature: + * CPUID Fn8000_001F[EAX] + * - Bit 0 - Secure Memory Encryption support + * - Bit 1 - Secure Encrypted Virtualization support + */ + eax = 0x8000001f; + ecx = 0; + native_cpuid(&eax, &ebx, &ecx, &edx); + /* Check whether SEV is supported */ + if (!(eax & BIT(1))) + return 0; + + boot_rdmsr(MSR_AMD64_SEV, &m); + return m.q; +} + +void sev_enable(struct boot_params *bp) +{ + unsigned int eax, ebx, ecx, edx; bool snp; /* @@ -358,37 +391,14 @@ void sev_enable(struct boot_params *bp) */ snp = snp_init(bp); - /* Check for the SME/SEV support leaf */ - eax = 0x80000000; - ecx = 0; - native_cpuid(&eax, &ebx, &ecx, &edx); - if (eax < 0x8000001f) - return; - - /* - * Check for the SME/SEV feature: - * CPUID Fn8000_001F[EAX] - * - Bit 0 - Secure Memory Encryption support - * - Bit 1 - Secure Encrypted Virtualization support - * CPUID Fn8000_001F[EBX] - * - Bits 5:0 - Pagetable bit position used to indicate encryption - */ - eax = 0x8000001f; - ecx = 0; - native_cpuid(&eax, &ebx, &ecx, &edx); - /* Check whether SEV is supported */ - if (!(eax & BIT(1))) { + /* Set the SME mask if this is an SEV guest. */ + sev_status = sev_get_status(); + if (!(sev_status & MSR_AMD64_SEV_ENABLED)) { if (snp) error("SEV-SNP support indicated by CC blob, but not CPUID."); return; } - /* Set the SME mask if this is an SEV guest. */ - boot_rdmsr(MSR_AMD64_SEV, &m); - sev_status = m.q; - if (!(sev_status & MSR_AMD64_SEV_ENABLED)) - return; - /* Negotiate the GHCB protocol version. */ if (sev_status & MSR_AMD64_SEV_ES_ENABLED) { if (!sev_es_negotiate_protocol()) @@ -409,6 +419,14 @@ void sev_enable(struct boot_params *bp) if (snp && !(sev_status & MSR_AMD64_SEV_SNP_ENABLED)) error("SEV-SNP supported indicated by CC blob, but not SEV status MSR."); + /* + * Check for the SME/SEV feature: + * CPUID Fn8000_001F[EBX] + * - Bits 5:0 - Pagetable bit position used to indicate encryption + */ + eax = 0x8000001f; + ecx = 0; + native_cpuid(&eax, &ebx, &ecx, &edx); sme_me_mask = BIT_ULL(ebx & 0x3f); } diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 13dc2a9d23c1eb25..e5aad673194698b8 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -202,6 +202,8 @@ void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); void __init __noreturn snp_abort(void); int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, struct snp_guest_request_ioctl *rio); +u64 snp_get_unsupported_features(u64 status); +u64 sev_get_status(void); #else static inline void sev_es_ist_enter(struct pt_regs *regs) { } static inline void sev_es_ist_exit(void) { } @@ -225,6 +227,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in { return -ENOTTY; } +static inline u64 snp_get_unsupported_features(u64 status) { return 0; } +static inline u64 sev_get_status(void) { return 0; } #endif #endif diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 2d3282d2ed6eb756..f9d203b5ee6236e8 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "efistub.h" #include "x86-stub.h" @@ -756,6 +757,19 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle) return EFI_SUCCESS; } +static bool have_unsupported_snp_features(void) +{ + u64 unsupported; + + unsupported = snp_get_unsupported_features(sev_get_status()); + if (unsupported) { + efi_err("Unsupported SEV-SNP features detected: 0x%llx\n", + unsupported); + return true; + } + return false; +} + static void __noreturn enter_kernel(unsigned long kernel_addr, struct boot_params *boot_params) { @@ -785,6 +799,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle, if (efi_system_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE) efi_exit(handle, EFI_INVALID_PARAMETER); + if (have_unsupported_snp_features()) + efi_exit(handle, EFI_UNSUPPORTED); + if (IS_ENABLED(CONFIG_EFI_DXE_MEM_ATTRIBUTES)) { efi_dxe_table = get_efi_config_table(EFI_DXE_SERVICES_TABLE_GUID); if (efi_dxe_table && From patchwork Fri Jun 2 10:13:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 688911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3788CC7EE29 for ; Fri, 2 Jun 2023 10:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235371AbjFBKZS (ORCPT ); Fri, 2 Jun 2023 06:25:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235541AbjFBKYR (ORCPT ); Fri, 2 Jun 2023 06:24:17 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FE1F171D; Fri, 2 Jun 2023 03:23:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DB83A64E80; Fri, 2 Jun 2023 10:23:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60956C433EF; Fri, 2 Jun 2023 10:23:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685701419; bh=x7IKVLjcFP3CYkwluyIAzVjkKiuNw7y/y12cM78AR+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JJwTqDrQGN6dJh9XDufzT2ymF63m/yrtttTQ5jQEnHkfZcTYh/6FdOt2ZBizC4/EY 3D1f3FobDS6xpip2z2/3iARDpiD2WRQN59Bs6MUMoWlroV7c5FdWWnGFtYYb9YZ619 Vapsx3IUKxNMrE4fFf+fMIe5wmnzejBuMMoqpSQ8/G2v8mfNo+NPlwgwT5+A6XmLCD ziAbvoMRWCa9RgIiOAe7/bbHEs1hR/vp7JWCoPSmpgWAtdDIJ0+kMPFu0hr/7dtcI2 LhJM1kqGc82S4Ap70aAjZnol4IOZj4Yi651wT1S/K4Np/GwNs6wBbQiovSo2bl/wtE 2AytDbT7k65KA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , Dave Young , Mario Limonciello , Kees Cook , Tom Lendacky , "Kirill A . Shutemov" , Linus Torvalds , Joerg Roedel Subject: [PATCH v4 21/21] x86/efistub: Avoid legacy decompressor when doing EFI boot Date: Fri, 2 Jun 2023 12:13:13 +0200 Message-Id: <20230602101313.3557775-22-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230602101313.3557775-1-ardb@kernel.org> References: <20230602101313.3557775-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=18379; i=ardb@kernel.org; h=from:subject; bh=x7IKVLjcFP3CYkwluyIAzVjkKiuNw7y/y12cM78AR+s=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIaXywE7jEv2vBoEvpSQVEpxndXBY/dtuvti63umWY/K3S zfDLhR1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIm03Gb4X90jqmU+5WDEilVn dGv177u2deyc8qlCYbv3E8f3OZ4vjzH8U/X++V7m30wvZ9/zc58k9qQ+vzRh2lIFtqKpv3yusVk eYAcA X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org The bare metal decompressor code was never really intended to run in a hosted environment such as the EFI boot services, and does a few things that are problematic in the context of EFI boot now that the logo requirements are getting tighter. In particular, the decompressor moves its own executable image around in memory, and relies on demand paging to populate the identity mappings, and these things are difficult to support in a context where memory is not permitted to be mapped writable and executable at the same time or, at the very least, is mapped non-executable by default, and needs special treatment for this restriction to be lifted. Since EFI already maps all of memory 1:1, it is unnecessary to create new page tables or handle page faults when decompressing the kernel. That means there is also no need to replace the special exception handlers for SEV. Generally, there is little need to do anything that the decompressor does beyond - initialize SEV encryption, if needed, - perform the 4/5 level paging switch, if needed, - decompress the kernel - relocate the kernel So do all of this from the EFI stub code, and avoid the bare metal decompressor altogether. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/Makefile | 5 + arch/x86/boot/compressed/efi_mixed.S | 55 ------- arch/x86/boot/compressed/head_32.S | 13 -- arch/x86/boot/compressed/head_64.S | 27 ---- arch/x86/include/asm/efi.h | 7 +- arch/x86/include/asm/sev.h | 2 + drivers/firmware/efi/libstub/x86-stub.c | 167 +++++++++----------- 7 files changed, 84 insertions(+), 192 deletions(-) diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index b13a580210867ffb..535608fe72e11265 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -74,6 +74,11 @@ LDFLAGS_vmlinux += -z noexecstack ifeq ($(CONFIG_LD_IS_BFD),y) LDFLAGS_vmlinux += $(call ld-option,--no-warn-rwx-segments) endif +ifeq ($(CONFIG_EFI_STUB),y) +# ensure that the static EFI stub library will be pulled in, even if it is +# never referenced explicitly from the startup code +LDFLAGS_vmlinux += -u efi_pe_entry +endif LDFLAGS_vmlinux += -T hostprogs := mkpiggy diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S index 8a02a151806df14c..f4e22ef774ab6b4a 100644 --- a/arch/x86/boot/compressed/efi_mixed.S +++ b/arch/x86/boot/compressed/efi_mixed.S @@ -269,10 +269,6 @@ SYM_FUNC_START_LOCAL(efi32_entry) jmp startup_32 SYM_FUNC_END(efi32_entry) -#define ST32_boottime 60 // offsetof(efi_system_table_32_t, boottime) -#define BS32_handle_protocol 88 // offsetof(efi_boot_services_32_t, handle_protocol) -#define LI32_image_base 32 // offsetof(efi_loaded_image_32_t, image_base) - /* * efi_status_t efi32_pe_entry(efi_handle_t image_handle, * efi_system_table_32_t *sys_table) @@ -280,8 +276,6 @@ SYM_FUNC_END(efi32_entry) SYM_FUNC_START(efi32_pe_entry) pushl %ebp movl %esp, %ebp - pushl %eax // dummy push to allocate loaded_image - pushl %ebx // save callee-save registers pushl %edi @@ -290,48 +284,8 @@ SYM_FUNC_START(efi32_pe_entry) movl $0x80000003, %eax // EFI_UNSUPPORTED jnz 2f - call 1f -1: pop %ebx - - /* Get the loaded image protocol pointer from the image handle */ - leal -4(%ebp), %eax - pushl %eax // &loaded_image - leal (loaded_image_proto - 1b)(%ebx), %eax - pushl %eax // pass the GUID address - pushl 8(%ebp) // pass the image handle - - /* - * Note the alignment of the stack frame. - * sys_table - * handle <-- 16-byte aligned on entry by ABI - * return address - * frame pointer - * loaded_image <-- local variable - * saved %ebx <-- 16-byte aligned here - * saved %edi - * &loaded_image - * &loaded_image_proto - * handle <-- 16-byte aligned for call to handle_protocol - */ - - movl 12(%ebp), %eax // sys_table - movl ST32_boottime(%eax), %eax // sys_table->boottime - call *BS32_handle_protocol(%eax) // sys_table->boottime->handle_protocol - addl $12, %esp // restore argument space - testl %eax, %eax - jnz 2f - movl 8(%ebp), %ecx // image_handle movl 12(%ebp), %edx // sys_table - movl -4(%ebp), %esi // loaded_image - movl LI32_image_base(%esi), %esi // loaded_image->image_base - leal (startup_32 - 1b)(%ebx), %ebp // runtime address of startup_32 - /* - * We need to set the image_offset variable here since startup_32() will - * use it before we get to the 64-bit efi_pe_entry() in C code. - */ - subl %esi, %ebp // calculate image_offset - movl %ebp, (image_offset - 1b)(%ebx) // save image_offset xorl %esi, %esi jmp efi32_entry // pass %ecx, %edx, %esi // no other registers remain live @@ -350,15 +304,6 @@ SYM_FUNC_START_NOALIGN(efi64_stub_entry) SYM_FUNC_END(efi64_stub_entry) #endif - .section ".rodata" - /* EFI loaded image protocol GUID */ - .balign 4 -SYM_DATA_START_LOCAL(loaded_image_proto) - .long 0x5b1b31a1 - .word 0x9562, 0x11d2 - .byte 0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b -SYM_DATA_END(loaded_image_proto) - .data .balign 8 SYM_DATA_START_LOCAL(efi32_boot_gdt) diff --git a/arch/x86/boot/compressed/head_32.S b/arch/x86/boot/compressed/head_32.S index beee858058df4403..cd9587fcd5084f22 100644 --- a/arch/x86/boot/compressed/head_32.S +++ b/arch/x86/boot/compressed/head_32.S @@ -84,19 +84,6 @@ SYM_FUNC_START(startup_32) #ifdef CONFIG_RELOCATABLE leal startup_32@GOTOFF(%edx), %ebx - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32() will be at an - * offset to the start of the space allocated for the image. efi_pe_entry() will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - subl image_offset@GOTOFF(%edx), %ebx -#endif - movl BP_kernel_alignment(%esi), %eax decl %eax addl %eax, %ebx diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 3074d278c7e665d8..aee518e71cdb7e75 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -146,19 +146,6 @@ SYM_FUNC_START(startup_32) #ifdef CONFIG_RELOCATABLE movl %ebp, %ebx - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32 will be at an - * offset to the start of the space allocated for the image. efi_pe_entry will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - subl rva(image_offset)(%ebp), %ebx -#endif - movl BP_kernel_alignment(%esi), %eax decl %eax addl %eax, %ebx @@ -335,20 +322,6 @@ SYM_CODE_START(startup_64) /* Start with the delta to where the kernel will run at. */ #ifdef CONFIG_RELOCATABLE leaq startup_32(%rip) /* - $startup_32 */, %rbp - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32 will be at an - * offset to the start of the space allocated for the image. efi_pe_entry will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - movl image_offset(%rip), %eax - subq %rax, %rbp -#endif - movl BP_kernel_alignment(%rsi), %eax decl %eax addq %rax, %rbp diff --git a/arch/x86/include/asm/efi.h b/arch/x86/include/asm/efi.h index 8b4be7cecdb8eb73..b0994ae3bc23f84d 100644 --- a/arch/x86/include/asm/efi.h +++ b/arch/x86/include/asm/efi.h @@ -90,6 +90,8 @@ static inline void efi_fpu_end(void) } #ifdef CONFIG_X86_32 +#define EFI_X86_KERNEL_ALLOC_LIMIT (SZ_512M - 1) + #define arch_efi_call_virt_setup() \ ({ \ efi_fpu_begin(); \ @@ -103,8 +105,7 @@ static inline void efi_fpu_end(void) }) #else /* !CONFIG_X86_32 */ - -#define EFI_LOADER_SIGNATURE "EL64" +#define EFI_X86_KERNEL_ALLOC_LIMIT EFI_ALLOC_LIMIT extern asmlinkage u64 __efi_call(void *fp, ...); @@ -218,6 +219,8 @@ efi_status_t efi_set_virtual_address_map(unsigned long memory_map_size, #ifdef CONFIG_EFI_MIXED +#define EFI_ALLOC_LIMIT (efi_is_64bit() ? ULONG_MAX : U32_MAX) + #define ARCH_HAS_EFISTUB_WRAPPERS static inline bool efi_is_64bit(void) diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index e5aad673194698b8..bb1ed2a8b8fb122d 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -157,6 +157,7 @@ static __always_inline void sev_es_nmi_complete(void) __sev_es_nmi_complete(); } extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd); +extern void sev_enable(struct boot_params *bp); static inline int rmpadjust(unsigned long vaddr, bool rmp_psize, unsigned long attrs) { @@ -210,6 +211,7 @@ static inline void sev_es_ist_exit(void) { } static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; } static inline void sev_es_nmi_complete(void) { } static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; } +static inline void sev_enable(struct boot_params *bp) { } static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) { return 0; } static inline int rmpadjust(unsigned long vaddr, bool rmp_psize, unsigned long attrs) { return 0; } static inline void setup_ghcb(void) { } diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index f9d203b5ee6236e8..ed91a2a4984d4bfb 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -15,17 +15,14 @@ #include #include #include +#include #include #include "efistub.h" #include "x86-stub.h" -/* Maximum physical address for 64-bit kernel with 4-level paging */ -#define MAXMEM_X86_64_4LEVEL (1ull << 46) - const efi_system_table_t *efi_system_table; const efi_dxe_services_table_t *efi_dxe_table; -u32 image_offset __section(".data"); static efi_loaded_image_t *image = NULL; static efi_memory_attribute_protocol_t *memattr; @@ -276,33 +273,9 @@ void efi_adjust_memory_range_protection(unsigned long start, } } -void startup_32(struct boot_params *boot_params); - -static void -setup_memory_protection(unsigned long image_base, unsigned long image_size) -{ -#ifdef CONFIG_64BIT - if (image_base != (unsigned long)startup_32) - efi_adjust_memory_range_protection(image_base, image_size); -#else - /* - * Clear protection flags on a whole range of possible - * addresses used for KASLR. We don't need to do that - * on x86_64, since KASLR/extraction is performed after - * dedicated identity page tables are built and we only - * need to remove possible protection on relocated image - * itself disregarding further relocations. - */ - efi_adjust_memory_range_protection(LOAD_PHYSICAL_ADDR, - KERNEL_IMAGE_SIZE - LOAD_PHYSICAL_ADDR); -#endif -} - static const efi_char16_t apple[] = L"Apple"; -static void setup_quirks(struct boot_params *boot_params, - unsigned long image_base, - unsigned long image_size) +static void setup_quirks(struct boot_params *boot_params) { efi_char16_t *fw_vendor = (efi_char16_t *)(unsigned long) efi_table_attr(efi_system_table, fw_vendor); @@ -311,9 +284,6 @@ static void setup_quirks(struct boot_params *boot_params, if (IS_ENABLED(CONFIG_APPLE_PROPERTIES)) retrieve_apple_device_properties(boot_params); } - - if (IS_ENABLED(CONFIG_EFI_DXE_MEM_ATTRIBUTES)) - setup_memory_protection(image_base, image_size); } /* @@ -466,7 +436,6 @@ efi_status_t __efiapi efi_pe_entry(efi_handle_t handle, } image_base = efi_table_attr(image, image_base); - image_offset = (void *)startup_32 - image_base; status = efi_allocate_pages(sizeof(struct boot_params), (unsigned long *)&boot_params, ULONG_MAX); @@ -770,6 +739,61 @@ static bool have_unsupported_snp_features(void) return false; } +static void efi_get_seed(void *seed, int size) +{ + efi_get_random_bytes(size, seed); + + /* + * This only updates seed[0] when running on 32-bit, but in that case, + * we don't use seed[1] anyway, as there is no virtual KASLR on 32-bit. + */ + *(unsigned long *)seed ^= kaslr_get_random_long("EFI"); +} + +static void error(char *str) +{ + efi_warn("Decompression failed: %s\n", str); +} + +static efi_status_t efi_decompress_kernel(unsigned long *kernel_entry) +{ + unsigned long virt_addr = LOAD_PHYSICAL_ADDR; + unsigned long addr, alloc_size, entry; + efi_status_t status; + u32 seed[2] = {}; + + /* determine the required size of the allocation */ + alloc_size = ALIGN(max((unsigned long)output_len, kernel_total_size), + MIN_KERNEL_ALIGN); + + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && !efi_nokaslr) { + u64 range = KERNEL_IMAGE_SIZE - LOAD_PHYSICAL_ADDR - kernel_total_size; + + efi_get_seed(seed, sizeof(seed)); + + virt_addr += (range * seed[1]) >> 32; + virt_addr &= ~(CONFIG_PHYSICAL_ALIGN - 1); + } + + status = efi_random_alloc(alloc_size, CONFIG_PHYSICAL_ALIGN, &addr, + seed[0], EFI_LOADER_CODE, + EFI_X86_KERNEL_ALLOC_LIMIT); + if (status != EFI_SUCCESS) + return status; + + entry = decompress_kernel((void *)addr, virt_addr, error); + if (entry == ULONG_MAX) { + efi_free(alloc_size, addr); + return EFI_LOAD_ERROR; + } + + *kernel_entry = addr + entry; + + efi_adjust_memory_range_protection(addr, kernel_total_size); + + return EFI_SUCCESS; +} + static void __noreturn enter_kernel(unsigned long kernel_addr, struct boot_params *boot_params) { @@ -788,10 +812,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle, struct boot_params *boot_params) { efi_guid_t guid = EFI_MEMORY_ATTRIBUTE_PROTOCOL_GUID; - unsigned long bzimage_addr = (unsigned long)startup_32; - unsigned long buffer_start, buffer_end; struct setup_header *hdr = &boot_params->hdr; const struct linux_efi_initrd *initrd = NULL; + unsigned long kernel_entry; efi_status_t status; efi_system_table = sys_table_arg; @@ -820,60 +843,6 @@ void __noreturn efi_stub_entry(efi_handle_t handle, goto fail; } - /* - * If the kernel isn't already loaded at a suitable address, - * relocate it. - * - * It must be loaded above LOAD_PHYSICAL_ADDR. - * - * The maximum address for 64-bit is 1 << 46 for 4-level paging. This - * is defined as the macro MAXMEM, but unfortunately that is not a - * compile-time constant if 5-level paging is configured, so we instead - * define our own macro for use here. - * - * For 32-bit, the maximum address is complicated to figure out, for - * now use KERNEL_IMAGE_SIZE, which will be 512MiB, the same as what - * KASLR uses. - * - * Also relocate it if image_offset is zero, i.e. the kernel wasn't - * loaded by LoadImage, but rather by a bootloader that called the - * handover entry. The reason we must always relocate in this case is - * to handle the case of systemd-boot booting a unified kernel image, - * which is a PE executable that contains the bzImage and an initrd as - * COFF sections. The initrd section is placed after the bzImage - * without ensuring that there are at least init_size bytes available - * for the bzImage, and thus the compressed kernel's startup code may - * overwrite the initrd unless it is moved out of the way. - */ - - buffer_start = ALIGN(bzimage_addr - image_offset, - hdr->kernel_alignment); - buffer_end = buffer_start + hdr->init_size; - - if ((buffer_start < LOAD_PHYSICAL_ADDR) || - (IS_ENABLED(CONFIG_X86_32) && buffer_end > KERNEL_IMAGE_SIZE) || - (IS_ENABLED(CONFIG_X86_64) && buffer_end > MAXMEM_X86_64_4LEVEL) || - (image_offset == 0)) { - extern char _bss[]; - - status = efi_relocate_kernel(&bzimage_addr, - (unsigned long)_bss - bzimage_addr, - hdr->init_size, - hdr->pref_address, - hdr->kernel_alignment, - LOAD_PHYSICAL_ADDR); - if (status != EFI_SUCCESS) { - efi_err("efi_relocate_kernel() failed!\n"); - goto fail; - } - /* - * Now that we've copied the kernel elsewhere, we no longer - * have a set up block before startup_32(), so reset image_offset - * to zero in case it was set earlier. - */ - image_offset = 0; - } - #ifdef CONFIG_CMDLINE_BOOL status = efi_parse_options(CONFIG_CMDLINE); if (status != EFI_SUCCESS) { @@ -891,6 +860,12 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } } + status = efi_decompress_kernel(&kernel_entry); + if (status != EFI_SUCCESS) { + efi_err("Failed to decompress kernel\n"); + goto fail; + } + /* * At this point, an initrd may already have been loaded by the * bootloader and passed via bootparams. We permit an initrd loaded @@ -930,7 +905,7 @@ void __noreturn efi_stub_entry(efi_handle_t handle, setup_efi_pci(boot_params); - setup_quirks(boot_params, bzimage_addr, buffer_end - buffer_start); + setup_quirks(boot_params); status = exit_boot(boot_params, handle); if (status != EFI_SUCCESS) { @@ -938,13 +913,15 @@ void __noreturn efi_stub_entry(efi_handle_t handle, goto fail; } + /* + * Call the SEV init code while still running with the firmware's + * GDT/IDT, so #VC exceptions will be handled by EFI. + */ + sev_enable(boot_params); + efi_5level_switch(); - if (IS_ENABLED(CONFIG_X86_64)) - /* add offset of startup_64() */ - bzimage_addr += 0x200; - - enter_kernel(bzimage_addr, boot_params); + enter_kernel(kernel_entry, boot_params); fail: efi_err("efi_stub_entry() failed!\n");