From patchwork Wed Oct 8 04:02:52 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roy Franz X-Patchwork-Id: 38445 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f197.google.com (mail-lb0-f197.google.com [209.85.217.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 0A28E202E7 for ; Wed, 8 Oct 2014 04:05:20 +0000 (UTC) Received: by mail-lb0-f197.google.com with SMTP id p9sf4773675lbv.4 for ; Tue, 07 Oct 2014 21:05:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:mime-version:in-reply-to:references :date:message-id:from:to:cc:subject:precedence:list-id :list-unsubscribe:list-post:list-help:list-subscribe:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:list-archive:content-type:content-transfer-encoding; bh=/ScuSpWlcwqtoXTCegl233eXaV5+VZDVJbSAGppe+N8=; b=D6s2YHnrsE5953dW1ZDPAYK99KX4yf1vm9qL4juhYUyHpyQsUq+6piogDxpJnLWGpo lERyA9vv9EQE6OuFeg0uluDnkKZ+ldd7EJIwgspYBlgoatsmTFwmb9DkAAfn6M+NGgV+ e5R4R/yVpQGIxteXMIMayMXUqdSnkGb+edoTlf7hzNISHytNseq5gT44g8i3OVoytDkP 3bRSW2AvN6meS/JE59YLXNJZq70r9PlnYl+TyXyFJhVlcTr5WkOwX9WyUner5w1EbSV9 mnxSQ5ZqwvsZJJOLAA/IPtu6P2CQBTn6q2gZwXsn8pWq/tjdtgpiPo8kQr39I3uLi7OT bDsg== X-Gm-Message-State: ALoCoQlRNor8TvN7j6mPHqG/wp4Pn3q2R9U8AZg4LZ15d59czC1XODej4vi2kOqdXP8mYf8keDUP X-Received: by 10.180.85.8 with SMTP id d8mr1171208wiz.0.1412741119477; Tue, 07 Oct 2014 21:05:19 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.5.134 with SMTP id s6ls13081las.108.gmail; Tue, 07 Oct 2014 21:05:19 -0700 (PDT) X-Received: by 10.112.150.106 with SMTP id uh10mr7864733lbb.11.1412741119050; Tue, 07 Oct 2014 21:05:19 -0700 (PDT) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mx.google.com with ESMTPS id lc7si22897911lac.55.2014.10.07.21.05.18 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 07 Oct 2014 21:05:19 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) client-ip=209.85.217.181; Received: by mail-lb0-f181.google.com with SMTP id l4so7217720lbv.40 for ; Tue, 07 Oct 2014 21:05:18 -0700 (PDT) X-Received: by 10.112.50.10 with SMTP id y10mr7880075lbn.0.1412741118922; Tue, 07 Oct 2014 21:05:18 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.130.169 with SMTP id of9csp456398lbb; Tue, 7 Oct 2014 21:05:18 -0700 (PDT) X-Received: by 10.180.212.48 with SMTP id nh16mr8852011wic.44.1412741117986; Tue, 07 Oct 2014 21:05:17 -0700 (PDT) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id ws1si23064445wjb.85.2014.10.07.21.05.17 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 07 Oct 2014 21:05:17 -0700 (PDT) Received-SPF: none (google.com: xen-devel-bounces@lists.xen.org does not designate permitted sender hosts) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1XbiSn-0007YW-7h; Wed, 08 Oct 2014 04:02:57 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1XbiSm-0007YQ-Er for xen-devel@lists.xen.org; Wed, 08 Oct 2014 04:02:56 +0000 Received: from [85.158.137.68:11061] by server-1.bemta-3.messagelabs.com id 16/7B-30185-F67B4345; Wed, 08 Oct 2014 04:02:55 +0000 X-Env-Sender: roy.franz@linaro.org X-Msg-Ref: server-15.tower-31.messagelabs.com!1412740973!11996118!1 X-Originating-IP: [209.85.220.176] X-SpamReason: No, hits=0.8 required=7.0 tests=BODY_RANDOM_LONG, RCVD_BY_IP X-StarScan-Received: X-StarScan-Version: 6.12.2; banners=-,-,- X-VirusChecked: Checked Received: (qmail 32655 invoked from network); 8 Oct 2014 04:02:54 -0000 Received: from mail-vc0-f176.google.com (HELO mail-vc0-f176.google.com) (209.85.220.176) by server-15.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 8 Oct 2014 04:02:54 -0000 Received: by mail-vc0-f176.google.com with SMTP id hq11so5982256vcb.35 for ; Tue, 07 Oct 2014 21:02:53 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.52.51.203 with SMTP id m11mr6357673vdo.72.1412740972979; Tue, 07 Oct 2014 21:02:52 -0700 (PDT) Received: by 10.220.110.78 with HTTP; Tue, 7 Oct 2014 21:02:52 -0700 (PDT) In-Reply-To: <5434973B.6090002@amd.com> References: <54335E33.6030005@amd.com> <1412673681.4972.10.camel@citrix.com> <5434973B.6090002@amd.com> Date: Tue, 7 Oct 2014 21:02:52 -0700 Message-ID: From: Roy Franz To: Suravee Suthikulanit Cc: Julien Grall , Ian Campbell , "stefano.stabellini@eu.citrix.com" , xen-devel Subject: Re: [Xen-devel] Failed to enable MMU when booting EFI on Seattle X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: roy.franz@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: On Tue, Oct 7, 2014 at 6:45 PM, Suravee Suthikulanit wrote: > On 10/7/2014 4:21 AM, Ian Campbell wrote: >> >> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote: >>> >>> Ian/Roy, >>> >>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and >>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to >>> address 0x2005F8, which has all zeros value. This is why we getting the >>> "Synchronous Exception" >>> >>> ...... >>> 1: >>> PRINT("- Turning on paging -\r\n") >>> >>> ldr x1, =paging /* Explicit vaddr, not >>> RIP-relative */ >>> mrs x0, SCTLR_EL2 >>> orr x0, x0, #SCTLR_M /* Enable MMU */ >>> orr x0, x0, #SCTLR_C /* Enable D-cache */ >>> dsb sy /* Flush PTE writes and finish >>> reads */ >>> b . >>> msr SCTLR_EL2, x0 /* now paging is enabled */ >>> isb /* Now, flush the icache */ >>> br x1 /* Get a proper vaddr into PC */ >>> paging: >>> ..... >>> >>> Notes: >>> * This is not the case when booting non-EFI. >>> >>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it >>> seems to branch to "paging:" fine. Although, I don't this this is the >>> right thing to do since it should already be using VA). >>> >>> So, I inspected the page tables and compare the ones from booting with >>> and without EFI. At this point, it seems that the contents of the page >>> tables are set up properly. The only thing different is the phys-offset >>> and the location of the page table: >>> >>> BOOTING WITH EFI: >>> >>> x20 (phys-offset) = 0x83fc2d0000 >>> x19 (start paddr) = 0x83fc4d0000 >>> (.text starts at 2MB = 0x200000 offset) >>> >>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000 >>> boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000 >>> boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000 >>> boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000 >>> >>> BOOTING WITHOUT EFI: >>> >>> x20 (phys-offset) = 0x8008500000 >>> x19 (start paddr) = 0x8008700000 >>> (.text starts at 2MB = 0x200000 offset) >>> >>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000 >>> boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000 >>> boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000 >>> boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000 >>> >>> Here is the memory map available from UEFI: >>> >>> Shell> memmap >>> Type Start End #pages Attributes >>> RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000 >>> 800000000000000F >>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE >>> 000000000000000F >>> BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002 >>> 000000000000000F >>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE >>> 000000000000000F >>> BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002 >>> 000000000000000F >>> Available 0000008020000000-000000802FFFFFFF 0000000000010000 >>> 000000000000000F >>> RT_Code 0000008030000000-0000008030000FFF 0000000000000001 >>> 800000000000000F >>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF >>> 000000000000000F >>> BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020 >>> 000000000000000F >>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3 >>> 000000000000000F >>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF >>> 000000000000000F >>> >>> However, it seems that the location falls in the "Available", which >>> should be ok to use. >> >> >> Right, I don't see anything wrong with the location of the page tables. >> >> What do the contents of these PTs look like? >> >> In particular for Xen's load address of 0x200000 I think entries at >> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are >> of interest. Do they correctly map their next level and/or the physical >> address with the actual Xen bits in? >> >> For completeness it would also be worth knowing the entries >> corresponding to the 1:1 pa mapping. >> >> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42, >> 0x100] (I think, but please do check my maths!). I think this will >> result in boot_pgtable[1] mapping boot_first_id which will contain a >> gigabyte page super mapping in slot [0]. >> >> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd]. >> Which I would again expect to result in boot_pgtable[1] mapping >> boot_first_id with a gigabyte mapping in slot 0xf this time. >> >> I expect the PA mapping to be correct, since as you say we are able to >> run off it, it's worth checking though in case we are somehow creating >> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead >> of boot_first_id when creating the PA mappings -- that sort of thing). >> >> Ian. > > > So, I have experimented with: > > /* > * Preserve x0 (fdt pointer) across call to __flush_dcache_area, > * restore for entry into Xen. > */ > mov x20, x0 > > /* > * Flush dcache covering current runtime addresses > * of xen text/data. Then flush all of icache. > */ > adrp x1, _start > add x1, x1, #:lo12:_start > mov x0, x1 <--- Fixed X0 > adrp x2, _end > add x2, x2, #:lo12:_end > sub x1, x2, x1 > > bl __flush_dcache_area > ic ialluis > tlbi alle2 <---- Added this > > Now, that I added the "tlbi alle2", it works after we enable MMU :) I guess > even though we set up the new page table, but it was still using the stale > info in the TLB. > > On to the next hurdle: > > Shell> FS0:xen -cfg=xen-seattle.cfg > Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty) EFI > loader > Image: 0x00000083fbbca000-0x00000083fc4ca890 > - UART enabled - > - CPU 00000000 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Zero BSS - > - Setting up control registers - > - Setup boot first - > - Setup boot second - > - Setup boot third - > - Turning on paging - > - Ready - > (XEN) Checking for initrd in /chosen > (XEN) RAM: 0000008001000000 - 0000008007ffdfff > (XEN) RAM: 0000008007ffe000 - 0000008007ffffff > (XEN) RAM: 0000008008000000 - 000000801fffdfff > (XEN) RAM: 000000801fffe000 - 000000801fffffff > (XEN) RAM: 0000008020000000 - 000000802fffffff > (XEN) RAM: 0000008030001000 - 00000083f0ffffff > (XEN) RAM: 00000083f1000000 - 00000083f101ffff > (XEN) RAM: 00000083f1020000 - 00000083fbbc7fff > (XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff > (XEN) RAM: 00000083fc6b2000 - 00000083fec25fff > (XEN) RAM: 00000083fec26000 - 00000083fee8bfff > (XEN) RAM: 00000083fee8c000 - 00000083ff225fff > (XEN) RAM: 00000083ff226000 - 00000083ff263fff > (XEN) RAM: 00000083ff265000 - 00000083ff2c4fff > (XEN) RAM: 00000083ffe70000 - 00000083ffffffff > (XEN) > (XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree > (XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel console=hvc0 > console=ttyAMA0,115200 earlycon=pl011,0xe1010000 show_styx_info > root=/dev/sda2 rootwait maxcpus=1 > (XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel > (XEN) > (XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x > dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2 > (XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000 > (XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 => > 000000802fe00000-000000802ff02d81 > (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44 > (XEN) CPU0: Unexpected Trap: Hypervisor > (XEN) ----[ Xen-4.5-unstable arm64 debug=y Not tainted ]---- > (XEN) CPU: 0 > (XEN) PC: 00000000002794b4 bootmem_region_add+0x194/0x1c4 > (XEN) LR: 00000000002794ac > (XEN) SP: 00000000002afd50 > (XEN) CPSR: 800003c9 MODE:64-bit EL2h (Hypervisor, handler) > (XEN) X0: 0000800000000000 X1: 0000800000000000 X2: 0000000000000000 > (XEN) X3: 0000800000000000 X4: ffffffffffffffff X5: 0000000000000000 > (XEN) X6: 0000800000000010 X7: 000000000000000a X8: 0000000000000000 > (XEN) X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038 > (XEN) X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030 > (XEN) X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc > (XEN) X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048 > (XEN) X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000 > (XEN) X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff > (XEN) X27: 0000000000298038 X28: 0000008007ffe000 FP: 00000000002afd50 > (XEN) > (XEN) VTCR_EL2: 80000000 > (XEN) VTTBR_EL2: 0000000000000000 > (XEN) > (XEN) SCTLR_EL2: 30cd183d > (XEN) HCR_EL2: 000000000038643f > (XEN) TTBR0_EL2: 000000802feec000 > (XEN) > (XEN) ESR_EL2: 96000044 > (XEN) HPFAR_EL2: 0000000000000000 > (XEN) FAR_EL2: 0000800000000000 > (XEN) > (XEN) Xen stack trace from sp=00000000002afd50: > (XEN) 00000000002afd80 0000000000279530 0000000000290048 0000000000000000 > (XEN) 0000008001000000 0000008001000000 00000000002afdd0 000000000024b154 > (XEN) 0000000000000000 0000000000000000 0000008001000000 0000008001000000 > (XEN) 0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac > (XEN) 00000000002afe20 0000000000283d50 0000000000298030 0000000000000418 > (XEN) 0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000 > (XEN) 0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690 > (XEN) 00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000 > (XEN) 0000000000400000 0000000000000000 0000000000000001 00000083fc544be0 > (XEN) 00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000 > (XEN) 00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 > (XEN) Xen call trace: > (XEN) [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC) > (XEN) [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR) > (XEN) [<0000000000279530>] init_boot_pages+0x4c/0x174 > (XEN) [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0 > (XEN) [<0000000000283d50>] start_xen+0xc38/0xc58 > (XEN) [<0000000000200690>] paging+0x88/0xc0 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) CPU0: Unexpected Trap: Hypervisor > (XEN) > (XEN) **************************************** > (XEN) > (XEN) Manual reset required ('noreboot' specified) > > Any thoughts?? > > Thanks, > > Suravee > > > > After running a bit with your patch (minus the TLB flushing), I found I was seeing some crashes with a corrupt FDT. From looking at the code, I didn't think we would need to flush the FDT, as I don't think it is accessed before mmu/caching is turned back on. This was an intermittent crash on the FVP model with cache state modelling enabled. The following hacky patch seems (in limited testing) to fix this: 1 MByte should cover the FDT - if we really do need to flush it I will do a proper flush of the correct size. The code where you are crashing it is examining the FDT to see if there are any mem-reserve regions, so if the FDT is corrupt it could be causing your crashes as well. (There won't be any mem-reserve regions, as the EFI boot code removes them along with the memory nodes from the FDT.) I'll be interested to hear if flushing the FDT fixes your crashes, and I need to further investigate why this flushing seems to be required. Roy diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 2b8998b..1c2346b 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -744,6 +744,8 @@ ENTRY(efi_xen_start) * restore for entry into Xen. */ mov x20, x0 + mov x1, 0x100000 + bl __flush_dcache_area /* * Flush dcache covering current runtime addresses