diff mbox

[Xen-devel] Failed to enable MMU when booting EFI on Seattle

Message ID CAFECyb9_kB2_=sVu2QD8cXmiVajMTi0roSCK-2qQg5xYjwaRZg@mail.gmail.com
State New
Headers show

Commit Message

Roy Franz Oct. 8, 2014, 4:02 a.m. UTC 
On Tue, Oct 7, 2014 at 6:45 PM, Suravee Suthikulanit
<suravee.suthikulpanit@amd.com> wrote:
> On 10/7/2014 4:21 AM, Ian Campbell wrote:
>>
>> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote:
>>>
>>> Ian/Roy,
>>>
>>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
>>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
>>> address 0x2005F8, which has all zeros value. This is why we getting the
>>> "Synchronous Exception"
>>>
>>> ......
>>> 1:
>>>           PRINT("- Turning on paging -\r\n")
>>>
>>>           ldr   x1, =paging            /* Explicit vaddr, not
>>> RIP-relative */
>>>           mrs   x0, SCTLR_EL2
>>>           orr   x0, x0, #SCTLR_M       /* Enable MMU */
>>>           orr   x0, x0, #SCTLR_C       /* Enable D-cache */
>>>           dsb   sy                     /* Flush PTE writes and finish
>>> reads */
>>>         b .
>>>           msr   SCTLR_EL2, x0          /* now paging is enabled */
>>>           isb                          /* Now, flush the icache */
>>>           br    x1                     /* Get a proper vaddr into PC */
>>> paging:
>>> .....
>>>
>>> Notes:
>>> * This is not the case when booting non-EFI.
>>>
>>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it
>>> seems to branch to "paging:" fine. Although, I don't this this is the
>>> right thing to do since it should already be using VA).
>>>
>>> So, I inspected the page tables and compare the ones from booting with
>>> and without EFI. At this point, it seems that the contents of the page
>>> tables are set up properly. The only thing different is the phys-offset
>>> and the location of the page table:
>>>
>>> BOOTING WITH EFI:
>>>
>>>         x20 (phys-offset) = 0x83fc2d0000
>>>         x19 (start paddr) = 0x83fc4d0000
>>>         (.text starts at 2MB = 0x200000 offset)
>>>
>>>         boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
>>>         boot_first (pa)   = x20 + 0x2f3000 = 0x83fc5c3000
>>>         boot_second(pa)   = x20 + 0x2f5000 = 0x83fc5c5000
>>>         boot_third (pa)   = x20 + 0x2f6000 = 0x83fc5c6000
>>>
>>> BOOTING WITHOUT EFI:
>>>
>>>         x20 (phys-offset) = 0x8008500000
>>>         x19 (start paddr) = 0x8008700000
>>>         (.text starts at 2MB = 0x200000 offset)
>>>
>>>         boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
>>>         boot_first (pa)   = x20 + 0x2f3000 = 0x80087f3000
>>>         boot_second(pa)   = x20 + 0x2f5000 = 0x80087f5000
>>>         boot_third (pa)   = x20 + 0x2f6000 = 0x80087f6000
>>>
>>> Here is the memory map available from UEFI:
>>>
>>> Shell> memmap
>>> Type      Start            End              #pages             Attributes
>>> RT_Data   0000008000000000-0000008000FFFFFF 0000000000001000
>>> 800000000000000F
>>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
>>> 000000000000000F
>>> BS_Code   0000008007FFE000-0000008007FFFFFF 0000000000000002
>>> 000000000000000F
>>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
>>> 000000000000000F
>>> BS_Data   000000801FFFE000-000000801FFFFFFF 0000000000000002
>>> 000000000000000F
>>> Available 0000008020000000-000000802FFFFFFF 0000000000010000
>>> 000000000000000F
>>> RT_Code   0000008030000000-0000008030000FFF 0000000000000001
>>> 800000000000000F
>>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
>>> 000000000000000F
>>> BS_Data   00000083F1000000-00000083F101FFFF 0000000000000020
>>> 000000000000000F
>>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
>>> 000000000000000F
>>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
>>> 000000000000000F
>>>
>>> However, it seems that the location falls in the "Available", which
>>> should be ok to use.
>>
>>
>> Right, I don't see anything wrong with the location of the page tables.
>>
>> What do the contents of these PTs look like?
>>
>> In particular for Xen's load address of 0x200000 I think entries at
>> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are
>> of interest. Do they correctly map their next level and/or the physical
>> address with the actual Xen bits in?
>>
>> For completeness it would also be worth knowing the entries
>> corresponding to the 1:1 pa mapping.
>>
>> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42,
>> 0x100] (I think, but please do check my maths!). I think this will
>> result in boot_pgtable[1] mapping boot_first_id which will contain a
>> gigabyte page super mapping in slot [0].
>>
>> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd].
>> Which I would again expect to result in boot_pgtable[1] mapping
>> boot_first_id with a gigabyte mapping in slot 0xf this time.
>>
>> I expect the PA mapping to be correct, since as you say we are able to
>> run off it, it's worth checking though in case we are somehow creating
>> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead
>> of boot_first_id when creating the PA mappings -- that sort of thing).
>>
>> Ian.
>
>
> So, I have experimented with:
>
>         /*
>          * Preserve x0 (fdt pointer) across call to __flush_dcache_area,
>          * restore for entry into Xen.
>          */
>         mov   x20, x0
>
>         /*
>          * Flush dcache covering current runtime addresses
>          * of xen text/data. Then flush all of icache.
>          */
>         adrp  x1, _start
>         add   x1, x1, #:lo12:_start
>         mov   x0, x1    <--- Fixed X0
>         adrp  x2, _end
>         add   x2, x2, #:lo12:_end
>         sub   x1, x2, x1
>
>         bl    __flush_dcache_area
>         ic    ialluis
>         tlbi  alle2     <---- Added this
>
> Now, that I added the "tlbi alle2", it works after we enable MMU :) I guess
> even though we set up the new page table, but it was still using the stale
> info in the TLB.
>
> On to the next hurdle:
>
> Shell> FS0:xen -cfg=xen-seattle.cfg
> Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty) EFI
> loader
> Image: 0x00000083fbbca000-0x00000083fc4ca890
> - UART enabled -
> - CPU 00000000 booting -
> - Current EL 00000008 -
> - Xen starting at EL2 -
> - Zero BSS -
> - Setting up control registers -
> - Setup boot first -
> - Setup boot second -
> - Setup boot third -
> - Turning on paging -
> - Ready -
> (XEN) Checking for initrd in /chosen
> (XEN) RAM: 0000008001000000 - 0000008007ffdfff
> (XEN) RAM: 0000008007ffe000 - 0000008007ffffff
> (XEN) RAM: 0000008008000000 - 000000801fffdfff
> (XEN) RAM: 000000801fffe000 - 000000801fffffff
> (XEN) RAM: 0000008020000000 - 000000802fffffff
> (XEN) RAM: 0000008030001000 - 00000083f0ffffff
> (XEN) RAM: 00000083f1000000 - 00000083f101ffff
> (XEN) RAM: 00000083f1020000 - 00000083fbbc7fff
> (XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff
> (XEN) RAM: 00000083fc6b2000 - 00000083fec25fff
> (XEN) RAM: 00000083fec26000 - 00000083fee8bfff
> (XEN) RAM: 00000083fee8c000 - 00000083ff225fff
> (XEN) RAM: 00000083ff226000 - 00000083ff263fff
> (XEN) RAM: 00000083ff265000 - 00000083ff2c4fff
> (XEN) RAM: 00000083ffe70000 - 00000083ffffffff
> (XEN)
> (XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree
> (XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel console=hvc0
> console=ttyAMA0,115200 earlycon=pl011,0xe1010000 show_styx_info
> root=/dev/sda2 rootwait maxcpus=1
> (XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel
> (XEN)
> (XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x
> dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2
> (XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000
> (XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 =>
> 000000802fe00000-000000802ff02d81
> (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN) ----[ Xen-4.5-unstable  arm64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) PC:     00000000002794b4 bootmem_region_add+0x194/0x1c4
> (XEN) LR:     00000000002794ac
> (XEN) SP:     00000000002afd50
> (XEN) CPSR:   800003c9 MODE:64-bit EL2h (Hypervisor, handler)
> (XEN)      X0: 0000800000000000  X1: 0000800000000000  X2: 0000000000000000
> (XEN)      X3: 0000800000000000  X4: ffffffffffffffff  X5: 0000000000000000
> (XEN)      X6: 0000800000000010  X7: 000000000000000a  X8: 0000000000000000
> (XEN)      X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038
> (XEN)     X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030
> (XEN)     X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc
> (XEN)     X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048
> (XEN)     X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000
> (XEN)     X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff
> (XEN)     X27: 0000000000298038 X28: 0000008007ffe000  FP: 00000000002afd50
> (XEN)
> (XEN)   VTCR_EL2: 80000000
> (XEN)  VTTBR_EL2: 0000000000000000
> (XEN)
> (XEN)  SCTLR_EL2: 30cd183d
> (XEN)    HCR_EL2: 000000000038643f
> (XEN)  TTBR0_EL2: 000000802feec000
> (XEN)
> (XEN)    ESR_EL2: 96000044
> (XEN)  HPFAR_EL2: 0000000000000000
> (XEN)    FAR_EL2: 0000800000000000
> (XEN)
> (XEN) Xen stack trace from sp=00000000002afd50:
> (XEN)    00000000002afd80 0000000000279530 0000000000290048 0000000000000000
> (XEN)    0000008001000000 0000008001000000 00000000002afdd0 000000000024b154
> (XEN)    0000000000000000 0000000000000000 0000008001000000 0000008001000000
> (XEN)    0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac
> (XEN)    00000000002afe20 0000000000283d50 0000000000298030 0000000000000418
> (XEN)    0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000
> (XEN)    0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690
> (XEN)    00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000
> (XEN)    0000000000400000 0000000000000000 0000000000000001 00000083fc544be0
> (XEN)    00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000
> (XEN)    00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC)
> (XEN)    [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR)
> (XEN)    [<0000000000279530>] init_boot_pages+0x4c/0x174
> (XEN)    [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0
> (XEN)    [<0000000000283d50>] start_xen+0xc38/0xc58
> (XEN)    [<0000000000200690>] paging+0x88/0xc0
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> Any thoughts??
>
> Thanks,
>
> Suravee
>
>
>
>

After running a bit with your patch (minus the TLB flushing), I found
I was seeing some crashes with a corrupt FDT.  From looking at the
code,
I didn't think we would need to flush the FDT, as I don't think it is
accessed before mmu/caching is turned back on.  This was an
intermittent crash
on the FVP model with cache state modelling enabled.

The following hacky patch seems (in limited testing) to fix this:


1 MByte should cover the FDT - if we really do need to flush it I will
do a proper flush of the correct size.

The code where you are crashing it is examining the FDT to see if
there are any mem-reserve regions, so if the FDT
is corrupt it could be causing your crashes as well.  (There won't be
any mem-reserve regions, as the EFI boot code
removes them along with the memory nodes from the FDT.)

I'll be interested to hear if flushing the FDT fixes your crashes, and
I need to further investigate why this flushing seems
to be required.

Roy
diff mbox

Patch

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 2b8998b..1c2346b 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -744,6 +744,8 @@  ENTRY(efi_xen_start)
          * restore for entry into Xen.
          */
         mov   x20, x0
+        mov   x1, 0x100000
+        bl    __flush_dcache_area

         /*
          * Flush dcache covering current runtime addresses