From patchwork Fri May 2 13:49:00 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 29552 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-vc0-f197.google.com (mail-vc0-f197.google.com [209.85.220.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 8DD99203F3 for ; Fri, 2 May 2014 13:52:06 +0000 (UTC) Received: by mail-vc0-f197.google.com with SMTP id if11sf8100593vcb.4 for ; Fri, 02 May 2014 06:52:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id:cc :precedence:list-id:list-unsubscribe:list-archive:list-post :list-help:list-subscribe:mime-version:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=ZuTcvnN8q9mtKWL7F2myqy+UfcMxREXfulujMnwgJbA=; b=HI4jGZEOPijaX0h1q2WS5wO52RcFiqwWHLRpjgI26xX6vCz2+PImB1vqjeEUck5SLz YV7GG2iSAO3wMqfAe4hRGvY8KhjP1wi08ORYIDViBaw5r653vRfr+my6vrqgDZLX0lXW tN9683/DvOeJwIrBT7+GNAn9cB6UQ5NETg7+bpfb8oQmUcGZRUUzWyIuhfx2d3MfB3/b /4g/O3vM3yfD/z1ZpMVvanlIkk/2l/R0TuuyoFar+3fb/DLyzD4lKHQS7Eh6JPXSlHl5 M9dM2ikWDFVZ0690B1Uc4KA8Rz8jYXjkoU0cJ5+bDHxReYuypB3CVJM+HYWoErlfPAlR YNLQ== X-Gm-Message-State: ALoCoQmhBSWWbhBL68t8IaDNMzwGqFeLupdyB7AjowdP5RnQlF4EiZUdglK5PkG8nRQ1t2nDyGwV X-Received: by 10.224.147.72 with SMTP id k8mr8784272qav.5.1399038726106; Fri, 02 May 2014 06:52:06 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.23.106 with SMTP id 97ls1511583qgo.34.gmail; Fri, 02 May 2014 06:52:06 -0700 (PDT) X-Received: by 10.58.111.163 with SMTP id ij3mr1924516veb.26.1399038725996; Fri, 02 May 2014 06:52:05 -0700 (PDT) Received: from mail-vc0-f177.google.com (mail-vc0-f177.google.com [209.85.220.177]) by mx.google.com with ESMTPS id tv3si6699259vdc.126.2014.05.02.06.52.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 02 May 2014 06:52:05 -0700 (PDT) Received-SPF: none (google.com: patch+caf_=patchwork-forward=linaro.org@linaro.org does not designate permitted sender hosts) client-ip=209.85.220.177; Received: by mail-vc0-f177.google.com with SMTP id if11so5343316vcb.22 for ; Fri, 02 May 2014 06:52:05 -0700 (PDT) X-Received: by 10.52.153.229 with SMTP id vj5mr586423vdb.34.1399038725780; Fri, 02 May 2014 06:52:05 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp104456vcb; Fri, 2 May 2014 06:52:05 -0700 (PDT) X-Received: by 10.140.47.206 with SMTP id m72mr20600726qga.21.1399038725053; Fri, 02 May 2014 06:52:05 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id m6si14042134qay.172.2014.05.02.06.52.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 May 2014 06:52:04 -0700 (PDT) Received-SPF: none (google.com: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org does not designate permitted sender hosts) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WgDqL-0003eB-Sy; Fri, 02 May 2014 13:49:37 +0000 Received: from mail-wi0-f169.google.com ([209.85.212.169]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WgDqI-0003bw-SN for linux-arm-kernel@lists.infradead.org; Fri, 02 May 2014 13:49:35 +0000 Received: by mail-wi0-f169.google.com with SMTP id hi5so2260674wib.0 for ; Fri, 02 May 2014 06:49:08 -0700 (PDT) X-Received: by 10.180.88.129 with SMTP id bg1mr3013490wib.51.1399038548324; Fri, 02 May 2014 06:49:08 -0700 (PDT) Received: from marmot.wormnet.eu (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id fx6sm5005704wib.4.2014.05.02.06.49.06 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 May 2014 06:49:07 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org Subject: [PATH V3] arm64: mm: Optimise tlb flush logic where we have >4K granule Date: Fri, 2 May 2014 14:49:00 +0100 Message-Id: <1399038540-6547-1-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.10.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140502_064935_214315_19C9F4D9 X-CRM114-Status: GOOD ( 17.67 ) X-Spam-Score: -0.7 (/) X-Spam-Report: SpamAssassin version 3.3.2 on bombadil.infradead.org summary: Content analysis details: (-0.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [209.85.212.169 listed in list.dnswl.org] Cc: Laura Abbott , catalin.marinas@arm.com, will.deacon@arm.com, Steve Capper X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: steve.capper@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: patch+caf_=patchwork-forward=linaro.org@linaro.org does not designate permitted sender hosts) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages. This patch re-works the logic s.t. we instead advance the loop by 1 << (PAGE_SHIFT - 12), so avoid repeating ourselves. Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations. The isb() has been removed from flush_tlb_kernel_range(.) as it is only needed when changing the execute permission of a mapping. If one needs to set an area of the kernel as execute/non-execute an isb() must be inserted after the call to flush_tlb_kernel_range. Cc: Laura Abbott Signed-off-by: Steve Capper Acked-by: Will Deacon --- Changed in V3: isb() taken out of flush_tlb_kernel_range, as it is only really needed when changing execute permissions. This has been documented in the commit log. --- arch/arm64/include/asm/tlbflush.h | 30 ++++++++++++++--- arch/arm64/mm/Makefile | 2 +- arch/arm64/mm/tlb.S | 71 --------------------------------------- 3 files changed, 26 insertions(+), 77 deletions(-) delete mode 100644 arch/arm64/mm/tlb.S diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 8b48203..7881d7d 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -98,11 +98,31 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(); } -/* - * Convert calls to our calling convention. - */ -#define flush_tlb_range(vma,start,end) __cpu_flush_user_tlb_range(start,end,vma) -#define flush_tlb_kernel_range(s,e) __cpu_flush_kern_tlb_range(s,e) +static inline void flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48; + unsigned long addr; + start = asid | (start >> 12); + end = asid | (end >> 12); + + dsb(ishst); + for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) + asm("tlbi vae1is, %0" : : "r"(addr)); + dsb(ish); +} + +static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + unsigned long addr; + start >>= 12; + end >>= 12; + + dsb(ishst); + for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) + asm("tlbi vaae1is, %0" : : "r"(addr)); + dsb(ish); +} /* * On AArch64, the cache coherency is handled via the set_pte_at() function. diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index b51d364..3ecb56c 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -1,5 +1,5 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ - context.o tlb.o proc.o + context.o proc.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o diff --git a/arch/arm64/mm/tlb.S b/arch/arm64/mm/tlb.S deleted file mode 100644 index 19da91e..0000000 --- a/arch/arm64/mm/tlb.S +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Based on arch/arm/mm/tlb.S - * - * Copyright (C) 1997-2002 Russell King - * Copyright (C) 2012 ARM Ltd. - * Written by Catalin Marinas - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see . - */ -#include -#include -#include -#include -#include -#include "proc-macros.S" - -/* - * __cpu_flush_user_tlb_range(start, end, vma) - * - * Invalidate a range of TLB entries in the specified address space. - * - * - start - start address (may not be aligned) - * - end - end address (exclusive, may not be aligned) - * - vma - vma_struct describing address range - */ -ENTRY(__cpu_flush_user_tlb_range) - vma_vm_mm x3, x2 // get vma->vm_mm - mmid w3, x3 // get vm_mm->context.id - dsb sy - lsr x0, x0, #12 // align address - lsr x1, x1, #12 - bfi x0, x3, #48, #16 // start VA and ASID - bfi x1, x3, #48, #16 // end VA and ASID -1: tlbi vae1is, x0 // TLB invalidate by address and ASID - add x0, x0, #1 - cmp x0, x1 - b.lo 1b - dsb sy - ret -ENDPROC(__cpu_flush_user_tlb_range) - -/* - * __cpu_flush_kern_tlb_range(start,end) - * - * Invalidate a range of kernel TLB entries. - * - * - start - start address (may not be aligned) - * - end - end address (exclusive, may not be aligned) - */ -ENTRY(__cpu_flush_kern_tlb_range) - dsb sy - lsr x0, x0, #12 // align address - lsr x1, x1, #12 -1: tlbi vaae1is, x0 // TLB invalidate by address - add x0, x0, #1 - cmp x0, x1 - b.lo 1b - dsb sy - isb - ret -ENDPROC(__cpu_flush_kern_tlb_range)