From patchwork Wed Sep 25 12:31:29 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 20568 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-qe0-f71.google.com (mail-qe0-f71.google.com [209.85.128.71]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id A2F5225E71 for ; Wed, 25 Sep 2013 12:31:40 +0000 (UTC) Received: by mail-qe0-f71.google.com with SMTP id a11sf5359278qen.2 for ; Wed, 25 Sep 2013 05:31:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-unsubscribe; bh=FC9W6UI3bR9FtKqEhGC3VbWdT+U8lVHh9BFRBCYvEuA=; b=iHDb2rmP4wNYKKcv3YjlHKUP+/2cgi4P4Y8TdtslCRtC6ieTXLOeNZA3JydRFVJmlj csHSp1/D+ZiFNFa4U0/BshH9m1wOkPH+Bj3x63Z0B2Yj51Ki8mNub3GchcaC4VludMBS KoPmjththoaQi6uTZ+KxbIbpYkb/Jy+PHWL1mDFLZitM0WjeihJy5bOM0yngHmBPAhGj lwygRbHnuTCIs1Jr5R8A0qbW/8O4LwRQgRTOoo8bCfEKedGQho2y/aZaU4cA9VThN+Sl Esj1zOOh2g2dgyllbupVVzCTcC/JwPZsW5Iipea7WqnNpN8VXqf9HLND/SoWKaD7BpZG oj/Q== X-Gm-Message-State: ALoCoQmsusNmHpC7yKkODQAJT+F3b+6i33iA5w+uCrNb6N2ZdjcY3+i+mDbnUXae5QTYPgYItbqu X-Received: by 10.236.147.18 with SMTP id s18mr10734437yhj.28.1380112299992; Wed, 25 Sep 2013 05:31:39 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.50.169 with SMTP id d9ls353757qeo.21.gmail; Wed, 25 Sep 2013 05:31:39 -0700 (PDT) X-Received: by 10.52.118.73 with SMTP id kk9mr27394473vdb.13.1380112299847; Wed, 25 Sep 2013 05:31:39 -0700 (PDT) Received: from mail-ve0-f176.google.com (mail-ve0-f176.google.com [209.85.128.176]) by mx.google.com with ESMTPS id 4si9702105vcq.122.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 25 Sep 2013 05:31:39 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.128.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.128.176; Received: by mail-ve0-f176.google.com with SMTP id jx11so4489140veb.7 for ; Wed, 25 Sep 2013 05:31:39 -0700 (PDT) X-Received: by 10.58.133.66 with SMTP id pa2mr17890269veb.18.1380112299704; Wed, 25 Sep 2013 05:31:39 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.174.196 with SMTP id u4csp302398vcz; Wed, 25 Sep 2013 05:31:38 -0700 (PDT) X-Received: by 10.194.158.67 with SMTP id ws3mr26980924wjb.5.1380112298454; Wed, 25 Sep 2013 05:31:38 -0700 (PDT) Received: from mail-we0-f173.google.com (mail-we0-f173.google.com [74.125.82.173]) by mx.google.com with ESMTPS id cw8si14721384wjb.27.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 25 Sep 2013 05:31:38 -0700 (PDT) Received-SPF: neutral (google.com: 74.125.82.173 is neither permitted nor denied by best guess record for domain of steve.capper@linaro.org) client-ip=74.125.82.173; Received: by mail-we0-f173.google.com with SMTP id w62so6073416wes.32 for ; Wed, 25 Sep 2013 05:31:38 -0700 (PDT) X-Received: by 10.180.108.199 with SMTP id hm7mr22734351wib.31.1380112297875; Wed, 25 Sep 2013 05:31:37 -0700 (PDT) Received: from marmot.wormnet.eu (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id ey2sm17470625wib.5.1969.12.31.16.00.00 (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 25 Sep 2013 05:31:37 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org Cc: linux@arm.linux.org.uk, nico@linaro.org, linaro-kernel@lists.linaro.org, patches@linaro.org, Steve Capper Subject: [PATCH V2] ARM: mm: make UACCESS_WITH_MEMCPY huge page aware Date: Wed, 25 Sep 2013 13:31:29 +0100 Message-Id: <1380112289-19572-1-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.10.4 X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: steve.capper@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.128.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The memory pinning code in uaccess_with_memcpy.c does not check for HugeTLB or THP pmds, and will enter an infinite loop should a __copy_to_user or __clear_user occur against a huge page. This patch adds detection code for huge pages to pin_page_for_write. As this code can be executed in a fast path it refers to the actual pmds rather than the vma. If a HugeTLB or THP is found (they have the same pmd representation on ARM), the page table spinlock is taken to prevent modification whilst the page is pinned. On ARM, huge pages are only represented as pmds, thus no huge pud checks are performed. (For huge puds one would lock the page table in a similar manner as in the pmd case). Two helper functions are introduced; pmd_thp_or_huge will check whether or not a page is huge or transparent huge (which have the same pmd layout on ARM), and pmd_hugewillfault will detect whether or not a page fault will occur on write to the page. Running the following test (with the chunking from read_zero removed): $ dd if=/dev/zero of=/dev/null bs=10M count=1024 Gave: 2.3 GB/s backed by normal pages, 2.9 GB/s backed by huge pages, 5.1 GB/s backed by huge pages, with page mask=HPAGE_MASK. After some discussion, it was decided not to adopt the HPAGE_MASK, as this would have a significant detrimental effect on the overall system latency due to page_table_lock being held for too long. This could be revisited if split huge page locks are adopted. Signed-off-by: Steve Capper Reviewed-by: Nicolas Pitre --- arch/arm/include/asm/pgtable-3level.h | 3 +++ arch/arm/lib/uaccess_with_memcpy.c | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 5689c18..39c54cf 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -206,6 +206,9 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr) #define __HAVE_ARCH_PMD_WRITE #define pmd_write(pmd) (!(pmd_val(pmd) & PMD_SECT_RDONLY)) +#define pmd_hugewillfault(pmd) (!pmd_young(pmd) || !pmd_write(pmd)) +#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT)) #define pmd_trans_splitting(pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING) diff --git a/arch/arm/lib/uaccess_with_memcpy.c b/arch/arm/lib/uaccess_with_memcpy.c index 025f742..3e58d71 100644 --- a/arch/arm/lib/uaccess_with_memcpy.c +++ b/arch/arm/lib/uaccess_with_memcpy.c @@ -18,6 +18,7 @@ #include /* for in_atomic() */ #include #include +#include #include #include @@ -40,7 +41,35 @@ pin_page_for_write(const void __user *_addr, pte_t **ptep, spinlock_t **ptlp) return 0; pmd = pmd_offset(pud, addr); - if (unlikely(pmd_none(*pmd) || pmd_bad(*pmd))) + if (unlikely(pmd_none(*pmd))) + return 0; + + /* + * A pmd can be bad if it refers to a HugeTLB or THP page. + * + * Both THP and HugeTLB pages have the same pmd layout + * and should not be manipulated by the pte functions. + * + * Lock the page table for the destination and check + * to see that it's still huge and whether or not we will + * need to fault on write, or if we have a splitting THP. + */ + if (unlikely(pmd_thp_or_huge(*pmd))) { + ptl = ¤t->mm->page_table_lock; + spin_lock(ptl); + if (unlikely(!pmd_thp_or_huge(*pmd) + || pmd_hugewillfault(*pmd) + || pmd_trans_splitting(*pmd))) { + spin_unlock(ptl); + return 0; + } + + *ptep = NULL; + *ptlp = ptl; + return 1; + } + + if (unlikely(pmd_bad(*pmd))) return 0; pte = pte_offset_map_lock(current->mm, pmd, addr, &ptl); @@ -94,7 +123,10 @@ __copy_to_user_memcpy(void __user *to, const void *from, unsigned long n) from += tocopy; n -= tocopy; - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); + else + spin_unlock(ptl); } if (!atomic) up_read(¤t->mm->mmap_sem); @@ -147,7 +179,10 @@ __clear_user_memset(void __user *addr, unsigned long n) addr += tocopy; n -= tocopy; - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); + else + spin_unlock(ptl); } up_read(¤t->mm->mmap_sem);