From patchwork Wed May 14 18:17:30 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 30188 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ie0-f197.google.com (mail-ie0-f197.google.com [209.85.223.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id E5C1A20446 for ; Wed, 14 May 2014 18:18:08 +0000 (UTC) Received: by mail-ie0-f197.google.com with SMTP id ar20sf10743049iec.0 for ; Wed, 14 May 2014 11:18:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe; bh=ms+2ESAtWoK+NjSt+ddKVdlnHBEoYMY3bez82Z3AXbw=; b=D7+e2TAcE17E4YIbeQ4qvUSwqLT9vboZ0EhDSffx0jj90cxOYHE9ZPsF4XdpJN6wHB 1YPsuLPuRHRjrYCT0uql7zgZ8aGs6cflUn6Q1S9aRZYCCaZ+9B5v9xanGw4qj+0ZWyo6 VdRH56h6j37BjgMSCzz4ki9boFvOR7gYj1KcOl1vH+jV9POb4tcvpG5msPLgl/cUXbK/ zqKV8uTcsciv0/jil0KsFLbR7D2OhBjhQXhkDvT1v4SZgfA1P+XjhnL8d3aGUToF+2c2 QI0W9FH7YIkMwO0z6xbtiUnNrVSePBK6JTZwNh84yZ1Po5IyzdiBD3QiL4uAK5wIyI9R xIBA== X-Gm-Message-State: ALoCoQmY3YNgJbc2RhDVGJEQzTkxLtreJXauYc4n4OEVrgWEtaghloJx3CpWRg0srtrgS3gEHpcu X-Received: by 10.50.130.102 with SMTP id od6mr501538igb.0.1400091488419; Wed, 14 May 2014 11:18:08 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.48.112 with SMTP id n103ls2583652qga.86.gmail; Wed, 14 May 2014 11:18:08 -0700 (PDT) X-Received: by 10.58.18.198 with SMTP id y6mr4187004ved.31.1400091488272; Wed, 14 May 2014 11:18:08 -0700 (PDT) Received: from mail-vc0-f177.google.com (mail-vc0-f177.google.com [209.85.220.177]) by mx.google.com with ESMTPS id ui2si461082vdc.64.2014.05.14.11.18.08 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 14 May 2014 11:18:08 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.177 as permitted sender) client-ip=209.85.220.177; Received: by mail-vc0-f177.google.com with SMTP id if17so2898363vcb.8 for ; Wed, 14 May 2014 11:18:08 -0700 (PDT) X-Received: by 10.58.216.163 with SMTP id or3mr9795vec.80.1400091488189; Wed, 14 May 2014 11:18:08 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp261281vcb; Wed, 14 May 2014 11:18:07 -0700 (PDT) X-Received: by 10.66.139.201 with SMTP id ra9mr2845139pab.84.1400091486864; Wed, 14 May 2014 11:18:06 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id qe5si1341172pbc.453.2014.05.14.11.18.06; Wed, 14 May 2014 11:18:06 -0700 (PDT) Received-SPF: none (google.com: linux-crypto-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751143AbaENSSF (ORCPT ); Wed, 14 May 2014 14:18:05 -0400 Received: from mail-yh0-f54.google.com ([209.85.213.54]:60112 "EHLO mail-yh0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751339AbaENSSE (ORCPT ); Wed, 14 May 2014 14:18:04 -0400 Received: by mail-yh0-f54.google.com with SMTP id i57so2031071yha.13 for ; Wed, 14 May 2014 11:18:03 -0700 (PDT) X-Received: by 10.236.23.163 with SMTP id v23mr7746405yhv.58.1400091483817; Wed, 14 May 2014 11:18:03 -0700 (PDT) Received: from ards-macbook-pro.swisscom.com ([12.153.182.133]) by mx.google.com with ESMTPSA id c66sm3641080yhk.23.2014.05.14.11.18.01 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 14 May 2014 11:18:02 -0700 (PDT) From: Ard Biesheuvel To: catalin.marinas@arm.com, jussi.kivilinna@iki.fi, herbert@gondor.apana.org.au Cc: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org, Ard Biesheuvel Subject: [PATCH v2 10/11] arm64/crypto: add voluntary preemption to Crypto Extensions SHA2 Date: Wed, 14 May 2014 11:17:30 -0700 Message-Id: <1400091451-9117-11-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1400091451-9117-1-git-send-email-ard.biesheuvel@linaro.org> References: <1400091451-9117-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.177 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The Crypto Extensions based SHA2 implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel Acked-by: Herbert Xu --- arch/arm64/crypto/sha2-ce-core.S | 19 ++++++++------- arch/arm64/crypto/sha2-ce-glue.c | 51 ++++++++++++++++++++++++++-------------- 2 files changed, 44 insertions(+), 26 deletions(-) diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S index 7f29fc031ea8..71c617cd57ca 100644 --- a/arch/arm64/crypto/sha2-ce-core.S +++ b/arch/arm64/crypto/sha2-ce-core.S @@ -73,8 +73,8 @@ .word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2 /* - * void sha2_ce_transform(int blocks, u8 const *src, u32 *state, - * u8 *head, long bytes) + * int sha2_ce_transform(int blocks, u8 const *src, u32 *state, + * u8 *head, long bytes, struct thread_info *ti) */ ENTRY(sha2_ce_transform) /* load round constants */ @@ -131,7 +131,14 @@ CPU_LE( rev32 v19.16b, v19.16b ) add dgbv.4s, dgbv.4s, dg1v.4s /* handled all input blocks? */ - cbnz w0, 0b + cbz w0, 4f + + /* should we exit early? */ + b_if_no_resched x5, x8, 0b + + /* store new state */ +3: stp dga, dgb, [x2] + ret /* * Final block: add padding and total bit count. @@ -139,7 +146,7 @@ CPU_LE( rev32 v19.16b, v19.16b ) * size was not a round multiple of the block size, and the padding is * handled by the C code. */ - cbz x4, 3f +4: cbz x4, 3b movi v17.2d, #0 mov x8, #0x80000000 movi v18.2d, #0 @@ -149,8 +156,4 @@ CPU_LE( rev32 v19.16b, v19.16b ) mov v19.d[0], xzr mov v19.d[1], x7 b 2b - - /* store new state */ -3: stp dga, dgb, [x2] - ret ENDPROC(sha2_ce_transform) diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c index c294e67d3925..3ab69e8f8604 100644 --- a/arch/arm64/crypto/sha2-ce-glue.c +++ b/arch/arm64/crypto/sha2-ce-glue.c @@ -21,7 +21,7 @@ MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); asmlinkage int sha2_ce_transform(int blocks, u8 const *src, u32 *state, - u8 *head, long bytes); + u8 *head, long bytes, struct thread_info *ti); static int sha224_init(struct shash_desc *desc) { @@ -49,6 +49,34 @@ static int sha256_init(struct shash_desc *desc) return 0; } +static u8 const *sha2_do_update(struct shash_desc *desc, const u8 *data, + int blocks, u8 *head, unsigned int len) +{ + struct sha256_state *sctx = shash_desc_ctx(desc); + struct thread_info *ti = NULL; + + /* + * Pass current's thread info pointer to sha2_ce_transform() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || IS_ENABLED(CONFIG_PREEMPT)) + && (desc->flags & CRYPTO_TFM_REQ_MAY_SLEEP)) + ti = current_thread_info(); + + do { + int rem; + + kernel_neon_begin_partial(28); + rem = sha2_ce_transform(blocks, data, sctx->state, head, 0, ti); + kernel_neon_end(); + + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + head = NULL; + } while (unlikely(ti && blocks > 0)); + return data; +} + static int sha2_update(struct shash_desc *desc, const u8 *data, unsigned int len) { @@ -58,8 +86,6 @@ static int sha2_update(struct shash_desc *desc, const u8 *data, sctx->count += len; if ((partial + len) >= SHA256_BLOCK_SIZE) { - int blocks; - if (partial) { int p = SHA256_BLOCK_SIZE - partial; @@ -68,15 +94,10 @@ static int sha2_update(struct shash_desc *desc, const u8 *data, len -= p; } - blocks = len / SHA256_BLOCK_SIZE; - len %= SHA256_BLOCK_SIZE; + data = sha2_do_update(desc, data, len / SHA256_BLOCK_SIZE, + partial ? sctx->buf : NULL, 0); - kernel_neon_begin_partial(28); - sha2_ce_transform(blocks, data, sctx->state, - partial ? sctx->buf : NULL, 0); - kernel_neon_end(); - - data += blocks * SHA256_BLOCK_SIZE; + len %= SHA256_BLOCK_SIZE; partial = 0; } if (len) @@ -131,7 +152,6 @@ static void sha2_finup(struct shash_desc *desc, const u8 *data, unsigned int len) { struct sha256_state *sctx = shash_desc_ctx(desc); - int blocks; if (sctx->count || !len || (len % SHA256_BLOCK_SIZE)) { sha2_update(desc, data, len); @@ -145,12 +165,7 @@ static void sha2_finup(struct shash_desc *desc, const u8 *data, * perform the entire digest calculation in a single invocation * of sha2_ce_transform() */ - blocks = len / SHA256_BLOCK_SIZE; - - kernel_neon_begin_partial(28); - sha2_ce_transform(blocks, data, sctx->state, NULL, len); - kernel_neon_end(); - data += blocks * SHA256_BLOCK_SIZE; + sha2_do_update(desc, data, len / SHA256_BLOCK_SIZE, NULL, len); } static int sha224_finup(struct shash_desc *desc, const u8 *data,