From patchwork Wed Dec 6 19:43:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120889 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468699qgn; Wed, 6 Dec 2017 11:44:23 -0800 (PST) X-Google-Smtp-Source: AGs4zMbu96T7mcOWO4y7gzpPYgQjfJW0x+3qk/RNOEINfyK1YLmKQmaH8tAQH4tYB2e/ljfnQfz1 X-Received: by 10.84.168.98 with SMTP id e89mr22689727plb.417.1512589463086; Wed, 06 Dec 2017 11:44:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589463; cv=none; d=google.com; s=arc-20160816; b=S6WvL0GKiHml7A3Bx0Be/UfTyyGU2dq4Hv+xcZF2FWuxXe9I1dWOrQXK2IExnMC66S wTcCAhUtM/eKuOgVkbQVgGVaacHqe9qkYrxlsiqwxI4VVyakGVt/dmxn4VXISLl3ryuv KOKDfJ/KdoyUMBFj2pQf2s2s80liuiGoXVKyRHCW2OCih9JtbZSarf/OuiLFvDLofuGa W56NDpF7rEAA+aXmbwLWPwTBqmQBio2A0w+mFZjOuMjFm+DnvpCn02G0Yyc8GUN2+vHz 37NYM5GG3VehKBp4tLdj3+4ASvhcpNNI3QD4GI4N6rw1BpRdcLdu/VSfOfqn1ezb04OD Ampg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ByEfG8S6mziH6dHyDov6YH26KAHAKkbH9eW3AIOrYSU=; b=tHHnn2ebh0a8txYTt7idy4wmRlSPDnQ6tDiboNaMMKiOf/0Lg3CVxx4m+gjWq4x9u9 VkI+3YAi/ndYAtKbFhqv/aiLuCCwrb5laShYyd5vqXd4KYg0REn2SPM9egKwqDFOBGWI /SV1jsG418rG+tPt7bTcDV+1LR+VJvGH543rdfINJVBmDwhUdbvptX5lmpIsrkNn4LMG leuK5vZaUzWG06/+3iysos9HWXh/f6n6Chb+284SAXLCVmTZ/WXphaRViX5GfKZTZAR0 rK13oPT5E89xYPdvCWWSDAmqBYpE+YKZd3HH6yDUafOj1rbvc+7n9Dx6Uo+h0SsBkUuT 7vvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=gPnc/IaL; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.22; Wed, 06 Dec 2017 11:44:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=gPnc/IaL; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752282AbdLFToV (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:21 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:41417 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752119AbdLFToQ (ORCPT ); Wed, 6 Dec 2017 14:44:16 -0500 Received: by mail-wr0-f194.google.com with SMTP id z18so5104136wrb.8 for ; Wed, 06 Dec 2017 11:44:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=gPnc/IaLPhCdaYOcGr4J+2RoP5OwvmoAXTKq9ICW09VxYkYF54b42E1vjjqzmKGLoM T7GLu3KWgqOXwI0iRN8boNafiqAXMmo+78Twwjs6gs3A3YmjVaKpmmOMOmWT0+WjGoUb R0uCMHgmXAQVpei9kx6u1t9YmKaE1qwLbasfo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=erCvdS1hCRkFn5sYiAmNNV3Cq5kYcJhug4TSarCw5pxWVVhlYIc190Hk5mPkd1rwHi UYyPFGBysUSegmrRjAQm8RHFv2a+ZIfxSSXnaJrKRn8IfVewXBdGsA0jPda3Tyz+yvBT I6UR0Irr/8/0TdvUWy1JX3VYElVkXUwX8TnEhu0RE3WXpRcSgWfcvlaFuaUw4pV1nN25 xGHc0uYVdn20A6/qHB+t/k/g3A6vxR2/9JyehH+SjWK/DP6Ha41WMzKSUWiI6u58y7gK h4/Un1G+OY9DCPM97LgAWiurMwqu1aLFR7Lzrq2PV6Zy/uMOMcziHyioRDgvBTiEH0RU Wr+Q== X-Gm-Message-State: AJaThX6yvHCWNPUl+Y0wgZtves/F8rm6alnVqerKwS+Huk4RkkokcNsf 4HgPq5Istrri+6+e/BYqARfiRw== X-Received: by 10.223.195.103 with SMTP id e36mr21193552wrg.10.1512589454918; Wed, 06 Dec 2017 11:44:14 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:13 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 07/20] crypto: arm64/aes-blk - add 4 way interleave to CBC encrypt path Date: Wed, 6 Dec 2017 19:43:33 +0000 Message-Id: <20171206194346.24393-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC encryption is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC encrypt which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 31 ++++++++++++++++---- 1 file changed, 25 insertions(+), 6 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 27a235b2ddee..e86535a1329d 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -94,17 +94,36 @@ AES_ENDPROC(aes_ecb_decrypt) */ AES_ENTRY(aes_cbc_encrypt) - ld1 {v0.16b}, [x5] /* get iv */ + ld1 {v4.16b}, [x5] /* get iv */ enc_prepare w3, x2, x6 -.Lcbcencloop: - ld1 {v1.16b}, [x1], #16 /* get next pt block */ - eor v0.16b, v0.16b, v1.16b /* ..and xor with iv */ +.Lcbcencloop4x: + subs w4, w4, #4 + bmi .Lcbcenc1x + ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + eor v0.16b, v0.16b, v4.16b /* ..and xor with iv */ encrypt_block v0, w3, x2, x6, w7 - st1 {v0.16b}, [x0], #16 + eor v1.16b, v1.16b, v0.16b + encrypt_block v1, w3, x2, x6, w7 + eor v2.16b, v2.16b, v1.16b + encrypt_block v2, w3, x2, x6, w7 + eor v3.16b, v3.16b, v2.16b + encrypt_block v3, w3, x2, x6, w7 + st1 {v0.16b-v3.16b}, [x0], #64 + mov v4.16b, v3.16b + b .Lcbcencloop4x +.Lcbcenc1x: + adds w4, w4, #4 + beq .Lcbcencout +.Lcbcencloop: + ld1 {v0.16b}, [x1], #16 /* get next pt block */ + eor v4.16b, v4.16b, v0.16b /* ..and xor with iv */ + encrypt_block v4, w3, x2, x6, w7 + st1 {v4.16b}, [x0], #16 subs w4, w4, #1 bne .Lcbcencloop - st1 {v0.16b}, [x5] /* return iv */ +.Lcbcencout: + st1 {v4.16b}, [x5] /* return iv */ ret AES_ENDPROC(aes_cbc_encrypt)