From patchwork Tue Dec 26 10:29:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122727 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp779120qgn; Tue, 26 Dec 2017 02:31:07 -0800 (PST) X-Google-Smtp-Source: ACJfBovmQ/dGSQ4PQ4O9z7H7jfIr/x3lRv7GvAtGWO60OoiJdM2c+F40pMI/zZE1v9kOkEssXgXw X-Received: by 10.101.96.1 with SMTP id m1mr22104434pgu.38.1514284267679; Tue, 26 Dec 2017 02:31:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514284267; cv=none; d=google.com; s=arc-20160816; b=o4MQeqFPVBDSUPl0w3hvHU+9yHRSxBvaJEHWSygF1dfioMcIzlYXm2TY11+uQpZfb5 k54zikUiJe/OIvVTbkz4Y+n4K2+05bqeXTOmsYwHxmK4Tt4Fs4zbr/IKyawnvY6pgNyV JiAB2U22EqZTmtsZSbH2ShcAoVV9Dqr+h3aeKVwNRE1W4gn7C/1/6vZNLzV+TEvpfcnp krlZE+r+biGjuhQNlTslGpCRfedCbEsrsZhe3Iiu4OCTD5lPdmqBwpbMcRiQWGJ0MPof FztEoT7OTCQocxEdSaCklx3JhjeuLpbHgHkSireyBHRx0KRYt1OBvPDuFemCehbtpGI8 knNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=4L8VBFJ8/vUZayezPEZFRvTau06CygKiThORT60NjPo=; b=zTLHmwbwxHvKdGyqUxcATOqLaJ3FGzSLYbcfFOJ8bGISyeUxPXCL8V9vnpHK1ukRk1 Yk5OO4vNBZJBLdpw1izRKVQCRwtLoigoYpT2c9W0L+W3lgwXlgiU+0vg2dQ81Brkda+r sHXOT5qgW/aKVoygX/YOsReSQXunEgqUeL4ksvOE7AsfFIUotnzFvz1jU1PsbYmj4BBr CJAgKShX2qGWXXDQp1WsApEeV3FF8RLae9zk7f3xdZHmhDa78+HlL/Qzgv/tzFsmdW75 +0M6U8urDl4irkGgZvfjyuEiW1xCNSxtUHp0sWaGrYBmQ6MKKf18OKnpLR/ypIVF+7EL rzxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=TeKl2C6U; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 1si22528060plw.770.2017.12.26.02.31.07; Tue, 26 Dec 2017 02:31:07 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=TeKl2C6U; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751168AbdLZKbF (ORCPT + 4 others); Tue, 26 Dec 2017 05:31:05 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35619 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751159AbdLZKbD (ORCPT ); Tue, 26 Dec 2017 05:31:03 -0500 Received: by mail-wr0-f193.google.com with SMTP id l19so21938867wrc.2 for ; Tue, 26 Dec 2017 02:31:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=TeKl2C6UwKzl/WOrBiYx6T4sb2jc0yg3gmsSB33y+uREsjUvSn8K77g3rmPuA19P3c Bgh3hap0Uhibh0J+rweXCUCjkdXoCwnu97OPMAnP1Nloe6W1Xwq02QXg058ORDUDr2Vz dr/C6ncn0CUKClZzhQBIu20rlL2nQ4HOKSv/E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=kaudKpSdZKG1Da56y+mr9qtqVxEtC6AaipoIXEcYiIBe6vRmKLUl+N4++8dAsJM/u6 FL3GNp7vrWpXm74NuFFd//vsc2Sw1bHr2dGB3gY/gwMFXtFlyoYjuwf9i5pUUcFj7BhK h5eOEqGhoN2W2xiRXWHR+fRKuzOBF2zBXRsa64PlNAszzK2OQrRTcQgqgQ7VZu000sgW /TDqXbFoEXoyZgWkX6IrMhvA1SOUr3OciUkHvUJosKFtczIDOKiBoG9tSUPOrzvf38pQ yFFozAjXCj/Xu7kSnqqs9fBKf5xEflreXS44a7PWyacvEu5HfkmNgJLKWfZ6vk3YmB+a Mi8g== X-Gm-Message-State: AKGB3mKs9KYAX9gdg1rC9WFNvlGFM8+lL3euB9qt3DIU9YHA1baseX+p hgd+gPGJqHiAEMY2XISrSE3HBA== X-Received: by 10.223.154.112 with SMTP id z103mr26711212wrb.210.1514284262010; Tue, 26 Dec 2017 02:31:02 -0800 (PST) Received: from localhost.localdomain ([160.171.216.245]) by smtp.gmail.com with ESMTPSA id l142sm13974036wmb.43.2017.12.26.02.30.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Dec 2017 02:31:01 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v4 08/20] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Tue, 26 Dec 2017 10:29:28 +0000 Message-Id: <20171226102940.26908-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171226102940.26908-1-ard.biesheuvel@linaro.org> References: <20171226102940.26908-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop