From patchwork Tue Feb 19 22:29:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 158748 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp4213473jaa; Tue, 19 Feb 2019 14:36:43 -0800 (PST) X-Google-Smtp-Source: AHgI3IZMZu960WPbTNfZY7DxPuNQ/8+TYkE9Khfo9YdsCDJX1c/LJP3jiiNCKc5S8mx3+49yONk3 X-Received: by 2002:a81:a342:: with SMTP id a63mr25895461ywh.362.1550615803051; Tue, 19 Feb 2019 14:36:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550615803; cv=none; d=google.com; s=arc-20160816; b=TJ2JLM3xOiqkMwAx88fZk8jwj3leMSHh6WcEKAMM9D5XrpdyiqxQklwiIjxjTYdZeX ZK6ktprNGp0J32nfX97bK+Z91b2IK/DpI2rhd0nlX5sh39Byd6Ofl6O5H+ZyDnKxvAwd /3KiLqAZSKyNLNiWTj4xXWoPHMRe7Zg2+Cay6HWXFWVINFsUGVU7Dev6+dPQgTkyZLZ1 AI9Ko97b8jPIPeEfQr3j5KcMAwFVLRTMZD9p9ygRr1JzLiuUgAZTlR0dd+pEfuTWnAY0 zTC+OFSqKxbUNUYDdLpl7evv0K/EQKWYUYUL3XfyL0fW6rmfBBqDZ4hcp4TASjAqtKSs 4TAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=cIgoi5ZJpcl3zdEbaiu1gS68e0aZGSDVr5lAbw6TYaY=; b=oYUQIucPN6c7JIAKVsYHg2gTjBY6IYcEv95shXnpcpkhAIpJPCfX9aL8cfv813pRA6 BvKp8bvEaJIbueLWoYqZ2PBi3WkFUku4RFXtXpNA4C0Ogd7raQDP3K+WUbSKGjWBWaHv nGYYzKgVNqst3ElcplZ1Jp4+1/pq1mRu9uwTZgttgIzsq4C5K9/c1wmd7HfMpB1CLgSS jedxtpjGhxDwQoBzaR1B9aYHbA+dC55MPyvSNs2Syjn558nxVrBJRSGRoNEAMyFD5tQv tHxX8HsU0tEQXsm0TLBoduH4nnGl21lAUui3XWdlCKy0ZaBFMFcTjQ7u2cHrTFCkjKq9 O9Lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kccKPx75; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id 187si7522907ybe.68.2019.02.19.14.36.42 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 19 Feb 2019 14:36:43 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kccKPx75; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1]:55969 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gwE0I-00049g-JF for patch@linaro.org; Tue, 19 Feb 2019 17:36:42 -0500 Received: from eggs.gnu.org ([209.51.188.92]:45890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gwDtw-0000Kl-A8 for qemu-devel@nongnu.org; Tue, 19 Feb 2019 17:30:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gwDts-0002VJ-OX for qemu-devel@nongnu.org; Tue, 19 Feb 2019 17:30:06 -0500 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:42714) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gwDtq-0002Qf-Ts for qemu-devel@nongnu.org; Tue, 19 Feb 2019 17:30:04 -0500 Received: by mail-pl1-x644.google.com with SMTP id s1so11105198plp.9 for ; Tue, 19 Feb 2019 14:29:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cIgoi5ZJpcl3zdEbaiu1gS68e0aZGSDVr5lAbw6TYaY=; b=kccKPx75yp+bom/DITnf27fcqRp7W+VaVMufEjA8i5djFgftda002uuZf4NLNbQoZQ pyzgiJXvtkdoDC5HxqqGguV9eUIvdlYlqbAeukUVBEZt1SF/A6yyDA9ekq16VSAmmKnt Z5k42+tRave9EQdQ5tzxeNXYtzSNOntkwhyWIoDxbwd9Mf9QgnjFUQvg2J/xqeSjaWZb m15aBdZO98kwQegmVf9P7UPQikzNjc8rBSmVy50UV+3QaGA7os2s8AkKUXj4vVGx5Ka0 jKN5566xiTEo2soB0D/8S5KC5OBgHpP7Jj4OJLvqTDmwWopf6K7sLO5R22r1+27IQrxO dl9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cIgoi5ZJpcl3zdEbaiu1gS68e0aZGSDVr5lAbw6TYaY=; b=HSCmcyJi9BmUhiUjCtzkQNGYwwvNgu0kAfiiWWgR9XD2szwRjRD7+SYezl42OZiMsR gG9WhT2eiwZYTembBMNEStnjzARAxg5h4HMhnSdiuJwQLrFkPlv199un4gUHIG6AAgQG 2aVjzdykIdh7NaddL1mEoOAkhZsHsr7Rcf0xLSPxd+98sUiwnn80NosmS0LWlzFqNdIz FczHbYF9K92u/thrVbrYjF4ALYimhg96OYOWz8sm16ziDIEJjGCFLaCUh80BLRgA9ZRU NCgbKFfEhMUDrAj26f7ITDQJssF9nMUketOKc7t0awQJ2uesvf+kTGCHjtti0csacZEL wWsw== X-Gm-Message-State: AHQUAua1V7pWTQjVYs2h/L845PJoqgdYEiLwzJMNBkJsfK594tvtjeB7 e0phPVZLsKnVX2m/9KhEjJVd5A+c2i0= X-Received: by 2002:a17:902:bc3:: with SMTP id 61mr33496699plr.15.1550615396528; Tue, 19 Feb 2019 14:29:56 -0800 (PST) Received: from cloudburst.twiddle.net (97-113-188-82.tukw.qwest.net. [97.113.188.82]) by smtp.gmail.com with ESMTPSA id 20sm12530778pfs.182.2019.02.19.14.29.55 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 19 Feb 2019 14:29:55 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 19 Feb 2019 14:29:48 -0800 Message-Id: <20190219222952.22183-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190219222952.22183-1-richard.henderson@linaro.org> References: <20190219222952.22183-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH v5 1/5] target/arm: Add helpers for FMLAL X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Note that float16_to_float32 rightly squashes SNaN to QNaN. But of course pickNaNMulAdd, for ARM, selects SNaNs first. So we have to preserve SNaN long enough for the correct NaN to be selected. Thus float16_to_float32_by_bits. Signed-off-by: Richard Henderson --- target/arm/helper.h | 9 +++ target/arm/vec_helper.c | 148 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 157 insertions(+) -- 2.17.2 diff --git a/target/arm/helper.h b/target/arm/helper.h index 747cb64d29..d363904278 100644 --- a/target/arm/helper.h +++ b/target/arm/helper.h @@ -677,6 +677,15 @@ DEF_HELPER_FLAGS_5(gvec_sqsub_s, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(gvec_sqsub_d, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmlal_a32, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmlal_a64, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmlal_idx_a32, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_5(gvec_fmlal_idx_a64, TCG_CALL_NO_RWG, + void, ptr, ptr, ptr, ptr, i32) + #ifdef TARGET_AARCH64 #include "helper-a64.h" #include "helper-sve.h" diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index dfc635cf9a..dedef62403 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -898,3 +898,151 @@ void HELPER(gvec_sqsub_d)(void *vd, void *vq, void *vn, } clear_tail(d, oprsz, simd_maxsz(desc)); } + +/* + * Convert float16 to float32, raising no exceptions and + * preserving exceptional values, including SNaN. + * This is effectively an unpack+repack operation. + */ +static float32 float16_to_float32_by_bits(uint32_t f16, bool fz16) +{ + const int f16_bias = 15; + const int f32_bias = 127; + uint32_t sign = extract32(f16, 15, 1); + uint32_t exp = extract32(f16, 10, 5); + uint32_t frac = extract32(f16, 0, 10); + + if (exp == 0x1f) { + /* Inf or NaN */ + exp = 0xff; + } else if (exp == 0) { + /* Zero or denormal. */ + if (frac != 0) { + if (fz16) { + frac = 0; + } else { + /* + * Denormal; these are all normal float32. + * Shift the fraction so that the msb is at bit 11, + * then remove bit 11 as the implicit bit of the + * normalized float32. Note that we still go through + * the shift for normal numbers below, to put the + * float32 fraction at the right place. + */ + int shift = clz32(frac) - 21; + frac = (frac << shift) & 0x3ff; + exp = f32_bias - f16_bias - shift + 1; + } + } + } else { + /* Normal number; adjust the bias. */ + exp += f32_bias - f16_bias; + } + sign <<= 31; + exp <<= 23; + frac <<= 23 - 10; + + return sign | exp | frac; +} + +static uint64_t load4_f16(uint64_t *ptr, int is_q, int is_2) +{ + /* + * Branchless load of u32[0], u64[0], u32[1], or u64[1]. + * Load the 2nd qword iff is_q & is_2. + * Shift to the 2nd dword iff !is_q & is_2. + * For !is_q & !is_2, the upper bits of the result are garbage. + */ + return ptr[is_q & is_2] >> ((is_2 & ~is_q) << 5); +} + +/* + * Note that FMLAL requires oprsz == 8 or oprsz == 16, + * as there is not yet SVE versions that might use blocking. + */ + +static void do_fmlal(float32 *d, void *vn, void *vm, float_status *fpst, + uint32_t desc, bool fz16) +{ + intptr_t i, oprsz = simd_oprsz(desc); + int is_s = extract32(desc, SIMD_DATA_SHIFT, 1); + int is_2 = extract32(desc, SIMD_DATA_SHIFT + 1, 1); + int is_q = oprsz == 16; + uint64_t n_4, m_4; + + /* Pre-load all of the f16 data, avoiding overlap issues. */ + n_4 = load4_f16(vn, is_q, is_2); + m_4 = load4_f16(vm, is_q, is_2); + + /* Negate all inputs for FMLSL at once. */ + if (is_s) { + n_4 ^= 0x8000800080008000ull; + } + + for (i = 0; i < oprsz / 4; i++) { + float32 n_1 = float16_to_float32_by_bits(n_4 >> (i * 16), fz16); + float32 m_1 = float16_to_float32_by_bits(m_4 >> (i * 16), fz16); + d[H4(i)] = float32_muladd(n_1, m_1, d[H4(i)], 0, fpst); + } + clear_tail(d, oprsz, simd_maxsz(desc)); +} + +void HELPER(gvec_fmlal_a32)(void *vd, void *vn, void *vm, + void *venv, uint32_t desc) +{ + CPUARMState *env = venv; + do_fmlal(vd, vn, vm, &env->vfp.standard_fp_status, desc, + get_flush_inputs_to_zero(&env->vfp.fp_status_f16)); +} + +void HELPER(gvec_fmlal_a64)(void *vd, void *vn, void *vm, + void *venv, uint32_t desc) +{ + CPUARMState *env = venv; + do_fmlal(vd, vn, vm, &env->vfp.fp_status, desc, + get_flush_inputs_to_zero(&env->vfp.fp_status_f16)); +} + +static void do_fmlal_idx(float32 *d, void *vn, void *vm, float_status *fpst, + uint32_t desc, bool fz16) +{ + intptr_t i, oprsz = simd_oprsz(desc); + int is_s = extract32(desc, SIMD_DATA_SHIFT, 1); + int is_2 = extract32(desc, SIMD_DATA_SHIFT + 1, 1); + int index = extract32(desc, SIMD_DATA_SHIFT + 2, 3); + int is_q = oprsz == 16; + uint64_t n_4; + float32 m_1; + + /* Pre-load all of the f16 data, avoiding overlap issues. */ + n_4 = load4_f16(vn, is_q, is_2); + + /* Negate all inputs for FMLSL at once. */ + if (is_s) { + n_4 ^= 0x8000800080008000ull; + } + + m_1 = float16_to_float32_by_bits(((float16 *)vm)[H2(index)], fz16); + + for (i = 0; i < oprsz / 4; i++) { + float32 n_1 = float16_to_float32_by_bits(n_4 >> (i * 16), fz16); + d[H4(i)] = float32_muladd(n_1, m_1, d[H4(i)], 0, fpst); + } + clear_tail(d, oprsz, simd_maxsz(desc)); +} + +void HELPER(gvec_fmlal_idx_a32)(void *vd, void *vn, void *vm, + void *venv, uint32_t desc) +{ + CPUARMState *env = venv; + do_fmlal_idx(vd, vn, vm, &env->vfp.standard_fp_status, desc, + get_flush_inputs_to_zero(&env->vfp.fp_status_f16)); +} + +void HELPER(gvec_fmlal_idx_a64)(void *vd, void *vn, void *vm, + void *venv, uint32_t desc) +{ + CPUARMState *env = venv; + do_fmlal_idx(vd, vn, vm, &env->vfp.fp_status, desc, + get_flush_inputs_to_zero(&env->vfp.fp_status_f16)); +}