From patchwork Thu May 18 04:40:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 683339 Delivered-To: patch@linaro.org Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp181581wrt; Wed, 17 May 2023 21:41:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ64jWLg0dqTtPFcOEjhnyiSzFf1ZdaUrNU6ASOOMn3VKZQumb607MqdP4IyJIeB5m2m+4IZ X-Received: by 2002:ac8:7f15:0:b0:3f5:2582:65d9 with SMTP id f21-20020ac87f15000000b003f5258265d9mr3346687qtk.26.1684384917572; Wed, 17 May 2023 21:41:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684384917; cv=none; d=google.com; s=arc-20160816; b=d0kUuEkbf/FbCBSSUeD5OXnaLX6hjNnXnJDHEIRnCMOFTJa9ClBCVcXsAG0EKuUdRg 2v3wKXpCXJqR+VbnnGfN6vLvBo03+5/iLuffXoot2CRzHV6s2UzMigQr970KwffV/2/g iBo7lk15VIB1/xpxPSFpCTamhdhMCIIv1Y8qJ/gWgF93VubCKhgAOj9o8j16sUEJ/VQS 9S7oG77ulZF5TVYf05Gh0XqTua8tKdr7v5BhsWF8FBvucq9rGGSBMOQ+Oya4LxjMmcC/ dc+o2W5I8OHIi9UPGKMUIxJVjua9tNf99NqWJWNaI1PW8Aunxy5F4Inx73Vy5ECz1uNH 2GvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Hvg0osdHLkOhj0PtXI1wjXd4d+316/OODP4J9ycDye8=; b=oBE36bjIqs6LDDua/HG9vSr3K2FQ09+1b6OPJ+Rr0zBSGZbJVzgmyPvDuBnOwbi8tN phN2lSjHLjbZeiQ7kymdy9uTUDsWLZD+5uTfjMn090gWB5bCyrbxx7OsaGDIawVz5VaB UrZ/i4m+qjfWwxrqYcGjPZN70HerMoShlvecYLCuBKqs66dxb1ofxsNLz6v0AgJfyiak DVVLSYIGTemcU8pCMe8LYI5o0ML4jHpYhmM+BUu0bvO20qOUYmVAPnh6nXN9O/U5D2U2 1BT1MGf4+9CtBomTTwCAoWzWyEuiu7BMqJGfneT1Lff3jT+ftEM9erZ/YfXXp0qgzPqb R2HA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rt+3qUBQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v10-20020a05622a130a00b003d90863314dsi402758qtk.699.2023.05.17.21.41.57 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 May 2023 21:41:57 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=rt+3qUBQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pzVRw-00068R-Vw; Thu, 18 May 2023 00:41:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pzVRu-00067U-RU for qemu-devel@nongnu.org; Thu, 18 May 2023 00:41:10 -0400 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pzVRr-00078K-EV for qemu-devel@nongnu.org; Thu, 18 May 2023 00:41:10 -0400 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-52cb78647ecso1057356a12.1 for ; Wed, 17 May 2023 21:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684384866; x=1686976866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hvg0osdHLkOhj0PtXI1wjXd4d+316/OODP4J9ycDye8=; b=rt+3qUBQZYYYaWkN6wOqCXvXrj8vFXx4cgP9nkOjXBx04byJNDPeXb/b46E+Vqggb2 rdSoPw6geyR/Zl9ZXy10NRd/YszfbWt+CgICwinooUec4lAKUojJXHWL6LbwUCW6QKn5 dO4VYU0ENXZJeanukcAPCqWmkkx5ZxOM6vKkJoJ7ZLu8zPM1jTl5N/Gok1Aqkg4axP9U HCWcA3kUmecVkYgKOWeUR86qnbJRPRmlXmyBWDO9WKUEny3ZLNMYNc8n3B0swFrnsoyq 3Y6j0q95D9d4B8b62Yw+178NYiL4296ti1QFGD0cj9XICEB9y3hYihuOKnz4fJD8SAT/ 8HAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684384866; x=1686976866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hvg0osdHLkOhj0PtXI1wjXd4d+316/OODP4J9ycDye8=; b=Xg00UGIWuNFxDhHp0++CLC9oGc6MjYHlc6clfv8hH/qzO4LUhoGZAXaxuE4g+dkFTN 27evlBza6ouedu+2f+hs84I4VjQAZuXQaESADhlqg9K4j2BjCwqZQOcHLKjGYhJ6xCNB lYO5Kzf0w6226+ehJ9omTRcfrez4h+PhkHwacIPMhNbTWjJMwY78R/PDPCOSdlbKfO+Y 0BfiIKL5a4XRTbQzJjPF2ByVXEAu72ftFVRDDawVfvxd5Nkh3/3UjzrnklCkqADc9vGl mSUfQXDmtbgZJi3HklxD/yFJAYSqs+2m0c3EMqjLkJxBx5HPMx2GAMoTkBAv+E5Endg3 cxqQ== X-Gm-Message-State: AC+VfDw4otP+jHZKB4uAmg6Rn6R6tokSDY42odVVCJiYGjZWeii5slGQ G+w/ZzCNlpnOWVEwJe9i0A/RbvpoUvGzPbEsqhs= X-Received: by 2002:a17:902:b18c:b0:1ae:5c80:5d61 with SMTP id s12-20020a170902b18c00b001ae5c805d61mr950434plr.28.1684384866136; Wed, 17 May 2023 21:41:06 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:17a4:1aa0:c49d:e8f7]) by smtp.gmail.com with ESMTPSA id t9-20020a170902e84900b001ab1d23bf5dsm225592plg.258.2023.05.17.21.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 May 2023 21:41:05 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 6/9] migration/xbzrle: Shuffle function order Date: Wed, 17 May 2023 21:40:55 -0700 Message-Id: <20230518044058.2777467-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230518044058.2777467-1-richard.henderson@linaro.org> References: <20230518044058.2777467-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::529; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x529.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org Place the CONFIG_AVX512BW_OPT block at the top, which will aid function selection in the next patch. Signed-off-by: Richard Henderson Reviewed-by: Juan Quintela --- Cc: Juan Quintela Cc: Peter Xu Cc: Leonardo Bras --- migration/xbzrle.c | 244 ++++++++++++++++++++++----------------------- 1 file changed, 122 insertions(+), 122 deletions(-) diff --git a/migration/xbzrle.c b/migration/xbzrle.c index 258e4959c9..751b5428f7 100644 --- a/migration/xbzrle.c +++ b/migration/xbzrle.c @@ -15,6 +15,128 @@ #include "qemu/host-utils.h" #include "xbzrle.h" +#if defined(CONFIG_AVX512BW_OPT) +#include + +int __attribute__((target("avx512bw"))) +xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen, + uint8_t *dst, int dlen) +{ + uint32_t zrun_len = 0, nzrun_len = 0; + int d = 0, i = 0, num = 0; + uint8_t *nzrun_start = NULL; + /* add 1 to include residual part in main loop */ + uint32_t count512s = (slen >> 6) + 1; + /* countResidual is tail of data, i.e., countResidual = slen % 64 */ + uint32_t count_residual = slen & 0b111111; + bool never_same = true; + uint64_t mask_residual = 1; + mask_residual <<= count_residual; + mask_residual -= 1; + __m512i r = _mm512_set1_epi32(0); + + while (count512s) { + int bytes_to_check = 64; + uint64_t mask = 0xffffffffffffffff; + if (count512s == 1) { + bytes_to_check = count_residual; + mask = mask_residual; + } + __m512i old_data = _mm512_mask_loadu_epi8(r, + mask, old_buf + i); + __m512i new_data = _mm512_mask_loadu_epi8(r, + mask, new_buf + i); + uint64_t comp = _mm512_cmpeq_epi8_mask(old_data, new_data); + count512s--; + + bool is_same = (comp & 0x1); + while (bytes_to_check) { + if (d + 2 > dlen) { + return -1; + } + if (is_same) { + if (nzrun_len) { + d += uleb128_encode_small(dst + d, nzrun_len); + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start = new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d += nzrun_len; + nzrun_len = 0; + } + /* 64 data at a time for speed */ + if (count512s && (comp == 0xffffffffffffffff)) { + i += 64; + zrun_len += 64; + break; + } + never_same = false; + num = ctz64(~comp); + num = (num < bytes_to_check) ? num : bytes_to_check; + zrun_len += num; + bytes_to_check -= num; + comp >>= num; + i += num; + if (bytes_to_check) { + /* still has different data after same data */ + d += uleb128_encode_small(dst + d, zrun_len); + zrun_len = 0; + } else { + break; + } + } + if (never_same || zrun_len) { + /* + * never_same only acts if + * data begins with diff in first count512s + */ + d += uleb128_encode_small(dst + d, zrun_len); + zrun_len = 0; + never_same = false; + } + /* has diff, 64 data at a time for speed */ + if ((bytes_to_check == 64) && (comp == 0x0)) { + i += 64; + nzrun_len += 64; + break; + } + num = ctz64(comp); + num = (num < bytes_to_check) ? num : bytes_to_check; + nzrun_len += num; + bytes_to_check -= num; + comp >>= num; + i += num; + if (bytes_to_check) { + /* mask like 111000 */ + d += uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start = new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d += nzrun_len; + nzrun_len = 0; + is_same = true; + } + } + } + + if (nzrun_len != 0) { + d += uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start = new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d += nzrun_len; + } + return d; +} +#endif + /* page = zrun nzrun | zrun nzrun page @@ -175,125 +297,3 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen) return d; } - -#if defined(CONFIG_AVX512BW_OPT) -#include - -int __attribute__((target("avx512bw"))) -xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen, - uint8_t *dst, int dlen) -{ - uint32_t zrun_len = 0, nzrun_len = 0; - int d = 0, i = 0, num = 0; - uint8_t *nzrun_start = NULL; - /* add 1 to include residual part in main loop */ - uint32_t count512s = (slen >> 6) + 1; - /* countResidual is tail of data, i.e., countResidual = slen % 64 */ - uint32_t count_residual = slen & 0b111111; - bool never_same = true; - uint64_t mask_residual = 1; - mask_residual <<= count_residual; - mask_residual -= 1; - __m512i r = _mm512_set1_epi32(0); - - while (count512s) { - int bytes_to_check = 64; - uint64_t mask = 0xffffffffffffffff; - if (count512s == 1) { - bytes_to_check = count_residual; - mask = mask_residual; - } - __m512i old_data = _mm512_mask_loadu_epi8(r, - mask, old_buf + i); - __m512i new_data = _mm512_mask_loadu_epi8(r, - mask, new_buf + i); - uint64_t comp = _mm512_cmpeq_epi8_mask(old_data, new_data); - count512s--; - - bool is_same = (comp & 0x1); - while (bytes_to_check) { - if (d + 2 > dlen) { - return -1; - } - if (is_same) { - if (nzrun_len) { - d += uleb128_encode_small(dst + d, nzrun_len); - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start = new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d += nzrun_len; - nzrun_len = 0; - } - /* 64 data at a time for speed */ - if (count512s && (comp == 0xffffffffffffffff)) { - i += 64; - zrun_len += 64; - break; - } - never_same = false; - num = ctz64(~comp); - num = (num < bytes_to_check) ? num : bytes_to_check; - zrun_len += num; - bytes_to_check -= num; - comp >>= num; - i += num; - if (bytes_to_check) { - /* still has different data after same data */ - d += uleb128_encode_small(dst + d, zrun_len); - zrun_len = 0; - } else { - break; - } - } - if (never_same || zrun_len) { - /* - * never_same only acts if - * data begins with diff in first count512s - */ - d += uleb128_encode_small(dst + d, zrun_len); - zrun_len = 0; - never_same = false; - } - /* has diff, 64 data at a time for speed */ - if ((bytes_to_check == 64) && (comp == 0x0)) { - i += 64; - nzrun_len += 64; - break; - } - num = ctz64(comp); - num = (num < bytes_to_check) ? num : bytes_to_check; - nzrun_len += num; - bytes_to_check -= num; - comp >>= num; - i += num; - if (bytes_to_check) { - /* mask like 111000 */ - d += uleb128_encode_small(dst + d, nzrun_len); - /* overflow */ - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start = new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d += nzrun_len; - nzrun_len = 0; - is_same = true; - } - } - } - - if (nzrun_len != 0) { - d += uleb128_encode_small(dst + d, nzrun_len); - /* overflow */ - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start = new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d += nzrun_len; - } - return d; -} -#endif