From patchwork Thu Sep 12 19:59:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 173726 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:ce:0:0:0:0 with SMTP id r14csp2599429ilq; Thu, 12 Sep 2019 13:00:37 -0700 (PDT) X-Google-Smtp-Source: APXvYqyAXfQ77Wm4urgFmi+Xz1PD4xQr8iJLnvFEdGXwesqjbNFIB7Nok82h0LjmNQRDN2zVQdJ4 X-Received: by 2002:a50:ad2c:: with SMTP id y41mr43586905edc.114.1568318437251; Thu, 12 Sep 2019 13:00:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1568318437; cv=none; d=google.com; s=arc-20160816; b=Tvviqc5u0TvB/rSXL+yqk7NgSmQc9zTmQdRWpj6YOlZXVEJ98lJRU/CJ9PT4ZM6MIx feEPI4gQTtlRPCIma4lg3RAMWC2DFXwJfvtLn2TLHd2B5F5gsOJxUVTPJDktklUIqqdl jo7+Gqr642ibFBlwaio2dlh/gIGilbGeTgNY+gPQIS9AcV9U2lD5U2Uvrqjtn6uZFKfV S4cKJrSOObl2Gm10qG3WCDDvNuBhNvaglDxrnzmxhepp1QalQl8pu0xkquSMf/4KqQb7 rtZ/7tcQQ4REuhG47jBrIZSNluKZfitSgYv7PyZFeccHvBcT02dJ/mNFJuYuCE5p8rDE zX5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=GVLFTA60hlmj+H2/7YDDPt5FNC57y3vvDPP1cxQ2fC0=; b=LCeQOc535haIZY8LW92h5TrHdPma5qWNcQzRox6YbdLaJeNG2MY0FZUyqfAbJEQsX8 /en0mByqyIohjJzE13y7ZAXAhqaP+QifZ6zkSiyIy5VjQA41SfAb2QR6jhiN6ACSQpqw I9qT89HaGTKIZLKGlHmetZl2W7VlY8nC+u0ttWxviOPYpjeFKOnjiB5GVHRcI3hAQUcu s7YUTQXrKcNDXvA8cdAfIo9r5Fx9yuul8EnJfSteqwVXRGnWe3maiJ+I7n5SR0unFrrE J3ICeR3MMqO1WIwKI/YZUH4jTWs7CYN6KukFIJzub40OngSdGNAnvH0p9QzBB36KE4py 0GVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UAjkkLxR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z55si6102749edb.41.2019.09.12.13.00.37 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 12 Sep 2019 13:00:37 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UAjkkLxR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:38548 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VGe-00016a-8s for patch@linaro.org; Thu, 12 Sep 2019 16:00:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46116) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VFk-00011R-NJ for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i8VFj-0005Is-O2 for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:40 -0400 Received: from mail-qk1-x733.google.com ([2607:f8b0:4864:20::733]:32991) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i8VFj-0005Id-Jx for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:39 -0400 Received: by mail-qk1-x733.google.com with SMTP id x134so25784694qkb.0 for ; Thu, 12 Sep 2019 12:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GVLFTA60hlmj+H2/7YDDPt5FNC57y3vvDPP1cxQ2fC0=; b=UAjkkLxRH9fiRL8QLbkLEsnY+SDmrDdWDNJx3wJcdb2Ga9eQl9yuayfALMFNe2gjMu ue5qJaiR/wDLfrLM4I0WDE9U4eLCzg2fK57XdaKI5PX15NikPbbtXXrKZGdBJ57vdxjQ 3+fIPmfxNRGSIbwHn/su8ygSIf4n0JSwf/n5O302Da0ZMSZs+XZLC9chUH8b11bHRjPQ YNP9CRJZ1K/UKOaagjoP4B0IS2jhzYYA1MRt083ywk2/loHZcTOkG/erLGQufpAmnydU q5eWVhD7lGAtbT+g8VbeAUt+GtJYv1ky1WBXVQy29DpMuDtkoF6/1swN1tZL3Ch8G/LD G+vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GVLFTA60hlmj+H2/7YDDPt5FNC57y3vvDPP1cxQ2fC0=; b=R+6dJ85/TXKyYWsX1aEK3pvBBLJG+oYKqiy0oi3LwerSvop4upcKlCvQkN18EJP5A/ /Vz0kZZWjhPAvAtwpfV55QIYcfxBwT7V23Xp714CFx8YeY8uiVJ1HBNsLpvs6DhaSRK3 SDPyI9XKIV0oVN+yGIxUyWVwlwduAs6Ixt64IReK3ChLi2PI0R3BUH+6dFmi3dt3mTm3 zWrUlfipITQ99TOUlUbviGmRgj4jIubFywUUwM4olTnl6GcmNp9GzSXQ1AOFz/Jo9QQe ll+YfkOzfcxmRCnTOAccswlqKpUI3HlXwp3YqQ+rOgXl/zcQizY52CY7NzBgWBeljccK cYSw== X-Gm-Message-State: APjAAAUHDyqKLRa/33hHdF8rDAwEZa0Ms5KW3GIQwwhAxikXjcEwCLkl KdFiJ2D/aKy0ZMLouYGkiynM9KL20WyyWw== X-Received: by 2002:a37:a48b:: with SMTP id n133mr37221462qke.398.1568318378422; Thu, 12 Sep 2019 12:59:38 -0700 (PDT) Received: from localhost.localdomain ([67.69.50.154]) by smtp.gmail.com with ESMTPSA id k11sm11140897qtp.26.2019.09.12.12.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2019 12:59:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Thu, 12 Sep 2019 15:59:32 -0400 Message-Id: <20190912195934.13502-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190912195934.13502-1-richard.henderson@linaro.org> References: <20190912195934.13502-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::733 Subject: [Qemu-devel] [PATCH v2 1/3] cputlb: Disable __always_inline__ without optimization X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tony.nguyen@bt.com, mark.cave-ayland@ilande.co.uk Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This forced inlining can result in missing symbols, which makes a debugging build harder to follow. Reported-by: Peter Maydell Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/qemu/compiler.h | 11 +++++++++++ accel/tcg/cputlb.c | 4 ++-- 2 files changed, 13 insertions(+), 2 deletions(-) -- 2.17.1 diff --git a/include/qemu/compiler.h b/include/qemu/compiler.h index 09fc44cca4..d6d400c523 100644 --- a/include/qemu/compiler.h +++ b/include/qemu/compiler.h @@ -170,6 +170,17 @@ # define QEMU_NONSTRING #endif +/* + * Forced inlining may be desired to encourage constant propagation + * of function parameters. However, it can also make debugging harder, + * so disable it for a non-optimizing build. + */ +#if defined(__OPTIMIZE__) && __has_attribute(always_inline) +#define QEMU_ALWAYS_INLINE __attribute__((always_inline)) +#else +#define QEMU_ALWAYS_INLINE +#endif + /* Implement C11 _Generic via GCC builtins. Example: * * QEMU_GENERIC(x, (float, sinf), (long double, sinl), sin) (x) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index abae79650c..2222b87764 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1281,7 +1281,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr, typedef uint64_t FullLoadHelper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr); -static inline uint64_t __attribute__((always_inline)) +static inline uint64_t QEMU_ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, FullLoadHelper *full_load) @@ -1530,7 +1530,7 @@ tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *env, target_ulong addr, * Store Helpers */ -static inline void __attribute__((always_inline)) +static inline void QEMU_ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op) { From patchwork Thu Sep 12 19:59:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 173727 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:ce:0:0:0:0 with SMTP id r14csp2602557ilq; Thu, 12 Sep 2019 13:03:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqzLvMBrveL8mNY5mvZTXWGhqKxZ1V+ZNI3TLAkfNOjxe0woZU7NoyUIhg+ie6Ncrxn64tfi X-Received: by 2002:aa7:c24b:: with SMTP id y11mr43844240edo.214.1568318588553; Thu, 12 Sep 2019 13:03:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1568318588; cv=none; d=google.com; s=arc-20160816; b=J+ICKOuYF8cVY74WQHMipNcBt/XBf0cXAaFSW8iR5NM74HzZY/Ns2wLpgMGUNSqiLh uRLhZ310Kr/iI1OsGVcn5UJKEU7XQAINja34FFYwd6lD0n4ZhT+vQR8Bx7u4dyjibNsF pcYyMn9Iut+o/7XQWiUW6NOPOyHyiqOszA3hdzfweRAB+w2VG8eId2v2X6Jepiitef5e zte92/ESMgwlassuqw3WSRIqEkliYRUQRcS46ssQHriS4ywarhwzvAzAPdgHWFGHJEEP XrH8cz5jU0U4oRvOTbtYgob2g19pS5MgLnMMvPQ+XXi5phEVSU7vGcsC3gJAYqXe6/jR j1Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=3cfrqeC7LyJB/ROXN1SmwKtiOawGwvTqMSrbdq/eHoM=; b=y5MQrhAJIYZj62ko/YKJ7hc01CGkFgBWllS7zSBwjDMSfNtFS+QVB+j/0EksUlPt3P +g2N/nPtRRjsJuPP0Gg5w7lmCV5ksxf3lTslgr6/XGH7s1FJRSMJivacr1dmwZQS2SHi 5XtIoiHdwKRKU1QQtUA/g9nUrWs/iHyFmGG7kXuKGzoiDM0kySjVY7B0DMi1P36TAsp/ sIiRfCIPrr4+qigcfjkNQQ4bFZpbHYjT1zwo/uz+2EtY+XMkU2lqtxePDBqA8hM4fG0x 56xyo94GGjvrvOiyRz69fK2PtgIrU8jhI4LbXAfEHaXekz0/LbY3FrAT98GpxiMLP07y Lr6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=xPOEHmqU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id r9si5589352eju.120.2019.09.12.13.03.08 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 12 Sep 2019 13:03:08 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=xPOEHmqU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:38654 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VJ5-0003wo-BY for patch@linaro.org; Thu, 12 Sep 2019 16:03:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46134) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VFn-00014n-64 for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i8VFl-0005JS-BR for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:43 -0400 Received: from mail-qt1-x843.google.com ([2607:f8b0:4864:20::843]:34287) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i8VFl-0005JJ-6G for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:41 -0400 Received: by mail-qt1-x843.google.com with SMTP id j1so18385629qth.1 for ; Thu, 12 Sep 2019 12:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3cfrqeC7LyJB/ROXN1SmwKtiOawGwvTqMSrbdq/eHoM=; b=xPOEHmqUwNDoV4uZBnI4rf2lRL/w4aaG/izxiSgujwOLkTQXVJXtbl5h0i4bjyj/pq jy92I9tdGZLk5aFvGeTRFxGJKnbkVoiM4g8dn3815rrjpMTE2GXFVed06yjejGtPoSp4 Ylh/d/Kk+cFHRumzs/+CYZqlmX5hwXcbq8nGLWHi7zDfXjScdq6NIK4jGLR1H9o84QRf KUxmpjYdTR64eEqOJNs+G/7lBqQZLQtAqUz6ebGdsvPobxb+O9LCqChQ28zl+k9tQ6ll vf16pXzrVg1lhYZlo5shWorGs3yy3sinhlnu+hHtK7PNh/wtdZLERtR2BCRT59IhaERY fubA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3cfrqeC7LyJB/ROXN1SmwKtiOawGwvTqMSrbdq/eHoM=; b=fAuhllzMOYgSMi3Sf3GyOoqLjC5KJba9HfNmSwUaPkta18nCtA3qyL787GbU3kAAIG ckAA5XOC7lIBIx3w8txMBOrJxVKoefrBJH+S1Ud6nX7tsYTLL2etZGwn/SQP8W2fAD4b gdPsI3dEG2r+9e/5wkVdQaBEnpZDn9VKD5h8pdE4QtnQx4oQCga0PmoDwcUFEX1bCGLA jfyuelBAcp1TQ8/P3YtN1RjI7pDqsjEaTeyBBH0FnRiV6XwCRz2WYlPwGlgldAXOmCEq CDslQ8gXPgolGuSEUUfzOXP+ApT3LQlbIiGb8S6wjIw3gFvnPcENQczI9U5SSGU1KdNy 0Mfg== X-Gm-Message-State: APjAAAVbTC8N2bZeO2FJ4TNmYoZucdTvnyq0TZ7usAdyXcwcg3szcVQh C43rlopprbuTimvMs0Q1x7+YD00oWZ/lXQ== X-Received: by 2002:ac8:6b12:: with SMTP id w18mr42599227qts.209.1568318379993; Thu, 12 Sep 2019 12:59:39 -0700 (PDT) Received: from localhost.localdomain ([67.69.50.154]) by smtp.gmail.com with ESMTPSA id k11sm11140897qtp.26.2019.09.12.12.59.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2019 12:59:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Thu, 12 Sep 2019 15:59:33 -0400 Message-Id: <20190912195934.13502-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190912195934.13502-1-richard.henderson@linaro.org> References: <20190912195934.13502-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::843 Subject: [Qemu-devel] [PATCH v2 2/3] cputlb: Replace switches in load/store_helper with callback X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tony.nguyen@bt.com, mark.cave-ayland@ilande.co.uk Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Add a function parameter to perform the actual load/store to ram. With optimization, this results in identical code. Signed-off-by: Richard Henderson --- accel/tcg/cputlb.c | 159 +++++++++++++++++++++++---------------------- 1 file changed, 83 insertions(+), 76 deletions(-) -- 2.17.1 diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 2222b87764..b4a63d3928 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1280,11 +1280,38 @@ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr, typedef uint64_t FullLoadHelper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr); +typedef uint64_t LoadHelper(const void *); + +/* Wrap the unaligned load helpers to that they have a common signature. */ +static inline uint64_t wrap_ldub(const void *haddr) +{ + return ldub_p(haddr); +} + +static inline uint64_t wrap_lduw_be(const void *haddr) +{ + return lduw_be_p(haddr); +} + +static inline uint64_t wrap_lduw_le(const void *haddr) +{ + return lduw_le_p(haddr); +} + +static inline uint64_t wrap_ldul_be(const void *haddr) +{ + return (uint32_t)ldl_be_p(haddr); +} + +static inline uint64_t wrap_ldul_le(const void *haddr) +{ + return (uint32_t)ldl_le_p(haddr); +} static inline uint64_t QEMU_ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, - FullLoadHelper *full_load) + FullLoadHelper *full_load, LoadHelper *direct) { uintptr_t mmu_idx = get_mmuidx(oi); uintptr_t index = tlb_index(env, mmu_idx, addr); @@ -1373,33 +1400,7 @@ load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, do_aligned_access: haddr = (void *)((uintptr_t)addr + entry->addend); - switch (op) { - case MO_UB: - res = ldub_p(haddr); - break; - case MO_BEUW: - res = lduw_be_p(haddr); - break; - case MO_LEUW: - res = lduw_le_p(haddr); - break; - case MO_BEUL: - res = (uint32_t)ldl_be_p(haddr); - break; - case MO_LEUL: - res = (uint32_t)ldl_le_p(haddr); - break; - case MO_BEQ: - res = ldq_be_p(haddr); - break; - case MO_LEQ: - res = ldq_le_p(haddr); - break; - default: - g_assert_not_reached(); - } - - return res; + return direct(haddr); } /* @@ -1415,7 +1416,8 @@ load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, static uint64_t full_ldub_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - return load_helper(env, addr, oi, retaddr, MO_UB, false, full_ldub_mmu); + return load_helper(env, addr, oi, retaddr, MO_UB, false, + full_ldub_mmu, wrap_ldub); } tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, @@ -1428,7 +1430,7 @@ static uint64_t full_le_lduw_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, false, - full_le_lduw_mmu); + full_le_lduw_mmu, wrap_lduw_le); } tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1441,7 +1443,7 @@ static uint64_t full_be_lduw_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, false, - full_be_lduw_mmu); + full_be_lduw_mmu, wrap_lduw_be); } tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1454,7 +1456,7 @@ static uint64_t full_le_ldul_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, false, - full_le_ldul_mmu); + full_le_ldul_mmu, wrap_ldul_le); } tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1467,7 +1469,7 @@ static uint64_t full_be_ldul_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, false, - full_be_ldul_mmu); + full_be_ldul_mmu, wrap_ldul_be); } tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1480,14 +1482,14 @@ uint64_t helper_le_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, false, - helper_le_ldq_mmu); + helper_le_ldq_mmu, ldq_le_p); } uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, false, - helper_be_ldq_mmu); + helper_be_ldq_mmu, ldq_be_p); } /* @@ -1530,9 +1532,38 @@ tcg_target_ulong helper_be_ldsl_mmu(CPUArchState *env, target_ulong addr, * Store Helpers */ +typedef void StoreHelper(void *, uint64_t); + +/* Wrap the unaligned store helpers to that they have a common signature. */ +static inline void wrap_stb(void *haddr, uint64_t val) +{ + stb_p(haddr, val); +} + +static inline void wrap_stw_be(void *haddr, uint64_t val) +{ + stw_be_p(haddr, val); +} + +static inline void wrap_stw_le(void *haddr, uint64_t val) +{ + stw_le_p(haddr, val); +} + +static inline void wrap_stl_be(void *haddr, uint64_t val) +{ + stl_be_p(haddr, val); +} + +static inline void wrap_stl_le(void *haddr, uint64_t val) +{ + stl_le_p(haddr, val); +} + static inline void QEMU_ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, - TCGMemOpIdx oi, uintptr_t retaddr, MemOp op) + TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, + StoreHelper *direct) { uintptr_t mmu_idx = get_mmuidx(oi); uintptr_t index = tlb_index(env, mmu_idx, addr); @@ -1657,74 +1688,49 @@ store_helper(CPUArchState *env, target_ulong addr, uint64_t val, do_aligned_access: haddr = (void *)((uintptr_t)addr + entry->addend); - switch (op) { - case MO_UB: - stb_p(haddr, val); - break; - case MO_BEUW: - stw_be_p(haddr, val); - break; - case MO_LEUW: - stw_le_p(haddr, val); - break; - case MO_BEUL: - stl_be_p(haddr, val); - break; - case MO_LEUL: - stl_le_p(haddr, val); - break; - case MO_BEQ: - stq_be_p(haddr, val); - break; - case MO_LEQ: - stq_le_p(haddr, val); - break; - default: - g_assert_not_reached(); - break; - } + direct(haddr, val); } void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_UB); + store_helper(env, addr, val, oi, retaddr, MO_UB, wrap_stb); } void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUW); + store_helper(env, addr, val, oi, retaddr, MO_LEUW, wrap_stw_le); } void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUW); + store_helper(env, addr, val, oi, retaddr, MO_BEUW, wrap_stw_be); } void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUL); + store_helper(env, addr, val, oi, retaddr, MO_LEUL, wrap_stl_le); } void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUL); + store_helper(env, addr, val, oi, retaddr, MO_BEUL, wrap_stl_be); } void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEQ); + store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p); } void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEQ); + store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p); } /* First set of helpers allows passing in of OI and RETADDR. This makes @@ -1789,7 +1795,8 @@ void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, static uint64_t full_ldub_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - return load_helper(env, addr, oi, retaddr, MO_8, true, full_ldub_cmmu); + return load_helper(env, addr, oi, retaddr, MO_8, true, + full_ldub_cmmu, wrap_ldub); } uint8_t helper_ret_ldb_cmmu(CPUArchState *env, target_ulong addr, @@ -1802,7 +1809,7 @@ static uint64_t full_le_lduw_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, true, - full_le_lduw_cmmu); + full_le_lduw_cmmu, wrap_lduw_le); } uint16_t helper_le_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1815,7 +1822,7 @@ static uint64_t full_be_lduw_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, true, - full_be_lduw_cmmu); + full_be_lduw_cmmu, wrap_lduw_be); } uint16_t helper_be_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1828,7 +1835,7 @@ static uint64_t full_le_ldul_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, true, - full_le_ldul_cmmu); + full_le_ldul_cmmu, wrap_ldul_le); } uint32_t helper_le_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1841,7 +1848,7 @@ static uint64_t full_be_ldul_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, true, - full_be_ldul_cmmu); + full_be_ldul_cmmu, wrap_ldul_be); } uint32_t helper_be_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1854,12 +1861,12 @@ uint64_t helper_le_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, true, - helper_le_ldq_cmmu); + helper_le_ldq_cmmu, ldq_le_p); } uint64_t helper_be_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, true, - helper_be_ldq_cmmu); + helper_be_ldq_cmmu, ldq_be_p); } From patchwork Thu Sep 12 19:59:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 173728 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:ce:0:0:0:0 with SMTP id r14csp2603414ilq; Thu, 12 Sep 2019 13:03:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqyJynN3KtKki1WTs0B7bXTbbFbrcc+Y4BpBEil9bk80+XBA54hepMbUEUzLjvtaLB2g9RC1 X-Received: by 2002:a17:906:3583:: with SMTP id o3mr13822311ejb.224.1568318630169; Thu, 12 Sep 2019 13:03:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1568318630; cv=none; d=google.com; s=arc-20160816; b=Ut3bzlzSxchfgXUvkukcp8W6rB2YJpfqncd/ZtAdx3IajFKZOxn3gNkNbSJlP2xz00 Jv6/wConMCKp7v+bzSJa4aml1OeIO1CBvkvorIxbiQzmx9dCuKCMX7Ze4TSjjf+oDf1j OSUqi6dcFzcoh5/yAFJ3b8A7gdzfG6u/7QHY2lLCi58RtUA142tWtuB2Jp8hqsFPPnN4 qN4iYAHZaMWC+cU6KZoMvU6W1V4W8VWpregIjVGM6dQqkkbdPV1N9xtMGp9PZeaypK/z DAu8NgHlNhBxeMxmFpty/OOVy16j7M3UkKcN0lGr2o8FRGyDrg6xIC5U6EADDa7dSdsB utGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=MeiDoldNRDTDe/ONw2aHiXEaBsl1OBQ2+CgBlsXUesg=; b=Uik/gqn8W5XthtU0kJfk8Kumh8/37MoXaYTVppZhot4CusMpW1KF0nyxd7Y4wObmqX bE3KrnWBigB5f3F9sU3ik2OQbbOB7TE7ut16X69mzD6z5hUdtAO1fGF/ZuX9ZX6zMTB8 v3w/36R4hdrKyNzbzd7XsDsUQYbqEGEQhVUXcHSXcwvstieooXt0hjcNUzf9pbJBYmzt y/LiZjiwEzzx4D/s26Wac/fUWEkzFiRz7BtI54jsD1Pv4TSpmYp9lSrBnnEJjSdCDKS7 LNZWBew676m/g6dXp5yK30OHmg6famLzXxzc87MHbKZmu9OJOlmdVPbBfNY0IYO0m+cl gfUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WmUglH4g; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z7si16452343edb.160.2019.09.12.13.03.49 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 12 Sep 2019 13:03:50 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WmUglH4g; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:38642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VJk-0003tq-H5 for patch@linaro.org; Thu, 12 Sep 2019 16:03:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46142) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i8VFo-00016t-EI for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i8VFm-0005Jj-J6 for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:44 -0400 Received: from mail-qk1-x743.google.com ([2607:f8b0:4864:20::743]:40667) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i8VFm-0005Je-DY for qemu-devel@nongnu.org; Thu, 12 Sep 2019 15:59:42 -0400 Received: by mail-qk1-x743.google.com with SMTP id y144so17722774qkb.7 for ; Thu, 12 Sep 2019 12:59:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MeiDoldNRDTDe/ONw2aHiXEaBsl1OBQ2+CgBlsXUesg=; b=WmUglH4gA5ukgBWxwQInZr+o45jdG7H2sk6Z4r3bCIqplTFcj+FTLOKWGpmHfJ+xgS DmCsBqG+jovfaxGqveYGx8OUFgJOn6H67UTttBahJwQF/14dt0qVm3a7943p+MpVRE02 9zXDMexrBF67xARaZkBxkM5xRlcHtWeG6+T7hlkS97PIZ18QScElbn+EbU8na7p7PjY8 5dmcavVNncJzNmCOFOjBxWErW2ofgqo+1GlJVJPy6Ah9mos1PfKgxmlQ/yVeDZCNPX8u 9t0gI39I8qo9yIKkzIdDlmr4rLTVErylp0K++vcOTVrnPhqFNDekf1QktojuZ58ye2rf ITyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MeiDoldNRDTDe/ONw2aHiXEaBsl1OBQ2+CgBlsXUesg=; b=dYVNFxEiP1ZF1J/HPQF8VtChJ8FT6w5L3e8FZM0u65moBzauU61/HKlhoKyD5Rv6to HCIt56bGiQlq+dsnzGdHMMUNhxj3Ntn37DRBU+eIX2+0NZTQyW7Bt2dC3rQg1SDU07MM uLTvqGQ8jkfUzrjFdBuOPzEgkylGUHVjJ2z/st02XxjFQhePcd6TqrggVGDOfogo7un5 0zBa8x8B97jlel0KptsCPYMtpLqmS0kAgBojZ1Eab9oOfKVc2tKd44bQ2LbG98Bbswje 86wGA1asmvRBfaQ9JAVqNzyoEHX+NKyLskCbeaj50xQ9+Jb9LqYmOyxndygpyAKfEP/N n/EQ== X-Gm-Message-State: APjAAAXTVZxiL3H6sGPBcjpJRoOgqvqV9XVbE8rp8ACwkiyc9PBIrCW+ Ay+MyoyHTBQdyQK2+dUSVoOUvuYrFLpokQ== X-Received: by 2002:a05:620a:15c5:: with SMTP id o5mr41404009qkm.16.1568318381261; Thu, 12 Sep 2019 12:59:41 -0700 (PDT) Received: from localhost.localdomain ([67.69.50.154]) by smtp.gmail.com with ESMTPSA id k11sm11140897qtp.26.2019.09.12.12.59.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2019 12:59:40 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Thu, 12 Sep 2019 15:59:34 -0400 Message-Id: <20190912195934.13502-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190912195934.13502-1-richard.henderson@linaro.org> References: <20190912195934.13502-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::743 Subject: [Qemu-devel] [PATCH v2 3/3] cputlb: Introduce TLB_BSWAP X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tony.nguyen@bt.com, mark.cave-ayland@ilande.co.uk Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Handle bswap on ram directly in load/store_helper. This fixes a bug with the previous implementation in that one cannot use the I/O path for RAM. Fixes: a26fc6f5152b47f1 Signed-off-by: Richard Henderson --- include/exec/cpu-all.h | 2 + accel/tcg/cputlb.c | 118 ++++++++++++++++++++--------------------- 2 files changed, 59 insertions(+), 61 deletions(-) -- 2.17.1 diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index d2d443c4f9..3928edab9a 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -331,6 +331,8 @@ CPUArchState *cpu_copy(CPUArchState *env); #define TLB_MMIO (1 << (TARGET_PAGE_BITS - 3)) /* Set if TLB entry contains a watchpoint. */ #define TLB_WATCHPOINT (1 << (TARGET_PAGE_BITS - 4)) +/* Set if TLB entry requires byte swap. */ +#define TLB_BSWAP (1 << (TARGET_PAGE_BITS - 5)) /* Use this mask to check interception with an alignment mask * in a TCG backend. diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index b4a63d3928..354a75927a 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -737,8 +737,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong vaddr, address |= TLB_INVALID_MASK; } if (attrs.byte_swap) { - /* Force the access through the I/O slow path. */ - address |= TLB_MMIO; + address |= TLB_BSWAP; } if (!memory_region_is_ram(section->mr) && !memory_region_is_romd(section->mr)) { @@ -901,10 +900,6 @@ static uint64_t io_readx(CPUArchState *env, CPUIOTLBEntry *iotlbentry, bool locked = false; MemTxResult r; - if (iotlbentry->attrs.byte_swap) { - op ^= MO_BSWAP; - } - section = iotlb_to_section(cpu, iotlbentry->addr, iotlbentry->attrs); mr = section->mr; mr_offset = (iotlbentry->addr & TARGET_PAGE_MASK) + addr; @@ -947,10 +942,6 @@ static void io_writex(CPUArchState *env, CPUIOTLBEntry *iotlbentry, bool locked = false; MemTxResult r; - if (iotlbentry->attrs.byte_swap) { - op ^= MO_BSWAP; - } - section = iotlb_to_section(cpu, iotlbentry->addr, iotlbentry->attrs); mr = section->mr; mr_offset = (iotlbentry->addr & TARGET_PAGE_MASK) + addr; @@ -1311,7 +1302,8 @@ static inline uint64_t wrap_ldul_le(const void *haddr) static inline uint64_t QEMU_ALWAYS_INLINE load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, bool code_read, - FullLoadHelper *full_load, LoadHelper *direct) + FullLoadHelper *full_load, LoadHelper *direct, + LoadHelper *direct_swap) { uintptr_t mmu_idx = get_mmuidx(oi); uintptr_t index = tlb_index(env, mmu_idx, addr); @@ -1361,26 +1353,27 @@ load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, /* On watchpoint hit, this will longjmp out. */ cpu_check_watchpoint(env_cpu(env), addr, size, iotlbentry->attrs, BP_MEM_READ, retaddr); - - /* The backing page may or may not require I/O. */ - tlb_addr &= ~TLB_WATCHPOINT; - if ((tlb_addr & ~TARGET_PAGE_MASK) == 0) { - goto do_aligned_access; - } } /* Handle I/O access. */ - return io_readx(env, iotlbentry, mmu_idx, addr, - retaddr, access_type, op); - } + if (likely(tlb_addr & TLB_MMIO)) { + return io_readx(env, iotlbentry, mmu_idx, addr, + retaddr, access_type, + op ^ (tlb_addr & TLB_BSWAP ? MO_BSWAP : 0)); + } - /* Handle slow unaligned access (it spans two pages or IO). */ - if (size > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 - >= TARGET_PAGE_SIZE)) { + if (unlikely(tlb_addr & TLB_BSWAP)) { + haddr = (void *)((uintptr_t)addr + entry->addend); + return direct_swap(haddr); + } + } else if (size > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 + >= TARGET_PAGE_SIZE)) { + /* Handle slow unaligned access (it spans two pages or IO). */ target_ulong addr1, addr2; uint64_t r1, r2; unsigned shift; + do_unaligned_access: addr1 = addr & ~((target_ulong)size - 1); addr2 = addr1 + size; @@ -1398,7 +1391,6 @@ load_helper(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, return res & MAKE_64BIT_MASK(0, size * 8); } - do_aligned_access: haddr = (void *)((uintptr_t)addr + entry->addend); return direct(haddr); } @@ -1417,7 +1409,7 @@ static uint64_t full_ldub_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_UB, false, - full_ldub_mmu, wrap_ldub); + full_ldub_mmu, wrap_ldub, wrap_ldub); } tcg_target_ulong helper_ret_ldub_mmu(CPUArchState *env, target_ulong addr, @@ -1430,7 +1422,7 @@ static uint64_t full_le_lduw_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, false, - full_le_lduw_mmu, wrap_lduw_le); + full_le_lduw_mmu, wrap_lduw_le, wrap_lduw_be); } tcg_target_ulong helper_le_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1443,7 +1435,7 @@ static uint64_t full_be_lduw_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, false, - full_be_lduw_mmu, wrap_lduw_be); + full_be_lduw_mmu, wrap_lduw_be, wrap_lduw_le); } tcg_target_ulong helper_be_lduw_mmu(CPUArchState *env, target_ulong addr, @@ -1456,7 +1448,7 @@ static uint64_t full_le_ldul_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, false, - full_le_ldul_mmu, wrap_ldul_le); + full_le_ldul_mmu, wrap_ldul_le, wrap_ldul_be); } tcg_target_ulong helper_le_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1469,7 +1461,7 @@ static uint64_t full_be_ldul_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, false, - full_be_ldul_mmu, wrap_ldul_be); + full_be_ldul_mmu, wrap_ldul_be, wrap_ldul_le); } tcg_target_ulong helper_be_ldul_mmu(CPUArchState *env, target_ulong addr, @@ -1482,14 +1474,14 @@ uint64_t helper_le_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, false, - helper_le_ldq_mmu, ldq_le_p); + helper_le_ldq_mmu, ldq_le_p, ldq_be_p); } uint64_t helper_be_ldq_mmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, false, - helper_be_ldq_mmu, ldq_be_p); + helper_be_ldq_mmu, ldq_be_p, ldq_le_p); } /* @@ -1563,7 +1555,7 @@ static inline void wrap_stl_le(void *haddr, uint64_t val) static inline void QEMU_ALWAYS_INLINE store_helper(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr, MemOp op, - StoreHelper *direct) + StoreHelper *direct, StoreHelper *direct_swap) { uintptr_t mmu_idx = get_mmuidx(oi); uintptr_t index = tlb_index(env, mmu_idx, addr); @@ -1608,23 +1600,24 @@ store_helper(CPUArchState *env, target_ulong addr, uint64_t val, /* On watchpoint hit, this will longjmp out. */ cpu_check_watchpoint(env_cpu(env), addr, size, iotlbentry->attrs, BP_MEM_WRITE, retaddr); - - /* The backing page may or may not require I/O. */ - tlb_addr &= ~TLB_WATCHPOINT; - if ((tlb_addr & ~TARGET_PAGE_MASK) == 0) { - goto do_aligned_access; - } } /* Handle I/O access. */ - io_writex(env, iotlbentry, mmu_idx, val, addr, retaddr, op); - return; - } + if (likely(tlb_addr & (TLB_MMIO | TLB_NOTDIRTY))) { + io_writex(env, iotlbentry, mmu_idx, val, addr, retaddr, + op ^ (tlb_addr & TLB_BSWAP ? MO_BSWAP : 0)); + return; + } - /* Handle slow unaligned access (it spans two pages or IO). */ - if (size > 1 - && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 - >= TARGET_PAGE_SIZE)) { + if (unlikely(tlb_addr & TLB_BSWAP)) { + haddr = (void *)((uintptr_t)addr + entry->addend); + direct_swap(haddr, val); + return; + } + } else if (size > 1 + && unlikely((addr & ~TARGET_PAGE_MASK) + size - 1 + >= TARGET_PAGE_SIZE)) { + /* Handle slow unaligned access (it spans two pages or IO). */ int i; uintptr_t index2; CPUTLBEntry *entry2; @@ -1686,7 +1679,6 @@ store_helper(CPUArchState *env, target_ulong addr, uint64_t val, return; } - do_aligned_access: haddr = (void *)((uintptr_t)addr + entry->addend); direct(haddr, val); } @@ -1694,43 +1686,47 @@ store_helper(CPUArchState *env, target_ulong addr, uint64_t val, void helper_ret_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_UB, wrap_stb); + store_helper(env, addr, val, oi, retaddr, MO_UB, wrap_stb, wrap_stb); } void helper_le_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUW, wrap_stw_le); + store_helper(env, addr, val, oi, retaddr, MO_LEUW, + wrap_stw_le, wrap_stw_be); } void helper_be_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUW, wrap_stw_be); + store_helper(env, addr, val, oi, retaddr, MO_BEUW, + wrap_stw_be, wrap_stw_le); } void helper_le_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEUL, wrap_stl_le); + store_helper(env, addr, val, oi, retaddr, MO_LEUL, + wrap_stl_le, wrap_stl_be); } void helper_be_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEUL, wrap_stl_be); + store_helper(env, addr, val, oi, retaddr, MO_BEUL, + wrap_stl_be, wrap_stl_le); } void helper_le_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p); + store_helper(env, addr, val, oi, retaddr, MO_LEQ, stq_le_p, stq_be_p); } void helper_be_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, TCGMemOpIdx oi, uintptr_t retaddr) { - store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p); + store_helper(env, addr, val, oi, retaddr, MO_BEQ, stq_be_p, stq_le_p); } /* First set of helpers allows passing in of OI and RETADDR. This makes @@ -1796,7 +1792,7 @@ static uint64_t full_ldub_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_8, true, - full_ldub_cmmu, wrap_ldub); + full_ldub_cmmu, wrap_ldub, wrap_ldub); } uint8_t helper_ret_ldb_cmmu(CPUArchState *env, target_ulong addr, @@ -1809,7 +1805,7 @@ static uint64_t full_le_lduw_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUW, true, - full_le_lduw_cmmu, wrap_lduw_le); + full_le_lduw_cmmu, wrap_lduw_le, wrap_lduw_be); } uint16_t helper_le_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1822,7 +1818,7 @@ static uint64_t full_be_lduw_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUW, true, - full_be_lduw_cmmu, wrap_lduw_be); + full_be_lduw_cmmu, wrap_lduw_be, wrap_lduw_le); } uint16_t helper_be_ldw_cmmu(CPUArchState *env, target_ulong addr, @@ -1835,7 +1831,7 @@ static uint64_t full_le_ldul_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEUL, true, - full_le_ldul_cmmu, wrap_ldul_le); + full_le_ldul_cmmu, wrap_ldul_le, wrap_ldul_be); } uint32_t helper_le_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1848,7 +1844,7 @@ static uint64_t full_be_ldul_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEUL, true, - full_be_ldul_cmmu, wrap_ldul_be); + full_be_ldul_cmmu, wrap_ldul_be, wrap_ldul_le); } uint32_t helper_be_ldl_cmmu(CPUArchState *env, target_ulong addr, @@ -1861,12 +1857,12 @@ uint64_t helper_le_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_LEQ, true, - helper_le_ldq_cmmu, ldq_le_p); + helper_le_ldq_cmmu, ldq_le_p, ldq_be_p); } uint64_t helper_be_ldq_cmmu(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { return load_helper(env, addr, oi, retaddr, MO_BEQ, true, - helper_be_ldq_cmmu, ldq_be_p); + helper_be_ldq_cmmu, ldq_be_p, ldq_le_p); }