From patchwork Tue May 8 13:25:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 135156 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp4386957lji; Tue, 8 May 2018 06:25:59 -0700 (PDT) X-Google-Smtp-Source: AB8JxZo+2i1BjTGa82zMWxE0mUSLfRagY31DRrvj3471ZAn8MO2edva+XCwuJUYhHUMkNBNJqDeb X-Received: by 2002:a17:902:bcc9:: with SMTP id o9-v6mr9824795pls.56.1525785959291; Tue, 08 May 2018 06:25:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525785959; cv=none; d=google.com; s=arc-20160816; b=kZKWk8Mae+ULHzXBZsBOFW8hbnKPJytWdorGkRcvjvlBDThOLdmIoLQRAhKiyapjsE 1djRLY68OCCNHW746Ddpaavf2+RORS5/N+3azmdl0hlIZIuawtl5DlTThxlHbJEhGKjc UWp3xc2ZLbFgDSZnaE9yunqfIZet1SCAgKuRgq0X55g58SUM75KAy6rFVWcwPXGFwpg9 ruSA+9Chy7lYVQlytlRpe6CluLL3pTJczOKjG6QgArbjB3hfvIgXJUTSrqqS4PYjrY2U 15pZ84mZ68T14lq1S5ruFndaoqe/pXUgO7OB+7c6DNlIH13AzETsR/uij+S+ajHjGy3z Xgig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:date:subject:mail-followup-to:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=nCUb1jkAf88gaAHiiPB/2+tVaFFoidhYR5exp2kDWio=; b=sUFXXkE1GaCoPJ3cUykYif7BCCTNeWu3Hk+WfCle7uXb5FYw0h5BEX+LVnWt1gV7M7 HUX3FLaauhLzlFe4h+2CNUM3/ekSJo1naFxv1M8Q0aRPcfw5ZZ3cP5oAkkfDvsf+yFeG FjbUFu0kpmFJ4zwcKHmWppqoprefv32MCGd46Z59efY93ehWMMmNbVVNfus0pzeqKsze obrjUH4XSU+Gaf23WS08q7ulDEg4ynGiTs1SbOYlzudZmuDlBiM9DaPW7ShuspdBhTyY f8r2GHFHsH3sR+f/kwZLSv3J7htiNaiPYKw2PcHzHPBJzmHiswc11dJcUgCw574WISO0 JoCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=hwWquXo5; spf=pass (google.com: domain of gcc-patches-return-477358-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-477358-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id t81-v6si19833075pgb.335.2018.05.08.06.25.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 06:25:59 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-477358-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=hwWquXo5; spf=pass (google.com: domain of gcc-patches-return-477358-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-477358-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; q=dns; s= default; b=onnsG9VPgqCiJgUoWLrJRNAdbsDpFBVUrcQCWVeDvWf9EOLrilIIl tyyurLF1/GsOsiFJBj+1Zb0pd+dR8s8knZCsobLPb6VYKjY1SOYF2R8Xu0IHzkgt y8RZrsoK4qjtc1q5HxN9Lf/vfAH/95WiE0p7AFdGbP0HIiVQvrFNbU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; s= default; bh=hC12o/iTq/QMrhaZea30GqmVObc=; b=hwWquXo5lJjxpkqYKGva 4BOXApXMwlSDs7+95XU3soGn3zlMiPgbruxh1ZIFXjYsCqxjIB0kYc5J1NGg43ng pWDgOdBWvUbcN1B9eHUH4UNg7XchVDbkt8V/TgyrPgDWiqE+9Qc4FzHx2owjpOIs kwCAKH5ckEk8rNfmZVnMxeE= Received: (qmail 2571 invoked by alias); 8 May 2018 13:25:47 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 2561 invoked by uid 89); 8 May 2018 13:25:47 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-9.2 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_NONE, SPF_PASS, UNSUBSCRIBE_BODY autolearn=ham version=3.3.2 spammy=expanding, H*r:sk:v5-v6so, assignments, edges X-HELO: mail-wr0-f173.google.com Received: from mail-wr0-f173.google.com (HELO mail-wr0-f173.google.com) (209.85.128.173) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 08 May 2018 13:25:45 +0000 Received: by mail-wr0-f173.google.com with SMTP id v5-v6so32302816wrf.9 for ; Tue, 08 May 2018 06:25:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:date:message-id :user-agent:mime-version; bh=nCUb1jkAf88gaAHiiPB/2+tVaFFoidhYR5exp2kDWio=; b=r9pagQzLkl94pzMZGhmBVKRg7c+a9lTCEESDZcfxBaQ3JSQDPkBzsxbcKcqe5g4HjW nZx2zuIX4tCNtR4Iehzd8U1JkQcrRQXcwKGfEQHSEnPnalOPorimbnpZy9WuVp0obyix Qcaolxz+r9TWe0FBLf1UUzL+N5WD9GxoTMo/lWuJmEUEkfNY33hUinzMGjLeu2tfPdo2 qcYANVJAnP8fhN2FxPe0z5AARhKpVMZEXZjh+ukDlEOBQ796k0d20xmnLdOUVlAqH/b5 iZ2peuGCgQkQ8qp8nCjH77KrY7KUwR8zPTpay18inDvf4OW6MMtM0OD+Jm+UE6HcneUD v1aQ== X-Gm-Message-State: ALQs6tAr0jKi3vR8RnB587ysffJK8tFcm1nfxk3mkbOh6LPeFncQLmH0 +WKJmWmn8Td32rwfgznqpdRAgMwO+0Q= X-Received: by 2002:adf:db85:: with SMTP id u5-v6mr31392644wri.278.1525785942580; Tue, 08 May 2018 06:25:42 -0700 (PDT) Received: from localhost (116.58.7.51.dyn.plus.net. [51.7.58.116]) by smtp.gmail.com with ESMTPSA id r10-v6sm12635243wrm.15.2018.05.08.06.25.41 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 08 May 2018 06:25:41 -0700 (PDT) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: Add clobbers around IFN_LOAD/STORE_LANES Date: Tue, 08 May 2018 14:25:41 +0100 Message-ID: <87vabyqnxm.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 We build up the input to IFN_STORE_LANES one vector at a time. In RTL, each of these vector assignments becomes a write to subregs of the form (subreg:VEC (reg:AGGR R)), where R is the eventual input to the store lanes instruction. The problem is that RTL isn't very good at tracking liveness when things are initialised piecemeal by subregs, so R tends to end up being live on all paths from the entry block to the store. This in turn leads to unnecessary spilling around calls, as well as to excess register pressure in vector loops. This patch adds gimple clobbers to indicate the liveness of the IFN_STORE_LANES variable and makes sure that gimple clobbers are expanded to rtl clobbers where useful. For consistency it also uses clobbers to mark the point at which an IFN_LOAD_LANES variable is no longer needed. Tested on aarch64-linux-gnu (with and without SVE), aaarch64_be-elf and x86_64-linux-gnu. OK to install? Thanks, Richard 2018-05-08 Richard Sandiford gcc/ * cfgexpand.c (expand_clobber): New function. (expand_gimple_stmt_1): Use it. * tree-vect-stmts.c (vect_clobber_variable): New function, split out from... (vectorizable_simd_clone_call): ...here. (vectorizable_store): Emit a clobber either side of an IFN_STORE_LANES sequence. (vectorizable_load): Emit a clobber after an IFN_LOAD_LANES sequence. gcc/testsuite/ * gcc.target/aarch64/store_lane_spill_1.c: New test. * gcc.target/aarch64/sve/store_lane_spill_1.c: Likewise. Index: gcc/cfgexpand.c =================================================================== --- gcc/cfgexpand.c 2018-05-08 09:42:02.974668379 +0100 +++ gcc/cfgexpand.c 2018-05-08 14:23:25.039856499 +0100 @@ -3582,6 +3582,20 @@ expand_return (tree retval, tree bounds) } } +/* Expand a clobber of LHS. If LHS is stored it in a register, tell + the rtl optimizers that its value is no longer needed. */ + +static void +expand_clobber (tree lhs) +{ + if (DECL_P (lhs)) + { + rtx decl_rtl = DECL_RTL_IF_SET (lhs); + if (decl_rtl && REG_P (decl_rtl)) + emit_clobber (decl_rtl); + } +} + /* A subroutine of expand_gimple_stmt, expanding one gimple statement STMT that doesn't require special handling for outgoing edges. That is no tailcalls and no GIMPLE_COND. */ @@ -3687,7 +3701,7 @@ expand_gimple_stmt_1 (gimple *stmt) if (TREE_CLOBBER_P (rhs)) /* This is a clobber to mark the going out of scope for this LHS. */ - ; + expand_clobber (lhs); else expand_assignment (lhs, rhs, gimple_assign_nontemporal_move_p ( Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2018-05-08 09:42:03.335655127 +0100 +++ gcc/tree-vect-stmts.c 2018-05-08 14:23:25.040856464 +0100 @@ -182,6 +182,18 @@ create_array_ref (tree type, tree ptr, t return mem_ref; } +/* Add a clobber of variable VAR to the vectorization of STMT. + Emit the clobber before *GSI. */ + +static void +vect_clobber_variable (gimple *stmt, gimple_stmt_iterator *gsi, tree var) +{ + tree clobber = build_constructor (TREE_TYPE (var), NULL); + TREE_THIS_VOLATILE (clobber) = 1; + gimple *new_stmt = gimple_build_assign (var, clobber); + vect_finish_stmt_generation (stmt, new_stmt, gsi); +} + /* Utility functions used by vect_mark_stmts_to_be_vectorized. */ /* Function vect_mark_relevant. @@ -4128,12 +4140,7 @@ vectorizable_simd_clone_call (gimple *st } if (ratype) - { - tree clobber = build_constructor (ratype, NULL); - TREE_THIS_VOLATILE (clobber) = 1; - new_stmt = gimple_build_assign (new_temp, clobber); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - } + vect_clobber_variable (stmt, gsi, new_temp); continue; } else if (simd_clone_subparts (vectype) > nunits) @@ -4156,10 +4163,7 @@ vectorizable_simd_clone_call (gimple *st CONSTRUCTOR_APPEND_ELT (ret_ctor_elts, NULL_TREE, gimple_assign_lhs (new_stmt)); } - tree clobber = build_constructor (ratype, NULL); - TREE_THIS_VOLATILE (clobber) = 1; - new_stmt = gimple_build_assign (new_temp, clobber); - vect_finish_stmt_generation (stmt, new_stmt, gsi); + vect_clobber_variable (stmt, gsi, new_temp); } else CONSTRUCTOR_APPEND_ELT (ret_ctor_elts, NULL_TREE, new_temp); @@ -4186,11 +4190,7 @@ vectorizable_simd_clone_call (gimple *st new_stmt = gimple_build_assign (make_ssa_name (vec_dest), t); vect_finish_stmt_generation (stmt, new_stmt, gsi); - tree clobber = build_constructor (ratype, NULL); - TREE_THIS_VOLATILE (clobber) = 1; - vect_finish_stmt_generation (stmt, - gimple_build_assign (new_temp, - clobber), gsi); + vect_clobber_variable (stmt, gsi, new_temp); } } @@ -6913,8 +6913,15 @@ vectorizable_store (gimple *stmt, gimple { tree vec_array; - /* Combine all the vectors into an array. */ + /* Get an array into which we can store the individual vectors. */ vec_array = create_vector_array (vectype, vec_num); + + /* Invalidate the current contents of VEC_ARRAY. This should + become an RTL clobber too, which prevents the vector registers + from being upward-exposed. */ + vect_clobber_variable (stmt, gsi, vec_array); + + /* Store the individual vectors into the array. */ for (i = 0; i < vec_num; i++) { vec_oprnd = dr_chain[i]; @@ -6953,6 +6960,9 @@ vectorizable_store (gimple *stmt, gimple gimple_call_set_nothrow (call, true); new_stmt = call; vect_finish_stmt_generation (stmt, new_stmt, gsi); + + /* Record that VEC_ARRAY is now dead. */ + vect_clobber_variable (stmt, gsi, vec_array); } else { @@ -8105,6 +8115,9 @@ vectorizable_load (gimple *stmt, gimple_ /* Record the mapping between SSA_NAMEs and statements. */ vect_record_grouped_load_vectors (stmt, dr_chain); + + /* Record that VEC_ARRAY is now dead. */ + vect_clobber_variable (stmt, gsi, vec_array); } else { Index: gcc/testsuite/gcc.target/aarch64/store_lane_spill_1.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/store_lane_spill_1.c 2018-05-08 14:23:25.039856499 +0100 @@ -0,0 +1,21 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#pragma GCC target "+nosve" + +int cont (void); + +void +f (int (*x)[3], int *a, int *b, int *c, int n) +{ + do + for (int i = 0; i < n; ++i) + { + x[i][0] = a[i] + 1; + x[i][1] = b[i] + 2; + x[i][2] = c[i] + 3; + } + while (cont ()); +} + +/* { dg-final { scan-assembler-not {\tst1\t} } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/store_lane_spill_1.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/store_lane_spill_1.c 2018-05-08 14:23:25.040856464 +0100 @@ -0,0 +1,19 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +int cont (void); + +void +f (int (*x)[3], int *a, int *b, int *c, int n) +{ + do + for (int i = 0; i < n; ++i) + { + x[i][0] = a[i] + 1; + x[i][1] = b[i] + 2; + x[i][2] = c[i] + 3; + } + while (cont ()); +} + +/* { dg-final { scan-assembler-not {\tstr\tz[0-9]} } } */