From patchwork Fri Aug 27 12:01:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503384 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197658jab; Fri, 27 Aug 2021 05:06:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwMte3NZQFFqxPNafzgnBMvxJ4b/0eCDK/zcHyaUI+Q8gKpct9Bm8zQwdEQAr/wJsPB8QEq X-Received: by 2002:aa7:d99a:: with SMTP id u26mr5520081eds.297.1630066016867; Fri, 27 Aug 2021 05:06:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066016; cv=none; d=google.com; s=arc-20160816; b=WVqK/vYLR5QlW90aFJBrt23+yoTsXyUurMfJsXbNa+yaw9VHLi+mP/dcdJDggbjYmE vbJQTe0uz57D8+DVocZ/mJVUuXoTslScKG59j8L/CY6vvKw6gQMVoBxCxYR/IORXbgxF rYfHEkPJXwwHNuaszyIxT7ITOrdZ6b3X0HKmw7RlDecAAnn6CzyqG/qz5xKXNfI7SF94 1TvZkeoCwvwBaZJKHB1hfNALPcakZD6NKbwFlbYEPsfbIfWqNo+zzm7QPhhkPuqSCut7 LWTSdH9qW9CtcA8Gt+nvJcjfUP4stppt125eRV3fKp2bN0G4gCkFsqORVc9ntUrhYJ3l 8nCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=QhTqrgsfYKlEolt2qG2LW8rHA8MA1o0d7NJ2xK9E5ic=; b=mg8A1juy/lfqMyyupZZjzgKCt+S6r2pl0TwAmvTbx+Tw0dORV6Voj5/L9zaGXY0TZk VktmVh5MTRLUiR6H3lPxscHBTtKym9kmobIufVrmvabD36VJAv+bjdFqt2BEp+2tcwNQ raoxTUfgTktqzdtGDkxx7uAU92IbeJRVVZ6AqXO1ay1XgXGt/uwzOhZYm4kudm0v9Zg2 oG9VnvIfR8gJLC7A9p2qTTEiam0qiSp/QU9OyWhxWENteLdaKmjtpzv70MLcB7MZjTvl 6FRqolP8b766iZGWkF2sxcqhXE7zyuHWEJDNWdlUTgi3pX/rw5+oQp3u40kEEyJRbeA9 g/dQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ne32si5450103ejc.27.2021.08.27.05.06.56 for ; Fri, 27 Aug 2021 05:06:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245103AbhH0MHl (ORCPT ); Fri, 27 Aug 2021 08:07:41 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3696 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245094AbhH0MHl (ORCPT ); Fri, 27 Aug 2021 08:07:41 -0400 Received: from fraeml701-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1n2F5Tz67fqW; Fri, 27 Aug 2021 20:05:33 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml701-chm.china.huawei.com (10.206.15.50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.8; Fri, 27 Aug 2021 14:06:50 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:48 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 01/13] blk-mq: Change rqs check in blk_mq_free_rqs() Date: Fri, 27 Aug 2021 20:01:52 +0800 Message-ID: <1630065724-69146-2-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org The original code in commit 24d2f90309b23 ("blk-mq: split out tag initialization, support shared tags") would check tags->rqs is non-NULL and then dereference tags->rqs[]. Then in commit 2af8cbe30531 ("blk-mq: split tag ->rqs[] into two"), we started to dereference tags->static_rqs[], but continued to check non-NULL tags->rqs. Check tags->static_rqs as non-NULL instead, which is more logical. Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 0a33d16a7298..7927e62d6395 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2348,7 +2348,7 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, { struct page *page; - if (tags->rqs && set->ops->exit_request) { + if (tags->static_rqs && set->ops->exit_request) { int i; for (i = 0; i < tags->nr_tags; i++) { From patchwork Fri Aug 27 12:01:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503385 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197701jab; Fri, 27 Aug 2021 05:06:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwUjvTWCVvpRi8scjqGP4SEOOA5g7qOprmkaeW9dhZHJKD2kqbFqRibEa1dqJiPzfBYb1Gp X-Received: by 2002:a17:907:10cc:: with SMTP id rv12mr9477368ejb.423.1630066018630; Fri, 27 Aug 2021 05:06:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066018; cv=none; d=google.com; s=arc-20160816; b=FPCycTs6z7mtVxcVC8awaEc3F9TNHmqmcZWfurnwN9TYrev3gMpahQoqWzUkPn2Wti 2/WUq2JtXFdxoroFkCHy8SgP99su0FlEP5YUwMrnEeSbPgFxp/Ka/X8VdzG6LPhBDFPb X10U9hDx3+hRDdiAe5M4IEGotxvAk4vA1qzHLPcTOYUpBkw1l1EOJ9X0NAHPU20An5dg 1Hk6NILdmbbXaQpLMIIPabsH5RNrOzXEHVSxL8XPtfBkx3YcOd17PYd4bBhTva8xZuZt p04bvncWgNo8yeAXfumA4sFVlvf4W/7ueJlyOPcc/wrj/XBPqMNgk+HAqYJuPIe6tmqC DROA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=gkfH0F0DDkkN6qt2+0c4lnzjJhY3CWO9UgP6w08HECA=; b=aLal04hbkDXnSB0nFPt7xAilJdoT1KsO4A20oqKO82/t5QYr5yxbQnSW24XO7O5WJL ++QM7saeoTJGQLNFi4qDtzafsX4a45pAnq4Ylp7i3LqwXS8rHhwXZPbF+n7nkUPlkMiI ymiHQsOKraACDRpyLmPzBaZ2SCZhMAwkhFjLZbMSTw0/w63aMqdE8dK/c7C8D1lG1xa5 JOUL+P+N7LKTj7dF0T7hdG7HzUHv3LlTugWvatlE3VH48t10NDKIDDf8x0nUtIxjl67g z4sPmLyvqOeY1pVyaXw9yz1hOvVe7cLzRP7eTMXFvcBvAAAM+IHg8tQxHqwZ3JyRe9JL I6rg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ne32si5450103ejc.27.2021.08.27.05.06.58 for ; Fri, 27 Aug 2021 05:06:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245108AbhH0MHo (ORCPT ); Fri, 27 Aug 2021 08:07:44 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3697 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245112AbhH0MHn (ORCPT ); Fri, 27 Aug 2021 08:07:43 -0400 Received: from fraeml745-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1q26Rvz67YT1; Fri, 27 Aug 2021 20:05:35 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml745-chm.china.huawei.com (10.206.15.226) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:06:52 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:50 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 02/13] block: Rename BLKDEV_MAX_RQ -> BLKDEV_DEFAULT_RQ Date: Fri, 27 Aug 2021 20:01:53 +0800 Message-ID: <1630065724-69146-3-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org It is a bit confusing that there is BLKDEV_MAX_RQ and MAX_SCHED_RQ, as the name BLKDEV_MAX_RQ would imply the max requests always, which it is not. Rename to BLKDEV_MAX_RQ to BLKDEV_DEFAULT_RQ, matching its usage - that being the default number of requests assigned when allocating a request queue. Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-core.c | 2 +- block/blk-mq-sched.c | 2 +- block/blk-mq-sched.h | 2 +- drivers/block/rbd.c | 2 +- include/linux/blkdev.h | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) -- 2.26.2 diff --git a/block/blk-core.c b/block/blk-core.c index 0874bc2fcdb4..93b585593ebd 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -569,7 +569,7 @@ struct request_queue *blk_alloc_queue(int node_id) blk_queue_dma_alignment(q, 511); blk_set_default_limits(&q->limits); - q->nr_requests = BLKDEV_MAX_RQ; + q->nr_requests = BLKDEV_DEFAULT_RQ; return q; diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 0f006cabfd91..2231fb0d4c35 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -606,7 +606,7 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) * Additionally, this is a per-hw queue depth. */ q->nr_requests = 2 * min_t(unsigned int, q->tag_set->queue_depth, - BLKDEV_MAX_RQ); + BLKDEV_DEFAULT_RQ); queue_for_each_hw_ctx(q, hctx, i) { ret = blk_mq_sched_alloc_tags(q, hctx, i); diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 5246ae040704..1e46be6c5178 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -5,7 +5,7 @@ #include "blk-mq.h" #include "blk-mq-tag.h" -#define MAX_SCHED_RQ (16 * BLKDEV_MAX_RQ) +#define MAX_SCHED_RQ (16 * BLKDEV_DEFAULT_RQ) void blk_mq_sched_assign_ioc(struct request *rq); diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index e65c9d706f6f..bf60aebd0cfb 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -836,7 +836,7 @@ struct rbd_options { u32 alloc_hint_flags; /* CEPH_OSD_OP_ALLOC_HINT_FLAG_* */ }; -#define RBD_QUEUE_DEPTH_DEFAULT BLKDEV_MAX_RQ +#define RBD_QUEUE_DEPTH_DEFAULT BLKDEV_DEFAULT_RQ #define RBD_ALLOC_SIZE_DEFAULT (64 * 1024) #define RBD_LOCK_TIMEOUT_DEFAULT 0 /* no timeout */ #define RBD_READ_ONLY_DEFAULT false diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index c9cb12483e12..fc3987eda2c1 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -44,7 +44,7 @@ struct blk_stat_callback; struct blk_keyslot_manager; #define BLKDEV_MIN_RQ 4 -#define BLKDEV_MAX_RQ 128 /* Default maximum */ +#define BLKDEV_DEFAULT_RQ 128 /* Must be consistent with blk_mq_poll_stats_bkt() */ #define BLK_MQ_POLL_STATS_BKTS 16 From patchwork Fri Aug 27 12:01:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503386 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197727jab; Fri, 27 Aug 2021 05:07:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJziXhrp3+m3CysVUtcdQFxtdZoBW4wDfQolZCvxPCmHDYALqItlCWTlpDUEfBUMc7dOS1Yb X-Received: by 2002:a17:906:498b:: with SMTP id p11mr9837297eju.295.1630066019972; Fri, 27 Aug 2021 05:06:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066019; cv=none; d=google.com; s=arc-20160816; b=zAsMV1v5MOcI2HGuN8ODdNstGz8MswdNLbDE19a5GcCzq2nPla9+PSOq9TgIe2lw+I /KP1m/ZNWEDOQfqjD4VUbff/k7ZFCd9puyFXviym/R+gGg7JAnKo8OjnC7pPhTj7L7JA iQoyDScuKIpJBRoLB8EsPLpTX4739aRw8bkuoBySzyRzn0k32loqv0cnZQJhP52dMdGC CKUa2aWOjzGH7iKhyrR/gKT5Q+HTFcmdoz0pBoIxq8c2jWfcxe5Hg/8xT1vU0MHnADDW t5291A7kSBiXq6W2rv4Li/ySBPeHGWRkn6Lm+wt45h2/nsnRw7tE/hWm5vLLj4UY7I7e B0FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=LaJPyEoYvppOPmRO/c219w018NBOXc40hPgIefY/Org=; b=NrEek15Y2shG2hf5JtTGVY/djFziSlwuzVpxnWUQyo5uMYZjmVB/G8KUCA0+00IiHB ydDNHGdeL0qDXxGkHry4PLvoVncaS3lMIQ1+Y5C1CNAauYfHASV44a3g3IRvHJoqgbC9 ak7KjlzhLUEGrsc6Nkf4hgwB3o6nZLbA4UxUuPk+CtRnEIR9mnD1YqIWUD4pPJSLFkkk qIh55z8V8og4JZralZ0hX12hAVDs6sm7VQnOjvYVg2svemencpE3NmyBV8Ud5384Db8P 07pGxQkvGqeykK/wTQ1wcLYWQlf0S62Us4Z9C3mezO2z4QVl294VUbswIlOngzaqiK6y zzIA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ne32si5450103ejc.27.2021.08.27.05.06.59 for ; Fri, 27 Aug 2021 05:06:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245124AbhH0MHq (ORCPT ); Fri, 27 Aug 2021 08:07:46 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3698 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245111AbhH0MHp (ORCPT ); Fri, 27 Aug 2021 08:07:45 -0400 Received: from fraeml744-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1j3qhbz67snn; Fri, 27 Aug 2021 20:05:29 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml744-chm.china.huawei.com (10.206.15.225) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:06:54 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:52 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 03/13] blk-mq: Relocate shared sbitmap resize in blk_mq_update_nr_requests() Date: Fri, 27 Aug 2021 20:01:54 +0800 Message-ID: <1630065724-69146-4-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org For shared sbitmap, if the call to blk_mq_tag_update_depth() was successful for any hctx when hctx->sched_tags is not set, then it would be successful for all (due to nature in which blk_mq_tag_update_depth() fails). As such, there is no need to call blk_mq_tag_resize_shared_sbitmap() for each hctx. So relocate the call until after the hctx iteration under the !q->elevator check, which is equivalent (to !hctx->sched_tags). Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-mq.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 7927e62d6395..2cc5b2952ceb 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3624,8 +3624,6 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) if (!hctx->sched_tags) { ret = blk_mq_tag_update_depth(hctx, &hctx->tags, nr, false); - if (!ret && blk_mq_is_sbitmap_shared(set->flags)) - blk_mq_tag_resize_shared_sbitmap(set, nr); } else { ret = blk_mq_tag_update_depth(hctx, &hctx->sched_tags, nr, true); @@ -3643,9 +3641,13 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) } if (!ret) { q->nr_requests = nr; - if (q->elevator && blk_mq_is_sbitmap_shared(set->flags)) - sbitmap_queue_resize(&q->sched_bitmap_tags, - nr - set->reserved_tags); + if (blk_mq_is_sbitmap_shared(set->flags)) { + if (q->elevator) + sbitmap_queue_resize(&q->sched_bitmap_tags, + nr - set->reserved_tags); + else + blk_mq_tag_resize_shared_sbitmap(set, nr); + } } blk_mq_unquiesce_queue(q); From patchwork Fri Aug 27 12:01:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503387 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197763jab; Fri, 27 Aug 2021 05:07:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwucg7hwjbl2hEu0WHg3eOWabm0WTh/9JrStQbD26WZDm0s7IZe4Ux1nCaY095x5XoVJ/39 X-Received: by 2002:a05:6512:ad6:: with SMTP id n22mr6827839lfu.532.1630066022123; Fri, 27 Aug 2021 05:07:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066022; cv=none; d=google.com; s=arc-20160816; b=tn2pXrB/UfJKk/5hEGapTgNcCUJv/gtoE9O7skVj3X4isoJ2BfMxLSTOpnpqkz5vi4 75BiZ7R0R3Q7E9pV9GdIQiqX0wV4iJrNu87Y9Sxk2DxrnnTj0JtGn11t0gncj25c7DGX cyPDoRaFGmUYfVNBc60VPS24VC5aTRFpv7Ne4O+dP3dOqageJq1fzc9i/WSPqh/K8zRI 0EvHJbRpdQKKq8CIkDwbE9Z7hpWy7d5AdThL6G+BLSOLHft+H57YrEHbcsCdCErezQg9 xrFqZmh+ukZGc08j6T08B1QWO8Va1Cds/8ZNr91vVjyAZgvkzf3judNgINzlY05yyj19 G1mA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=68KKQh/W8q8WGWkB2m5t/AE64TsmaNufMN+FLJIiOoc=; b=v3cNOQfv3bM6aWHX/IcRg2Wwli1XMu8muVbQkw2AK/osFz0ILKvUmWEO6f4uLbr93C DtJJ7Mo/RZg58bCDCBsdtT3e3S+lDUa9txbjmeHbIXKLpB84vGLKOHNawkki0cXhIQEz Wggmr+RviyFkMvkjun99+UJg8gqjlyu1+iH6YawRm1uKMRnk69yqwP7Z8dgdkoqlXUzH 36vtl4Jkfd9B90kv3w4FxUnpZz5yMukXFSj8JR/gR8+2NUSRODHXejxJvNQ58MWbieQo 9+/vXGTK+jl7LKt6XeTANG9WLmLwClIGVEpIVIEc15GYSbOrqdOrQzy6A1gbw8fYA/pN U5Xg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ca20si5772834edb.178.2021.08.27.05.07.01 for ; Fri, 27 Aug 2021 05:07:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245111AbhH0MHs (ORCPT ); Fri, 27 Aug 2021 08:07:48 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3699 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245125AbhH0MHr (ORCPT ); Fri, 27 Aug 2021 08:07:47 -0400 Received: from fraeml743-chm.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1v3Ftdz67YT1; Fri, 27 Aug 2021 20:05:39 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml743-chm.china.huawei.com (10.206.15.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:06:56 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:54 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 04/13] blk-mq: Invert check in blk_mq_update_nr_requests() Date: Fri, 27 Aug 2021 20:01:55 +0800 Message-ID: <1630065724-69146-5-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org It's easier to read: if (x) X; else Y; over: if (!x) Y; else X; No functional change intended. Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-mq.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 2cc5b2952ceb..b51d59ec7d7a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3621,18 +3621,18 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) * If we're using an MQ scheduler, just update the scheduler * queue depth. This is similar to what the old code would do. */ - if (!hctx->sched_tags) { - ret = blk_mq_tag_update_depth(hctx, &hctx->tags, nr, - false); - } else { + if (hctx->sched_tags) { ret = blk_mq_tag_update_depth(hctx, &hctx->sched_tags, - nr, true); + nr, true); if (blk_mq_is_sbitmap_shared(set->flags)) { hctx->sched_tags->bitmap_tags = &q->sched_bitmap_tags; hctx->sched_tags->breserved_tags = &q->sched_breserved_tags; } + } else { + ret = blk_mq_tag_update_depth(hctx, &hctx->tags, nr, + false); } if (ret) break; From patchwork Fri Aug 27 12:01:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503388 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197793jab; Fri, 27 Aug 2021 05:07:04 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyero/QuTc07ylT3itcr46biyLAq2KkNulXLv6wAOCGGgJesy6JwzSZ2nKS2cuNE4YKhiMz X-Received: by 2002:a05:6402:1d05:: with SMTP id dg5mr9212208edb.375.1630066024006; Fri, 27 Aug 2021 05:07:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066024; cv=none; d=google.com; s=arc-20160816; b=Wnt+XDHOTEYSoLCJHQN0Mn35WzEhGBlFNsMZdpaQmZ5oNIrgOI/cWUWVv1edHYPEDs aSZwK0Jy5OjPFYFouPYbS6U6AnjTYafxo2yCnmwRiXvZFGlvGjdo6J+vn/dCRWnBVBI5 /4+uy3qakF031fXKXR2FnSJpRuoYQOJ6bKklvsx8TtMqyG+TiO6WBa5FAupahyH+s+BA kBlV4PxBSsdGpp/3uN2eJVDJF59jor7lR1sFhG5iUpQbu1tFqL6xtMJi0MGRWSaUR/r0 QowgNrgtd3Z4Avr4I2ChI4dvxvsZSaQ16kfT2EIxgt8AUzAkwZqLGZHcXT/dc7VZhgGi cf8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=Paz5e9pxEoNv7W8lcrWO623CLc7ymBcnFvyvtNCROIk=; b=WTPtnuCrM35rlLmgAFrJUTVRbuNwEnZyJ9JaUgopuuPqlRHNAdZ1A+WtDbJxk2W2q0 cv4fxMk4Ivg5nyuRGZjPk1lScKJoUlrFV8QTEH7ohwrcBx7ENJ7BBuH1Y44w3XWuJAwC hg+surBlcNj/yMsnOsy4aLDlFtzEym2tUAxN4m+9EyPWv2S/QTBsKDob0HaviPcmSQvw xcHLxsENimX6Em9Uj5CmLDkYSltH0QL9lsY4oTCfVzwNymMDeK3ifAfMF4rNawJ6rZYY yoH55KDS1JDdOVuEmMFZFaSnKUARjZAU8yFyKj+2p5dlGB1Eqh4e9IiW+ne/JJTFb32h +jlg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id qa36si4642051ejc.337.2021.08.27.05.07.03 for ; Fri, 27 Aug 2021 05:07:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245152AbhH0MHu (ORCPT ); Fri, 27 Aug 2021 08:07:50 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3700 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245138AbhH0MHt (ORCPT ); Fri, 27 Aug 2021 08:07:49 -0400 Received: from fraeml742-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1n6FHhz67w17; Fri, 27 Aug 2021 20:05:33 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml742-chm.china.huawei.com (10.206.15.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:06:58 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:56 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 05/13] blk-mq-sched: Rename blk_mq_sched_alloc_{tags -> map_and_rqs}() Date: Fri, 27 Aug 2021 20:01:56 +0800 Message-ID: <1630065724-69146-6-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Function blk_mq_sched_alloc_tags() does same as __blk_mq_alloc_map_and_request(), so give a similar name to be consistent. Similarly rename label err_free_tags -> err_free_map_and_rqs. Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-mq-sched.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) -- 2.26.2 diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 2231fb0d4c35..5f340203e6e5 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -515,9 +515,9 @@ void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx, percpu_ref_put(&q->q_usage_counter); } -static int blk_mq_sched_alloc_tags(struct request_queue *q, - struct blk_mq_hw_ctx *hctx, - unsigned int hctx_idx) +static int blk_mq_sched_alloc_map_and_rqs(struct request_queue *q, + struct blk_mq_hw_ctx *hctx, + unsigned int hctx_idx) { struct blk_mq_tag_set *set = q->tag_set; int ret; @@ -609,15 +609,15 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) BLKDEV_DEFAULT_RQ); queue_for_each_hw_ctx(q, hctx, i) { - ret = blk_mq_sched_alloc_tags(q, hctx, i); + ret = blk_mq_sched_alloc_map_and_rqs(q, hctx, i); if (ret) - goto err_free_tags; + goto err_free_map_and_rqs; } if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) { ret = blk_mq_init_sched_shared_sbitmap(q); if (ret) - goto err_free_tags; + goto err_free_map_and_rqs; } ret = e->ops.init_sched(q, e); @@ -645,8 +645,8 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) err_free_sbitmap: if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) blk_mq_exit_sched_shared_sbitmap(q); -err_free_tags: blk_mq_sched_free_requests(q); +err_free_map_and_rqs: blk_mq_sched_tags_teardown(q); q->elevator = NULL; return ret; From patchwork Fri Aug 27 12:01:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503389 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197817jab; Fri, 27 Aug 2021 05:07:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwBu1tXkLLeBhJXxrwHeW+8/n8ON8ENEWjMLKC4LKFRwRxQ57DMicFe2OTIn3J847YXHfV7 X-Received: by 2002:a17:906:2745:: with SMTP id a5mr9544415ejd.215.1630066025776; Fri, 27 Aug 2021 05:07:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066025; cv=none; d=google.com; s=arc-20160816; b=EfIhNaDZoSPV44xMZH24gCvcLaz+/4ay73kvCYINJVhx1mmj+ij4YSzWeYH3CC+AEX yy633TtVqn3jgregZTpnwEdnBZ78ObEs0rigTnXSAzTLcEXoLIAlzkCoTPPeTrvth8th /HBD/8BV+xDLXIZEBMXKmmOgzfkhoE/XPw0bljdT0Sw3lNVCEJ2z57HT+LtIbNExY+MF 3V+/sQIfB4QTg6pgrSKg4Oe4MlDBwRoVIvZadckoi7Z2WA6YEnSTrgqGmZbLtRnoPLJN VyqK43xnt3CV2aFGg9gILpfD74ArDJTJ3cJ4ohPWWKYjOw1IEu3wSiIJaAc9Lu0/wHUO ePZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=PZBpUe8q0RLURk8mAc4IoOk7QGFUatZE+Gfi+vTGqyk=; b=jZiWd4y+MNjdrDmcl5BPbte8Ej1gmWNXFk7OIr9+Suk/bc7uu5MR9SY582vOTiqNLV OiA4sPB6ohrM3jNAAhXDLCXrqo4Ji90Aihx/VUkmluH3h/uirFJLAA8XdwvxymFUazVT QNqyeJsCAZSizXZrqKupxiuDX7V4vZs+fyMHyZTDSM4zqCoETyQazOdzLTE2rmbm+dPA Vp1laSkmYQGw91Bx9uo3cjsTWU16key1v4uJpzNY+1Fm9R81lw/+U1mUbMgZlvVy/vKy 7Sz/5Hn4EkAymYgN04Hxfk371n/yWlpTLbxm8iCuWuBtfIlfD3yidbfhQedYM5OJiU9E Cdwg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id qa36si4642051ejc.337.2021.08.27.05.07.05 for ; Fri, 27 Aug 2021 05:07:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245138AbhH0MHw (ORCPT ); Fri, 27 Aug 2021 08:07:52 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3701 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245151AbhH0MHv (ORCPT ); Fri, 27 Aug 2021 08:07:51 -0400 Received: from fraeml741-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1z4LpVz67g00; Fri, 27 Aug 2021 20:05:43 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml741-chm.china.huawei.com (10.206.15.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:00 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:06:58 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 06/13] blk-mq-sched: Rename blk_mq_sched_free_{requests -> rqs}() Date: Fri, 27 Aug 2021 20:01:57 +0800 Message-ID: <1630065724-69146-7-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org To be more concise and consistent in naming, rename blk_mq_sched_free_requests() -> blk_mq_sched_free_rqs(). Signed-off-by: John Garry --- block/blk-core.c | 2 +- block/blk-mq-sched.c | 6 +++--- block/blk-mq-sched.h | 2 +- block/blk.h | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) -- 2.26.2 diff --git a/block/blk-core.c b/block/blk-core.c index 93b585593ebd..d8e36b08eb2b 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -407,7 +407,7 @@ void blk_cleanup_queue(struct request_queue *q) */ mutex_lock(&q->sysfs_lock); if (q->elevator) - blk_mq_sched_free_requests(q); + blk_mq_sched_free_rqs(q); mutex_unlock(&q->sysfs_lock); percpu_ref_exit(&q->q_usage_counter); diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 5f340203e6e5..3ab26154f0ea 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -631,7 +631,7 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) ret = e->ops.init_hctx(hctx, i); if (ret) { eq = q->elevator; - blk_mq_sched_free_requests(q); + blk_mq_sched_free_rqs(q); blk_mq_exit_sched(q, eq); kobject_put(&eq->kobj); return ret; @@ -645,7 +645,7 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) err_free_sbitmap: if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) blk_mq_exit_sched_shared_sbitmap(q); - blk_mq_sched_free_requests(q); + blk_mq_sched_free_rqs(q); err_free_map_and_rqs: blk_mq_sched_tags_teardown(q); q->elevator = NULL; @@ -656,7 +656,7 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) * called in either blk_queue_cleanup or elevator_switch, tagset * is required for freeing requests */ -void blk_mq_sched_free_requests(struct request_queue *q) +void blk_mq_sched_free_rqs(struct request_queue *q) { struct blk_mq_hw_ctx *hctx; int i; diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 1e46be6c5178..e70748d18754 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -28,7 +28,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx); int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e); void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e); -void blk_mq_sched_free_requests(struct request_queue *q); +void blk_mq_sched_free_rqs(struct request_queue *q); static inline bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, diff --git a/block/blk.h b/block/blk.h index bbbcc1a64a2d..76c59ae436d8 100644 --- a/block/blk.h +++ b/block/blk.h @@ -204,7 +204,7 @@ static inline void elevator_exit(struct request_queue *q, { lockdep_assert_held(&q->sysfs_lock); - blk_mq_sched_free_requests(q); + blk_mq_sched_free_rqs(q); __elevator_exit(q, e); } From patchwork Fri Aug 27 12:01:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503390 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197845jab; Fri, 27 Aug 2021 05:07:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxMHioZ2rQXTk3z0+Yq1ow76UuWl/Y9riSllhZfdwPQrQH3d9C3ab5H3VSOh3fvynxLwjcc X-Received: by 2002:a50:da47:: with SMTP id a7mr9335645edk.315.1630066027251; Fri, 27 Aug 2021 05:07:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066027; cv=none; d=google.com; s=arc-20160816; b=tWUBAH/XrDIfb3Nsv/beoXoxqe0B/jW3J4G+yh2nz10hoDuGLYxKl0Tn7o+n/XDTm7 t6ixb5LtcNyszpI6klLHpL6SRFpaehkbC+vbmIt2SHMsUo1Cup+Dn8PDaEHK5Ds03uOQ 7xgNyjuhnTVYJnc6u0buX3NUH/5UgLykmveL0jqXIm3i/kbAKYMCE6lVfgtyymfScxYf RDNRmnLSLSd2BVcftQ+7CQGdEmreMDWaOiFWPoS9XHTmWn8/VqBPkXGmzYX4oQMyNlFJ gWMSILcIwWTDdafcDPO4/oWGBXyuottPUI1elh3et/xR2KjAxBRj/FkHYvyIqktWnvRh qw4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=+QQIwdl/rc4+6A4Yg5pwiWFvchHx3Yrw6P2IOc7/vF0=; b=ut99WPHYQRzYnmXGT0nbZaG2Ti4PCvkfB9/4EVKaMYB9qfsB6ZlgAGgDxv9/IVWDdz G6N5oPTnsSho3oy0ALte1ssBDTzlAsRZQWqYWuTEg0CfeimYHzb1uLS7F9ILnzCcIXys jtXEX3IDwf6+EtQULIhdFQi31N0tgqBJBarm/hptpMFK/vxunroKDe+PJwsFkRMjkoOn 6TdPnLQP9qHMiYCYAW4avPibQydeXjDqIZdZG8IX8W+9scNG1Hf21IXfTeY8Q20Y7L40 LfQpSxaH0usrUq+c2Fnu7mQ31D96vv6gDZj7ewWjGL2SlW8YPKjr5NxllTCmN2qlrcb+ zReg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id du10si5674681ejc.547.2021.08.27.05.07.07 for ; Fri, 27 Aug 2021 05:07:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245169AbhH0MHy (ORCPT ); Fri, 27 Aug 2021 08:07:54 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3702 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245162AbhH0MHx (ORCPT ); Fri, 27 Aug 2021 08:07:53 -0400 Received: from fraeml739-chm.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1t0ZZTz67wp5; Fri, 27 Aug 2021 20:05:38 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml739-chm.china.huawei.com (10.206.15.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:02 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:00 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 07/13] blk-mq: Pass driver tags to blk_mq_clear_rq_mapping() Date: Fri, 27 Aug 2021 20:01:58 +0800 Message-ID: <1630065724-69146-8-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Function blk_mq_clear_rq_mapping() will be used for shared sbitmap tags in future, so pass a driver tags pointer instead of the tagset container and HW queue index. Signed-off-by: John Garry --- block/blk-mq.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index b51d59ec7d7a..556c203e6b7c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2310,10 +2310,9 @@ static size_t order_to_size(unsigned int order) } /* called before freeing request pool in @tags */ -static void blk_mq_clear_rq_mapping(struct blk_mq_tag_set *set, - struct blk_mq_tags *tags, unsigned int hctx_idx) +static void blk_mq_clear_rq_mapping(struct blk_mq_tags *drv_tags, + struct blk_mq_tags *tags) { - struct blk_mq_tags *drv_tags = set->tags[hctx_idx]; struct page *page; unsigned long flags; @@ -2322,7 +2321,7 @@ static void blk_mq_clear_rq_mapping(struct blk_mq_tag_set *set, unsigned long end = start + order_to_size(page->private); int i; - for (i = 0; i < set->queue_depth; i++) { + for (i = 0; i < drv_tags->nr_tags; i++) { struct request *rq = drv_tags->rqs[i]; unsigned long rq_addr = (unsigned long)rq; @@ -2346,8 +2345,11 @@ static void blk_mq_clear_rq_mapping(struct blk_mq_tag_set *set, void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, unsigned int hctx_idx) { + struct blk_mq_tags *drv_tags; struct page *page; + drv_tags = set->tags[hctx_idx]; + if (tags->static_rqs && set->ops->exit_request) { int i; @@ -2361,7 +2363,7 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, } } - blk_mq_clear_rq_mapping(set, tags, hctx_idx); + blk_mq_clear_rq_mapping(drv_tags, tags); while (!list_empty(&tags->page_list)) { page = list_first_entry(&tags->page_list, struct page, lru); From patchwork Fri Aug 27 12:01:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503391 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1197940jab; Fri, 27 Aug 2021 05:07:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwpX6T82uh3Hvb7iWoYm9NqNgaybruDHBaK1GaU4ag39zlN2MgSGJMTB+SJ8fbOqYnYs3Oe X-Received: by 2002:aa7:dc57:: with SMTP id g23mr382289edu.71.1630066032793; Fri, 27 Aug 2021 05:07:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066032; cv=none; d=google.com; s=arc-20160816; b=RWuYbdZMQ/urSWeNHJccnbpL2A6npZhd16WUKKnYtjOT4ihs8Ypk7CzuEYNOwyo60P CT9bspiy5nImAvhB4dg69AdN+APmUDg5Jj0HcYz/ZN13u4oL6oATpAU84wBxcu6GosQ3 8pqjMLlr/zNOQ/AWXyXHN51OIGHxSXLIbG1sbA6W7Vkk27891PtacyhJ1og5ZDlUA3Fv VZHCk+GfjgmDIeGD5OmYFPziKONsdt204JyY3ykKpYzSCa/YO+qMArqHLGs/LOeeVhBh yJDuZlLyI1NPt07F6jmq1y6+OReUMqfYQwAutXVUDUz9ZHZaB7TNMNsugYx0U57q7DJL WegQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=z14+AzHwEI+Tvp4MnHyb0BpHtPLiSMmCtOo1OJZGLjo=; b=JVJyls9fnsFuTvY/v5S4BkwBOBkoZVzf0bZ2yevWV527cOVn/rOXaHp6eD0ei8YEWJ yJUDsUYcbCRihQRoNTm1CTQExyG+ytzFn2usi4S7AfX+RthJxvAqg6UfO07w+eGgsiPy 5k+VUVr4VbRjMl2vHtnK6p9mRpAkKTTnFS8xwd3EbtgJYSlJ8aVeL/WcKlOlzKTMRZdT EKNqzzKaMHRZcfopsgyEMmF/JMykBAE4vo0CeZHs5y3OzLFIFIgLUsLQwdWgnSI4XXog /v2WaUejDBAsRG00uPR3CyziWCjTLbSQk5Iqh5LhJ4YWNGmFnvUfjrHQTbqsP2IzcOrq CKMg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id e29si2098515ejl.212.2021.08.27.05.07.12 for ; Fri, 27 Aug 2021 05:07:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245196AbhH0MIA (ORCPT ); Fri, 27 Aug 2021 08:08:00 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3703 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245171AbhH0MHz (ORCPT ); Fri, 27 Aug 2021 08:07:55 -0400 Received: from fraeml740-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1w2XDmz67wgc; Fri, 27 Aug 2021 20:05:40 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml740-chm.china.huawei.com (10.206.15.221) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:05 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:02 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 08/13] blk-mq: Don't clear driver tags own mapping Date: Fri, 27 Aug 2021 20:01:59 +0800 Message-ID: <1630065724-69146-9-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Function blk_mq_clear_rq_mapping() is required to clear the sched tags mappings in driver tags rqs[]. But there is no need for a driver tags to clear its own mapping, so skip clearing the mapping in this scenario. Signed-off-by: John Garry --- block/blk-mq.c | 4 ++++ 1 file changed, 4 insertions(+) -- 2.26.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 556c203e6b7c..09380274d2d6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2316,6 +2316,10 @@ static void blk_mq_clear_rq_mapping(struct blk_mq_tags *drv_tags, struct page *page; unsigned long flags; + /* There is no need to clear a driver tags own mapping */ + if (drv_tags == tags) + return; + list_for_each_entry(page, &tags->page_list, lru) { unsigned long start = (unsigned long)page_address(page); unsigned long end = start + order_to_size(page->private); From patchwork Fri Aug 27 12:02:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503395 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1198185jab; Fri, 27 Aug 2021 05:07:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy+dbqZ7XfRUsislsCQb8XAgqihHYh4O1WqGB5//p8Iw8UBxyTvXjOrfiJpjyCfI/giYb/b X-Received: by 2002:a17:906:a0c9:: with SMTP id bh9mr9466162ejb.51.1630066043891; Fri, 27 Aug 2021 05:07:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066043; cv=none; d=google.com; s=arc-20160816; b=GI5SbRjCVwMvxgYiDrjr7e22rwKOfBtidFXX2hgVl0eKv998dPWAcwE+bxIj4TuMWX /IFB/fWvUi2Y49auQBlekxMaJn9mZeuaXEgIY7pSMdIEt73yRJO77A5NHkazwY2MeQia Qt1UCd8hb8PAMdoAThLOJkgpmNrmZeW/UKOrYghEIODxUDJfZuaykrs2pPWNhD+2zMxD 9JBDZk8+9I5mIuLbB2ylNN13n41gYbM8m99P8kp9/E3UDs3/W1OAPsbL7Z23n8yYlcob +j5T856W7mMYVlGjDolMi4sobZS4QnL6gEf+ZRNIqR9M3+bCoVrRpaq1nLogMsCzNPbT ldQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=w47oROLbeKgSQZQtaGN6VyGUZGxKnmAOt7WociPMhbI=; b=p+Fl90/ghCnPFgsoLbetzLr5uA1R2Y2cGSCmlC1X5uM1XwGQlVk/wvCgAMFbYlSzj2 t3aO2Ok7MTaUP8h0SbZnKJRw1Vaq/x2J3JJOFurCFiSxj31WDKCvgiHo+cliswbuEmgh f9hlmngW9WxPcNQY9DuSBHfV8qG75DJw++aOMd8Y6jbNsPYZ6FM0Tkg3KDN2+xwrVfIL nM9OajQfVtmQ8m7hYYwo8M6DJq0Od7hWi8y9SaYArzSK9MeN3Ujn20uRsbGbh+s2QCMF WVSerZdsb/lSX75VC47V8314AvCWr80tYKEc4I+dXmHR+GrV2wCkTE1ItARARTMbN+9R kiww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a15si6560684edr.429.2021.08.27.05.07.23 for ; Fri, 27 Aug 2021 05:07:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245209AbhH0MIK (ORCPT ); Fri, 27 Aug 2021 08:08:10 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3704 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245172AbhH0MH5 (ORCPT ); Fri, 27 Aug 2021 08:07:57 -0400 Received: from fraeml738-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1y1qlfz67svh; Fri, 27 Aug 2021 20:05:42 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml738-chm.china.huawei.com (10.206.15.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:07 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:04 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 09/13] blk-mq: Add blk_mq_tag_update_sched_shared_sbitmap() Date: Fri, 27 Aug 2021 20:02:00 +0800 Message-ID: <1630065724-69146-10-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Put the functionality to update the sched shared sbitmap size in a common function. Since the same formula is always used to resize, and it can be got from the request queue argument, so just pass the request queue pointer. Signed-off-by: John Garry Reviewed-by: Ming Lei --- block/blk-mq-sched.c | 3 +-- block/blk-mq-tag.c | 6 ++++++ block/blk-mq-tag.h | 1 + block/blk-mq.c | 3 +-- 4 files changed, 9 insertions(+), 4 deletions(-) -- 2.26.2 diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 3ab26154f0ea..a3b5a5399bc8 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -575,8 +575,7 @@ static int blk_mq_init_sched_shared_sbitmap(struct request_queue *queue) &queue->sched_breserved_tags; } - sbitmap_queue_resize(&queue->sched_bitmap_tags, - queue->nr_requests - set->reserved_tags); + blk_mq_tag_update_sched_shared_sbitmap(queue); return 0; } diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 86f87346232a..5f06ad6efc8f 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -634,6 +634,12 @@ void blk_mq_tag_resize_shared_sbitmap(struct blk_mq_tag_set *set, unsigned int s sbitmap_queue_resize(&set->__bitmap_tags, size - set->reserved_tags); } +void blk_mq_tag_update_sched_shared_sbitmap(struct request_queue *q) +{ + sbitmap_queue_resize(&q->sched_bitmap_tags, + q->nr_requests - q->tag_set->reserved_tags); +} + /** * blk_mq_unique_tag() - return a tag that is unique queue-wide * @rq: request for which to compute a unique tag diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 8ed55af08427..88f3c6485543 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -48,6 +48,7 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, unsigned int depth, bool can_grow); extern void blk_mq_tag_resize_shared_sbitmap(struct blk_mq_tag_set *set, unsigned int size); +extern void blk_mq_tag_update_sched_shared_sbitmap(struct request_queue *q); extern void blk_mq_tag_wakeup_all(struct blk_mq_tags *tags, bool); void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn, diff --git a/block/blk-mq.c b/block/blk-mq.c index 09380274d2d6..d82d477c9c9d 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3649,8 +3649,7 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) q->nr_requests = nr; if (blk_mq_is_sbitmap_shared(set->flags)) { if (q->elevator) - sbitmap_queue_resize(&q->sched_bitmap_tags, - nr - set->reserved_tags); + blk_mq_tag_update_sched_shared_sbitmap(q); else blk_mq_tag_resize_shared_sbitmap(set, nr); } From patchwork Fri Aug 27 12:02:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503394 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1198100jab; Fri, 27 Aug 2021 05:07:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxiF9f0hn+Qy7VMNn++q9dAfdIJc0r6gZBHCtmFwwPhXleBYLzm1bWMEi/cXuUinDv/XZsR X-Received: by 2002:a19:c315:: with SMTP id t21mr6216142lff.107.1630066042598; Fri, 27 Aug 2021 05:07:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066042; cv=none; d=google.com; s=arc-20160816; b=C9p4VufQ98ZDiIm0FR3R0ygwz73Kw6t2teca3/11xPhmiVBHgL306TRV2JVE04i/lW 32VWiJTkcEDIOH3DLLbDrtfAfHAvMiwfEUZtHl1hGZlsUNyX6Dk0dgoDQqvjl9SjFY+e l4pr6+fzSio5xwwPqJS/U0+oFjPT3eBuv1fNzbpeuOPwKT+TT/uyC6CY2QWphRaFzS8G HS3PvUDrhynl83r5Vg4GeoDa2KouBmSv1a95sykxJdplbcIoSGaudNskXmmO3VAGGfm9 f0+1uoJSLiLwLdeDuFZHZizAoB8aUwOVK4NckGH7CoygK6nMBtWrCR13iBOCpRG2BAI3 qCrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=5ra+UKfcBfp3AbJURPmiJQaN6g8eZTgomVfLlsnNns0=; b=nOmVe/t2OPc06qDg0H0/XzymyF9+JVjjYgrtOnzgZJxlWfl+MMftFS5EnHLVoA9w6H QlRYP6FQTUQk7y/lnAK+50Uu4o2GqEp37CThhKg+yooRwY1Q46lXIYSeDXrNbHbAHXAy TwKEnX0Ww4VrtSGemOZPFTzE/A6OwB8ZypjfrFOSQmI01hj06RU+4MNUKAiZzO678dKC GH7sO/RoG/VZlocKm8BF5U/QYVB8oDyvc6dnsUNlDWFdlXtThObrq6sS2S94uKasM7g9 vFo6PsrHnuX7KIhrbfYCHQB6fAaizxU8dR1M7ALBgLWyXMzqdeB0+kNkVY5DUVM4eTG0 37EA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a15si6560684edr.429.2021.08.27.05.07.22 for ; Fri, 27 Aug 2021 05:07:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245205AbhH0MIJ (ORCPT ); Fri, 27 Aug 2021 08:08:09 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3705 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245190AbhH0MH7 (ORCPT ); Fri, 27 Aug 2021 08:07:59 -0400 Received: from fraeml736-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz1z16W2z67gYW; Fri, 27 Aug 2021 20:05:43 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml736-chm.china.huawei.com (10.206.15.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:09 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:06 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 10/13] blk-mq: Add blk_mq_alloc_map_and_rqs() Date: Fri, 27 Aug 2021 20:02:01 +0800 Message-ID: <1630065724-69146-11-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add a function to combine allocating tags and the associated requests, and factor out common patterns to use this new function. Some functions only call blk_mq_alloc_map_and_rqs() now, but more functionality will be added later to those functions. Also make blk_mq_alloc_rq_map() and blk_mq_alloc_rqs() static since they are only used in blk-mq.c, and finally rename some functions for conciseness and consistency with other function names: - __blk_mq_alloc_map_and_{request -> rqs}() - blk_mq_alloc_{map_and_requests -> set_map_and_rqs}() Suggested-by: Ming Lei Signed-off-by: John Garry --- block/blk-mq-sched.c | 15 +++-------- block/blk-mq-tag.c | 9 +------ block/blk-mq.c | 62 +++++++++++++++++++++++++------------------- block/blk-mq.h | 9 ++----- 4 files changed, 42 insertions(+), 53 deletions(-) -- 2.26.2 diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index a3b5a5399bc8..17752f39e144 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -519,21 +519,12 @@ static int blk_mq_sched_alloc_map_and_rqs(struct request_queue *q, struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) { - struct blk_mq_tag_set *set = q->tag_set; - int ret; + hctx->sched_tags = blk_mq_alloc_map_and_rqs(q->tag_set, hctx_idx, + q->nr_requests); - hctx->sched_tags = blk_mq_alloc_rq_map(set, hctx_idx, q->nr_requests, - set->reserved_tags, set->flags); if (!hctx->sched_tags) return -ENOMEM; - - ret = blk_mq_alloc_rqs(set, hctx->sched_tags, hctx_idx, q->nr_requests); - if (ret) { - blk_mq_free_rq_map(hctx->sched_tags, set->flags); - hctx->sched_tags = NULL; - } - - return ret; + return 0; } /* called in queue's release handler, tagset has gone away */ diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 5f06ad6efc8f..d0b5e52be3c8 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -592,7 +592,6 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, if (tdepth > tags->nr_tags) { struct blk_mq_tag_set *set = hctx->queue->tag_set; struct blk_mq_tags *new; - bool ret; if (!can_grow) return -EINVAL; @@ -604,15 +603,9 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, if (tdepth > MAX_SCHED_RQ) return -EINVAL; - new = blk_mq_alloc_rq_map(set, hctx->queue_num, tdepth, - tags->nr_reserved_tags, set->flags); + new = blk_mq_alloc_map_and_rqs(set, hctx->queue_num, tdepth); if (!new) return -ENOMEM; - ret = blk_mq_alloc_rqs(set, new, hctx->queue_num, tdepth); - if (ret) { - blk_mq_free_rq_map(new, set->flags); - return -ENOMEM; - } blk_mq_free_rqs(set, *tagsptr, hctx->queue_num); blk_mq_free_rq_map(*tagsptr, set->flags); diff --git a/block/blk-mq.c b/block/blk-mq.c index d82d477c9c9d..6759914823e3 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2391,11 +2391,11 @@ void blk_mq_free_rq_map(struct blk_mq_tags *tags, unsigned int flags) blk_mq_free_tags(tags, flags); } -struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, - unsigned int hctx_idx, - unsigned int nr_tags, - unsigned int reserved_tags, - unsigned int flags) +static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, + unsigned int hctx_idx, + unsigned int nr_tags, + unsigned int reserved_tags, + unsigned int flags) { struct blk_mq_tags *tags; int node; @@ -2443,8 +2443,9 @@ static int blk_mq_init_request(struct blk_mq_tag_set *set, struct request *rq, return 0; } -int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, - unsigned int hctx_idx, unsigned int depth) +static int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, + struct blk_mq_tags *tags, + unsigned int hctx_idx, unsigned int depth) { unsigned int i, j, entries_per_page, max_order = 4; size_t rq_size, left; @@ -2855,25 +2856,34 @@ static void blk_mq_init_cpu_queues(struct request_queue *q, } } -static bool __blk_mq_alloc_map_and_request(struct blk_mq_tag_set *set, - int hctx_idx) +struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, + unsigned int hctx_idx, + unsigned int depth) { - unsigned int flags = set->flags; - int ret = 0; + struct blk_mq_tags *tags; + int ret; - set->tags[hctx_idx] = blk_mq_alloc_rq_map(set, hctx_idx, - set->queue_depth, set->reserved_tags, flags); - if (!set->tags[hctx_idx]) - return false; + tags = blk_mq_alloc_rq_map(set, hctx_idx, depth, set->reserved_tags, + set->flags); + if (!tags) + return NULL; - ret = blk_mq_alloc_rqs(set, set->tags[hctx_idx], hctx_idx, - set->queue_depth); - if (!ret) - return true; + ret = blk_mq_alloc_rqs(set, tags, hctx_idx, depth); + if (ret) { + blk_mq_free_rq_map(tags, set->flags); + return NULL; + } - blk_mq_free_rq_map(set->tags[hctx_idx], flags); - set->tags[hctx_idx] = NULL; - return false; + return tags; +} + +static bool __blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, + int hctx_idx) +{ + set->tags[hctx_idx] = blk_mq_alloc_map_and_rqs(set, hctx_idx, + set->queue_depth); + + return set->tags[hctx_idx]; } static void blk_mq_free_map_and_requests(struct blk_mq_tag_set *set, @@ -2918,7 +2928,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) hctx_idx = set->map[j].mq_map[i]; /* unmapped hw queue can be remapped after CPU topo changed */ if (!set->tags[hctx_idx] && - !__blk_mq_alloc_map_and_request(set, hctx_idx)) { + !__blk_mq_alloc_map_and_rqs(set, hctx_idx)) { /* * If tags initialization fail for some hctx, * that hctx won't be brought online. In this @@ -3351,7 +3361,7 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) int i; for (i = 0; i < set->nr_hw_queues; i++) { - if (!__blk_mq_alloc_map_and_request(set, i)) + if (!__blk_mq_alloc_map_and_rqs(set, i)) goto out_unwind; cond_resched(); } @@ -3370,7 +3380,7 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) * may reduce the depth asked for, if memory is tight. set->queue_depth * will be updated to reflect the allocated depth. */ -static int blk_mq_alloc_map_and_requests(struct blk_mq_tag_set *set) +static int blk_mq_alloc_set_map_and_rqs(struct blk_mq_tag_set *set) { unsigned int depth; int err; @@ -3536,7 +3546,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) if (ret) goto out_free_mq_map; - ret = blk_mq_alloc_map_and_requests(set); + ret = blk_mq_alloc_set_map_and_rqs(set); if (ret) goto out_free_mq_map; diff --git a/block/blk-mq.h b/block/blk-mq.h index d08779f77a26..83585a344568 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -55,13 +55,8 @@ void blk_mq_put_rq_ref(struct request *rq); void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, unsigned int hctx_idx); void blk_mq_free_rq_map(struct blk_mq_tags *tags, unsigned int flags); -struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, - unsigned int hctx_idx, - unsigned int nr_tags, - unsigned int reserved_tags, - unsigned int flags); -int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, - unsigned int hctx_idx, unsigned int depth); +struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, + unsigned int hctx_idx, unsigned int depth); /* * Internal helpers for request insertion into sw queues From patchwork Fri Aug 27 12:02:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503392 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1198050jab; Fri, 27 Aug 2021 05:07:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwPDgHC9Ps4Uthn71clVMcwsCeW2hxZ4G5Q6Pqv9HJmvej/K0AsrlhsX6fOrH7ZKPM4p/WL X-Received: by 2002:a2e:84c7:: with SMTP id q7mr7319833ljh.61.1630066039703; Fri, 27 Aug 2021 05:07:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066039; cv=none; d=google.com; s=arc-20160816; b=EV6lBDbFHdt3PE5inksO1yBTAPBwiKyQRqCE+wYw7ej+7SWDkWyzuoRCVyo68ahMS5 FepJSMnPnI0iz0XCY5ZwzPBLobMq1qYZz28o7CYtB50ZiZ3GITdU5MtVfSV2kbzfg6sz OUbfFsz9zwJLK5noeyZAQLJ30aSsnFDB4v+QEtHujUUBu63OkxNygNbjltYrG3njrVRe Q7GO8jnud3FYkMBG9+2nB5KAApeVDH9zSH3Dn3aPQWP1yAulRUMszXSeCc2gpL5DjrUx hTBeAvBNZu3dBPYp8WPZYaVUJAhHDwOhsWQxRoDcIX4hHGAlxtNGtqDhVo2pvR7ZvQeZ 9Bbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=sHsjOOlNksPLP3xvp3eTeszKWPHaSxB0oQC0pLmilbM=; b=C4Y5HrwTohmJXODlf7ZYMwjPu2NBQS4VL9HArZW9mUtVdn6LIRrm9KDNEo9JfxvJgg lkj2R/RqhKFVHz+LlrmYUqu+mLVZpMYe4MWaUjWYNM2LrRfsKgShEmUGLUVyG6sNk2iR Wkx8SXRoFgFRXeiSdjBnXmTVRLppFoQwp0ZCgrw5uiPXXaqj4LKy8ETP5XNSfMJPN6Ov P5YVirh1AZ8Kap4bRuDd+6ITstjK7wdeMGg1Xhb7cxqHB7XI2fXHseMkbN0ybNDALYqB 72yeZoizLpOEOdedLwHH0FMkKErYGPe8tLSq8KI3z8qwZJ+oNc8gvlwXoAbHguQToukH 5bzQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a15si6560684edr.429.2021.08.27.05.07.19 for ; Fri, 27 Aug 2021 05:07:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245174AbhH0MIH (ORCPT ); Fri, 27 Aug 2021 08:08:07 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3706 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245171AbhH0MIC (ORCPT ); Fri, 27 Aug 2021 08:08:02 -0400 Received: from fraeml737-chm.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz2B2NVcz67gLl; Fri, 27 Aug 2021 20:05:54 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml737-chm.china.huawei.com (10.206.15.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:11 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:09 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 11/13] blk-mq: Refactor and rename blk_mq_free_map_and_{requests->rqs}() Date: Fri, 27 Aug 2021 20:02:02 +0800 Message-ID: <1630065724-69146-12-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Refactor blk_mq_free_map_and_requests() such that it can be used at many sites at which the tag map and rqs are freed. Also rename to blk_mq_free_map_and_rqs(), which is shorter and matches the alloc equivalent. Suggested-by: Ming Lei Signed-off-by: John Garry --- block/blk-mq-tag.c | 3 +-- block/blk-mq.c | 40 ++++++++++++++++++++++++---------------- block/blk-mq.h | 4 +++- 3 files changed, 28 insertions(+), 19 deletions(-) -- 2.26.2 diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index d0b5e52be3c8..fcc33581ce5e 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -607,8 +607,7 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, if (!new) return -ENOMEM; - blk_mq_free_rqs(set, *tagsptr, hctx->queue_num); - blk_mq_free_rq_map(*tagsptr, set->flags); + blk_mq_free_map_and_rqs(set, *tagsptr, hctx->queue_num); *tagsptr = new; } else { /* diff --git a/block/blk-mq.c b/block/blk-mq.c index 6759914823e3..af6bd375b6a5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2886,15 +2886,15 @@ static bool __blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, return set->tags[hctx_idx]; } -static void blk_mq_free_map_and_requests(struct blk_mq_tag_set *set, - unsigned int hctx_idx) +void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, + struct blk_mq_tags *tags, + unsigned int hctx_idx) { unsigned int flags = set->flags; - if (set->tags && set->tags[hctx_idx]) { - blk_mq_free_rqs(set, set->tags[hctx_idx], hctx_idx); - blk_mq_free_rq_map(set->tags[hctx_idx], flags); - set->tags[hctx_idx] = NULL; + if (tags) { + blk_mq_free_rqs(set, tags, hctx_idx); + blk_mq_free_rq_map(tags, flags); } } @@ -2975,8 +2975,10 @@ static void blk_mq_map_swqueue(struct request_queue *q) * fallback in case of a new remap fails * allocation */ - if (i && set->tags[i]) - blk_mq_free_map_and_requests(set, i); + if (i && set->tags[i]) { + blk_mq_free_map_and_rqs(set, set->tags[i], i); + set->tags[i] = NULL; + } hctx->tags = NULL; continue; @@ -3270,8 +3272,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, struct blk_mq_hw_ctx *hctx = hctxs[j]; if (hctx) { - if (hctx->tags) - blk_mq_free_map_and_requests(set, j); + blk_mq_free_map_and_rqs(set, set->tags[j], j); + set->tags[j] = NULL; blk_mq_exit_hctx(q, set, hctx, j); hctxs[j] = NULL; } @@ -3369,8 +3371,10 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) return 0; out_unwind: - while (--i >= 0) - blk_mq_free_map_and_requests(set, i); + while (--i >= 0) { + blk_mq_free_map_and_rqs(set, set->tags[i], i); + set->tags[i] = NULL; + } return -ENOMEM; } @@ -3565,8 +3569,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) return 0; out_free_mq_rq_maps: - for (i = 0; i < set->nr_hw_queues; i++) - blk_mq_free_map_and_requests(set, i); + for (i = 0; i < set->nr_hw_queues; i++) { + blk_mq_free_map_and_rqs(set, set->tags[i], i); + set->tags[i] = NULL; + } out_free_mq_map: for (i = 0; i < set->nr_maps; i++) { kfree(set->map[i].mq_map); @@ -3598,8 +3604,10 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) { int i, j; - for (i = 0; i < set->nr_hw_queues; i++) - blk_mq_free_map_and_requests(set, i); + for (i = 0; i < set->nr_hw_queues; i++) { + blk_mq_free_map_and_rqs(set, set->tags[i], i); + set->tags[i] = NULL; + } if (blk_mq_is_sbitmap_shared(set->flags)) blk_mq_exit_shared_sbitmap(set); diff --git a/block/blk-mq.h b/block/blk-mq.h index 83585a344568..bcb0ca89d37a 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -57,7 +57,9 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, void blk_mq_free_rq_map(struct blk_mq_tags *tags, unsigned int flags); struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, unsigned int hctx_idx, unsigned int depth); - +void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, + struct blk_mq_tags *tags, + unsigned int hctx_idx); /* * Internal helpers for request insertion into sw queues */ From patchwork Fri Aug 27 12:02:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503393 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1198062jab; Fri, 27 Aug 2021 05:07:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzXkwklB1psR2NRv69JHUyppmKn9g/efSN5az6bmbceHMpXjY0MN55i4yYKT3IGcyi4DkpN X-Received: by 2002:a17:906:4413:: with SMTP id x19mr9723137ejo.234.1630066040218; Fri, 27 Aug 2021 05:07:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066040; cv=none; d=google.com; s=arc-20160816; b=ibL4kEL4DT8pcNwdgueaaZIEXBHvN4VsaoOrf8gz/YKSA5GDEljQiXw2SAwer8HFt5 TVv0VDUzBh6FOmeuZShiz7CReKft8zE7HW7oPOvAy0ar3A3iuJ6VgGmEiGYkhi/y5DvC 3y7YW5ckCIhEXI+SIYN3UAOIdA8lYMeINdSqeuDJ6ToouWXqBSdJw1PJinwBs+W+g1hX xnEx0IFsBxpGDUUDDS7haK9F0iaD8zv0MM0jrGTf+HwpqVNL6mWGxeUhdZb2yUkeDQLt pp2ORIbfBxyGOg10vL/wvRdjNqDw/Bg4/BpwSaoNXBTR/uPdlYvd3G9WpGZS/eNZ9DwT Apsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=DdXn2lLATdOfzoviTsUErHjuG9k5NR1usAuGYgzTDdw=; b=H/FoP0bwcSuljhcwUzWyhefI/JIJcnFkRqvhvV2zNKabJM2qQTT4k/R+65lZlvBWsw ty3+oGdH/IvJxApA9fdDXQCPAXQ7CKQJVMCb9Eurp9Vg0WXfDfbfcvWRmvfiKhnYRPvX +IsaZMSQZztQnu3+Ii7Wb5lZf4TIFSPvYGavJejfVdnFKfs8fjq+mPmniH20lG7Xk21o Gl+1w9trpTxOZwpFAX9kwKh8Xe/TJPbJc6anhGVhB6wZ+hR9Dd0xOMCA+/S3y6d0i4iA CXVIQ6GDrqdESPfqRklbsoyXiCpOC/yY9SYTDELpY7YXCsMXEyL/4VGeFy1gJjqBLPTH 8S9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a15si6560684edr.429.2021.08.27.05.07.20 for ; Fri, 27 Aug 2021 05:07:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245192AbhH0MIH (ORCPT ); Fri, 27 Aug 2021 08:08:07 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3707 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245158AbhH0MIE (ORCPT ); Fri, 27 Aug 2021 08:08:04 -0400 Received: from fraeml735-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz2D2c7Xz67c4B; Fri, 27 Aug 2021 20:05:56 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml735-chm.china.huawei.com (10.206.15.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:13 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:11 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 12/13] blk-mq: Use shared tags for shared sbitmap support Date: Fri, 27 Aug 2021 20:02:03 +0800 Message-ID: <1630065724-69146-13-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Currently we use separate sbitmap pairs and active_queues atomic_t for shared sbitmap support. However a full sets of static requests are used per HW queue, which is quite wasteful, considering that the total number of requests usable at any given time across all HW queues is limited by the shared sbitmap depth. As such, it is considerably more memory efficient in the case of shared sbitmap to allocate a set of static rqs per tag set or request queue, and not per HW queue. So replace the sbitmap pairs and active_queues atomic_t with a shared tags per tagset and request queue, which will hold a set of shared static rqs. Since there is now no valid HW queue index to be passed to the blk_mq_ops .init and .exit_request callbacks, pass an invalid index token. This changes the semantics of the APIs, such that the callback would need to validate the HW queue index before using it. Currently no user of shared sbitmap actually uses the HW queue index (as would be expected). Continue to use term "shared sbitmap" for now, as the meaning is known. Signed-off-by: John Garry --- block/blk-mq-sched.c | 84 +++++++++++++++++++------------------- block/blk-mq-tag.c | 61 +++++++++------------------- block/blk-mq-tag.h | 6 +-- block/blk-mq.c | 92 ++++++++++++++++++++++++------------------ block/blk-mq.h | 5 ++- include/linux/blk-mq.h | 15 ++++--- include/linux/blkdev.h | 3 +- 7 files changed, 127 insertions(+), 139 deletions(-) -- 2.26.2 diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 17752f39e144..428da4949d80 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -519,6 +519,11 @@ static int blk_mq_sched_alloc_map_and_rqs(struct request_queue *q, struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) { + if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) { + hctx->sched_tags = q->shared_sbitmap_tags; + return 0; + } + hctx->sched_tags = blk_mq_alloc_map_and_rqs(q->tag_set, hctx_idx, q->nr_requests); @@ -527,61 +532,54 @@ static int blk_mq_sched_alloc_map_and_rqs(struct request_queue *q, return 0; } +static void blk_mq_exit_sched_shared_sbitmap(struct request_queue *queue) +{ + blk_mq_free_rq_map(queue->shared_sbitmap_tags); + queue->shared_sbitmap_tags = NULL; +} + /* called in queue's release handler, tagset has gone away */ -static void blk_mq_sched_tags_teardown(struct request_queue *q) +static void blk_mq_sched_tags_teardown(struct request_queue *q, unsigned int flags) { struct blk_mq_hw_ctx *hctx; int i; queue_for_each_hw_ctx(q, hctx, i) { if (hctx->sched_tags) { - blk_mq_free_rq_map(hctx->sched_tags, hctx->flags); + if (!blk_mq_is_sbitmap_shared(q->tag_set->flags)) + blk_mq_free_rq_map(hctx->sched_tags); hctx->sched_tags = NULL; } } + + if (blk_mq_is_sbitmap_shared(flags)) + blk_mq_exit_sched_shared_sbitmap(q); } static int blk_mq_init_sched_shared_sbitmap(struct request_queue *queue) { struct blk_mq_tag_set *set = queue->tag_set; - int alloc_policy = BLK_MQ_FLAG_TO_ALLOC_POLICY(set->flags); - struct blk_mq_hw_ctx *hctx; - int ret, i; /* * Set initial depth at max so that we don't need to reallocate for * updating nr_requests. */ - ret = blk_mq_init_bitmaps(&queue->sched_bitmap_tags, - &queue->sched_breserved_tags, - MAX_SCHED_RQ, set->reserved_tags, - set->numa_node, alloc_policy); - if (ret) - return ret; - - queue_for_each_hw_ctx(queue, hctx, i) { - hctx->sched_tags->bitmap_tags = - &queue->sched_bitmap_tags; - hctx->sched_tags->breserved_tags = - &queue->sched_breserved_tags; - } + queue->shared_sbitmap_tags = blk_mq_alloc_map_and_rqs(set, + BLK_MQ_NO_HCTX_IDX, + MAX_SCHED_RQ); + if (!queue->shared_sbitmap_tags) + return -ENOMEM; blk_mq_tag_update_sched_shared_sbitmap(queue); return 0; } -static void blk_mq_exit_sched_shared_sbitmap(struct request_queue *queue) -{ - sbitmap_queue_free(&queue->sched_bitmap_tags); - sbitmap_queue_free(&queue->sched_breserved_tags); -} - int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) { + unsigned int i, flags = q->tag_set->flags; struct blk_mq_hw_ctx *hctx; struct elevator_queue *eq; - unsigned int i; int ret; if (!e) { @@ -598,21 +596,21 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) q->nr_requests = 2 * min_t(unsigned int, q->tag_set->queue_depth, BLKDEV_DEFAULT_RQ); - queue_for_each_hw_ctx(q, hctx, i) { - ret = blk_mq_sched_alloc_map_and_rqs(q, hctx, i); + if (blk_mq_is_sbitmap_shared(flags)) { + ret = blk_mq_init_sched_shared_sbitmap(q); if (ret) - goto err_free_map_and_rqs; + return ret; } - if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) { - ret = blk_mq_init_sched_shared_sbitmap(q); + queue_for_each_hw_ctx(q, hctx, i) { + ret = blk_mq_sched_alloc_map_and_rqs(q, hctx, i); if (ret) goto err_free_map_and_rqs; } ret = e->ops.init_sched(q, e); if (ret) - goto err_free_sbitmap; + goto err_free_map_and_rqs; blk_mq_debugfs_register_sched(q); @@ -632,12 +630,10 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) return 0; -err_free_sbitmap: - if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) - blk_mq_exit_sched_shared_sbitmap(q); - blk_mq_sched_free_rqs(q); err_free_map_and_rqs: - blk_mq_sched_tags_teardown(q); + blk_mq_sched_free_rqs(q); + blk_mq_sched_tags_teardown(q, flags); + q->elevator = NULL; return ret; } @@ -651,9 +647,15 @@ void blk_mq_sched_free_rqs(struct request_queue *q) struct blk_mq_hw_ctx *hctx; int i; - queue_for_each_hw_ctx(q, hctx, i) { - if (hctx->sched_tags) - blk_mq_free_rqs(q->tag_set, hctx->sched_tags, i); + if (blk_mq_is_sbitmap_shared(q->tag_set->flags)) { + blk_mq_free_rqs(q->tag_set, q->shared_sbitmap_tags, + BLK_MQ_NO_HCTX_IDX); + } else { + queue_for_each_hw_ctx(q, hctx, i) { + if (hctx->sched_tags) + blk_mq_free_rqs(q->tag_set, + hctx->sched_tags, i); + } } } @@ -674,8 +676,6 @@ void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e) blk_mq_debugfs_unregister_sched(q); if (e->type->ops.exit_sched) e->type->ops.exit_sched(e); - blk_mq_sched_tags_teardown(q); - if (blk_mq_is_sbitmap_shared(flags)) - blk_mq_exit_sched_shared_sbitmap(q); + blk_mq_sched_tags_teardown(q, flags); q->elevator = NULL; } diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index fcc33581ce5e..29b93ea0ea27 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -27,10 +27,11 @@ bool __blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx) if (blk_mq_is_sbitmap_shared(hctx->flags)) { struct request_queue *q = hctx->queue; struct blk_mq_tag_set *set = q->tag_set; + struct blk_mq_tags *tags = set->shared_sbitmap_tags; if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags) && !test_and_set_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags)) - atomic_inc(&set->active_queues_shared_sbitmap); + atomic_inc(&tags->active_queues); } else { if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state) && !test_and_set_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) @@ -61,10 +62,12 @@ void __blk_mq_tag_idle(struct blk_mq_hw_ctx *hctx) struct blk_mq_tag_set *set = q->tag_set; if (blk_mq_is_sbitmap_shared(hctx->flags)) { + struct blk_mq_tags *tags = set->shared_sbitmap_tags; + if (!test_and_clear_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags)) return; - atomic_dec(&set->active_queues_shared_sbitmap); + atomic_dec(&tags->active_queues); } else { if (!test_and_clear_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) return; @@ -510,38 +513,10 @@ static int blk_mq_init_bitmap_tags(struct blk_mq_tags *tags, return 0; } -int blk_mq_init_shared_sbitmap(struct blk_mq_tag_set *set) -{ - int alloc_policy = BLK_MQ_FLAG_TO_ALLOC_POLICY(set->flags); - int i, ret; - - ret = blk_mq_init_bitmaps(&set->__bitmap_tags, &set->__breserved_tags, - set->queue_depth, set->reserved_tags, - set->numa_node, alloc_policy); - if (ret) - return ret; - - for (i = 0; i < set->nr_hw_queues; i++) { - struct blk_mq_tags *tags = set->tags[i]; - - tags->bitmap_tags = &set->__bitmap_tags; - tags->breserved_tags = &set->__breserved_tags; - } - - return 0; -} - -void blk_mq_exit_shared_sbitmap(struct blk_mq_tag_set *set) -{ - sbitmap_queue_free(&set->__bitmap_tags); - sbitmap_queue_free(&set->__breserved_tags); -} - struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, unsigned int reserved_tags, - int node, unsigned int flags) + int node, int alloc_policy) { - int alloc_policy = BLK_MQ_FLAG_TO_ALLOC_POLICY(flags); struct blk_mq_tags *tags; if (total_tags > BLK_MQ_TAG_MAX) { @@ -557,9 +532,6 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, tags->nr_reserved_tags = reserved_tags; spin_lock_init(&tags->lock); - if (blk_mq_is_sbitmap_shared(flags)) - return tags; - if (blk_mq_init_bitmap_tags(tags, node, alloc_policy) < 0) { kfree(tags); return NULL; @@ -567,12 +539,10 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, return tags; } -void blk_mq_free_tags(struct blk_mq_tags *tags, unsigned int flags) +void blk_mq_free_tags(struct blk_mq_tags *tags) { - if (!blk_mq_is_sbitmap_shared(flags)) { - sbitmap_queue_free(tags->bitmap_tags); - sbitmap_queue_free(tags->breserved_tags); - } + sbitmap_queue_free(tags->bitmap_tags); + sbitmap_queue_free(tags->breserved_tags); kfree(tags); } @@ -603,6 +573,13 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, if (tdepth > MAX_SCHED_RQ) return -EINVAL; + /* + * Only the sbitmap needs resizing since we allocated the max + * initially. + */ + if (blk_mq_is_sbitmap_shared(set->flags)) + return 0; + new = blk_mq_alloc_map_and_rqs(set, hctx->queue_num, tdepth); if (!new) return -ENOMEM; @@ -623,12 +600,14 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, void blk_mq_tag_resize_shared_sbitmap(struct blk_mq_tag_set *set, unsigned int size) { - sbitmap_queue_resize(&set->__bitmap_tags, size - set->reserved_tags); + struct blk_mq_tags *tags = set->shared_sbitmap_tags; + + sbitmap_queue_resize(&tags->__bitmap_tags, size - set->reserved_tags); } void blk_mq_tag_update_sched_shared_sbitmap(struct request_queue *q) { - sbitmap_queue_resize(&q->sched_bitmap_tags, + sbitmap_queue_resize(q->shared_sbitmap_tags->bitmap_tags, q->nr_requests - q->tag_set->reserved_tags); } diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 88f3c6485543..e433e39a9cfa 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -30,16 +30,14 @@ struct blk_mq_tags { extern struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags, unsigned int reserved_tags, - int node, unsigned int flags); -extern void blk_mq_free_tags(struct blk_mq_tags *tags, unsigned int flags); + int node, int alloc_policy); +extern void blk_mq_free_tags(struct blk_mq_tags *tags); extern int blk_mq_init_bitmaps(struct sbitmap_queue *bitmap_tags, struct sbitmap_queue *breserved_tags, unsigned int queue_depth, unsigned int reserved, int node, int alloc_policy); -extern int blk_mq_init_shared_sbitmap(struct blk_mq_tag_set *set); -extern void blk_mq_exit_shared_sbitmap(struct blk_mq_tag_set *set); extern unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data); extern void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, unsigned int tag); diff --git a/block/blk-mq.c b/block/blk-mq.c index af6bd375b6a5..958761c0b26b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2352,7 +2352,10 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, struct blk_mq_tags *drv_tags; struct page *page; - drv_tags = set->tags[hctx_idx]; + if (blk_mq_is_sbitmap_shared(set->flags)) + drv_tags = set->shared_sbitmap_tags; + else + drv_tags = set->tags[hctx_idx]; if (tags->static_rqs && set->ops->exit_request) { int i; @@ -2362,6 +2365,7 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, if (!rq) continue; + set->ops->exit_request(set, rq, hctx_idx); tags->static_rqs[i] = NULL; } @@ -2381,21 +2385,20 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, } } -void blk_mq_free_rq_map(struct blk_mq_tags *tags, unsigned int flags) +void blk_mq_free_rq_map(struct blk_mq_tags *tags) { kfree(tags->rqs); tags->rqs = NULL; kfree(tags->static_rqs); tags->static_rqs = NULL; - blk_mq_free_tags(tags, flags); + blk_mq_free_tags(tags); } static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, unsigned int hctx_idx, unsigned int nr_tags, - unsigned int reserved_tags, - unsigned int flags) + unsigned int reserved_tags) { struct blk_mq_tags *tags; int node; @@ -2404,7 +2407,8 @@ static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, if (node == NUMA_NO_NODE) node = set->numa_node; - tags = blk_mq_init_tags(nr_tags, reserved_tags, node, flags); + tags = blk_mq_init_tags(nr_tags, reserved_tags, node, + BLK_MQ_FLAG_TO_ALLOC_POLICY(set->flags)); if (!tags) return NULL; @@ -2412,7 +2416,7 @@ static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node); if (!tags->rqs) { - blk_mq_free_tags(tags, flags); + blk_mq_free_tags(tags); return NULL; } @@ -2421,7 +2425,7 @@ static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, node); if (!tags->static_rqs) { kfree(tags->rqs); - blk_mq_free_tags(tags, flags); + blk_mq_free_tags(tags); return NULL; } @@ -2863,14 +2867,13 @@ struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags; int ret; - tags = blk_mq_alloc_rq_map(set, hctx_idx, depth, set->reserved_tags, - set->flags); + tags = blk_mq_alloc_rq_map(set, hctx_idx, depth, set->reserved_tags); if (!tags) return NULL; ret = blk_mq_alloc_rqs(set, tags, hctx_idx, depth); if (ret) { - blk_mq_free_rq_map(tags, set->flags); + blk_mq_free_rq_map(tags); return NULL; } @@ -2880,6 +2883,12 @@ struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, static bool __blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, int hctx_idx) { + if (blk_mq_is_sbitmap_shared(set->flags)) { + set->tags[hctx_idx] = set->shared_sbitmap_tags; + + return true; + } + set->tags[hctx_idx] = blk_mq_alloc_map_and_rqs(set, hctx_idx, set->queue_depth); @@ -2890,14 +2899,22 @@ void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, unsigned int hctx_idx) { - unsigned int flags = set->flags; - if (tags) { blk_mq_free_rqs(set, tags, hctx_idx); - blk_mq_free_rq_map(tags, flags); + blk_mq_free_rq_map(tags); } } +static void __blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, + struct blk_mq_tags *tags, + unsigned int hctx_idx) +{ + if (blk_mq_is_sbitmap_shared(set->flags)) + return; + + blk_mq_free_map_and_rqs(set, tags, hctx_idx); +} + static void blk_mq_map_swqueue(struct request_queue *q) { unsigned int i, j, hctx_idx; @@ -2976,7 +2993,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) * allocation */ if (i && set->tags[i]) { - blk_mq_free_map_and_rqs(set, set->tags[i], i); + __blk_mq_free_map_and_rqs(set, set->tags[i], i); set->tags[i] = NULL; } @@ -3272,7 +3289,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, struct blk_mq_hw_ctx *hctx = hctxs[j]; if (hctx) { - blk_mq_free_map_and_rqs(set, set->tags[j], j); + __blk_mq_free_map_and_rqs(set, set->tags[j], j); set->tags[j] = NULL; blk_mq_exit_hctx(q, set, hctx, j); hctxs[j] = NULL; @@ -3362,6 +3379,14 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) { int i; + if (blk_mq_is_sbitmap_shared(set->flags)) { + set->shared_sbitmap_tags = blk_mq_alloc_map_and_rqs(set, + BLK_MQ_NO_HCTX_IDX, + set->queue_depth); + if (!set->shared_sbitmap_tags) + return -ENOMEM; + } + for (i = 0; i < set->nr_hw_queues; i++) { if (!__blk_mq_alloc_map_and_rqs(set, i)) goto out_unwind; @@ -3372,10 +3397,15 @@ static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) out_unwind: while (--i >= 0) { - blk_mq_free_map_and_rqs(set, set->tags[i], i); + __blk_mq_free_map_and_rqs(set, set->tags[i], i); set->tags[i] = NULL; } + if (blk_mq_is_sbitmap_shared(set->flags)) { + blk_mq_free_map_and_rqs(set, set->shared_sbitmap_tags, + BLK_MQ_NO_HCTX_IDX); + } + return -ENOMEM; } @@ -3554,25 +3584,11 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) if (ret) goto out_free_mq_map; - if (blk_mq_is_sbitmap_shared(set->flags)) { - atomic_set(&set->active_queues_shared_sbitmap, 0); - - if (blk_mq_init_shared_sbitmap(set)) { - ret = -ENOMEM; - goto out_free_mq_rq_maps; - } - } - mutex_init(&set->tag_list_lock); INIT_LIST_HEAD(&set->tag_list); return 0; -out_free_mq_rq_maps: - for (i = 0; i < set->nr_hw_queues; i++) { - blk_mq_free_map_and_rqs(set, set->tags[i], i); - set->tags[i] = NULL; - } out_free_mq_map: for (i = 0; i < set->nr_maps; i++) { kfree(set->map[i].mq_map); @@ -3605,12 +3621,14 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) int i, j; for (i = 0; i < set->nr_hw_queues; i++) { - blk_mq_free_map_and_rqs(set, set->tags[i], i); + __blk_mq_free_map_and_rqs(set, set->tags[i], i); set->tags[i] = NULL; } - if (blk_mq_is_sbitmap_shared(set->flags)) - blk_mq_exit_shared_sbitmap(set); + if (blk_mq_is_sbitmap_shared(set->flags)) { + blk_mq_free_map_and_rqs(set, set->shared_sbitmap_tags, + BLK_MQ_NO_HCTX_IDX); + } for (j = 0; j < set->nr_maps; j++) { kfree(set->map[j].mq_map); @@ -3648,12 +3666,6 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) if (hctx->sched_tags) { ret = blk_mq_tag_update_depth(hctx, &hctx->sched_tags, nr, true); - if (blk_mq_is_sbitmap_shared(set->flags)) { - hctx->sched_tags->bitmap_tags = - &q->sched_bitmap_tags; - hctx->sched_tags->breserved_tags = - &q->sched_breserved_tags; - } } else { ret = blk_mq_tag_update_depth(hctx, &hctx->tags, nr, false); diff --git a/block/blk-mq.h b/block/blk-mq.h index bcb0ca89d37a..b34385211e0a 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -54,7 +54,7 @@ void blk_mq_put_rq_ref(struct request *rq); */ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, unsigned int hctx_idx); -void blk_mq_free_rq_map(struct blk_mq_tags *tags, unsigned int flags); +void blk_mq_free_rq_map(struct blk_mq_tags *tags); struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set, unsigned int hctx_idx, unsigned int depth); void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, @@ -331,10 +331,11 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, if (blk_mq_is_sbitmap_shared(hctx->flags)) { struct request_queue *q = hctx->queue; struct blk_mq_tag_set *set = q->tag_set; + struct blk_mq_tags *tags = set->shared_sbitmap_tags; if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags)) return true; - users = atomic_read(&set->active_queues_shared_sbitmap); + users = atomic_read(&tags->active_queues); } else { if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state)) return true; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 13ba1861e688..808854a8ebc4 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -232,13 +232,11 @@ enum hctx_type { * @flags: Zero or more BLK_MQ_F_* flags. * @driver_data: Pointer to data owned by the block driver that created this * tag set. - * @active_queues_shared_sbitmap: - * number of active request queues per tag set. - * @__bitmap_tags: A shared tags sbitmap, used over all hctx's - * @__breserved_tags: - * A shared reserved tags sbitmap, used over all hctx's * @tags: Tag sets. One tag set per hardware queue. Has @nr_hw_queues * elements. + * @shared_sbitmap_tags: + * Shared sbitmap set of tags. Has @nr_hw_queues elements. If + * set, shared by all @tags. * @tag_list_lock: Serializes tag_list accesses. * @tag_list: List of the request queues that use this tag set. See also * request_queue.tag_set_list. @@ -255,12 +253,11 @@ struct blk_mq_tag_set { unsigned int timeout; unsigned int flags; void *driver_data; - atomic_t active_queues_shared_sbitmap; - struct sbitmap_queue __bitmap_tags; - struct sbitmap_queue __breserved_tags; struct blk_mq_tags **tags; + struct blk_mq_tags *shared_sbitmap_tags; + struct mutex tag_list_lock; struct list_head tag_list; }; @@ -432,6 +429,8 @@ enum { ((policy & ((1 << BLK_MQ_F_ALLOC_POLICY_BITS) - 1)) \ << BLK_MQ_F_ALLOC_POLICY_START_BIT) +#define BLK_MQ_NO_HCTX_IDX (-1U) + struct gendisk *__blk_mq_alloc_disk(struct blk_mq_tag_set *set, void *queuedata, struct lock_class_key *lkclass); #define blk_mq_alloc_disk(set, queuedata) \ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index fc3987eda2c1..76f75274081e 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -466,8 +466,7 @@ struct request_queue { atomic_t nr_active_requests_shared_sbitmap; - struct sbitmap_queue sched_bitmap_tags; - struct sbitmap_queue sched_breserved_tags; + struct blk_mq_tags *shared_sbitmap_tags; struct list_head icq_list; #ifdef CONFIG_BLK_CGROUP From patchwork Fri Aug 27 12:02:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 503396 Delivered-To: patch@linaro.org Received: by 2002:a02:6f15:0:0:0:0:0 with SMTP id x21csp1198268jab; Fri, 27 Aug 2021 05:07:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzJ6YTGAQo38AiQSncelofUkXllsyj+myZJkhm/YQrQgCS1wULworSfNc7t9YBqdzbzZKyY X-Received: by 2002:a05:6402:51cf:: with SMTP id r15mr9527657edd.211.1630066051722; Fri, 27 Aug 2021 05:07:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1630066051; cv=none; d=google.com; s=arc-20160816; b=EDuJA3dYH5RZg67OIB+ZhcL2UG/wBwaU4qtAlZ6E2SjzWQkwzOx7UbUhjR1H/piKM5 Zn1fRYygVUtiEUSWOrwIfDskwqHteWXJJOf+sI1ne2r/+IsslLTjg9/xiZeSOXjkONMe tPNKeTnL01ROn4O6Q6dqayfTZPD6wYVLpvDh7TVVHF/w2lm5RtkUCo/SzRwD3CH4B36W d6epBr0UQmTQFncX4ajH/a/g4GwkVn1jyD4CQEhwm7NJ2tpHJ49EgpkdaAJzPmEV7BfD b+KnyvKQ8ZmS8CufXBYAEQUonoRlQ6KdfSZbaLF3sAIpCNbMMwREjVY7ibKhO++mYuNm 90Cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from; bh=CuCQ2HEZbxkRoC9PJxtSFC3TEclADwO+CFOQt3944yc=; b=hKNx8PRYLLTaYdv8QvvdrLEtpKjwmszU+XMFcTbm6UbQTBhJDHvZ4oJT58OIyCSKkp Csj5NV/TVQxgsaEtQsXQofmheqzeY6UriQqjMzP9rVdIw3nBCC3oTObjLgyGfGCJXQOm k3MEVPjRo20Z0V1XnENABY6wNVZwwsOsoFRSaXkC/gk2ZrPCubKpm0ncPc2dQc5VUcvz ih3ukSTfBvJUL1zBcTNhI8JS4IDBPabroiK25ZRwmd76v9LFQmLBCrZ0KIPbWPITFxie mO8oy5YSzMdqj4mktL7ipMKxy3qZE4t/DmhU2PbwWZ8u5CQLgybReKud5nCUUKmpHwq6 RrwA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id n15si5476460ejl.353.2021.08.27.05.07.31 for ; Fri, 27 Aug 2021 05:07:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-scsi-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-scsi-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245235AbhH0MIS (ORCPT ); Fri, 27 Aug 2021 08:08:18 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:3708 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245231AbhH0MIO (ORCPT ); Fri, 27 Aug 2021 08:08:14 -0400 Received: from fraeml734-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Gwz2P68vwz67gT9; Fri, 27 Aug 2021 20:06:05 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml734-chm.china.huawei.com (10.206.15.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 14:07:15 +0200 Received: from localhost.localdomain (10.69.192.58) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Fri, 27 Aug 2021 13:07:13 +0100 From: John Garry To: CC: , , , , John Garry Subject: [PATCH v3 13/13] blk-mq: Stop using pointers for blk_mq_tags bitmap tags Date: Fri, 27 Aug 2021 20:02:04 +0800 Message-ID: <1630065724-69146-14-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1630065724-69146-1-git-send-email-john.garry@huawei.com> References: <1630065724-69146-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Now that we use shared tags for shared sbitmap support, we don't require the tags sbitmap pointers, so drop them. This essentially reverts commit 222a5ae03cdd ("blk-mq: Use pointers for blk_mq_tags bitmap tags"). Function blk_mq_init_bitmap_tags() is removed also, since it would be only a wrappper for blk_mq_init_bitmaps(). Reviewed-by: Ming Lei Signed-off-by: John Garry --- block/bfq-iosched.c | 4 +-- block/blk-mq-debugfs.c | 8 +++--- block/blk-mq-tag.c | 56 ++++++++++++++-------------------------- block/blk-mq-tag.h | 7 ++--- block/blk-mq.c | 8 +++--- block/kyber-iosched.c | 4 +-- block/mq-deadline-main.c | 2 +- 7 files changed, 35 insertions(+), 54 deletions(-) -- 2.26.2 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 480e1a134859..cb3a59883c4c 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6884,8 +6884,8 @@ static void bfq_depth_updated(struct blk_mq_hw_ctx *hctx) struct blk_mq_tags *tags = hctx->sched_tags; unsigned int min_shallow; - min_shallow = bfq_update_depths(bfqd, tags->bitmap_tags); - sbitmap_queue_min_shallow_depth(tags->bitmap_tags, min_shallow); + min_shallow = bfq_update_depths(bfqd, &tags->bitmap_tags); + sbitmap_queue_min_shallow_depth(&tags->bitmap_tags, min_shallow); } static int bfq_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int index) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 4b66d2776eda..4000376330c9 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -452,11 +452,11 @@ static void blk_mq_debugfs_tags_show(struct seq_file *m, atomic_read(&tags->active_queues)); seq_puts(m, "\nbitmap_tags:\n"); - sbitmap_queue_show(tags->bitmap_tags, m); + sbitmap_queue_show(&tags->bitmap_tags, m); if (tags->nr_reserved_tags) { seq_puts(m, "\nbreserved_tags:\n"); - sbitmap_queue_show(tags->breserved_tags, m); + sbitmap_queue_show(&tags->breserved_tags, m); } } @@ -487,7 +487,7 @@ static int hctx_tags_bitmap_show(void *data, struct seq_file *m) if (res) goto out; if (hctx->tags) - sbitmap_bitmap_show(&hctx->tags->bitmap_tags->sb, m); + sbitmap_bitmap_show(&hctx->tags->bitmap_tags.sb, m); mutex_unlock(&q->sysfs_lock); out: @@ -521,7 +521,7 @@ static int hctx_sched_tags_bitmap_show(void *data, struct seq_file *m) if (res) goto out; if (hctx->sched_tags) - sbitmap_bitmap_show(&hctx->sched_tags->bitmap_tags->sb, m); + sbitmap_bitmap_show(&hctx->sched_tags->bitmap_tags.sb, m); mutex_unlock(&q->sysfs_lock); out: diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 29b93ea0ea27..a313e6869639 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -46,9 +46,9 @@ bool __blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx) */ void blk_mq_tag_wakeup_all(struct blk_mq_tags *tags, bool include_reserve) { - sbitmap_queue_wake_all(tags->bitmap_tags); + sbitmap_queue_wake_all(&tags->bitmap_tags); if (include_reserve) - sbitmap_queue_wake_all(tags->breserved_tags); + sbitmap_queue_wake_all(&tags->breserved_tags); } /* @@ -104,10 +104,10 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) WARN_ON_ONCE(1); return BLK_MQ_NO_TAG; } - bt = tags->breserved_tags; + bt = &tags->breserved_tags; tag_offset = 0; } else { - bt = tags->bitmap_tags; + bt = &tags->bitmap_tags; tag_offset = tags->nr_reserved_tags; } @@ -153,9 +153,9 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) data->ctx); tags = blk_mq_tags_from_data(data); if (data->flags & BLK_MQ_REQ_RESERVED) - bt = tags->breserved_tags; + bt = &tags->breserved_tags; else - bt = tags->bitmap_tags; + bt = &tags->bitmap_tags; /* * If destination hw queue is changed, fake wake up on @@ -189,10 +189,10 @@ void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, const int real_tag = tag - tags->nr_reserved_tags; BUG_ON(real_tag >= tags->nr_tags); - sbitmap_queue_clear(tags->bitmap_tags, real_tag, ctx->cpu); + sbitmap_queue_clear(&tags->bitmap_tags, real_tag, ctx->cpu); } else { BUG_ON(tag >= tags->nr_reserved_tags); - sbitmap_queue_clear(tags->breserved_tags, tag, ctx->cpu); + sbitmap_queue_clear(&tags->breserved_tags, tag, ctx->cpu); } } @@ -343,9 +343,9 @@ static void __blk_mq_all_tag_iter(struct blk_mq_tags *tags, WARN_ON_ONCE(flags & BT_TAG_ITER_RESERVED); if (tags->nr_reserved_tags) - bt_tags_for_each(tags, tags->breserved_tags, fn, priv, + bt_tags_for_each(tags, &tags->breserved_tags, fn, priv, flags | BT_TAG_ITER_RESERVED); - bt_tags_for_each(tags, tags->bitmap_tags, fn, priv, flags); + bt_tags_for_each(tags, &tags->bitmap_tags, fn, priv, flags); } /** @@ -462,8 +462,8 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn, continue; if (tags->nr_reserved_tags) - bt_for_each(hctx, tags->breserved_tags, fn, priv, true); - bt_for_each(hctx, tags->bitmap_tags, fn, priv, false); + bt_for_each(hctx, &tags->breserved_tags, fn, priv, true); + bt_for_each(hctx, &tags->bitmap_tags, fn, priv, false); } blk_queue_exit(q); } @@ -495,24 +495,6 @@ int blk_mq_init_bitmaps(struct sbitmap_queue *bitmap_tags, return -ENOMEM; } -static int blk_mq_init_bitmap_tags(struct blk_mq_tags *tags, - int node, int alloc_policy) -{ - int ret; - - ret = blk_mq_init_bitmaps(&tags->__bitmap_tags, - &tags->__breserved_tags, - tags->nr_tags, tags->nr_reserved_tags, - node, alloc_policy); - if (ret) - return ret; - - tags->bitmap_tags = &tags->__bitmap_tags; - tags->breserved_tags = &tags->__breserved_tags; - - return 0; -} - struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, unsigned int reserved_tags, int node, int alloc_policy) @@ -532,7 +514,9 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, tags->nr_reserved_tags = reserved_tags; spin_lock_init(&tags->lock); - if (blk_mq_init_bitmap_tags(tags, node, alloc_policy) < 0) { + if (blk_mq_init_bitmaps(&tags->bitmap_tags, &tags->breserved_tags, + total_tags, reserved_tags, node, + alloc_policy) < 0) { kfree(tags); return NULL; } @@ -541,8 +525,8 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags, void blk_mq_free_tags(struct blk_mq_tags *tags) { - sbitmap_queue_free(tags->bitmap_tags); - sbitmap_queue_free(tags->breserved_tags); + sbitmap_queue_free(&tags->bitmap_tags); + sbitmap_queue_free(&tags->breserved_tags); kfree(tags); } @@ -591,7 +575,7 @@ int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, * Don't need (or can't) update reserved tags here, they * remain static and should never need resizing. */ - sbitmap_queue_resize(tags->bitmap_tags, + sbitmap_queue_resize(&tags->bitmap_tags, tdepth - tags->nr_reserved_tags); } @@ -602,12 +586,12 @@ void blk_mq_tag_resize_shared_sbitmap(struct blk_mq_tag_set *set, unsigned int s { struct blk_mq_tags *tags = set->shared_sbitmap_tags; - sbitmap_queue_resize(&tags->__bitmap_tags, size - set->reserved_tags); + sbitmap_queue_resize(&tags->bitmap_tags, size - set->reserved_tags); } void blk_mq_tag_update_sched_shared_sbitmap(struct request_queue *q) { - sbitmap_queue_resize(q->shared_sbitmap_tags->bitmap_tags, + sbitmap_queue_resize(&q->shared_sbitmap_tags->bitmap_tags, q->nr_requests - q->tag_set->reserved_tags); } diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index e433e39a9cfa..23747ea2bb53 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -11,11 +11,8 @@ struct blk_mq_tags { atomic_t active_queues; - struct sbitmap_queue *bitmap_tags; - struct sbitmap_queue *breserved_tags; - - struct sbitmap_queue __bitmap_tags; - struct sbitmap_queue __breserved_tags; + struct sbitmap_queue bitmap_tags; + struct sbitmap_queue breserved_tags; struct request **rqs; struct request **static_rqs; diff --git a/block/blk-mq.c b/block/blk-mq.c index 958761c0b26b..1013be6675d2 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1082,14 +1082,14 @@ static inline unsigned int queued_to_index(unsigned int queued) static bool __blk_mq_get_driver_tag(struct request *rq) { - struct sbitmap_queue *bt = rq->mq_hctx->tags->bitmap_tags; + struct sbitmap_queue *bt = &rq->mq_hctx->tags->bitmap_tags; unsigned int tag_offset = rq->mq_hctx->tags->nr_reserved_tags; int tag; blk_mq_tag_busy(rq->mq_hctx); if (blk_mq_tag_is_reserved(rq->mq_hctx->sched_tags, rq->internal_tag)) { - bt = rq->mq_hctx->tags->breserved_tags; + bt = &rq->mq_hctx->tags->breserved_tags; tag_offset = 0; } else { if (!hctx_may_queue(rq->mq_hctx, bt)) @@ -1132,7 +1132,7 @@ static int blk_mq_dispatch_wake(wait_queue_entry_t *wait, unsigned mode, struct sbitmap_queue *sbq; list_del_init(&wait->entry); - sbq = hctx->tags->bitmap_tags; + sbq = &hctx->tags->bitmap_tags; atomic_dec(&sbq->ws_active); } spin_unlock(&hctx->dispatch_wait_lock); @@ -1150,7 +1150,7 @@ static int blk_mq_dispatch_wake(wait_queue_entry_t *wait, unsigned mode, static bool blk_mq_mark_tag_wait(struct blk_mq_hw_ctx *hctx, struct request *rq) { - struct sbitmap_queue *sbq = hctx->tags->bitmap_tags; + struct sbitmap_queue *sbq = &hctx->tags->bitmap_tags; struct wait_queue_head *wq; wait_queue_entry_t *wait; bool ret; diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 81e3279ecd57..d6f9f17e40b4 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -451,11 +451,11 @@ static void kyber_depth_updated(struct blk_mq_hw_ctx *hctx) { struct kyber_queue_data *kqd = hctx->queue->elevator->elevator_data; struct blk_mq_tags *tags = hctx->sched_tags; - unsigned int shift = tags->bitmap_tags->sb.shift; + unsigned int shift = tags->bitmap_tags.sb.shift; kqd->async_depth = (1U << shift) * KYBER_ASYNC_PERCENT / 100U; - sbitmap_queue_min_shallow_depth(tags->bitmap_tags, kqd->async_depth); + sbitmap_queue_min_shallow_depth(&tags->bitmap_tags, kqd->async_depth); } static int kyber_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) diff --git a/block/mq-deadline-main.c b/block/mq-deadline-main.c index 294be0c0db65..23e50296dfbb 100644 --- a/block/mq-deadline-main.c +++ b/block/mq-deadline-main.c @@ -554,7 +554,7 @@ static void dd_depth_updated(struct blk_mq_hw_ctx *hctx) dd->async_depth = max(1UL, 3 * q->nr_requests / 4); - sbitmap_queue_min_shallow_depth(tags->bitmap_tags, dd->async_depth); + sbitmap_queue_min_shallow_depth(&tags->bitmap_tags, dd->async_depth); } /* Called by blk_mq_init_hctx() and blk_mq_init_sched(). */