From patchwork Mon Jun 16 22:33:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 897275 Received: from 003.mia.mailroute.net (003.mia.mailroute.net [199.89.3.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CDBA2185A6; Mon, 16 Jun 2025 22:34:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=199.89.3.6 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750113247; cv=none; b=FdNN+61CrKNbP9gApQA0vYvVb+tQVEnpZg+nArfQC29Kd8BGsPNgkacaNDP20XXkuSZVf1THPpfjd3tGukaw9+KjX4S2qyvcknRbZ9b/VtJLZfi2avB47j8nFo6d+oh+yqrWqcX7qLTd5lMgtIGdNxG3ASYiD770WGnn9sTMuCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750113247; c=relaxed/simple; bh=foBTuhJNTBLZYdwB/mMltl58qgIg3yPqBZbJw8g9k4E=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=Pay0zCLjTrUuYyOo6JvjJgCMXLblqP56jKpme4eyVLAiYQ5DaNDu/NKM+lGiLoiR+x0llf+52u6Il7VQYNkrDd58yNdZu3zKiAY06oQzAULkEb3xxRJOkMNX30Q6ACDncmcn6zeeuUgA9b/35LG55axyXyE4XJ9o5L7FRERQyjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org; spf=pass smtp.mailfrom=acm.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b=raxKDSTL; arc=none smtp.client-ip=199.89.3.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=acm.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=acm.org header.i=@acm.org header.b="raxKDSTL" Received: from localhost (localhost [127.0.0.1]) by 003.mia.mailroute.net (Postfix) with ESMTP id 4bLlCp13s6zlgqxn; Mon, 16 Jun 2025 22:33:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acm.org; h= content-transfer-encoding:mime-version:x-mailer:message-id:date :date:subject:subject:from:from:received:received; s=mr01; t= 1750113236; x=1752705237; bh=tgxIs9nhOFRBPAENFrdoyFOru1rWaFeH7JC NcSLAaK4=; b=raxKDSTLR+reCmdBiz7FMh0li2Gnzn5nujucGTk0qm/Yp7qteNO 63XrDHnezhlBdXfefyZpsSHSeDvBbKfyosNOMq1Wl5BdAR89lUIsl6rINglERVby TlQOnLBbr9GS1PhlnxqdfUzuycKVgrz0zmphqJDrPsQrYxyFbXQeQMAXVs6enRYB mnt0Gonpv+rBknWVf79pPdOq+3wahM2bO1jd9/DqGsn7lUsVrK4sfwjsNx5pTpH+ PHXGeyBq5P7jFvJrhYkj1tK/X2uX+y+ykzZWDbkMCDaHCt4CCVJPzs+4CpYp02en IibupfGEP0KpF0PdN/WDRKFVtlNlSkLHOGA== X-Virus-Scanned: by MailRoute Received: from 003.mia.mailroute.net ([127.0.0.1]) by localhost (003.mia [127.0.0.1]) (mroute_mailscanner, port 10029) with LMTP id y2JTc7z1nLQ4; Mon, 16 Jun 2025 22:33:56 +0000 (UTC) Received: from bvanassche.mtv.corp.google.com (unknown [104.135.204.82]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: bvanassche@acm.org) by 003.mia.mailroute.net (Postfix) with ESMTPSA id 4bLlCh48L4zlgqVx; Mon, 16 Jun 2025 22:33:51 +0000 (UTC) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, Christoph Hellwig , Damien Le Moal , Bart Van Assche Subject: [PATCH v18 00/12] Improve write performance for zoned UFS devices Date: Mon, 16 Jun 2025 15:33:00 -0700 Message-ID: <20250616223312.1607638-1-bvanassche@acm.org> X-Mailer: git-send-email 2.50.0.rc2.692.g299adb8693-goog Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Hi Jens, This patch series improves small write IOPS by a factor of two for zoned UFS devices on my test setup. The changes included in this patch series are as follows: - A new request queue limits flag is introduced that allows block drivers to declare whether or not they preserve the request order per hardware queue. - The order of zoned writes is preserved in the block layer by submitting all zoned writes from the same CPU core as long as any zoned writes are pending. A new member 'from_cpu' is introduced in the per-zone data structure 'blk_zone_wplug' to track from which CPU to submit zoned writes. This data member is reset to -1 after all pending zoned writes for a zone have completed. - The retry count for zoned writes is increased in the SCSI core to deal with reordering caused by unit attention conditions or the SCSI error handler. - New functionality is added in the null_blk and scsi_debug drivers to make it easier to test the changes introduced by this patch series. Please consider this patch series for the next merge window. Thanks, Bart. Changes compared to v17: - Rebased the patch series on top of kernel v6.16-rc1. - Dropped support for UFSHCI 3.0 controllers because the UFSHCI 3.0 auto- hibernation mechanism causes request reordering. UFSHCI 4.0 controllers remain supported. - Removed the error handling and write pointer tracking mechanisms again from block/blk-zoned.c. - Dropped the dm-linear patch from this patch series since I'm not aware of any use cases for write pipelining and dm-linear. Changes compared to v16: - Rebased the entire patch series on top of Jens' for-next branch. Compared to when v16 of this series was posted, the BLK_ZONE_WPLUG_NEED_WP_UPDATE flag has been introduced and support for REQ_NOWAIT has been fixed. - The behavior for SMR disks is preserved: if .driver_preserves_write_order has not been set, BLK_ZONE_WPLUG_NEED_WP_UPDATE is still set if a write error has been encountered. If .driver_preserves_write_order has not been set, the write pointer is restored and the failed zoned writes are retried. - The superfluous "disk->zone_wplugs_hash_bits != 0" tests have been removed. Changes compared to v15: - Reworked this patch series on top of the zone write plugging approach. - Moved support for requeuing requests from the SCSI core into the block layer core. - In the UFS driver, instead of disabling write pipelining if auto-hibernation is enabled, rely on the requeuing mechanism to handle reordering caused by resuming from auto-hibernation. Changes compared to v14: - Removed the drivers/scsi/Kconfig.kunit and drivers/scsi/Makefile.kunit files. Instead, modified drivers/scsi/Kconfig and added #include "*_test.c" directives in the appropriate .c files. Removed the EXPORT_SYMBOL() directives that were added to make the unit tests link. - Fixed a double free in a unit test. Changes compared to v13: - Reworked patch "block: Preserve the order of requeued zoned writes". - Addressed a performance concern by removing the eh_needs_prepare_resubmit SCSI driver callback and by introducing the SCSI host template flag .needs_prepare_resubmit instead. - Added a patch that adds a 'host' argument to scsi_eh_flush_done_q(). - Made the code in unit tests less repetitive. Changes compared to v12: - Added two new patches: "block: Preserve the order of requeued zoned writes" and "scsi: sd: Add a unit test for sd_cmp_sector()" - Restricted the number of zoned write retries. To my surprise I had to add "&& scmd->retries <= scmd->allowed" in the SCSI error handler to limit the number of retries. - In patch "scsi: ufs: Inform the block layer about write ordering", only set ELEVATOR_F_ZBD_SEQ_WRITE for zoned block devices. Changes compared to v11: - Fixed a NULL pointer dereference that happened when booting from an ATA device by adding an scmd->device != NULL check in scsi_needs_preparation(). - Updated Reviewed-by tags. Changes compared to v10: - Dropped the UFS MediaTek and HiSilicon patches because these are not correct and because it is safe to drop these patches. - Updated Acked-by / Reviewed-by tags. Changes compared to v9: - Introduced an additional scsi_driver callback: .eh_needs_prepare_resubmit(). - Renamed the scsi_debug kernel module parameter 'no_zone_write_lock' into 'preserves_write_order'. - Fixed an out-of-bounds access in the unit scsi_call_prepare_resubmit() unit test. - Wrapped ufshcd_auto_hibern8_update() calls in UFS host drivers with WARN_ON_ONCE() such that a kernel stack appears in case an error code is returned. - Elaborated a comment in the UFSHCI driver. Changes compared to v8: - Fixed handling of 'driver_preserves_write_order' and 'use_zone_write_lock' in blk_stack_limits(). - Added a comment in disk_set_zoned(). - Modified blk_req_needs_zone_write_lock() such that it returns false if q->limits.use_zone_write_lock is false. - Modified disk_clear_zone_settings() such that it clears q->limits.use_zone_write_lock. - Left out one change from the mq-deadline patch that became superfluous due to the blk_req_needs_zone_write_lock() change. - Modified scsi_call_prepare_resubmit() such that it only calls list_sort() if zoned writes have to be resubmitted for which zone write locking is disabled. - Added an additional unit test for scsi_call_prepare_resubmit(). - Modified the sorting code in the sd driver such that only those SCSI commands are sorted for which write locking is disabled. - Modified sd_zbc.c such that ELEVATOR_F_ZBD_SEQ_WRITE is only set if the write order is not preserved. - Included three patches for UFS host drivers that rework code that wrote directly to the auto-hibernation controller register. - Modified the UFS driver such that enabling auto-hibernation is not allowed if a zoned logical unit is present and if the controller operates in legacy mode. - Also in the UFS driver, simplified ufshcd_auto_hibern8_update(). Changes compared to v7: - Split the queue_limits member variable `use_zone_write_lock' into two member variables: `use_zone_write_lock' (set by disk_set_zoned()) and `driver_preserves_write_order' (set by the block driver or SCSI LLD). This should clear up the confusion about the purpose of this variable. - Moved the code for sorting SCSI commands by LBA from the SCSI error handler into the SCSI disk (sd) driver as requested by Christoph. Changes compared to v6: - Removed QUEUE_FLAG_NO_ZONE_WRITE_LOCK and instead introduced a flag in the request queue limits data structure. Changes compared to v5: - Renamed scsi_cmp_lba() into scsi_cmp_sector(). - Improved several source code comments. Changes compared to v4: - Dropped the patch that introduces the REQ_NO_ZONE_WRITE_LOCK flag. - Dropped the null_blk patch and added two scsi_debug patches instead. - Dropped the f2fs patch. - Split the patch for the UFS driver into two patches. - Modified several patch descriptions and source code comments. - Renamed dd_use_write_locking() into dd_use_zone_write_locking(). - Moved the list_sort() call from scsi_unjam_host() into scsi_eh_flush_done_q() such that sorting happens just before reinserting. - Removed the scsi_cmd_retry_allowed() call from scsi_check_sense() to make sure that the retry counter is adjusted once per retry instead of twice. Changes compared to v3: - Restored the patch that introduces QUEUE_FLAG_NO_ZONE_WRITE_LOCK. That patch had accidentally been left out from v2. - In patch "block: Introduce the flag REQ_NO_ZONE_WRITE_LOCK", improved the patch description and added the function blk_no_zone_write_lock(). - In patch "block/mq-deadline: Only use zone locking if necessary", moved the blk_queue_is_zoned() call into dd_use_write_locking(). - In patch "fs/f2fs: Disable zone write locking", set REQ_NO_ZONE_WRITE_LOCK from inside __bio_alloc() instead of in f2fs_submit_write_bio(). Changes compared to v2: - Renamed the request queue flag for disabling zone write locking. - Introduced a new request flag for disabling zone write locking. - Modified the mq-deadline scheduler such that zone write locking is only disabled if both flags are set. - Added an F2FS patch that sets the request flag for disabling zone write locking. - Only disable zone write locking in the UFS driver if auto-hibernation is disabled. Changes compared to v1: - Left out the patches that are already upstream. - Switched the approach in patch "scsi: Retry unaligned zoned writes" from retrying immediately to sending unaligned write commands to the SCSI error handler. Bart Van Assche (12): block: Support block drivers that preserve the order of write requests block: Rework request allocation in blk_mq_submit_bio() block: Support allocating from a specific software queue blk-mq: Restore the zoned write order when requeuing blk-zoned: Add an argument to blk_zone_plug_bio() blk-zoned: Support pipelining of zoned writes null_blk: Add the preserves_write_order attribute scsi: core: Retry unaligned zoned writes scsi: sd: Increase retry count for zoned writes scsi: scsi_debug: Add the preserves_write_order module parameter scsi: scsi_debug: Support injecting unaligned write errors scsi: ufs: Inform the block layer about write ordering block/bfq-iosched.c | 2 + block/blk-mq.c | 73 +++++++++++++++++++------------ block/blk-mq.h | 3 ++ block/blk-settings.c | 2 + block/blk-zoned.c | 34 ++++++++++---- block/kyber-iosched.c | 2 + block/mq-deadline.c | 7 ++- drivers/block/null_blk/main.c | 3 ++ drivers/block/null_blk/null_blk.h | 1 + drivers/md/dm.c | 2 +- drivers/scsi/scsi_debug.c | 21 ++++++++- drivers/scsi/scsi_error.c | 16 +++++++ drivers/scsi/sd.c | 7 +++ drivers/ufs/core/ufshcd.c | 6 +++ include/linux/blk-mq.h | 13 +++++- include/linux/blkdev.h | 7 ++- 16 files changed, 158 insertions(+), 41 deletions(-) Reviewed-by: Avri Altman