| Message ID | 20250320180450.539-3-shiju.jose@huawei.com |
|---|---|
| State | New |
| Headers | show |
| Series | cxl: support CXL memory RAS features | expand |
>-----Original Message----- >From: Jonathan Cameron <jonathan.cameron@huawei.com> >Sent: 21 March 2025 10:03 >To: Shiju Jose <shiju.jose@huawei.com> >Cc: linux-cxl@vger.kernel.org; dan.j.williams@intel.com; dave@stgolabs.net; >dave.jiang@intel.com; alison.schofield@intel.com; vishal.l.verma@intel.com; >ira.weiny@intel.com; david@redhat.com; Vilas.Sridharan@amd.com; linux- >edac@vger.kernel.org; linux-acpi@vger.kernel.org; linux-mm@kvack.org; linux- >kernel@vger.kernel.org; bp@alien8.de; tony.luck@intel.com; rafael@kernel.org; >lenb@kernel.org; mchehab@kernel.org; leo.duran@amd.com; >Yazen.Ghannam@amd.com; rientjes@google.com; jiaqiyan@google.com; >Jon.Grimm@amd.com; dave.hansen@linux.intel.com; >naoya.horiguchi@nec.com; james.morse@arm.com; jthoughton@google.com; >somasundaram.a@hpe.com; erdemaktas@google.com; pgonda@google.com; >duenwen@google.com; gthelen@google.com; >wschwartz@amperecomputing.com; dferguson@amperecomputing.com; >wbs@os.amperecomputing.com; nifan.cxl@gmail.com; tanxiaofei ><tanxiaofei@huawei.com>; Zengtao (B) <prime.zeng@hisilicon.com>; Roberto >Sassu <roberto.sassu@huawei.com>; kangkang.shen@futurewei.com; >wanghuiqiang <wanghuiqiang@huawei.com>; Linuxarm ><linuxarm@huawei.com> >Subject: Re: [PATCH v2 2/8] EDAC: Update documentation for the CXL memory >patrol scrub control feature > >On Thu, 20 Mar 2025 18:04:39 +0000 ><shiju.jose@huawei.com> wrote: > >> From: Shiju Jose <shiju.jose@huawei.com> >> >> Update the Documentation/edac/scrub.rst to include descriptions and >> policies for CXL memory device-based and CXL region-based patrol scrub >> control. >> >> Note: This may require inputs from CXL memory experts regarding >> region-based scrubbing policies. > >So I suggested the region interfaces in the first place. It's all about usecases and >'why' we might increase the scrub rate. >Ultimately the hardware is controlled in a device wide way, so we could have >made it complex userspace problem to deal with it on a perf device. >The region interfaces are there as a simplification not because they are strictly >necessary. > >Anyhow, the use cases: > >1) Scrubbing because a device is showing unexpectedly high errors. That > control needs to be at device granularity. If one device in an interleave > set (backing a region) is dodgy, why make them all do more work? > >2) Scrubbing may apply to memory that isn't online at all yet. Nice to know > if we have a problem before we start using it! Likely this is setting > system wide defaults on boot. > >3) Scrubbing at higher rate because software has decided that we want > more reliability for particular data. I've been calling this > Differentiated Reliability. That data sits in a region which > may cover part of multiple devices. The region interfaces are about > supporting this use case. > >So now the question is what do we do if both interfaces are poked because >someone cares simultaneously about 1 and 3? > >I'd suggest just laying out a set for rules on how to set the scrub rates for any >mixture of requirements, rather than making the driver work out the optimum >combination. > >> >> Signed-off-by: Shiju Jose <shiju.jose@huawei.com> >> --- >> Documentation/edac/scrub.rst | 47 >> ++++++++++++++++++++++++++++++++++++ >> 1 file changed, 47 insertions(+) >> >> diff --git a/Documentation/edac/scrub.rst >> b/Documentation/edac/scrub.rst index daab929cdba1..d1c02bd90090 100644 >> --- a/Documentation/edac/scrub.rst >> +++ b/Documentation/edac/scrub.rst >> @@ -264,3 +264,51 @@ Sysfs files are documented in >> `Documentation/ABI/testing/sysfs-edac-scrub` >> >> `Documentation/ABI/testing/sysfs-edac-ecs` >> + >> +Examples >> +-------- >> + >> +The usage takes the form shown in these examples: >> + >> +1. CXL memory device patrol scrubber >> + >> +1.1 Device based scrubbing >> + >> +CXL memory is exposed to memory management subsystem and ultimately >> +userspace via CXL devices. >> + >> +For cases where hardware interleave controls do not directly map to >> +regions of Physical Address space, perhaps due to interleave the >> +approach described in >> +1.2 Region based scrubbing section, which is specific to CXL regions >> +should be followed. > >These sentences end up a bit unwieldy. Perhaps simply a forwards reference. > >When combining control via the device interfaces and region interfaces see >1.2 Region bases scrubbing. > > > >> In those cases settings on the presented interface may interact with >> +direct control via a device instance specific interface and care must be taken. >> + >> +Sysfs files for scrubbing are documented in >> +`Documentation/ABI/testing/sysfs-edac-scrub` >> + >> +1.2. Region based scrubbing >> + >> +CXL memory is exposed to memory management subsystem and ultimately >> +userspace via CXL regions. CXL Regions represent mapped memory >> +capacity in system physical address space. These can incorporate one >> +or more parts of multiple CXL memory devices with traffic interleaved >> +across them. The user may want to control the scrub rate via this >> +more abstract region instead of having to figure out the constituent >> +devices and program them separately. The scrub rate for each device >> +covers the whole device. Thus if multiple regions use parts of that >> +device then requests for scrubbing of other regions may result in a higher >scrub rate than requested for this specific region. >> + >> +1. When user sets scrub rate for a memory region, the scrub rate for all the >CXL >> + memory devices interleaved under that region is updated with the same >scrub >> + rate. > >Note that this may affect multiple regions. > >> + >> +2. When user sets scrub rate for a memory device, only the scrub rate for >that >> + memory devices is updated though device may be part of a memory region >and >> + does not change scrub rate of other memory devices of that memory >region. >> + >> +3. Scrub rate of a CXL memory device may be set via EDAC device or region >scrub >> + interface simultaneously. Care must be taken to prevent a race condition, >or >> + only region-based setting may be allowed. > >So is this saying if you want to mix and match, set region first then device next? >Can we just lay out the rules to set up a weird mixture. We could add more >smarts to the driver but do we care as mixing 1 and 3 above is probably >unusual? > >1. Taking each region in turn from lowest desired scrub rate to highest and set > their scrub rates. Later regions may override the scrub rate on individual > devices (and hence potentially whole regions). > >2. Take each device for which enhanced scrubbing is required (higher rate) and > set those scrub rates. This will override the scrub rates of individual devices > leaving any that are not specifically set to scrub at the maximum rate required > for any of the regions they are involved in backing. Thanks. Will incorporate these info and rules in the next version. > > >> + >> +Sysfs files for scrubbing are documented in >> +`Documentation/ABI/testing/sysfs-edac-scrub` Shiju
diff --git a/Documentation/edac/scrub.rst b/Documentation/edac/scrub.rst index daab929cdba1..d1c02bd90090 100644 --- a/Documentation/edac/scrub.rst +++ b/Documentation/edac/scrub.rst @@ -264,3 +264,51 @@ Sysfs files are documented in `Documentation/ABI/testing/sysfs-edac-scrub` `Documentation/ABI/testing/sysfs-edac-ecs` + +Examples +-------- + +The usage takes the form shown in these examples: + +1. CXL memory device patrol scrubber + +1.1 Device based scrubbing + +CXL memory is exposed to memory management subsystem and ultimately userspace +via CXL devices. + +For cases where hardware interleave controls do not directly map to regions of +Physical Address space, perhaps due to interleave the approach described in +1.2 Region based scrubbing section, which is specific to CXL regions should be +followed. In those cases settings on the presented interface may interact with +direct control via a device instance specific interface and care must be taken. + +Sysfs files for scrubbing are documented in +`Documentation/ABI/testing/sysfs-edac-scrub` + +1.2. Region based scrubbing + +CXL memory is exposed to memory management subsystem and ultimately userspace +via CXL regions. CXL Regions represent mapped memory capacity in system +physical address space. These can incorporate one or more parts of multiple CXL +memory devices with traffic interleaved across them. The user may want to control +the scrub rate via this more abstract region instead of having to figure out the +constituent devices and program them separately. The scrub rate for each device +covers the whole device. Thus if multiple regions use parts of that device then +requests for scrubbing of other regions may result in a higher scrub rate than +requested for this specific region. + +1. When user sets scrub rate for a memory region, the scrub rate for all the CXL + memory devices interleaved under that region is updated with the same scrub + rate. + +2. When user sets scrub rate for a memory device, only the scrub rate for that + memory devices is updated though device may be part of a memory region and + does not change scrub rate of other memory devices of that memory region. + +3. Scrub rate of a CXL memory device may be set via EDAC device or region scrub + interface simultaneously. Care must be taken to prevent a race condition, or + only region-based setting may be allowed. + +Sysfs files for scrubbing are documented in +`Documentation/ABI/testing/sysfs-edac-scrub`