| Message ID | 20220105193910.25678-1-mario.limonciello@amd.com |
|---|---|
| Headers | show |
| Series | On AMD platforms only offer s2idle w/ proper FW | expand |
On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello <mario.limonciello@amd.com> wrote: > > Currently the Linux kernel will offer s2idle regardless of whether the FADT > indicates the system should use or on X86 if the LPS0 ACPI device has been > activated. > > On some non-AMD platforms s2idle can be offered even without proper > firmware support. The power consumption may be higher in these instances > but the system otherwise properly suspends and resumes. Well, the idea is that s2idle should not require FW support at all. It may not be possible to reach the minimum power level of the platform without FW support, but that should not prevent s2idle from being used. > On AMD platforms however when the FW has been configured not to offer > s2idle some different hardware initialization has occurred such that the > system won't properly resume. That's rather unfortunate. Can you please share some details on what's going on in those cases? Technically, without FW support there should be no difference between the platform state reachable via s2idle and the platform state reachable via runtime idle.
+Alex On 1/11/2022 09:52, Rafael J. Wysocki wrote: > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > <mario.limonciello@amd.com> wrote: >> >> Currently the Linux kernel will offer s2idle regardless of whether the FADT >> indicates the system should use or on X86 if the LPS0 ACPI device has been >> activated. >> >> On some non-AMD platforms s2idle can be offered even without proper >> firmware support. The power consumption may be higher in these instances >> but the system otherwise properly suspends and resumes. > > Well, the idea is that s2idle should not require FW support at all. > May I ask - why? It's an intentional design decision? > It may not be possible to reach the minimum power level of the > platform without FW support, but that should not prevent s2idle from > being used. > >> On AMD platforms however when the FW has been configured not to offer >> s2idle some different hardware initialization has occurred such that the >> system won't properly resume. > > That's rather unfortunate. > > Can you please share some details on what's going on in those cases? > > Technically, without FW support there should be no difference between > the platform state reachable via s2idle and the platform state > reachable via runtime idle. During resume there is a number of page faults that occur and during initialization the ring tests fail. The graphics is unusable at this time as a result. The amdgpu code actually *does* distinguish between the 3 different cases of S3, S0ix, and runtime suspend. The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to be used during the suspend routine. In this particular case that FADT doesn't set the low power idle bit and that function returns false meaning the s3 codepath is taken but the hardware didn't go through a reset. It *might* also be possible to solve this by mandating an ASIC reset in such a case (we didn't try). However it comes back to my first upleveveled question - is this a case we really want to support and encourage? This type of bug and combination of codepaths is not a case that is going to be well tested. This patch series will align the kernel behavior to only what AMD validates.
On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario <mario.limonciello@amd.com> wrote: > > +Alex > > On 1/11/2022 09:52, Rafael J. Wysocki wrote: > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > > <mario.limonciello@amd.com> wrote: > >> > >> Currently the Linux kernel will offer s2idle regardless of whether the FADT > >> indicates the system should use or on X86 if the LPS0 ACPI device has been > >> activated. > >> > >> On some non-AMD platforms s2idle can be offered even without proper > >> firmware support. The power consumption may be higher in these instances > >> but the system otherwise properly suspends and resumes. > > > > Well, the idea is that s2idle should not require FW support at all. > > > May I ask - why? It's an intentional design decision? Yes, it is. > > It may not be possible to reach the minimum power level of the > > platform without FW support, but that should not prevent s2idle from > > being used. > > > >> On AMD platforms however when the FW has been configured not to offer > >> s2idle some different hardware initialization has occurred such that the > >> system won't properly resume. > > > > That's rather unfortunate. > > > > Can you please share some details on what's going on in those cases? > > > > Technically, without FW support there should be no difference between > > the platform state reachable via s2idle and the platform state > > reachable via runtime idle. > > During resume there is a number of page faults that occur and during > initialization the ring tests fail. The graphics is unusable at this > time as a result. > > The amdgpu code actually *does* distinguish between the 3 different > cases of S3, S0ix, and runtime suspend. But s2idle doesn't guarantee S0ix in any case. > The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to > be used during the suspend routine. Well, as I said, s2idle need not mean S0ix. > In this particular case that FADT doesn't set the low power idle bit > and that function returns false meaning the s3 codepath is taken but > the hardware didn't go through a reset. If there is a separate S3 code path, taking it when pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect. > It *might* also be possible to solve this by mandating an ASIC reset in > such a case (we didn't try). I'd rather do a PM-runtime path equivalent if the target sleep state is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > However it comes back to my first upleveveled question - is this a case > we really want to support and encourage? This type of bug and > combination of codepaths is not a case that is going to be well tested. > This patch series will align the kernel behavior to only what AMD validates. But this does not follow the definition of s2idle and its documentation.
On 1/11/2022 11:05, Rafael J. Wysocki wrote: > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario > <mario.limonciello@amd.com> wrote: >> >> +Alex >> >> On 1/11/2022 09:52, Rafael J. Wysocki wrote: >>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello >>> <mario.limonciello@amd.com> wrote: >>>> >>>> Currently the Linux kernel will offer s2idle regardless of whether the FADT >>>> indicates the system should use or on X86 if the LPS0 ACPI device has been >>>> activated. >>>> >>>> On some non-AMD platforms s2idle can be offered even without proper >>>> firmware support. The power consumption may be higher in these instances >>>> but the system otherwise properly suspends and resumes. >>> >>> Well, the idea is that s2idle should not require FW support at all. > >> >> May I ask - why? It's an intentional design decision? > > Yes, it is. > >>> It may not be possible to reach the minimum power level of the >>> platform without FW support, but that should not prevent s2idle from >>> being used. >>> >>>> On AMD platforms however when the FW has been configured not to offer >>>> s2idle some different hardware initialization has occurred such that the >>>> system won't properly resume. >>> >>> That's rather unfortunate. >>> >>> Can you please share some details on what's going on in those cases? >>> >>> Technically, without FW support there should be no difference between >>> the platform state reachable via s2idle and the platform state >>> reachable via runtime idle. >> >> During resume there is a number of page faults that occur and during >> initialization the ring tests fail. The graphics is unusable at this >> time as a result. >> >> The amdgpu code actually *does* distinguish between the 3 different >> cases of S3, S0ix, and runtime suspend. > > But s2idle doesn't guarantee S0ix in any case. > >> The function "amdgpu_acpi_is_s0ix_active" causes different codepaths to >> be used during the suspend routine. > > Well, as I said, s2idle need not mean S0ix. > >> In this particular case that FADT doesn't set the low power idle bit >> and that function returns false meaning the s3 codepath is taken but >> the hardware didn't go through a reset. > > If there is a separate S3 code path, taking it when > pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect. > >> It *might* also be possible to solve this by mandating an ASIC reset in >> such a case (we didn't try). > > I'd rather do a PM-runtime path equivalent if the target sleep state > is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > >> However it comes back to my first upleveveled question - is this a case >> we really want to support and encourage? This type of bug and >> combination of codepaths is not a case that is going to be well tested. >> This patch series will align the kernel behavior to only what AMD validates. > > But this does not follow the definition of s2idle and its documentation. Very well, we'll work out in amdgpu what we can do. Please close the kernel bugzilla (I don't have permissions).
[AMD Official Use Only] > -----Original Message----- > From: Rafael J. Wysocki <rafael@kernel.org> > Sent: Tuesday, January 11, 2022 12:06 PM > To: Limonciello, Mario <Mario.Limonciello@amd.com> > Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki > <rjw@rjwysocki.net>; ACPI Devel Maling List <linux-acpi@vger.kernel.org>; > S-k, Shyam-sundar <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > <Basavaraj.Natikar@amd.com>; Deucher, Alexander > <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW > > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario > <mario.limonciello@amd.com> wrote: > > > > +Alex > > > > On 1/11/2022 09:52, Rafael J. Wysocki wrote: > > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > > > <mario.limonciello@amd.com> wrote: > > >> > > >> Currently the Linux kernel will offer s2idle regardless of whether > > >> the FADT indicates the system should use or on X86 if the LPS0 ACPI > > >> device has been activated. > > >> > > >> On some non-AMD platforms s2idle can be offered even without > proper > > >> firmware support. The power consumption may be higher in these > > >> instances but the system otherwise properly suspends and resumes. > > > > > > Well, the idea is that s2idle should not require FW support at all. > > > > > > > > May I ask - why? It's an intentional design decision? > > Yes, it is. > > > > It may not be possible to reach the minimum power level of the > > > platform without FW support, but that should not prevent s2idle from > > > being used. > > > > > >> On AMD platforms however when the FW has been configured not to > > >> offer s2idle some different hardware initialization has occurred > > >> such that the system won't properly resume. > > > > > > That's rather unfortunate. > > > > > > Can you please share some details on what's going on in those cases? > > > > > > Technically, without FW support there should be no difference > > > between the platform state reachable via s2idle and the platform > > > state reachable via runtime idle. > > > > During resume there is a number of page faults that occur and during > > initialization the ring tests fail. The graphics is unusable at this > > time as a result. > > > > The amdgpu code actually *does* distinguish between the 3 different > > cases of S3, S0ix, and runtime suspend. > > But s2idle doesn't guarantee S0ix in any case. > > > The function "amdgpu_acpi_is_s0ix_active" causes different codepaths > > to be used during the suspend routine. > > Well, as I said, s2idle need not mean S0ix. > > > In this particular case that FADT doesn't set the low power idle bit > > and that function returns false meaning the s3 codepath is taken but > > the hardware didn't go through a reset. > > If there is a separate S3 code path, taking it when pm_suspend_target_state > == PM_SUSPEND_TO_IDLE is incorrect. > > > It *might* also be possible to solve this by mandating an ASIC reset > > in such a case (we didn't try). > > I'd rather do a PM-runtime path equivalent if the target sleep state is > PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > > > However it comes back to my first upleveveled question - is this a > > case we really want to support and encourage? This type of bug and > > combination of codepaths is not a case that is going to be well tested. > > This patch series will align the kernel behavior to only what AMD validates. > > But this does not follow the definition of s2idle and its documentation. At least for devices integrated into the SoC, the power rails are controlled by the firmware in the SoC. For S3, the power rails are cut by the FW when the platform enters S3. For S0ix, the power rails are cut when all of the devices on the rail suspended and various conditions are met. Also, in the case of some devices, the device has to be in a very specific state for s0ix to work properly. The GPU is the big one here. For S3, the entire GPU has to be re-initialized at resume. For S0ix, the GPU's state is largely handled by the firmware and attempting to re-initialize it won't work unless you reset it. Integrated AMD graphics don't support runtime power management, only dGPUs do. For integrated graphics the firmware dynamically controls the power at runtime so there is no need to do anything special for runtime pm. For dGPUs we support d3cold either via ACPI on platforms like all-in-ones and laptops or via a driver initiated sequence for add-in-cards. What does S2idle ultimately do when all devices have suspended? Does it enter S0ix or S3 at the end when it want to ultimately suspend the platform, or is the assumption that if all devices have suspended, the that is equivalent to S0ix or S3? For AMD platforms, either S3 or S0ix needs to be entered for the platform to actually power down most of the power rails. It's not clear to me what we should do for s2idle. Alex
On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander <Alexander.Deucher@amd.com> wrote: > > [AMD Official Use Only] > > > -----Original Message----- > > From: Rafael J. Wysocki <rafael@kernel.org> > > Sent: Tuesday, January 11, 2022 12:06 PM > > To: Limonciello, Mario <Mario.Limonciello@amd.com> > > Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki > > <rjw@rjwysocki.net>; ACPI Devel Maling List <linux-acpi@vger.kernel.org>; > > S-k, Shyam-sundar <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > > <Basavaraj.Natikar@amd.com>; Deucher, Alexander > > <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com > > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW > > > > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario > > <mario.limonciello@amd.com> wrote: > > > > > > +Alex > > > > > > On 1/11/2022 09:52, Rafael J. Wysocki wrote: > > > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > > > > <mario.limonciello@amd.com> wrote: > > > >> > > > >> Currently the Linux kernel will offer s2idle regardless of whether > > > >> the FADT indicates the system should use or on X86 if the LPS0 ACPI > > > >> device has been activated. > > > >> > > > >> On some non-AMD platforms s2idle can be offered even without > > proper > > > >> firmware support. The power consumption may be higher in these > > > >> instances but the system otherwise properly suspends and resumes. > > > > > > > > Well, the idea is that s2idle should not require FW support at all. > > > > > > > > > > > May I ask - why? It's an intentional design decision? > > > > Yes, it is. > > > > > > It may not be possible to reach the minimum power level of the > > > > platform without FW support, but that should not prevent s2idle from > > > > being used. > > > > > > > >> On AMD platforms however when the FW has been configured not to > > > >> offer s2idle some different hardware initialization has occurred > > > >> such that the system won't properly resume. > > > > > > > > That's rather unfortunate. > > > > > > > > Can you please share some details on what's going on in those cases? > > > > > > > > Technically, without FW support there should be no difference > > > > between the platform state reachable via s2idle and the platform > > > > state reachable via runtime idle. > > > > > > During resume there is a number of page faults that occur and during > > > initialization the ring tests fail. The graphics is unusable at this > > > time as a result. > > > > > > The amdgpu code actually *does* distinguish between the 3 different > > > cases of S3, S0ix, and runtime suspend. > > > > But s2idle doesn't guarantee S0ix in any case. > > > > > The function "amdgpu_acpi_is_s0ix_active" causes different codepaths > > > to be used during the suspend routine. > > > > Well, as I said, s2idle need not mean S0ix. > > > > > In this particular case that FADT doesn't set the low power idle bit > > > and that function returns false meaning the s3 codepath is taken but > > > the hardware didn't go through a reset. > > > > If there is a separate S3 code path, taking it when pm_suspend_target_state > > == PM_SUSPEND_TO_IDLE is incorrect. > > > > > It *might* also be possible to solve this by mandating an ASIC reset > > > in such a case (we didn't try). > > > > I'd rather do a PM-runtime path equivalent if the target sleep state is > > PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > > > > > However it comes back to my first upleveveled question - is this a > > > case we really want to support and encourage? This type of bug and > > > combination of codepaths is not a case that is going to be well tested. > > > This patch series will align the kernel behavior to only what AMD validates. > > > > But this does not follow the definition of s2idle and its documentation. > > At least for devices integrated into the SoC, the power rails are controlled by the firmware in the SoC. For S3, the power rails are cut by the FW when the platform enters S3. For S0ix, the power rails are cut when all of the devices on the rail suspended and various conditions are met. Also, in the case of some devices, the device has to be in a very specific state for s0ix to work properly. The GPU is the big one here. For S3, the entire GPU has to be re-initialized at resume. For S0ix, the GPU's state is largely handled by the firmware and attempting to re-initialize it won't work unless you reset it. Integrated AMD graphics don't support runtime power management, only dGPUs do. For integrated graphics the firmware dynamically controls the power at runtime so there is no need to do anything special for runtime pm. For dGPUs we support d3cold either via ACPI on platforms like all-in-ones and laptops or via a driver initiated sequence for add-in-cards. > > What does S2idle ultimately do when all devices have suspended? Does it enter S0ix or S3 at the end when it want to ultimately suspend the platform, or is the assumption that if all devices have suspended, the that is equivalent to S0ix or S3? For AMD platforms, either S3 or S0ix needs to be entered for the platform to actually power down most of the power rails. It's not clear to me what we should do for s2idle. s2idle will never enter S3, because that requires platform support and generally is a different thing (eg. some devices need special upfront preparation for S3, wakeup may need to be configured in a special way etc.). It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into the deepest available idle state and stay there until an interrupt (or equivalent) triggers. Physically, at a device level, s2idle is more similar to runtime suspend than to S3, but it uses system-wide suspend callbacks and it requires wakeup to be disabled for the devices that are not allowed to wake up the system by user space, that is device_may_wakeup(0 returns false (PM-runtime assumes wakeup to always be enabled for the suspended devices that can signal wakeup). In any case, the ACPI system state for s2idle is always S0.
[Public] > -----Original Message----- > From: Rafael J. Wysocki <rafael@kernel.org> > Sent: Tuesday, January 11, 2022 12:45 PM > To: Deucher, Alexander <Alexander.Deucher@amd.com> > Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario > <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>; > ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar > <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW > > On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander > <Alexander.Deucher@amd.com> wrote: > > > > [AMD Official Use Only] > > > > > -----Original Message----- > > > From: Rafael J. Wysocki <rafael@kernel.org> > > > Sent: Tuesday, January 11, 2022 12:06 PM > > > To: Limonciello, Mario <Mario.Limonciello@amd.com> > > > Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki > > > <rjw@rjwysocki.net>; ACPI Devel Maling List > > > <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar > > > <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > > > <Basavaraj.Natikar@amd.com>; Deucher, Alexander > > > <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com > > > Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ > > > proper FW > > > > > > On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario > > > <mario.limonciello@amd.com> wrote: > > > > > > > > +Alex > > > > > > > > On 1/11/2022 09:52, Rafael J. Wysocki wrote: > > > > > On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > > > > > <mario.limonciello@amd.com> wrote: > > > > >> > > > > >> Currently the Linux kernel will offer s2idle regardless of > > > > >> whether the FADT indicates the system should use or on X86 if > > > > >> the LPS0 ACPI device has been activated. > > > > >> > > > > >> On some non-AMD platforms s2idle can be offered even without > > > proper > > > > >> firmware support. The power consumption may be higher in these > > > > >> instances but the system otherwise properly suspends and > resumes. > > > > > > > > > > Well, the idea is that s2idle should not require FW support at all. > > > > > > > > > > > > > > May I ask - why? It's an intentional design decision? > > > > > > Yes, it is. > > > > > > > > It may not be possible to reach the minimum power level of the > > > > > platform without FW support, but that should not prevent s2idle > > > > > from being used. > > > > > > > > > >> On AMD platforms however when the FW has been configured not > to > > > > >> offer s2idle some different hardware initialization has > > > > >> occurred such that the system won't properly resume. > > > > > > > > > > That's rather unfortunate. > > > > > > > > > > Can you please share some details on what's going on in those cases? > > > > > > > > > > Technically, without FW support there should be no difference > > > > > between the platform state reachable via s2idle and the platform > > > > > state reachable via runtime idle. > > > > > > > > During resume there is a number of page faults that occur and > > > > during initialization the ring tests fail. The graphics is > > > > unusable at this time as a result. > > > > > > > > The amdgpu code actually *does* distinguish between the 3 > > > > different cases of S3, S0ix, and runtime suspend. > > > > > > But s2idle doesn't guarantee S0ix in any case. > > > > > > > The function "amdgpu_acpi_is_s0ix_active" causes different > > > > codepaths to be used during the suspend routine. > > > > > > Well, as I said, s2idle need not mean S0ix. > > > > > > > In this particular case that FADT doesn't set the low power idle > > > > bit and that function returns false meaning the s3 codepath is > > > > taken but the hardware didn't go through a reset. > > > > > > If there is a separate S3 code path, taking it when > > > pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect. > > > > > > > It *might* also be possible to solve this by mandating an ASIC > > > > reset in such a case (we didn't try). > > > > > > I'd rather do a PM-runtime path equivalent if the target sleep state > > > is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > > > > > > > However it comes back to my first upleveveled question - is this a > > > > case we really want to support and encourage? This type of bug > > > > and combination of codepaths is not a case that is going to be well > tested. > > > > This patch series will align the kernel behavior to only what AMD > validates. > > > > > > But this does not follow the definition of s2idle and its documentation. > > > > At least for devices integrated into the SoC, the power rails are controlled > by the firmware in the SoC. For S3, the power rails are cut by the FW when > the platform enters S3. For S0ix, the power rails are cut when all of the > devices on the rail suspended and various conditions are met. Also, in the > case of some devices, the device has to be in a very specific state for s0ix to > work properly. The GPU is the big one here. For S3, the entire GPU has to > be re-initialized at resume. For S0ix, the GPU's state is largely handled by the > firmware and attempting to re-initialize it won't work unless you reset it. > Integrated AMD graphics don't support runtime power management, only > dGPUs do. For integrated graphics the firmware dynamically controls the > power at runtime so there is no need to do anything special for runtime pm. > For dGPUs we support d3cold either via ACPI on platforms like all-in-ones > and laptops or via a driver initiated sequence for add-in-cards. > > > > What does S2idle ultimately do when all devices have suspended? Does it > enter S0ix or S3 at the end when it want to ultimately suspend the platform, > or is the assumption that if all devices have suspended, the that is equivalent > to S0ix or S3? For AMD platforms, either S3 or S0ix needs to be entered for > the platform to actually power down most of the power rails. It's not clear to > me what we should do for s2idle. > > s2idle will never enter S3, because that requires platform support and > generally is a different thing (eg. some devices need special upfront > preparation for S3, wakeup may need to be configured in a special way etc.). > > It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into > the deepest available idle state and stay there until an interrupt (or > equivalent) triggers. > > Physically, at a device level, s2idle is more similar to runtime suspend than to > S3, but it uses system-wide suspend callbacks and it requires wakeup to be > disabled for the devices that are not allowed to wake up the system by user > space, that is device_may_wakeup(0 returns false (PM-runtime assumes > wakeup to always be enabled for the suspended devices that can signal > wakeup). > > In any case, the ACPI system state for s2idle is always S0. Thanks. In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system. Does it even make sense to expose it? It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle. Alex
On 1/11/2022 12:30, Deucher, Alexander wrote: > [Public] > >> -----Original Message----- >> From: Rafael J. Wysocki <rafael@kernel.org> >> Sent: Tuesday, January 11, 2022 12:45 PM >> To: Deucher, Alexander <Alexander.Deucher@amd.com> >> Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario >> <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>; >> ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar >> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj >> <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com >> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW >> >> On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander >> <Alexander.Deucher@amd.com> wrote: >>> >>> [AMD Official Use Only] >>> >>>> -----Original Message----- >>>> From: Rafael J. Wysocki <rafael@kernel.org> >>>> Sent: Tuesday, January 11, 2022 12:06 PM >>>> To: Limonciello, Mario <Mario.Limonciello@amd.com> >>>> Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki >>>> <rjw@rjwysocki.net>; ACPI Devel Maling List >>>> <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar >>>> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj >>>> <Basavaraj.Natikar@amd.com>; Deucher, Alexander >>>> <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com >>>> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ >>>> proper FW >>>> >>>> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario >>>> <mario.limonciello@amd.com> wrote: >>>>> >>>>> +Alex >>>>> >>>>> On 1/11/2022 09:52, Rafael J. Wysocki wrote: >>>>>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello >>>>>> <mario.limonciello@amd.com> wrote: >>>>>>> >>>>>>> Currently the Linux kernel will offer s2idle regardless of >>>>>>> whether the FADT indicates the system should use or on X86 if >>>>>>> the LPS0 ACPI device has been activated. >>>>>>> >>>>>>> On some non-AMD platforms s2idle can be offered even without >>>> proper >>>>>>> firmware support. The power consumption may be higher in these >>>>>>> instances but the system otherwise properly suspends and >> resumes. >>>>>> >>>>>> Well, the idea is that s2idle should not require FW support at all. >>>>>>> >>>>> >>>>> May I ask - why? It's an intentional design decision? >>>> >>>> Yes, it is. >>>> >>>>>> It may not be possible to reach the minimum power level of the >>>>>> platform without FW support, but that should not prevent s2idle >>>>>> from being used. >>>>>> >>>>>>> On AMD platforms however when the FW has been configured not >> to >>>>>>> offer s2idle some different hardware initialization has >>>>>>> occurred such that the system won't properly resume. >>>>>> >>>>>> That's rather unfortunate. >>>>>> >>>>>> Can you please share some details on what's going on in those cases? >>>>>> >>>>>> Technically, without FW support there should be no difference >>>>>> between the platform state reachable via s2idle and the platform >>>>>> state reachable via runtime idle. >>>>> >>>>> During resume there is a number of page faults that occur and >>>>> during initialization the ring tests fail. The graphics is >>>>> unusable at this time as a result. >>>>> >>>>> The amdgpu code actually *does* distinguish between the 3 >>>>> different cases of S3, S0ix, and runtime suspend. >>>> >>>> But s2idle doesn't guarantee S0ix in any case. >>>> >>>>> The function "amdgpu_acpi_is_s0ix_active" causes different >>>>> codepaths to be used during the suspend routine. >>>> >>>> Well, as I said, s2idle need not mean S0ix. >>>> >>>>> In this particular case that FADT doesn't set the low power idle >>>>> bit and that function returns false meaning the s3 codepath is >>>>> taken but the hardware didn't go through a reset. >>>> >>>> If there is a separate S3 code path, taking it when >>>> pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect. >>>> >>>>> It *might* also be possible to solve this by mandating an ASIC >>>>> reset in such a case (we didn't try). >>>> >>>> I'd rather do a PM-runtime path equivalent if the target sleep state >>>> is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. >>>> >>>>> However it comes back to my first upleveveled question - is this a >>>>> case we really want to support and encourage? This type of bug >>>>> and combination of codepaths is not a case that is going to be well >> tested. >>>>> This patch series will align the kernel behavior to only what AMD >> validates. >>>> >>>> But this does not follow the definition of s2idle and its documentation. >>> >>> At least for devices integrated into the SoC, the power rails are controlled >> by the firmware in the SoC. For S3, the power rails are cut by the FW when >> the platform enters S3. For S0ix, the power rails are cut when all of the >> devices on the rail suspended and various conditions are met. Also, in the >> case of some devices, the device has to be in a very specific state for s0ix to >> work properly. The GPU is the big one here. For S3, the entire GPU has to >> be re-initialized at resume. For S0ix, the GPU's state is largely handled by the >> firmware and attempting to re-initialize it won't work unless you reset it. >> Integrated AMD graphics don't support runtime power management, only >> dGPUs do. For integrated graphics the firmware dynamically controls the >> power at runtime so there is no need to do anything special for runtime pm. >> For dGPUs we support d3cold either via ACPI on platforms like all-in-ones >> and laptops or via a driver initiated sequence for add-in-cards. >>> >>> What does S2idle ultimately do when all devices have suspended? Does it >> enter S0ix or S3 at the end when it want to ultimately suspend the platform, >> or is the assumption that if all devices have suspended, the that is equivalent >> to S0ix or S3? For AMD platforms, either S3 or S0ix needs to be entered for >> the platform to actually power down most of the power rails. It's not clear to >> me what we should do for s2idle. >> >> s2idle will never enter S3, because that requires platform support and >> generally is a different thing (eg. some devices need special upfront >> preparation for S3, wakeup may need to be configured in a special way etc.). >> >> It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into >> the deepest available idle state and stay there until an interrupt (or >> equivalent) triggers. >> >> Physically, at a device level, s2idle is more similar to runtime suspend than to >> S3, but it uses system-wide suspend callbacks and it requires wakeup to be >> disabled for the devices that are not allowed to wake up the system by user >> space, that is device_may_wakeup(0 returns false (PM-runtime assumes >> wakeup to always be enabled for the suspended devices that can signal >> wakeup). >> >> In any case, the ACPI system state for s2idle is always S0. > > Thanks. In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system. Does it even make sense to expose it? It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle. > > Alex That's why I thought my patch series made sense. I guess another (antisocial) response would be to to return false when the suspend callback for amdgpu happens and dev_err mentioning that firmware support is needed for suspend.
On Tue, Jan 11, 2022 at 7:36 PM Limonciello, Mario <mario.limonciello@amd.com> wrote: > > On 1/11/2022 12:30, Deucher, Alexander wrote: > > [Public] > > > >> -----Original Message----- > >> From: Rafael J. Wysocki <rafael@kernel.org> > >> Sent: Tuesday, January 11, 2022 12:45 PM > >> To: Deucher, Alexander <Alexander.Deucher@amd.com> > >> Cc: Rafael J. Wysocki <rafael@kernel.org>; Limonciello, Mario > >> <Mario.Limonciello@amd.com>; Rafael J . Wysocki <rjw@rjwysocki.net>; > >> ACPI Devel Maling List <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar > >> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > >> <Basavaraj.Natikar@amd.com>; bjoern.daase@gmail.com > >> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ proper FW > >> > >> On Tue, Jan 11, 2022 at 6:32 PM Deucher, Alexander > >> <Alexander.Deucher@amd.com> wrote: > >>> > >>> [AMD Official Use Only] > >>> > >>>> -----Original Message----- > >>>> From: Rafael J. Wysocki <rafael@kernel.org> > >>>> Sent: Tuesday, January 11, 2022 12:06 PM > >>>> To: Limonciello, Mario <Mario.Limonciello@amd.com> > >>>> Cc: Rafael J. Wysocki <rafael@kernel.org>; Rafael J . Wysocki > >>>> <rjw@rjwysocki.net>; ACPI Devel Maling List > >>>> <linux-acpi@vger.kernel.org>; S-k, Shyam-sundar > >>>> <Shyam-sundar.S-k@amd.com>; Natikar, Basavaraj > >>>> <Basavaraj.Natikar@amd.com>; Deucher, Alexander > >>>> <Alexander.Deucher@amd.com>; bjoern.daase@gmail.com > >>>> Subject: Re: [PATCH 0/3] On AMD platforms only offer s2idle w/ > >>>> proper FW > >>>> > >>>> On Tue, Jan 11, 2022 at 5:23 PM Limonciello, Mario > >>>> <mario.limonciello@amd.com> wrote: > >>>>> > >>>>> +Alex > >>>>> > >>>>> On 1/11/2022 09:52, Rafael J. Wysocki wrote: > >>>>>> On Wed, Jan 5, 2022 at 8:39 PM Mario Limonciello > >>>>>> <mario.limonciello@amd.com> wrote: > >>>>>>> > >>>>>>> Currently the Linux kernel will offer s2idle regardless of > >>>>>>> whether the FADT indicates the system should use or on X86 if > >>>>>>> the LPS0 ACPI device has been activated. > >>>>>>> > >>>>>>> On some non-AMD platforms s2idle can be offered even without > >>>> proper > >>>>>>> firmware support. The power consumption may be higher in these > >>>>>>> instances but the system otherwise properly suspends and > >> resumes. > >>>>>> > >>>>>> Well, the idea is that s2idle should not require FW support at all. > >>>>>>> > >>>>> > >>>>> May I ask - why? It's an intentional design decision? > >>>> > >>>> Yes, it is. > >>>> > >>>>>> It may not be possible to reach the minimum power level of the > >>>>>> platform without FW support, but that should not prevent s2idle > >>>>>> from being used. > >>>>>> > >>>>>>> On AMD platforms however when the FW has been configured not > >> to > >>>>>>> offer s2idle some different hardware initialization has > >>>>>>> occurred such that the system won't properly resume. > >>>>>> > >>>>>> That's rather unfortunate. > >>>>>> > >>>>>> Can you please share some details on what's going on in those cases? > >>>>>> > >>>>>> Technically, without FW support there should be no difference > >>>>>> between the platform state reachable via s2idle and the platform > >>>>>> state reachable via runtime idle. > >>>>> > >>>>> During resume there is a number of page faults that occur and > >>>>> during initialization the ring tests fail. The graphics is > >>>>> unusable at this time as a result. > >>>>> > >>>>> The amdgpu code actually *does* distinguish between the 3 > >>>>> different cases of S3, S0ix, and runtime suspend. > >>>> > >>>> But s2idle doesn't guarantee S0ix in any case. > >>>> > >>>>> The function "amdgpu_acpi_is_s0ix_active" causes different > >>>>> codepaths to be used during the suspend routine. > >>>> > >>>> Well, as I said, s2idle need not mean S0ix. > >>>> > >>>>> In this particular case that FADT doesn't set the low power idle > >>>>> bit and that function returns false meaning the s3 codepath is > >>>>> taken but the hardware didn't go through a reset. > >>>> > >>>> If there is a separate S3 code path, taking it when > >>>> pm_suspend_target_state == PM_SUSPEND_TO_IDLE is incorrect. > >>>> > >>>>> It *might* also be possible to solve this by mandating an ASIC > >>>>> reset in such a case (we didn't try). > >>>> > >>>> I'd rather do a PM-runtime path equivalent if the target sleep state > >>>> is PM_SUSPEND_TO_IDLE and there is no FW support for S0ix. > >>>> > >>>>> However it comes back to my first upleveveled question - is this a > >>>>> case we really want to support and encourage? This type of bug > >>>>> and combination of codepaths is not a case that is going to be well > >> tested. > >>>>> This patch series will align the kernel behavior to only what AMD > >> validates. > >>>> > >>>> But this does not follow the definition of s2idle and its documentation. > >>> > >>> At least for devices integrated into the SoC, the power rails are controlled > >> by the firmware in the SoC. For S3, the power rails are cut by the FW when > >> the platform enters S3. For S0ix, the power rails are cut when all of the > >> devices on the rail suspended and various conditions are met. Also, in the > >> case of some devices, the device has to be in a very specific state for s0ix to > >> work properly. The GPU is the big one here. For S3, the entire GPU has to > >> be re-initialized at resume. For S0ix, the GPU's state is largely handled by the > >> firmware and attempting to re-initialize it won't work unless you reset it. > >> Integrated AMD graphics don't support runtime power management, only > >> dGPUs do. For integrated graphics the firmware dynamically controls the > >> power at runtime so there is no need to do anything special for runtime pm. > >> For dGPUs we support d3cold either via ACPI on platforms like all-in-ones > >> and laptops or via a driver initiated sequence for add-in-cards. > >>> > >>> What does S2idle ultimately do when all devices have suspended? Does it > >> enter S0ix or S3 at the end when it want to ultimately suspend the platform, > >> or is the assumption that if all devices have suspended, the that is equivalent > >> to S0ix or S3? For AMD platforms, either S3 or S0ix needs to be entered for > >> the platform to actually power down most of the power rails. It's not clear to > >> me what we should do for s2idle. > >> > >> s2idle will never enter S3, because that requires platform support and > >> generally is a different thing (eg. some devices need special upfront > >> preparation for S3, wakeup may need to be configured in a special way etc.). > >> > >> It will attempt to enter S0ix if possible, but otherwise it will just put CPUs into > >> the deepest available idle state and stay there until an interrupt (or > >> equivalent) triggers. > >> > >> Physically, at a device level, s2idle is more similar to runtime suspend than to > >> S3, but it uses system-wide suspend callbacks and it requires wakeup to be > >> disabled for the devices that are not allowed to wake up the system by user > >> space, that is device_may_wakeup(0 returns false (PM-runtime assumes > >> wakeup to always be enabled for the suspended devices that can signal > >> wakeup). > >> > >> In any case, the ACPI system state for s2idle is always S0. > > > > Thanks. In that case, it's kind of pointless on AMD platforms then since the power rails will never be turned off for most devices on the system. Does it even make sense to expose it? It just gives users a false sense of suspend and then they will probably complain that it uses too much power when in s2idle. > > > > Alex > > That's why I thought my patch series made sense. > > I guess another (antisocial) response would be to to return false when > the suspend callback for amdgpu happens and dev_err mentioning that > firmware support is needed for suspend. If you want to avoid exposing s2idle at all in some cases, you need to change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle" in mem_sleep or "freeze" in state at all. That's much more intrusive than just failing every s2idle attempt (which is what the patches do AFAICS). Do you want to do that for platforms supporting S3, or for platforms that don't support any kind of suspend at all?
>> >> That's why I thought my patch series made sense. >> >> I guess another (antisocial) response would be to to return false when >> the suspend callback for amdgpu happens and dev_err mentioning that >> firmware support is needed for suspend. > > If you want to avoid exposing s2idle at all in some cases, you need to > change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle" > in mem_sleep or "freeze" in state at all. Right now they make it so that s2idle isn't exposed in "mem_sleep", but "freeze" is still exposed in "state". I'd be worried about breaking userspace with taking "freeze" out of state. > That's much more intrusive > than just failing every s2idle attempt (which is what the patches do > AFAICS) As they stand right now when the BIOS is set to S3: # cat /sys/power/mem_sleep [deep] # echo freeze > /sys/power/state Returns a failure though. And then when it's set to Modern Standby : # cat /sys/power/mem_sleep [s2idle] # echo freeze > /sys/power/state Works > > Do you want to do that for platforms supporting S3, or for platforms > that don't support any kind of suspend at all? You know since this series went out 6dc8265f9803ccb7e5da804e01601f0c14f270e0 was merged. This will probably have cleaned up the problem state that Bjoern found in the first place. So this is really a philosophical discussion now if it's worth offering s2idle in /sys/power/mem_sleep on AMD systems. Admittedly this has been an APU notebook oriented discussion. Platforms that don't support any kind of suspend (like servers) will get caught up in this and could no longer do s2idle either. With the assumption that 6dc8265f9803ccb7e5da804e01601f0c14f270e0 helps the problem state I'm leaning back on we should add some warnings to let people know how to help themselves for power consumption if they did this. We can probably put those in amdgpu though.
On Tue, Jan 11, 2022 at 8:35 PM Limonciello, Mario <mario.limonciello@amd.com> wrote: > > >> > >> That's why I thought my patch series made sense. > >> > >> I guess another (antisocial) response would be to to return false when > >> the suspend callback for amdgpu happens and dev_err mentioning that > >> firmware support is needed for suspend. > > > > If you want to avoid exposing s2idle at all in some cases, you need to > > change the sysfs I/F in /sys/power/ so that it doesn't expose "s2idle" > > in mem_sleep or "freeze" in state at all. > > Right now they make it so that s2idle isn't exposed in "mem_sleep", but > "freeze" is still exposed in "state". > > I'd be worried about breaking userspace with taking "freeze" out of state. > > > That's much more intrusive > > than just failing every s2idle attempt (which is what the patches do > > AFAICS) > As they stand right now when the BIOS is set to S3: > # cat /sys/power/mem_sleep > [deep] > > # echo freeze > /sys/power/state > Returns a failure though. > > And then when it's set to Modern Standby : > > # cat /sys/power/mem_sleep > [s2idle] > > # echo freeze > /sys/power/state > Works > > > > > Do you want to do that for platforms supporting S3, or for platforms > > that don't support any kind of suspend at all? > > You know since this series went out > 6dc8265f9803ccb7e5da804e01601f0c14f270e0 was merged. This will probably > have cleaned up the problem state that Bjoern found in the first place. > > So this is really a philosophical discussion now if it's worth offering > s2idle in /sys/power/mem_sleep on AMD systems. Well, if "freeze" is present in "state", then "s2idle" needs to be there in "mem_sleep", because they are clearly documented as alternative paths to the same functionality. > Admittedly this has been an APU notebook oriented discussion. Platforms > that don't support any kind of suspend (like servers) will get caught up > in this and could no longer do s2idle either. > > With the assumption that 6dc8265f9803ccb7e5da804e01601f0c14f270e0 helps > the problem state I'm leaning back on we should add some warnings to let > people know how to help themselves for power consumption if they did this. > > We can probably put those in amdgpu though. Sounds like a better direction to me, honestly.