Message ID | cover.1594289009.git.viresh.kumar@linaro.org |
---|---|
Headers | show |
Series | cpufreq: cppc: Add support for frequency invariance | expand |
Hi Viresh, I'll put all my comments here for now, as they refer more to the design of the solution. I hope it won't be too repetitive compared to what we previously discussed offline. I understand you want to get additional points of view. On Thursday 09 Jul 2020 at 15:43:32 (+0530), Viresh Kumar wrote: > Hello, > > CPPC cpufreq driver is used for ARM servers and this patch series tries > to provide frequency invariance support for them. The same is also > provided using a specific hardware extension, known as AMU (Activity > Monitors Unit), but that is optional for platforms and at least few of > them don't have it. > > This patchset allows multiple parts of the kernel to provide the same > functionality, by registering with the topology core. > > This is tested with some hacks, as I didn't have access to the right > hardware, on the ARM64 hikey board to check the overall functionality > and that works fine. > > Ionela/Peter Puhov, it would be nice if you guys can give this a shot. > I believe the code is unnecessarily invasive for the functionality it tries to introduce and it does break existing functionality. - (1) From code readability and design point of view, this switching between an architectural method and a driver method complicates an already complicated situation. We already have code that chooses between a cpufreq-based method and a counter based method for frequency invariance. This would basically introduce a choice between a cpufreq-based method through arch_set_freq_scale(), an architectural counter-based method through arch_set_freq_tick(), and another cpufreq-based method that piggy-backs on the architectural arch_set_freq_tick(). As discussed offline, before I even try to begin accepting the possibility of this complicated mix, I would like to know why methods of obtaining the same thing by using the cpufreq arch_set_freq_scale() or even the more invasive wrapping of the counter read functions is not working. I believe those solutions would brings only a fraction of the complexity added through this set. - (2) For 1/3, the presence of AMU counters does not guarantee their usability for frequency invariance. I know you wanted to avoid the complications of AMUs being marked as supporting invariance after the cpufreq driver init function, but this breaks the scenario in which the maximum frequency is invalid. - (3) For 2/3, currently we support platforms that have partial support for AMUs, while this would not be supported here. The suggestions at (1) would give us this for free. While I'm keen on having invariance support for CPPC when lacking part or full support for AMUs, I think we should explore alternative designs. I can try to come up with some code suggestions, but it will take a few days as I have many other things in the air :). I'm happy to test this when the design is agreed on. Hope it helps, Ionela.
Thanks for the quick reply Ionela. On 09-07-20, 13:43, Ionela Voinescu wrote: > I'll put all my comments here for now, as they refer more to the design > of the solution. > > I hope it won't be too repetitive compared to what we previously discussed > offline. > I understand you want to get additional points of view. Not necessarily, I knew you would be one of the major reviewers here :) I posted so you don't need to review in private anymore and then the code is somewhat updated since the previous time. > On Thursday 09 Jul 2020 at 15:43:32 (+0530), Viresh Kumar wrote: > I believe the code is unnecessarily invasive for the functionality it > tries to introduce and it does break existing functionality. > > > - (1) From code readability and design point of view, this switching > between an architectural method and a driver method complicates > an already complicated situation. We already have code that > chooses between a cpufreq-based method and a counter based method > for frequency invariance. This would basically introduce a choice > between a cpufreq-based method through arch_set_freq_scale(), an > architectural counter-based method through arch_set_freq_tick(), > and another cpufreq-based method that piggy-backs on the > architectural arch_set_freq_tick(). I agree. > As discussed offline, before I even try to begin accepting the > possibility of this complicated mix, I would like to know why > methods of obtaining the same thing by using the cpufreq > arch_set_freq_scale() The problem is same as that was in case of x86, we don't know the real frequency the CPU may be running at and we need something that fires up periodically in a guaranteed way to capture the freq-scale. Though I am thinking now if we can trust the target_index() helper and keep updating the freq-scale based on the delta between last call to it and the latest call. I am not sure if it will be sufficient. > or even the more invasive wrapping of the > counter read functions is not working. I am not sure I understood this one. > - (2) For 1/3, the presence of AMU counters does not guarantee their > usability for frequency invariance. I know you wanted to avoid > the complications of AMUs being marked as supporting invariance > after the cpufreq driver init function, but this breaks the > scenario in which the maximum frequency is invalid. Is that really a scenario ? i.e. Invalid maximum frequency ? Why would that ever happen ? And I am not sure if this breaks anything which already exists, because all we are doing in this case now is not registering cppc for FI, which should be fine. > - (3) For 2/3, currently we support platforms that have partial support > for AMUs, while this would not be supported here. The suggestions > at (1) would give us this for free. As both were counter based mechanisms, I thought it would be better and more consistent if only one of them is picked. Though partial support of AMUs would still work without the CPPC driver. -- viresh
On Fri, 10 Jul 2020 at 05:00, Viresh Kumar <viresh.kumar@linaro.org> wrote: > > Thanks for the quick reply Ionela. > > On 09-07-20, 13:43, Ionela Voinescu wrote: > > I'll put all my comments here for now, as they refer more to the design > > of the solution. > > > > I hope it won't be too repetitive compared to what we previously discussed > > offline. > > > I understand you want to get additional points of view. > > Not necessarily, I knew you would be one of the major reviewers here > :) > > I posted so you don't need to review in private anymore and then the > code is somewhat updated since the previous time. > > > On Thursday 09 Jul 2020 at 15:43:32 (+0530), Viresh Kumar wrote: > > I believe the code is unnecessarily invasive for the functionality it > > tries to introduce and it does break existing functionality. > > > > > > - (1) From code readability and design point of view, this switching > > between an architectural method and a driver method complicates > > an already complicated situation. We already have code that > > chooses between a cpufreq-based method and a counter based method > > for frequency invariance. This would basically introduce a choice > > between a cpufreq-based method through arch_set_freq_scale(), an > > architectural counter-based method through arch_set_freq_tick(), > > and another cpufreq-based method that piggy-backs on the > > architectural arch_set_freq_tick(). > > I agree. > > > As discussed offline, before I even try to begin accepting the > > possibility of this complicated mix, I would like to know why > > methods of obtaining the same thing by using the cpufreq > > arch_set_freq_scale() > > The problem is same as that was in case of x86, we don't know the real > frequency the CPU may be running at and we need something that fires > up periodically in a guaranteed way to capture the freq-scale. Yeah it's exactly the same behavior as x86 and re using the same mechanism seems the best solution The main problem is that AMU currently assumes that it will be the only to support such tick based mechanism whereas others like cppc can provides similar information > > Though I am thinking now if we can trust the target_index() helper and > keep updating the freq-scale based on the delta between last call to > it and the latest call. I am not sure if it will be sufficient. > > > or even the more invasive wrapping of the > > counter read functions is not working. > > I am not sure I understood this one. > > > - (2) For 1/3, the presence of AMU counters does not guarantee their > > usability for frequency invariance. I know you wanted to avoid > > the complications of AMUs being marked as supporting invariance > > after the cpufreq driver init function, but this breaks the > > scenario in which the maximum frequency is invalid. > > Is that really a scenario ? i.e. Invalid maximum frequency ? Why would > that ever happen ? > > And I am not sure if this breaks anything which already exists, > because all we are doing in this case now is not registering cppc for > FI, which should be fine. IIUC, AMU must wait for cpufreq drivers to be registered in order to get the maximum freq and being able to enable its FIE support. Could we have a sequence like: cppc register its scale_freq_tick function AMU can then override the tick function for cpu which supports AMU > > > - (3) For 2/3, currently we support platforms that have partial support > > for AMUs, while this would not be supported here. The suggestions > > at (1) would give us this for free. > > As both were counter based mechanisms, I thought it would be better > and more consistent if only one of them is picked. Though partial > support of AMUs would still work without the CPPC driver. > > -- > viresh
On 24-07-20, 11:38, Vincent Guittot wrote: > Yeah it's exactly the same behavior as x86 and re using the same > mechanism seems the best solution > > The main problem is that AMU currently assumes that it will be the > only to support such tick based mechanism whereas others like cppc can > provides similar information Ionela do you have some feedback to Vincent's email? We can't handle this apart from the tick handler, i.e. capture this data at every tick like it is done for x86 or AMU. -- viresh
Hi guys, On Friday 24 Jul 2020 at 11:38:59 (+0200), Vincent Guittot wrote: > On Fri, 10 Jul 2020 at 05:00, Viresh Kumar <viresh.kumar@linaro.org> wrote: > > > > Thanks for the quick reply Ionela. > > > > On 09-07-20, 13:43, Ionela Voinescu wrote: > > > I'll put all my comments here for now, as they refer more to the design > > > of the solution. > > > > > > I hope it won't be too repetitive compared to what we previously discussed > > > offline. > > > > > I understand you want to get additional points of view. > > > > Not necessarily, I knew you would be one of the major reviewers here > > :) > > > > I posted so you don't need to review in private anymore and then the > > code is somewhat updated since the previous time. > > > > > On Thursday 09 Jul 2020 at 15:43:32 (+0530), Viresh Kumar wrote: > > > I believe the code is unnecessarily invasive for the functionality it > > > tries to introduce and it does break existing functionality. > > > > > > > > > - (1) From code readability and design point of view, this switching > > > between an architectural method and a driver method complicates > > > an already complicated situation. We already have code that > > > chooses between a cpufreq-based method and a counter based method > > > for frequency invariance. This would basically introduce a choice > > > between a cpufreq-based method through arch_set_freq_scale(), an > > > architectural counter-based method through arch_set_freq_tick(), > > > and another cpufreq-based method that piggy-backs on the > > > architectural arch_set_freq_tick(). > > > > I agree. > > > > > As discussed offline, before I even try to begin accepting the > > > possibility of this complicated mix, I would like to know why > > > methods of obtaining the same thing by using the cpufreq > > > arch_set_freq_scale() > > > > The problem is same as that was in case of x86, we don't know the real > > frequency the CPU may be running at and we need something that fires > > up periodically in a guaranteed way to capture the freq-scale. > > Yeah it's exactly the same behavior as x86 and re using the same > mechanism seems the best solution > > The main problem is that AMU currently assumes that it will be the > only to support such tick based mechanism whereas others like cppc can > provides similar information > Yes, I agree that a similar method to the use of AMUs or APERF/MPERF would result in a more accurate frequency scale factor. > > > > Though I am thinking now if we can trust the target_index() helper and > > keep updating the freq-scale based on the delta between last call to > > it and the latest call. I am not sure if it will be sufficient. > > > > > or even the more invasive wrapping of the > > > counter read functions is not working. > > > > I am not sure I understood this one. > > I've been putting some more thought/code into this one and I believe something as follows might look nicer as well as cover a few corner cases (ignore implementation details for now, they can be changed): - Have a per cpu value that marks the use of either AMUs, CPPC, or cpufreq for freq invariance (can be done with per-cpu variable or with cpumasks) - arch_topology.c: initialization code as follows: for_each_present_cpu(cpu) { if (freq_inv_amus_valid(cpu) && !freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu) * 1000, arch_timer_get_rate(), cpu)) { per_cpu(inv_source, cpu) = INV_AMU_COUNTERS; continue; } if (freq_inv_cppc_counters_valid(cpu) && !freq_inv_set_max_ratio(cppc_max_perf, cppc_ref_perf, cpu)) { per_cpu(inv_source, cpu) = INV_CPPC_COUNTERS; continue; } if (!cpufreq_supports_freq_invariance() || freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu), 1, cpu)) { pr_info("FIE disabled: no valid source for CPU%d.", cpu); return 0; } } This would live in an equivalent of the init function we have now for AMU counters only (init_amu_fie), made to handle more sources and moved to arch_topology.c. The freq_inv_set_max_ratio() would be a generic version of what is now validate_cpu_freq_invariance_counters() (only the ratio computation and setting). - Finally, void freq_inv_update_counter_refs(void) { this_cpu_write(arch_core_cycles_prev, read_corecnt()); this_cpu_write(arch_const_cycles_prev, read_constcnt()); } This would be an equivalent of init_cpu_freq_invariance_counters(). There is the option of either read_{corecnt/constcnt}() to either do AMU reads or CPPC counter reads depending on inv_source, or for either arm64 or cppc code to define the entire freq_inv_update_counter_refs(). - Given all of the above, topology_scale_freq_tick() can then be made generic prev_const_cnt = this_cpu_read(arch_const_cycles_prev); prev_core_cnt = this_cpu_read(arch_core_cycles_prev); freq_inv_update_counter_refs(); const_cnt = this_cpu_read(arch_const_cycles_prev); core_cnt = this_cpu_read(arch_core_cycles_prev); I have some versions of code that do this generalisation for AMUs and cpufreq with a common topology_set_freq_scale() that is to be used for both arch_set_freq_scale() and arch_set_freq_tick(). But it's written with this usecase in mind so it can be extended to use CPPC counters as source as well, as detailed above. So, this is basically what I had in mind when recommending "wrapping of the counter read functions" :). This would basically reuse much of what is now the AMU invariance code while allowing for CPPC counters as a possible source. I'll stop here for now to see what you guys think about this. Thanks, Ionela. > > > - (2) For 1/3, the presence of AMU counters does not guarantee their > > > usability for frequency invariance. I know you wanted to avoid > > > the complications of AMUs being marked as supporting invariance > > > after the cpufreq driver init function, but this breaks the > > > scenario in which the maximum frequency is invalid. > > > > Is that really a scenario ? i.e. Invalid maximum frequency ? Why would > > that ever happen ? > > > > And I am not sure if this breaks anything which already exists, > > because all we are doing in this case now is not registering cppc for > > FI, which should be fine. > > IIUC, AMU must wait for cpufreq drivers to be registered in order to > get the maximum freq and being able to enable its FIE support. > Could we have a sequence like: > cppc register its scale_freq_tick function > AMU can then override the tick function for cpu which supports AMU > > > > > > - (3) For 2/3, currently we support platforms that have partial support > > > for AMUs, while this would not be supported here. The suggestions > > > at (1) would give us this for free. > > > > As both were counter based mechanisms, I thought it would be better > > and more consistent if only one of them is picked. Though partial > > support of AMUs would still work without the CPPC driver. > > > > -- > > viresh
On 25-08-20, 10:56, Ionela Voinescu wrote: > I've been putting some more thought/code into this one and I believe > something as follows might look nicer as well as cover a few corner cases > (ignore implementation details for now, they can be changed): I saw the other patchset you sent where AMU can be used as the backend for CPPC driver, which means that if AMU IP is present on the platform it will be used by the CPPC to get the perf counts, right ? > - Have a per cpu value that marks the use of either AMUs, CPPC, or > cpufreq for freq invariance (can be done with per-cpu variable or with > cpumasks) > > - arch_topology.c: initialization code as follows: > > for_each_present_cpu(cpu) { > if (freq_inv_amus_valid(cpu) && > !freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu) * 1000, > arch_timer_get_rate(), cpu)) { > per_cpu(inv_source, cpu) = INV_AMU_COUNTERS; > continue; > } > if (freq_inv_cppc_counters_valid(cpu) && > !freq_inv_set_max_ratio(cppc_max_perf, cppc_ref_perf, cpu)) { > per_cpu(inv_source, cpu) = INV_CPPC_COUNTERS; > continue; > } > if (!cpufreq_supports_freq_invariance() || > freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu), > 1, cpu)) { > pr_info("FIE disabled: no valid source for CPU%d.", cpu); > return 0; > } > } Based on that (your other patchset), I think this can get further simplified to whomsoever can register first for freq invariance. i.e. if CPPC registers for it first then there is no need to check AMUs further (as CPPC will be using AMUs anyway), else we will fallback to AMU, else cpufreq. Is that understanding correct ? -- viresh
Hi Viresh, On Thursday 27 Aug 2020 at 13:21:49 (+0530), Viresh Kumar wrote: > On 25-08-20, 10:56, Ionela Voinescu wrote: > > I've been putting some more thought/code into this one and I believe > > something as follows might look nicer as well as cover a few corner cases > > (ignore implementation details for now, they can be changed): > > I saw the other patchset you sent where AMU can be used as the backend > for CPPC driver, which means that if AMU IP is present on the platform > it will be used by the CPPC to get the perf counts, right ? > > > - Have a per cpu value that marks the use of either AMUs, CPPC, or > > cpufreq for freq invariance (can be done with per-cpu variable or with > > cpumasks) > > > > - arch_topology.c: initialization code as follows: > > > > for_each_present_cpu(cpu) { > > if (freq_inv_amus_valid(cpu) && > > !freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu) * 1000, > > arch_timer_get_rate(), cpu)) { > > per_cpu(inv_source, cpu) = INV_AMU_COUNTERS; > > continue; > > } > > if (freq_inv_cppc_counters_valid(cpu) && > > !freq_inv_set_max_ratio(cppc_max_perf, cppc_ref_perf, cpu)) { > > per_cpu(inv_source, cpu) = INV_CPPC_COUNTERS; > > continue; > > } > > if (!cpufreq_supports_freq_invariance() || > > freq_inv_set_max_ratio(cpufreq_get_hw_max_freq(cpu), > > 1, cpu)) { > > pr_info("FIE disabled: no valid source for CPU%d.", cpu); > > return 0; > > } > > } > > Based on that (your other patchset), I think this can get further > simplified to whomsoever can register first for freq invariance. > I don't see it as anyone registering for freq invariance, rather the freq invariance framework chooses its source of information (AMU, CPPC, cpufreq). > i.e. if CPPC registers for it first then there is no need to check > AMUs further (as CPPC will be using AMUs anyway), else we will ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Not necessarily. Even if AMUs are present, they are only used for CPPC's delivered and reference performance counters if the ACPI _CPC entry specifies FFH as method: ResourceTemplate(){Register(FFixedHW, 0x40, 0, 1, 0x4)}, ResourceTemplate(){Register(FFixedHW, 0x40, 0, 0, 0x4)}, > fallback to AMU, else cpufreq. > > Is that understanding correct ? While I understand your point (accessing AMUs through CPPC's read functions to implement invariance) I don't think it's worth tying the two together. I see the two functionalities as independent: - frequency invariance with whichever source of information is valid (AMUs, cpufreq, etc) is separate from - CPPC's delivered and reference performance counters, which currently are used in cpufreq's .get() function. Therefore, taking each of the scenarios one by one: - All CPUs support AMUs: the freq invariance initialisation code will find AMUs valid and it will use them to set the scale factor; completely independently, if the FFH method is specified for CPPC's delivered and reference performance counters, it will also use AMUs, even if, let's say, invariance is disabled. - None of the CPUs support AMUs, but the _CPC entry specifies some platform specific counters for delivered and reference performance. With the current mainline code neither cpufreq or counter based invariance is supported, but the CPPC counters can be used in the cppc_cpufreq driver for the .get() function. But with the above new functionality we can detect that AMUs are not supported and expose the CPPC counters to replace them in implementing invariance. - Mixed scenarios are also supported if we play our cards right and implement the above per-cpu. I'm thinking that having some well defined invariance sources might work well: it will simplify the init function (go through all registered sources and choose (per-cpu) the one that's valid) and allow for otherwise generic invariance support. Something like: enum freq_inv_source {INV_CPUFREQ, INV_AMU_COUNTERS, INV_CPPC_COUNTERS}; struct freq_inv_source { enum freq_inv_source source; bool (*valid)(int cpu); u64 (*read_corecnt)(int cpu); u64 (*read_constcnt)(int cpu); u64 (*max_rate)(int cpu); u64 (*ref_rate)(int cpu); } I am in the middle of unifying AMU counter and cpufreq invariance through something like this, so if you like the idea and you don't think I'm stepping too much on your toes with this, I can consider the usecase in my (what should be) generic support. So in the end this might end up being just a matter of adding a new invariance source (CPPC counters). My only worry is that while I know how a cpufreq source behaves and how AMU counters behave, I'm not entirely sure what to expect from CPPC counters: if they are always appropriate for updates on the tick (not blocking), if they both stop during idle, if there is save/restore functionality before/after idle, etc. What do you think? Regards, Ionela. > -- > viresh
On 27-08-20, 12:27, Ionela Voinescu wrote: > I don't see it as anyone registering for freq invariance, rather the > freq invariance framework chooses its source of information (AMU, CPPC, > cpufreq). Yeah, either way is fine for me. > > i.e. if CPPC registers for it first then there is no need to check > > AMUs further (as CPPC will be using AMUs anyway), else we will > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Not necessarily. Even if AMUs are present, they are only used for CPPC's > delivered and reference performance counters if the ACPI _CPC entry > specifies FFH as method: > > ResourceTemplate(){Register(FFixedHW, 0x40, 0, 1, 0x4)}, > ResourceTemplate(){Register(FFixedHW, 0x40, 0, 0, 0x4)}, Right. > While I understand your point (accessing AMUs through CPPC's read > functions to implement invariance) I don't think it's worth tying the > two together. > > I see the two functionalities as independent: > - frequency invariance with whichever source of information is valid > (AMUs, cpufreq, etc) is separate from > - CPPC's delivered and reference performance counters, which currently > are used in cpufreq's .get() function. > > Therefore, taking each of the scenarios one by one: > - All CPUs support AMUs: the freq invariance initialisation code will > find AMUs valid and it will use them to set the scale factor; > completely independently, if the FFH method is specified for CPPC's > delivered and reference performance counters, it will also use > AMUs, even if, let's say, invariance is disabled. > > - None of the CPUs support AMUs, but the _CPC entry specifies some > platform specific counters for delivered and reference performance. > With the current mainline code neither cpufreq or counter based > invariance is supported, but the CPPC counters can be used in the > cppc_cpufreq driver for the .get() function. > > But with the above new functionality we can detect that AMUs are not > supported and expose the CPPC counters to replace them in > implementing invariance. > > - Mixed scenarios are also supported if we play our cards right and > implement the above per-cpu. > > > I'm thinking that having some well defined invariance sources might work > well: it will simplify the init function (go through all registered > sources and choose (per-cpu) the one that's valid) and allow for > otherwise generic invariance support. Something like: > > enum freq_inv_source {INV_CPUFREQ, INV_AMU_COUNTERS, INV_CPPC_COUNTERS}; > > struct freq_inv_source { > enum freq_inv_source source; > bool (*valid)(int cpu); > u64 (*read_corecnt)(int cpu); > u64 (*read_constcnt)(int cpu); > u64 (*max_rate)(int cpu); > u64 (*ref_rate)(int cpu); > } > > I am in the middle of unifying AMU counter and cpufreq invariance through > something like this, so if you like the idea and you don't think I'm > stepping too much on your toes with this, I can consider the usecase in > my (what should be) generic support. So in the end this might end up > being just a matter of adding a new invariance source (CPPC counters). Okay, if you have already started working on that, no issues from my side. I can just get the relevant stuff from CPPC added once you provide that layer.. > My only worry is that while I know how a cpufreq source behaves and how > AMU counters behave, I'm not entirely sure what to expect from CPPC Neither do I :) > counters: if they are always appropriate for updates on the tick (not > blocking), The update stuff may sleep here and so I had to do stuff in the irq-work handler in my patch. > if they both stop during idle, if there is save/restore > functionality before/after idle, etc. This I will check. -- viresh
On 27-08-20, 12:27, Ionela Voinescu wrote: > I am in the middle of unifying AMU counter and cpufreq invariance through > something like this, so if you like the idea and you don't think I'm > stepping too much on your toes with this, I can consider the usecase in > my (what should be) generic support. So in the end this might end up > being just a matter of adding a new invariance source (CPPC counters). Any update on this ?
Hi Viresh, On Monday 05 Oct 2020 at 13:28:22 (+0530), Viresh Kumar wrote: > On 27-08-20, 12:27, Ionela Voinescu wrote: > > I am in the middle of unifying AMU counter and cpufreq invariance through > > something like this, so if you like the idea and you don't think I'm > > stepping too much on your toes with this, I can consider the usecase in > > my (what should be) generic support. So in the end this might end up > > being just a matter of adding a new invariance source (CPPC counters). > > Any update on this ? > I have some code for this, but not yet in the final state I wanted to bring it to. The code has some dependencies/conflicts with the FFH support and in small part with the new BL_SWITCHER patches so I wanted to get those through first. I'm in the middle of some distractions now, so probably it will take a around two-three more weeks before I submit the code for review. Sorry for the delay, Ionela. > -- > viresh