[v6,0/8] Support H264 multi-core encoder on MT8195

MT8195 has two H264 encoder hardware, named core0 and core1, this two
cores can encode two input frames separately at the same time to achieve
higher performance.

This series of patches are used to enable the two H264 encoder cores,
the difference between encoding process before and after enable two
cores is just like as below:
As-Is: Synchronous
V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder IRQ
-->
encoding done with result --> job_finish
V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder IRQ
-->
encoding done with result --> job_finish
...
To-Be: Asynchronous
V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish
..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish
(venc core0 may encode done here, done the encoding result to client)
V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish.

There is a tast test on chromeOS can measure the encoder fps summary:
command line:
tast run $DUT_IP video.PlatformEncoding.v4l2_h264*
 +-------------+------+----------+------+----------+------+----------+
 | fps summary | 180p | 180_meet | 360p | 360_meet | 720p | 720_meet |
 +-------------+------+----------+------+----------+------+----------+
 |  w/o patch  | 1913 |   1935   | 554  |    571   | 159  |    189   |
 +-------------+------+----------+------+----------+------+----------+
 | apply patch | 2236 |   2337   | 820  |    830   | 324  |    390   |
 +-------------+------+----------+------+----------+------+----------+

changes compared with v5:
- rebase to the newer linux media_stage tree.

changes compared with v4:
- reabse to the newer linux media stage tree.
- remove "mediatek,venc-multi-core" property since sub-device can
  be probed by of_platform_populate api directly.
- some modifications for patch v4's review comments.

changes compared with v3:
- rebase to the newer linux media stage.
- add a capability to indicate scp firmware support multi-core.
- probe core0 as main device, core1 as sub-device.

changes compared with v2:
- update venc core dt-bindings, add two new properties for current
  usage.
- parse venc multi_core mode from device tree.
- rebase to the newer linux media stage.

changes compared with v1:
- of_platform_populate was used in place of the component framework.
- new yaml file for venc cores.
- some modifications for patch v1's review comments.
---

Irui Wang (8):
  dt-bindings: media: mediatek: vcodec: Adds encoder cores dt-bindings
    for mt8195
  media: mediatek: vcodec: Enable venc dual core usage
  media: mediatek: vcodec: Refactor venc power manage function
  media: mediatek: vcodec: Add more extra processing for multi-core
    encoding
  media: mediatek: vcodec: Add venc power on/off function
  media: mediatek: vcodec: Refactor encoder clock on/off function
  media: mediatek: vcodec: Add multi-core encoding process
  media: mediatek: vcodec: Return encoding result in asynchronous mode

 .../media/mediatek,vcodec-encoder-core.yaml   | 217 ++++++++++++++++++
 .../media/mediatek,vcodec-encoder.yaml        |   1 -
 .../media/platform/mediatek/vcodec/Makefile   |   4 +-
 .../platform/mediatek/vcodec/mtk_vcodec_drv.h |  27 ++-
 .../platform/mediatek/vcodec/mtk_vcodec_enc.c | 113 ++++++---
 .../platform/mediatek/vcodec/mtk_vcodec_enc.h |  11 +-
 .../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  44 +++-
 .../mediatek/vcodec/mtk_vcodec_enc_hw.c       | 156 +++++++++++++
 .../mediatek/vcodec/mtk_vcodec_enc_hw.h       |  34 +++
 .../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 178 ++++++++++++--
 .../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  11 +-
 .../mediatek/vcodec/mtk_vcodec_util.c         |  19 ++
 .../mediatek/vcodec/mtk_vcodec_util.h         |   3 +
 .../mediatek/vcodec/venc/venc_h264_if.c       | 143 +++++++++---
 .../mediatek/vcodec/venc/venc_vp8_if.c        |   3 +-
 .../platform/mediatek/vcodec/venc_drv_if.c    |  75 ++++--
 .../platform/mediatek/vcodec/venc_drv_if.h    |   6 +
 .../platform/mediatek/vcodec/venc_vpu_if.c    |  25 +-
 .../platform/mediatek/vcodec/venc_vpu_if.h    |   3 +-
 19 files changed, 958 insertions(+), 115 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-core.yaml
 create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.c
 create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.h

On Sat, 1 Oct 2022, Irui Wang wrote:

> when enable multi-core encoding, all available encoder cores' power need
> to be on/off, add new functions for encoder cores' power management.
> 
> Signed-off-by: Irui Wang <irui.wang@mediatek.com>
> ---

> diff --git a/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c b/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> index 75de5031d292..213c3f50e9eb 100644
> --- a/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> +++ b/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> @@ -9,6 +9,7 @@
>  #include <linux/of_platform.h>
>  #include <linux/pm_runtime.h>
>  
> +#include "mtk_vcodec_enc_hw.h"
>  #include "mtk_vcodec_enc_pm.h"
>  #include "mtk_vcodec_util.h"
>  
> @@ -56,6 +57,88 @@ int mtk_vcodec_init_enc_clk(struct platform_device *pdev,
>  }
>  EXPORT_SYMBOL_GPL(mtk_vcodec_init_enc_clk);
>  
> +static int mtk_enc_core_power_on(struct mtk_vcodec_ctx *ctx)
> +{
> +	struct mtk_venc_hw_dev *sub_core;
> +	int ret, i;
> +
> +	/* multi-core encoding need power on all available cores */
> +	for (i = 0; i < MTK_VENC_HW_MAX; i++) {
> +		sub_core = (struct mtk_venc_hw_dev *)ctx->dev->enc_hw_dev[i];
> +		if (!sub_core)
> +			continue;
> +
> +		ret = pm_runtime_resume_and_get(&sub_core->plat_dev->dev);
> +		if (ret) {
> +			mtk_v4l2_err("power on sub_core[%d] fail %d", i, ret);
> +			goto pm_on_fail;
> +		}
> +	}
> +	return ret;
> +
> +pm_on_fail:
> +	for (i -= 1; i >= 0; i--) {

while (i--) {

achieves the same.

> +		sub_core = (struct mtk_venc_hw_dev *)ctx->dev->enc_hw_dev[i];
> +		pm_runtime_put_sync(&sub_core->plat_dev->dev);
> +	}
> +	return ret;
> +}

On Sat, 1 Oct 2022, Irui Wang wrote:

> when enable multi-core encoding, encoder cores use their own clock,
> refactor clock management functions with used encoder hardware id.
> 
> Signed-off-by: Irui Wang <irui.wang@mediatek.com>
> ---
>  .../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 89 ++++++++++++++++---
>  .../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  6 +-
>  .../platform/mediatek/vcodec/venc_drv_if.c    |  4 +-
>  3 files changed, 84 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c b/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> index 213c3f50e9eb..2f83aade779a 100644
> --- a/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> +++ b/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_pm.c
> @@ -60,7 +60,9 @@ EXPORT_SYMBOL_GPL(mtk_vcodec_init_enc_clk);
>  static int mtk_enc_core_power_on(struct mtk_vcodec_ctx *ctx)
>  {
>  	struct mtk_venc_hw_dev *sub_core;
> +	struct mtk_vcodec_clk *clk;
>  	int ret, i;
> +	int j = 0;
>  
>  	/* multi-core encoding need power on all available cores */
>  	for (i = 0; i < MTK_VENC_HW_MAX; i++) {
> @@ -73,12 +75,27 @@ static int mtk_enc_core_power_on(struct mtk_vcodec_ctx *ctx)
>  			mtk_v4l2_err("power on sub_core[%d] fail %d", i, ret);
>  			goto pm_on_fail;
>  		}
> +
> +		clk = &sub_core->pm.venc_clk;
> +		for (j = 0; j < clk->clk_num; j++) {
> +			ret = clk_prepare(clk->clk_info[j].vcodec_clk);
> +			if (ret) {
> +				mtk_v4l2_err("prepare clk [%s] fail %d",
> +					     clk->clk_info[j].clk_name, ret);
> +				goto pm_on_fail;
> +			}
> +		}
>  	}
>  	return ret;
>  
>  pm_on_fail:
>  	for (i -= 1; i >= 0; i--) {
>  		sub_core = (struct mtk_venc_hw_dev *)ctx->dev->enc_hw_dev[i];
> +
> +		clk = &sub_core->pm.venc_clk;
> +		for (j -= 1; j >= 0; j--)
> +			clk_unprepare(clk->clk_info[j].vcodec_clk);
> +
>  		pm_runtime_put_sync(&sub_core->plat_dev->dev);

There's more than one thing wrong here.

pm_runtime_put_sync() won't be called for the ith entry when the later 
goto pm_on_fail is taken because the loop decrements i right at the 
start.

Similarly, i and j will mismatch for the ith entry because i was 
decremented.

Third, j does not start from clk->clk_num - 1 for the other entries 
(for those lower "i"s).