| Message ID | 20220317082230.23622-1-irui.wang@mediatek.com |
|---|---|
| Headers | show |
| Series | Enable two H264 encoder core on MT8195 | expand |
Il 17/03/22 09:22, Irui Wang ha scritto: > MT8195 has two H264 encoder cores, they have their own power-domains, > clocks, interrupts, register base. The two H264 encoder cores can work > together to achieve higher performance, it's a core mode called > frame-racing, one core has 4K@30fps performance, two cores can achieve > 4K@60fps. > The two encoder core encoding process looks like this: > > VENC Core0: frm#0....frm#2....frm#4.... > VENC Core1: ..frm#1....frm#3....frm#5.... > > This series of patches are used to enable the two H264 encoder cores, > encoding process will be changed: > As-Is: Synchronous > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder IRQ --> > encoding done with result --> job_finish > V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder IRQ --> > encoding done with result --> job_finish > ... > > To-Be: Asynchronous > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish > ..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish > (venc core0 may encode done here, done the encoding result to client) > V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish. > > There is no "wait encoder IRQ" synchronous call during frame-racing mode > encoding process, it can full use the two encoder cores to achieve higher > performance. > Hello Irui, like v1 and v2, also this version of the series will crash the kernel when trying to encode. Before pushing the next series, you shall ensure that it is not crashing the entire kernel, hence, please test your code with gstreamer. Please use the following pipeline to test: gst-launch-1.0 videotestsrc num-buffers=100 ! v4l2h264enc ! queue ! h264parse ! qtmux ! filesink location=test.mov Regards, Angelo
Dear Angelo, Thanks for your comments. On Fri, 2022-03-18 at 15:04 +0100, AngeloGioacchino Del Regno wrote: > Il 17/03/22 09:22, Irui Wang ha scritto: > > MT8195 has two H264 encoder cores, they have their own power- > > domains, > > clocks, interrupts, register base. The two H264 encoder cores can > > work > > together to achieve higher performance, it's a core mode called > > frame-racing, one core has 4K@30fps performance, two cores can > > achieve > > 4K@60fps. > > The two encoder core encoding process looks like this: > > > > VENC Core0: frm#0....frm#2....frm#4.... > > VENC Core1: ..frm#1....frm#3....frm#5.... > > > > This series of patches are used to enable the two H264 encoder > > cores, > > encoding process will be changed: > > As-Is: Synchronous > > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder > > IRQ --> > > encoding done with result --> job_finish > > V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder > > IRQ --> > > encoding done with result --> job_finish > > ... > > > > To-Be: Asynchronous > > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish > > ..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish > > (venc core0 may encode done here, done the encoding result to > > client) > > V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish. > > > > There is no "wait encoder IRQ" synchronous call during frame-racing > > mode > > encoding process, it can full use the two encoder cores to achieve > > higher > > performance. > > > > Hello Irui, > like v1 and v2, also this version of the series will crash the kernel > when > trying to encode. As mentioned before, dual core encoding need more encoding working buffers in patch-8: media: mtk-vcodec: Add more extra processing for venc_multi_core mode a new structure, struct venc_multi_core_vsi defined for it, encoding processing like this: kernel --> ipi message --> scp firmware(userspace), vsi is a share memory to transfer data between kernel and user, so, the definition of vsi must be the same in kernel and userspace. otherwise, errors will be occurred, for example, kernel crash. When we enable dual core encoding in this patch, the venc_multi_core_vsi used in kernel, but scp(userspace) still use the older one vsi(struct venc_h264_vsi), the crash happened. Maybe we would find out how to resolve the limitations, many thanks for your review. > > Before pushing the next series, you shall ensure that it is not > crashing > the entire kernel, hence, please test your code with gstreamer. > > > Please use the following pipeline to test: > > gst-launch-1.0 videotestsrc num-buffers=100 ! v4l2h264enc ! queue ! > h264parse ! > qtmux ! filesink location=test.mov > > > Regards, > Angelo Thanks Best Regards
MT8195 has two H264 encoder cores, they have their own power-domains, clocks, interrupts, register base. The two H264 encoder cores can work together to achieve higher performance, it's a core mode called frame-racing, one core has 4K@30fps performance, two cores can achieve 4K@60fps. The two encoder core encoding process looks like this: VENC Core0: frm#0....frm#2....frm#4.... VENC Core1: ..frm#1....frm#3....frm#5.... This series of patches are used to enable the two H264 encoder cores, encoding process will be changed: As-Is: Synchronous V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder IRQ --> encoding done with result --> job_finish V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder IRQ --> encoding done with result --> job_finish ... To-Be: Asynchronous V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish ..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish (venc core0 may encode done here, done the encoding result to client) V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish. There is no "wait encoder IRQ" synchronous call during frame-racing mode encoding process, it can full use the two encoder cores to achieve higher performance. --- This series patches dependent on: media_stage tree: [1] https://git.linuxtv.org/media_stage.git/commit/?id=b3627647f9ea7473d10fb08a95fd7c4133a17ca4 mt8192 decoder: [2] https://patchwork.kernel.org/project/linux-mediatek/list/?series=616991 patch3 new venc dt-bindings included files [3] MM IOMMU binding: https://patchwork.kernel.org/project/linux-mediatek/patch/20220217113453.13658-2-yong.wu@mediatek.com/ [4] MT8195 power domain: https://patchwork.kernel.org/project/linux-mediatek/list/?series=580579 changes compared with v2: - update venc core dt-bindings, add two new properties for current usage. - parse venc multi_core mode from device tree. - rebase to the newer linux media stage. changes compared with v1: - of_platform_populate was used in place of the component framework. - new yaml file for venc cores. - some modifications for patch v1's review comments. --- Irui Wang (10): media: mtk-vcodec: Use core type to indicate h264 and vp8 enc media: mtk-vcodec: export encoder functions dt-bindings: media: mtk-vcodec: Adds encoder cores dt-bindings for mt8195 media: mtk-vcodec: Enable venc dual core usage media: mtk-vcodec: mtk-vcodec: Rewrite venc power manage interface media: mtk-vcodec: Add venc power on/off interface media: mtk-vcodec: Rewrite venc clock interface media: mtk-vcodec: Add more extra processing for venc_multi_core mode media: mtk-vcodec: Add venc_multi_core mode encode process media: mtk-vcodec: Done encode result to client .../media/mediatek,vcodec-encoder-core.yaml | 181 +++++++++++++++ .../media/mediatek,vcodec-encoder.yaml | 1 - drivers/media/platform/mtk-vcodec/Makefile | 4 +- .../platform/mtk-vcodec/mtk_vcodec_drv.h | 36 ++- .../platform/mtk-vcodec/mtk_vcodec_enc.c | 113 +++++++--- .../platform/mtk-vcodec/mtk_vcodec_enc.h | 7 +- .../platform/mtk-vcodec/mtk_vcodec_enc_core.c | 169 ++++++++++++++ .../platform/mtk-vcodec/mtk_vcodec_enc_core.h | 36 +++ .../platform/mtk-vcodec/mtk_vcodec_enc_drv.c | 107 +++++---- .../platform/mtk-vcodec/mtk_vcodec_enc_pm.c | 183 +++++++++++++-- .../platform/mtk-vcodec/mtk_vcodec_enc_pm.h | 11 +- .../platform/mtk-vcodec/mtk_vcodec_util.c | 19 ++ .../platform/mtk-vcodec/mtk_vcodec_util.h | 5 + .../platform/mtk-vcodec/venc/venc_h264_if.c | 212 +++++++++++++++--- .../platform/mtk-vcodec/venc/venc_vp8_if.c | 3 +- .../media/platform/mtk-vcodec/venc_drv_if.c | 79 +++++-- .../media/platform/mtk-vcodec/venc_drv_if.h | 7 + .../media/platform/mtk-vcodec/venc_vpu_if.c | 11 +- .../media/platform/mtk-vcodec/venc_vpu_if.h | 3 +- 19 files changed, 1026 insertions(+), 161 deletions(-) create mode 100644 Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-core.yaml create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_core.c create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_core.h