Message ID | 20180409151919.9133-1-ross.burton@intel.com |
---|---|
State | Accepted |
Commit | ce1cfa57d9828c0898f52e736f671ce8db534031 |
Headers | show |
Series | populate_sdk_base: use xz -T instead of pixz | expand |
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: > xz has native support for threaded compression now and SDK creation was the only > part of oe-core which is using pixz instead of xz. > > Not only does this remove pixz-native from the SDK dependencies, but in my > limited testing xz -T0 is slightly faster and produces smaller archives than > pixz for the same input. > this is using nativesdk-xz ? > Signed-off-by: Ross Burton <ross.burton@intel.com> > --- > meta/classes/populate_sdk_base.bbclass | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass > index 77ec8aaec27..79984d7914f 100644 > --- a/meta/classes/populate_sdk_base.bbclass > +++ b/meta/classes/populate_sdk_base.bbclass > @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" > TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" > > SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" > -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" > +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" > SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" > > # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it > @@ -225,7 +225,7 @@ fakeroot tar_sdk() { > # Package it up > mkdir -p ${SDKDEPLOYDIR} > cd ${SDK_OUTPUT}/${SDKPATH} > - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > } > > fakeroot create_shar() { > -- > 2.11.0 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). Ross On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote: > On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >> xz has native support for threaded compression now and SDK creation was the only >> part of oe-core which is using pixz instead of xz. >> >> Not only does this remove pixz-native from the SDK dependencies, but in my >> limited testing xz -T0 is slightly faster and produces smaller archives than >> pixz for the same input. >> > > this is using nativesdk-xz ? > >> Signed-off-by: Ross Burton <ross.burton@intel.com> >> --- >> meta/classes/populate_sdk_base.bbclass | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass >> index 77ec8aaec27..79984d7914f 100644 >> --- a/meta/classes/populate_sdk_base.bbclass >> +++ b/meta/classes/populate_sdk_base.bbclass >> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" >> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" >> >> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" >> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" >> >> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >> # Package it up >> mkdir -p ${SDKDEPLOYDIR} >> cd ${SDK_OUTPUT}/${SDKPATH} >> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> } >> >> fakeroot create_shar() { >> -- >> 2.11.0 >> >> -- >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross <ross.burton@intel.com> wrote: > No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). > OK then we need to make sure that all supported distros will have the threaded xz version > Ross > > On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote: >> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >>> xz has native support for threaded compression now and SDK creation was the only >>> part of oe-core which is using pixz instead of xz. >>> >>> Not only does this remove pixz-native from the SDK dependencies, but in my >>> limited testing xz -T0 is slightly faster and produces smaller archives than >>> pixz for the same input. >>> >> >> this is using nativesdk-xz ? >> >>> Signed-off-by: Ross Burton <ross.burton@intel.com> >>> --- >>> meta/classes/populate_sdk_base.bbclass | 4 ++-- >>> 1 file changed, 2 insertions(+), 2 deletions(-) >>> >>> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass >>> index 77ec8aaec27..79984d7914f 100644 >>> --- a/meta/classes/populate_sdk_base.bbclass >>> +++ b/meta/classes/populate_sdk_base.bbclass >>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" >>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" >>> >>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" >>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" >>> >>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>> # Package it up >>> mkdir -p ${SDKDEPLOYDIR} >>> cd ${SDK_OUTPUT}/${SDKPATH} >>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> } >>> >>> fakeroot create_shar() { >>> -- >>> 2.11.0 >>> >>> -- >>> _______________________________________________ >>> Openembedded-core mailing list >>> Openembedded-core@lists.openembedded.org >>> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
xz-native isn't in ASSUME_PROVIDED, we built it. Ross On 9 April 2018 at 19:44, Khem Raj <raj.khem@gmail.com> wrote: > On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross <ross.burton@intel.com> wrote: >> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk). >> > > OK then we need to make sure that all supported distros will have the > threaded xz version > >> Ross >> >> On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote: >>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >>>> xz has native support for threaded compression now and SDK creation was the only >>>> part of oe-core which is using pixz instead of xz. >>>> >>>> Not only does this remove pixz-native from the SDK dependencies, but in my >>>> limited testing xz -T0 is slightly faster and produces smaller archives than >>>> pixz for the same input. >>>> >>> >>> this is using nativesdk-xz ? >>> >>>> Signed-off-by: Ross Burton <ross.burton@intel.com> >>>> --- >>>> meta/classes/populate_sdk_base.bbclass | 4 ++-- >>>> 1 file changed, 2 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass >>>> index 77ec8aaec27..79984d7914f 100644 >>>> --- a/meta/classes/populate_sdk_base.bbclass >>>> +++ b/meta/classes/populate_sdk_base.bbclass >>>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" >>>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" >>>> >>>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" >>>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >>>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" >>>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" >>>> >>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>>> # Package it up >>>> mkdir -p ${SDKDEPLOYDIR} >>>> cd ${SDK_OUTPUT}/${SDKPATH} >>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>>> } >>>> >>>> fakeroot create_shar() { >>>> -- >>>> 2.11.0 >>>> >>>> -- >>>> _______________________________________________ >>>> Openembedded-core mailing list >>>> Openembedded-core@lists.openembedded.org >>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: > xz has native support for threaded compression now and SDK creation was the only > part of oe-core which is using pixz instead of xz. > > Not only does this remove pixz-native from the SDK dependencies, but in my > limited testing xz -T0 is slightly faster and produces smaller archives than > pixz for the same input. > > Signed-off-by: Ross Burton <ross.burton@intel.com> > --- > > # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it > @@ -225,7 +225,7 @@ fakeroot tar_sdk() { > # Package it up > mkdir -p ${SDKDEPLOYDIR} > cd ${SDK_OUTPUT}/${SDKPATH} > - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz Since -T 0 will use all available CPUs (ie any attempts the user may have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE will be ignored), perhaps it's worth adding something like "--memlimit=70%" to try to give some protection for environments with lots of CPUs but not so much DRAM? -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote: > On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >> xz has native support for threaded compression now and SDK creation was the only >> part of oe-core which is using pixz instead of xz. >> >> Not only does this remove pixz-native from the SDK dependencies, but in my >> limited testing xz -T0 is slightly faster and produces smaller archives than >> pixz for the same input. >> >> Signed-off-by: Ross Burton <ross.burton@intel.com> >> --- >> >> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >> # Package it up >> mkdir -p ${SDKDEPLOYDIR} >> cd ${SDK_OUTPUT}/${SDKPATH} >> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz > > Since -T 0 will use all available CPUs (ie any attempts the user may > have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE > will be ignored), perhaps it's worth adding something like > "--memlimit=70%" to try to give some protection for environments with > lots of CPUs but not so much DRAM? There's a few places where -T0 is passed already: tar.xz image creation and opkg creation, so I guess we need to centralise this somewhere too. Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross <ross.burton@intel.com> wrote: > On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote: >> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >>> xz has native support for threaded compression now and SDK creation was the only >>> part of oe-core which is using pixz instead of xz. >>> >>> Not only does this remove pixz-native from the SDK dependencies, but in my >>> limited testing xz -T0 is slightly faster and produces smaller archives than >>> pixz for the same input. >>> >>> Signed-off-by: Ross Burton <ross.burton@intel.com> >>> --- >>> >>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>> # Package it up >>> mkdir -p ${SDKDEPLOYDIR} >>> cd ${SDK_OUTPUT}/${SDKPATH} >>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> >> Since -T 0 will use all available CPUs (ie any attempts the user may >> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE >> will be ignored), perhaps it's worth adding something like >> "--memlimit=70%" to try to give some protection for environments with >> lots of CPUs but not so much DRAM? > > There's a few places where -T0 is passed already: tar.xz image > creation and opkg creation, so I guess we need to centralise this > somewhere too. > May be use XZ_OPT env bariable and param to -T could use the bitbake calculated number of CPUs > Ross > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 9 April 2018 at 22:20, Khem Raj <raj.khem@gmail.com> wrote: > On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross <ross.burton@intel.com> wrote: >> On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote: >>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote: >>>> xz has native support for threaded compression now and SDK creation was the only >>>> part of oe-core which is using pixz instead of xz. >>>> >>>> Not only does this remove pixz-native from the SDK dependencies, but in my >>>> limited testing xz -T0 is slightly faster and produces smaller archives than >>>> pixz for the same input. >>>> >>>> Signed-off-by: Ross Burton <ross.burton@intel.com> >>>> --- >>>> >>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it >>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() { >>>> # Package it up >>>> mkdir -p ${SDKDEPLOYDIR} >>>> cd ${SDK_OUTPUT}/${SDKPATH} >>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >>> >>> Since -T 0 will use all available CPUs (ie any attempts the user may >>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE >>> will be ignored), perhaps it's worth adding something like >>> "--memlimit=70%" to try to give some protection for environments with >>> lots of CPUs but not so much DRAM? >> >> There's a few places where -T0 is passed already: tar.xz image >> creation and opkg creation, so I guess we need to centralise this >> somewhere too. >> > > May be use XZ_OPT env bariable and param to -T could use the bitbake > calculated number of CPUs -T0 is use as many threads as there are cores, which is the same as the bitbake function. Re-using BB_NUMBER_THREADS might work. Ross -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass index 77ec8aaec27..79984d7914f 100644 --- a/meta/classes/populate_sdk_base.bbclass +++ b/meta/classes/populate_sdk_base.bbclass @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= "" TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}" SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}" -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross" SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale" # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it @@ -225,7 +225,7 @@ fakeroot tar_sdk() { # Package it up mkdir -p ${SDKDEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } fakeroot create_shar() {
xz has native support for threaded compression now and SDK creation was the only part of oe-core which is using pixz instead of xz. Not only does this remove pixz-native from the SDK dependencies, but in my limited testing xz -T0 is slightly faster and produces smaller archives than pixz for the same input. Signed-off-by: Ross Burton <ross.burton@intel.com> --- meta/classes/populate_sdk_base.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.11.0 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core